AI Sec Watch

A researcher tested a machine learning model called Husky AI by creating simple test images (all black, all white, and random pixels) and sending them through an HTTP API to see if the model would make incorrect predictions. The white canvas image successfully tricked the model into incorrectly classifying it as a husky, demonstrating a perturbation attack (where slightly modified or unusual inputs fool an AI into making wrong predictions).

This post explains threat modeling for machine learning systems, which is a process to systematically identify potential security attacks. The author uses Microsoft's Threat Modeling tool and STRIDE (a framework categorizing threats into spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege) to identify vulnerabilities in a machine learning system called 'Husky AI', and notes that perturbation attacks (where attackers query the model to trick it into making wrong predictions) are a particular concern for ML systems.

Operationalizing an ML model (putting it into production so it can be used by real applications) involves deploying the trained model to a web server so it can make predictions. The author found that integrating TensorFlow (a popular ML framework) with Golang was unexpectedly complicated, so they chose Python instead for their web server.

This post describes how the author built Husky AI, a machine learning system that classifies images as huskies or non-huskies, using a convolutional neural network (CNN, a type of AI model designed to process images). The author gathered about 1,300 husky images and 3,000 other images using Bing Image Search, then organized them into separate training and validation folders to build and test the model. The post notes a potential security risk: attackers could poison either the training or validation image sets to cause the model to perform poorly.

This post introduces the machine learning pipeline, which consists of sequential steps from collecting training images, pre-processing data, defining and training a model, evaluating performance, and finally deploying it to production as an API (application programming interface, a way for software to communicate). The author uses a "Husky AI" example application that identifies whether uploaded images contain huskies, and explains that understanding this pipeline's components is important for identifying potential security attacks on machine learning systems.

A security researcher describes their year-long study of machine learning and AI fundamentals, with the goal of understanding how to build and then attack ML systems. The post outlines their learning approach, courses, and materials for others interested in starting adversarial machine learning (attacking ML systems).

Race conditions in ACL (access control list, the rules that determine who can access files) application occur when a system creates a sensitive file but there is a time gap before permissions are applied to protect it, potentially allowing attackers to access the file during that window. This type of vulnerability exploits the timing between file creation and permission lockdown to expose sensitive information.

Telemetry (data collected about how users interact with software) is often used by companies to make business decisions, but telemetry pipelines (the systems that collect and process this data) can be vulnerable to attacks. A red team security test demonstrated this by spoofing telemetry requests to falsely show a Commodore 64 as the most popular operating system, which could mislead companies into making poor decisions based on fake usage data.

This article discusses how to measure the maturity and effectiveness of security testing programs, particularly red teaming (simulated attacks to find vulnerabilities). The author suggests using existing frameworks like CMMI (Capability Maturity Model Integration, a system developed by Carnegie Mellon University that rates how well-organized software processes are on a scale of one to five) that can be adapted to evaluate offensive security programs.