AI Sec Watch

This post demonstrates how to use the Adversarial Robustness Toolbox (ART, an open-source library created by IBM for testing machine learning security) to generate adversarial examples, which are modified images designed to trick AI models into making wrong predictions. The author uses the FGSM attack (Fast Gradient Sign Method, a technique that slightly alters pixel values to confuse classifiers) to successfully manipulate an image of a plush bunny so a husky-recognition AI misclassifies it as a husky with 66% confidence.

TensorFlow versions before 2.4.0 have a bug in the `tf.image.crop_and_resize` function where very large values in the `boxes` argument are converted to NaN (a special floating point value meaning "not a number"), causing undefined behavior and a segmentation fault (a crash from illegal memory access). This vulnerability affects the CPU implementation of the function.

In TensorFlow before version 2.4.0, an attacker can provide an invalid `axis` parameter (a setting that specifies which dimension of data to work with) to a quantization function, causing the program to access memory outside the bounds of an array, which crashes the system. The vulnerability exists because the code only uses DCHECK (a debug-only validation that is disabled in normal builds) rather than proper runtime validation.

This item is promotional content for a conference talk about attacking and defending machine learning systems, presented at GrayHat 2020's Red Team Village. The speaker created an introductory video for a session titled 'Learning by doing: Building and breaking a machine learning system,' scheduled for October 31st, 2020.

The VS Code Python extension had a vulnerability where HTML and JavaScript code could be injected through error messages (called tracebacks, which show where a program failed) in Jupyter Notebooks, potentially allowing attackers to steal user information or take control of their computer. The vulnerability occurred because strings in error messages were not properly escaped (prevented from being interpreted as code), and could be triggered by modifying a notebook file directly or by having the notebook connect to a remote server controlled by an attacker.

Attackers can steal machine learning model files through direct approaches like compromising systems to find model files (often with .h5 extensions), or through indirect approaches like model stealing where attackers build similar models themselves. One specific attack vector involves SSH agent hijacking (exploiting SSH keys stored in memory on compromised machines), which allows attackers to access production systems containing model files without needing the original passphrases.

This is an announcement for a conference talk about attacking and defending machine learning systems, covering practical threats like brute forcing predictions (testing many inputs to guess outputs), perturbations (small changes to data that fool AI), and backdooring models (secretly poisoning training data). The speaker will discuss both ML-specific attacks and traditional security breaches, as well as defenses to protect these systems.

TensorFlow Lite versions before 2.2.1 and 2.3.1 have a bug where the segment sum operation (a function that groups and sums data) crashes or causes memory corruption if the segment IDs (labels that organize the data) are not sorted in increasing order. The code incorrectly assumes the IDs are sorted, so it allocates too little memory, leading to a segmentation fault (a crash caused by accessing memory it shouldn't).

TensorFlow Lite (a lightweight version of TensorFlow used on mobile and embedded devices) before versions 2.2.1 and 2.3.1 has a vulnerability where attackers can crash an application by making it try to allocate too much memory through the segment sum operation (a function that groups and sums data). The vulnerability works because the code uses the largest value in the input data to determine how much memory to request, so an attacker can provide a very large number to exhaust available memory.