AI Sec Watch

In TensorFlow before version 2.4.0, an attacker can provide an invalid `axis` parameter (a setting that specifies which dimension of data to work with) to a quantization function, causing the program to access memory outside the bounds of an array, which crashes the system. The vulnerability exists because the code only uses DCHECK (a debug-only validation that is disabled in normal builds) rather than proper runtime validation.

This item is promotional content for a conference talk about attacking and defending machine learning systems, presented at GrayHat 2020's Red Team Village. The speaker created an introductory video for a session titled 'Learning by doing: Building and breaking a machine learning system,' scheduled for October 31st, 2020.

The VS Code Python extension had a vulnerability where HTML and JavaScript code could be injected through error messages (called tracebacks, which show where a program failed) in Jupyter Notebooks, potentially allowing attackers to steal user information or take control of their computer. The vulnerability occurred because strings in error messages were not properly escaped (prevented from being interpreted as code), and could be triggered by modifying a notebook file directly or by having the notebook connect to a remote server controlled by an attacker.

Attackers can steal machine learning model files through direct approaches like compromising systems to find model files (often with .h5 extensions), or through indirect approaches like model stealing where attackers build similar models themselves. One specific attack vector involves SSH agent hijacking (exploiting SSH keys stored in memory on compromised machines), which allows attackers to access production systems containing model files without needing the original passphrases.

This is an announcement for a conference talk about attacking and defending machine learning systems, covering practical threats like brute forcing predictions (testing many inputs to guess outputs), perturbations (small changes to data that fool AI), and backdooring models (secretly poisoning training data). The speaker will discuss both ML-specific attacks and traditional security breaches, as well as defenses to protect these systems.

TensorFlow Lite versions before 2.2.1 and 2.3.1 have a bug where the segment sum operation (a function that groups and sums data) crashes or causes memory corruption if the segment IDs (labels that organize the data) are not sorted in increasing order. The code incorrectly assumes the IDs are sorted, so it allocates too little memory, leading to a segmentation fault (a crash caused by accessing memory it shouldn't).

TensorFlow Lite (a lightweight version of TensorFlow used on mobile and embedded devices) before versions 2.2.1 and 2.3.1 has a vulnerability where attackers can crash an application by making it try to allocate too much memory through the segment sum operation (a function that groups and sums data). The vulnerability works because the code uses the largest value in the input data to determine how much memory to request, so an attacker can provide a very large number to exhaust available memory.

TensorFlow Lite versions before 2.2.1 and 2.3.1 have a vulnerability where negative values in the segment_ids tensor (an array of numbers used to group data) can cause the software to write data outside its allocated memory area, potentially crashing the program or corrupting memory. This vulnerability can be exploited by anyone who can modify the segment_ids data.

TensorFlow Lite (a machine learning framework for mobile devices) versions before 1.15.4, 2.0.3, 2.1.2, 2.2.1, and 2.3.1 have a vulnerability in how they validate saved models. The framework uses a special index value of -1 to mark optional inputs, but this value is incorrectly accepted for all operators and even output tensors, allowing attackers to read and write data outside the intended memory boundaries.