AI Sec Watch

A vulnerability in Libsvm v324 (a machine learning library used by scikit-learn 0.23.2) allows attackers to crash a program by sending a specially crafted machine learning model with an extremely large value in the _n_support array, causing a segmentation fault (a type of crash where the program tries to access memory it shouldn't). The scikit-learn developers noted this only happens if an application violates the library's API by modifying private attributes.

Repudiation is a security threat where someone denies performing an action, such as replacing an AI model file with a malicious version. The source explains how to use auditd (a Linux auditing tool) and centralized monitoring systems like Splunk or Elastic Stack to create audit logs that track who accessed or modified files and when, helping prove or investigate whether specific accounts made changes.

This is a YouTube talk about building and breaking machine learning systems, presented at a security conference (GrayHat Red Team Village). The speaker is exploring whether to develop this content into a hands-on workshop where participants could practice these concepts.

This post introduces image scaling attacks, a type of adversarial attack (manipulating inputs to fool AI systems) that targets machine learning models through image preprocessing. The author discovered this attack concept while preparing demos and references academic research on understanding and preventing these attacks.

This post demonstrates how to use the Adversarial Robustness Toolbox (ART, an open-source library created by IBM for testing machine learning security) to generate adversarial examples, which are modified images designed to trick AI models into making wrong predictions. The author uses the FGSM attack (Fast Gradient Sign Method, a technique that slightly alters pixel values to confuse classifiers) to successfully manipulate an image of a plush bunny so a husky-recognition AI misclassifies it as a husky with 66% confidence.

TensorFlow versions before 2.4.0 have a bug in the `tf.image.crop_and_resize` function where very large values in the `boxes` argument are converted to NaN (a special floating point value meaning "not a number"), causing undefined behavior and a segmentation fault (a crash from illegal memory access). This vulnerability affects the CPU implementation of the function.

In TensorFlow before version 2.4.0, an attacker can provide an invalid `axis` parameter (a setting that specifies which dimension of data to work with) to a quantization function, causing the program to access memory outside the bounds of an array, which crashes the system. The vulnerability exists because the code only uses DCHECK (a debug-only validation that is disabled in normal builds) rather than proper runtime validation.

This item is promotional content for a conference talk about attacking and defending machine learning systems, presented at GrayHat 2020's Red Team Village. The speaker created an introductory video for a session titled 'Learning by doing: Building and breaking a machine learning system,' scheduled for October 31st, 2020.

The VS Code Python extension had a vulnerability where HTML and JavaScript code could be injected through error messages (called tracebacks, which show where a program failed) in Jupyter Notebooks, potentially allowing attackers to steal user information or take control of their computer. The vulnerability occurred because strings in error messages were not properly escaped (prevented from being interpreted as code), and could be triggered by modifying a notebook file directly or by having the notebook connect to a remote server controlled by an attacker.