AI Sec Watch

TensorFlow versions before 2.2.1 and 2.3.1 have a vulnerability in the `dlpack.to_dlpack` function where it can be tricked into using uninitialized memory (memory that hasn't been set to a known value), leading to further memory corruption. The problem occurs because the code assumes the input is a TensorFlow tensor, but an attacker can pass in a regular Python object instead, causing a faulty type conversion that accesses memory incorrectly.

TensorFlow versions before 2.2.1 and 2.3.1 have a memory leak (wasted computer memory that isn't freed) when users pass a list of strings to a function called `dlpack.to_dlpack`. The bug happens because the code doesn't properly check for error conditions during validation, so it continues running even when it should stop and clean up.

TensorFlow versions before 2.2.1 and 2.3.1 have a bug where invalid arguments to `dlpack.to_dlpack` (a function that converts data between formats) cause the code to create null pointers (memory references that point to nothing) without properly checking for errors. This can lead to the program crashing or behaving unpredictably when it tries to use these invalid pointers.

TensorFlow versions before 1.15.4, 2.0.3, 2.1.2, 2.2.1, and 2.3.1 have a bug in the `tf.raw_ops.Switch` operation where it tries to access a null pointer (a reference to nothing), causing the program to crash. The problem occurs because the operation outputs two tensors (data structures in machine learning frameworks) but only one is actually created, leaving the other as an undefined reference that shouldn't be accessed.

This article describes a participant's experience in Microsoft and CUJO AI's Machine Learning Security Evasion Competition, where the goal was to modify malware samples to bypass machine learning models (AI systems trained to detect malicious files) while keeping them functional. The participant attempted two main evasion techniques: hiding data in binaries using steganography (concealing information within files), which had minimal impact, and signing binaries with fake Microsoft certificates using Authenticode (a digital signature system that verifies software authenticity), which showed more promise.

This post discusses backdooring attacks on machine learning models, where an adversary gains access to a model file (the trained AI system used in production) and overwrites it with malicious code. The threat was identified during threat modeling, which is a security planning process where teams imagine potential attacks to prepare defenses. The post indicates it will cover attacks, mitigations, and how Husky AI was built to address this risk.

This post discusses a machine learning attack technique where researchers modify existing images through small changes (perturbations, or slight adjustments to pixels) to trick an AI model into misclassifying them. For example, they aim to alter a picture of a plush bunny so that an image recognition model incorrectly identifies it as a husky dog.

This post is part of a series about machine learning security attacks, with sections covering how an AI system called Husky AI was built and threat-modeled, plus investigations into attacks against it. The previous post demonstrated basic techniques to fool an image recognition model (a type of AI trained to identify what's in pictures) by generating images with solid colors or random pixels.

A researcher tested a machine learning model called Husky AI by creating simple test images (all black, all white, and random pixels) and sending them through an HTTP API to see if the model would make incorrect predictions. The white canvas image successfully tricked the model into incorrectly classifying it as a husky, demonstrating a perturbation attack (where slightly modified or unusual inputs fool an AI into making wrong predictions).