New tools, products, platforms, funding rounds, and company developments in AI security.
AI and machine learning systems have caused serious problems in real-world situations, including Amazon's recruiting tool that discriminated against women, Microsoft's chatbot that became racist and sexist, IBM's cancer treatment recommendation system that doctors criticized, and Facebook's AI that made incorrect translations leading to someone's arrest. These examples show that AI systems can develop and spread biased predictions and failures with harmful consequences. The article highlights the importance of addressing bias when building and deploying AI systems responsibly.
NAT Slipstreaming is a technique where visiting a malicious website can punch a hole through your router's firewall by exploiting the Application Layer Gateway (ALG, a feature that helps protocols like SIP, Session Initiation Protocol, work with firewalls). The attack works because the ALG is designed to allow devices inside a network to open firewall ports, but an attacker can abuse this intended functionality.
During a Red Team Operation (a simulated attack where security testers try to break into a company's systems), researchers discovered that Blue Team infrastructure, like endpoint agents (software that monitors and controls devices on a network), can be exploited for remote code execution (running commands on systems without authorization) if not properly protected. Companies often lack adequate security controls like MFA (multi-factor authentication, requiring multiple verification steps) and monitoring to prevent unauthorized access to these agents.
Offensive security teams are groups that test and challenge an organization's defenses by simulating attacks from an adversary's perspective. Rather than debating terminology like 'red team' or 'pentest' (security testing where authorized people attempt to break into systems), the source suggests defining these teams by the services they provide to customers within the organization, including business groups, defensive teams, developers, and employees.
The VS Code Python extension had a vulnerability where HTML and JavaScript code could be injected through error messages (called tracebacks, which show where a program failed) in Jupyter Notebooks, potentially allowing attackers to steal user information or take control of their computer. The vulnerability occurred because strings in error messages were not properly escaped (prevented from being interpreted as code), and could be triggered by modifying a notebook file directly or by having the notebook connect to a remote server controlled by an attacker.
Attackers can steal machine learning model files through direct approaches like compromising systems to find model files (often with .h5 extensions), or through indirect approaches like model stealing where attackers build similar models themselves. One specific attack vector involves SSH agent hijacking (exploiting SSH keys stored in memory on compromised machines), which allows attackers to access production systems containing model files without needing the original passphrases.
This item describes a presentation about 'Shadowbunny,' a technique that uses virtual machines (software that simulates a complete computer inside another computer) to hide malware and avoid detection by security tools. The content provided is primarily background information about the presentation's origin and does not detail the actual technical attack or defense mechanisms.
Operationalizing an ML model (putting it into production so it can be used by real applications) involves deploying the trained model to a web server so it can make predictions. The author found that integrating TensorFlow (a popular ML framework) with Golang was unexpectedly complicated, so they chose Python instead for their web server.
Repudiation is a security threat where someone denies performing an action, such as replacing an AI model file with a malicious version. The source explains how to use auditd (a Linux auditing tool) and centralized monitoring systems like Splunk or Elastic Stack to create audit logs that track who accessed or modified files and when, helping prove or investigate whether specific accounts made changes.
Fix: To mitigate repudiation threats, the source recommends: (1) installing and configuring auditd on Linux using 'sudo apt install auditd', (2) adding file monitoring rules with auditctl (example: 'sudo auditctl -w /path/to/file -p rwa -k keyword' to audit read, write, and append operations), and (3) pushing audit logs to centralized monitoring systems such as Splunk, Elastic Stack, or Azure Sentinel for analysis and visualization.
Embrace The RedThis is a YouTube talk about building and breaking machine learning systems, presented at a security conference (GrayHat Red Team Village). The speaker is exploring whether to develop this content into a hands-on workshop where participants could practice these concepts.
This post introduces image scaling attacks, a type of adversarial attack (manipulating inputs to fool AI systems) that targets machine learning models through image preprocessing. The author discovered this attack concept while preparing demos and references academic research on understanding and preventing these attacks.
This post demonstrates how to use the Adversarial Robustness Toolbox (ART, an open-source library created by IBM for testing machine learning security) to generate adversarial examples, which are modified images designed to trick AI models into making wrong predictions. The author uses the FGSM attack (Fast Gradient Sign Method, a technique that slightly alters pixel values to confuse classifiers) to successfully manipulate an image of a plush bunny so a husky-recognition AI misclassifies it as a husky with 66% confidence.
This item is promotional content for a conference talk about attacking and defending machine learning systems, presented at GrayHat 2020's Red Team Village. The speaker created an introductory video for a session titled 'Learning by doing: Building and breaking a machine learning system,' scheduled for October 31st, 2020.
Fix: Microsoft Security Response Center (MSRC) confirmed the vulnerability and fixed it, with the fix released in October 2020 as documented in their security bulletin.
Embrace The RedThis is an announcement for a conference talk about attacking and defending machine learning systems, covering practical threats like brute forcing predictions (testing many inputs to guess outputs), perturbations (small changes to data that fool AI), and backdooring models (secretly poisoning training data). The speaker will discuss both ML-specific attacks and traditional security breaches, as well as defenses to protect these systems.
This article describes a participant's experience in Microsoft and CUJO AI's Machine Learning Security Evasion Competition, where the goal was to modify malware samples to bypass machine learning models (AI systems trained to detect malicious files) while keeping them functional. The participant attempted two main evasion techniques: hiding data in binaries using steganography (concealing information within files), which had minimal impact, and signing binaries with fake Microsoft certificates using Authenticode (a digital signature system that verifies software authenticity), which showed more promise.
This post discusses backdooring attacks on machine learning models, where an adversary gains access to a model file (the trained AI system used in production) and overwrites it with malicious code. The threat was identified during threat modeling, which is a security planning process where teams imagine potential attacks to prepare defenses. The post indicates it will cover attacks, mitigations, and how Husky AI was built to address this risk.
This post discusses a machine learning attack technique where researchers modify existing images through small changes (perturbations, or slight adjustments to pixels) to trick an AI model into misclassifying them. For example, they aim to alter a picture of a plush bunny so that an image recognition model incorrectly identifies it as a husky dog.
This post is part of a series about machine learning security attacks, with sections covering how an AI system called Husky AI was built and threat-modeled, plus investigations into attacks against it. The previous post demonstrated basic techniques to fool an image recognition model (a type of AI trained to identify what's in pictures) by generating images with solid colors or random pixels.
A researcher tested a machine learning model called Husky AI by creating simple test images (all black, all white, and random pixels) and sending them through an HTTP API to see if the model would make incorrect predictions. The white canvas image successfully tricked the model into incorrectly classifying it as a husky, demonstrating a perturbation attack (where slightly modified or unusual inputs fool an AI into making wrong predictions).
This post explains threat modeling for machine learning systems, which is a process to systematically identify potential security attacks. The author uses Microsoft's Threat Modeling tool and STRIDE (a framework categorizing threats into spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege) to identify vulnerabilities in a machine learning system called 'Husky AI', and notes that perturbation attacks (where attackers query the model to trick it into making wrong predictions) are a particular concern for ML systems.