AI Sec Watch

Generative AI (systems that create new text, images, or other content) is transforming many industries but raises ethical concerns like data privacy (protecting personal information), bias (unfair treatment of certain groups), transparency (being open about how the AI works), and accountability (responsibility for the AI's actions). Researchers propose a trust framework based on transparency, fairness, accountability, and privacy to help ensure generative AI is developed and used responsibly.

Large language models (LLMs, AI systems trained on huge amounts of text to generate human-like responses) can now mimic not just general human language but also unusual, individual-specific human behaviors. This ability could lead to LLMs being used more widely in research studies and potentially reduce the role of actual humans, which raises concerns about AI alignment (ensuring AI systems behave in ways humans intend and approve of) and how this technology affects society.

This research addresses a problem in federated learning (a method where multiple computers train an AI model together without sharing raw data) combined with adversarial training (a technique that makes AI models resistant to intentionally tricky inputs). The authors found that simply combining these two approaches causes the model's accuracy to drop because adversarial training increases differences in the data across different computers, making the federated learning less effective. They propose SFAT (Slack Federated Adversarial Training), which uses a relaxation mechanism to adjust how the computers combine their learning results, reducing the harmful effects of data differences and improving overall performance.

This research proposes SIAMD, a framework for detecting social media bots (automated accounts that spread misinformation) before they cause harm. The system analyzes patterns in how user accounts interact with messages, uses structural entropy (a measure of uncertainty in data patterns) to identify bot-like behavior, and generates synthetic bot messages with large language models (AI systems trained on text data) to test and improve detection systems.

Federated Learning (FL, a method where multiple computers train an AI model together without sharing raw data) can leak private information through gradient inversion attacks (GIA, techniques that reconstruct sensitive data from the mathematical updates used in training). This paper reviews three types of GIA methods and finds that while optimization-based GIA is most practical, generation-based and analytics-based GIA have significant limitations, and proposes a three-stage defense pipeline for FL frameworks.

Langflow, a tool for building AI-powered agents and workflows, has a vulnerability in versions before 1.7.0 where an attacker can specify any file path in a request to create or overwrite files anywhere on the server. The vulnerability exists because the server doesn't restrict or validate the file paths, allowing attackers to write files to sensitive locations like system directories.

Langflow, a tool for building AI-powered agents and workflows, has a vulnerability in versions before 1.7.0 where its API Request component can make arbitrary HTTP requests to internal network addresses. An attacker with an API key could exploit this SSRF (server-side request forgery, where a server is tricked into making requests to unintended targets) to access sensitive internal resources like databases and metadata services, potentially stealing information or preparing further attacks.

The article argues that while AI language models (LLMs, systems trained on large amounts of text to generate responses) and traditional programming languages both increase abstraction, they differ fundamentally in a critical way: compilers are deterministic (they reliably produce the same output every time), while LLMs are nondeterministic (they produce different outputs for the same input). This matters for software security and correctness because compilers preserve the programmer's intended meaning through the translation process, but LLMs cannot guarantee they will generate code that does what you actually need.

The AIBOM Generator, an open-source tool that creates an AI Software Bill of Materials (AIBOM, a structured document listing key information about an AI model like its data sources and configurations), has been moved to OWASP (a nonprofit focused on software security) to enable broader community collaboration and development. The tool helps organizations understand what's inside AI models, where they came from, and how trustworthy their documentation is, addressing a gap between rapid AI adoption and lagging transparency practices. The project is now part of the OWASP GenAI Security Project and will continue improving AI supply chain visibility through community-driven enhancements.