人工智能黑客
AI 安全资源

人工智能系统威胁

AI特定漏洞和攻击向量的综合目录

严重威胁

可能造成严重影响的直接风险。需要紧急修复。

高风险

应及时解决严重漏洞以减少暴露。

防御策略

减少AI威胁暴露的最佳实践和缓解措施。

威胁类别

Prompt Injection

Critical

Crafted inputs designed to manipulate model behavior, override safeguards, or extract sensitive information.

测试方法
  • Craft adversarial prompts with hidden instructions or special characters
  • Attempt multi-turn injection chaining
  • Test for jailbreak bypass of alignment filters
  • Evaluate output sanitization and safety layers

Training Data Poisoning

Critical

Malicious or biased data introduced into training pipelines, compromising model integrity and reliability.

测试方法
  • Analyze data provenance and supply chain
  • Inject poisoned samples and assess downstream effects
  • Test resilience to mislabeled or manipulated data
  • Review validation and anomaly detection mechanisms

Model Inversion

High

Reconstructing training data or sensitive attributes from model outputs, leading to privacy breaches.

测试方法
  • Attempt to recover representative training samples
  • Test susceptibility to membership inference attacks
  • Evaluate differential privacy protections
  • Assess risk of leaking PII from embeddings

Adversarial Examples

High

Inputs intentionally perturbed to cause misclassification, hallucinations, or other erroneous outputs.

测试方法
  • Generate gradient-based adversarial examples
  • Apply noise and perturbation attacks
  • Check model consistency across variations
  • Evaluate robustness against transfer attacks

Model Stealing

High

Extraction of model functionality or parameters through repeated queries or side-channel analysis.

测试方法
  • Simulate query-based model extraction
  • Analyze API rate limits and response variability
  • Check for fingerprinting vulnerabilities
  • Test throttling and monitoring protections

Data Memorization Leakage

High

Sensitive information unintentionally memorized by AI models, retrievable via crafted prompts.

测试方法
  • Probe for known secret patterns in outputs
  • Test for repeated exposure of sensitive training data
  • Assess risk of accidental PII disclosure

Model Misuse & Malicious Automation

High

AI leveraged to perform tasks outside intended scope, enabling social engineering, spam, or automated attacks.

测试方法
  • Simulate misuse scenarios using sandbox models
  • Test AI output moderation and guardrails
  • Assess monitoring alerts for abnormal behaviors

测试和防御最佳实践

安全测试环境

  • 使用沙箱或副本实例进行测试
  • 切勿在生产系统上执行未经授权的测试
  • 实施监控、日志记录和回滚功能

文档和可观察性

  • 维护详细的测试日志和证据
  • 捕获模型响应以实现可重复性
  • 对测试用例进行标记、分类和组织以供将来审核

法律与道德合规性

  • 保持在授权范围和合同范围内
  • 尊重数据保护、隐私法和知识产权
  • 遵循负责任的披露和协调的漏洞披露

监控和缓解

  • 对异常人工智能输出实施异常检测
  • 定期检查速率限制、API 访问和查询模式
  • 集成关键威胁的实时警报
AH
AI Hacking Team

The AI Hacking team researches and documents AI/LLM security vulnerabilities, red teaming techniques, and defensive strategies. Our guides are based on real-world pentesting experience and continuous monitoring of the AI security landscape.

Frequently Asked Questions

What are the most common AI/LLM security threats?
The most common AI/LLM security threats include prompt injection (direct and indirect), data poisoning, model theft and extraction, supply chain attacks, denial of service against model endpoints, insecure output handling, and sensitive information disclosure through LLM responses.
What is prompt injection and how does it work?
Prompt injection is an attack where malicious input is crafted to override or bypass an AI system's safety instructions. Direct prompt injection targets the system prompt, while indirect prompt injection embeds instructions in external content like web pages or documents that the AI processes.
What is model theft and data poisoning in AI systems?
Model theft involves extracting an AI model's parameters, architecture, or training data through API queries or side-channel attacks. Data poisoning occurs when an attacker injects malicious data into the training set, causing the model to learn incorrect behaviors or backdoors that can be triggered later.
How can organizations mitigate AI supply chain attacks?
Organizations can mitigate AI supply chain attacks by vetting model sources and verifying checksums, scanning models for backdoors using tools like ModelScan, implementing strict dependency pinning, monitoring for typosquatting packages, and maintaining a software bill of materials (SBOM) for AI components.

Stay Ahead of AI Security

Get the latest AI/LLM security research, OWASP updates, and new vulnerabilities delivered straight to your inbox.