AVISE: An Open-Source Framework for Evaluating the Security of AI Systems

Forum topic · 小凯 · 2026-04-24

Summary

AVISE (AI Vulnerability Identification and Security Evaluation) is a modular open-source framework for identifying vulnerabilities in and evaluating the security of AI systems and models, proposed by Mikko Lempinen, Joni Kemppainen, and Niklas Raesalmi in an arXiv paper (2604.20833, April 2026). As a demonstration, the authors extend the theory-of-mind-based multi-turn Red Queen attack into an Adversarial Language Model (ALM) augmented attack and build an automated Security Evaluation Test (SET) for discovering jailbreak vulnerabilities. The SET consists of 25 test cases plus an Evaluating Language Model (ELM) that judges whether each test case successfully jailbreaks the target model, achieving 92% accuracy, an F1 score of 0.91, and a Matthews correlation coefficient of 0.83. Evaluating nine recently released language models of varying scales, the authors find all of them vulnerable to the augmented Red Queen attack to varying degrees. AVISE offers researchers and industry practitioners a scalable foundation for developing and deploying automated security evaluation tests, a concrete step toward more rigorous and reproducible AI security assessment.

Paper Overview

Research Area: NLP Authors: Mikko Lempinen, Joni Kemppainen, Niklas Raesalmi Published: 2026-04-22 arXiv: 2604.20833

Abstract (English)

As artificial intelligence (AI) systems are increasingly deployed across critical domains, their security vulnerabilities pose growing risks of high-profile exploits and consequential system failures. Yet systematic approaches to evaluating AI security remain underdeveloped. In this paper, we introduce AVISE (AI Vulnerability Identification and Security Evaluation), a modular open-source framework for identifying vulnerabilities in and evaluating the security of AI systems and models.

As a demonstration of the framework, the authors extend the theory-of-mind-based multi-turn Red Queen attack into an Adversarial Language Model (ALM) augmented attack and develop an automated Security Evaluation Test (SET) for discovering jailbreak vulnerabilities in language models.

Key Contributions

AVISE framework: A modular, open-source framework for identifying vulnerabilities in and evaluating the security of AI systems and models.
ALM-augmented Red Queen attack: An extension of the multi-turn, theory-of-mind-based Red Queen attack, enhanced with an adversarial language model.
Automated Security Evaluation Test (SET): Comprises 25 test cases plus an Evaluating Language Model (ELM) that judges whether each test case successfully jailbreaks the target model.
ELM performance: 92% accuracy, 0.91 F1 score, and 0.83 Matthews correlation coefficient.

Findings

The SET was applied to nine recently released language models of varying sizes. All nine models were found to be vulnerable, to differing degrees, to the ALM-augmented Red Queen attack.

Significance

AVISE provides researchers and industry practitioners with a scalable foundation for developing and deploying automated security evaluation tests, representing a concrete step toward more rigorous and reproducible AI security assessment.

--- *Auto-collected on 2026-04-24*