What AI Can Learn from Aviation Safety
In a recent security test, a generative AI banking chatbot, crafted to support customers in loan applications, was manipulated to disclose sensitive financial data. Testers managed to circumvent security controls, extracting a comprehensive array of loan approvals, complete with customer names.
This cautionary tale highlights a fundamental issue: while generative AI could potentially revolutionize entire industries, its lack of solid safety protocols poses potentially disastrous outcomes. The traditional safety models no longer suffice. Transformative technologies like generative AI demand a new, holistic approach to cybersecurity.
Aviation provides a valuable model. Much like supersonic jets, generative AI is a transformational technology with immense potential. However, without trained operators, well-crafted systems, and robust safeguards, the risk of catastrophic failure cannot be ignored.
Thanks to rigorous safety protocols, air travel has evolved into one of the safest transportation modes. Similarly, the potential of AI is undeniable, but its future relies heavily on addressing inherent safety risks. A recent BCG study indicated that three-quarters of business executives view cybersecurity as a significant obstacle to AI scalability.
Unlike traditional software, generative AI operates based on probabilities, leading to unpredictable outcomes. Large language models (LLMs) exhibit indeterministic behaviour, creating cybersecurity blind spots. The dependency on natural language inputs, adaptive learning, and extensive integrations with various tools and services makes these models uniquely vulnerable.
Just as aviation requires a comprehensive, multifaceted approach to safety, cybersecurity must be embedded at every layer of AI. This includes everything from its architecture to data management and human oversight. Without such a foundation, the future of AI remains uncertain.
AI Vulnerabilities: Risks and Responses
A primary vulnerability of AI systems is the prompt-injection attacks, allowing attackers to manipulate a model into divulging sensitive data or altering its decision-making logic.
The recent banking chatbot test exposed an equally alarming risk: privilege escalation. Here, testers impersonated an administrator, approving unauthorized loans and manipulating backend data.
In healthcare, AI assistants have been similarly compromised. Security researchers successfully extracted confidential patient records by subtly rephrasing their queries. Instead of directly requesting medical histories, attackers framed their requests to mimic legitimate doctor inquiries. This exploit revealed a significant weakness: AI often prioritizes linguistic logic over access controls.
These vulnerabilities extend well beyond the sectors of banking and healthcare. Many AI applications utilize agentic systems that autonomously make decisions based on real-time data, creating further opportunities for exploitation. For example, a security assessment of an AI-driven customer service chatbot indicated that attackers exploited a weak API validation to manipulate an LLM into exposing internal discount codes and inventory details.
AI’s adaptability opens other avenues for exploitation through phenomena like context poisoning. By gradually influencing a model’s responses over time, attackers can guide its suggestions toward incorrect or even dangerous recommendations. In one cited experiment, a spa chatbot, constantly fed with inputs that framed unsafe ingredients as beneficial, began endorsing harmful skincare products.
As AI systems inundate traditional infrastructure with automated requests, they can lead to systemic collapse—a phenomenon known as legacy contamination. Avoiding this outcome necessitates organizations implementing adversarial training, continually exposing AI models to deceptive inputs to fortify their resilience.
Real-time anomaly detection—both automated and manual—can intercept unusual AI behaviours before manipulated data influences its outputs. Just like flight-control systems trust in independent backups, generative AI security must be anchored with layered safeguards. This includes automated anomaly detection to flag irregular actions, redundant access validation to thwart unauthorized interactions, and real-time rollback mechanisms to undo harmful changes.
Embedding Security in AI Development
While global spending on AI is expected to exceed $631bn by 2028, many such investments could fail to yield significant returns unless foundational cybersecurity challenges are handled. Most critically, AI security must evolve beyond being an “add-on” feature to become a core component, fully embedded in system architectures, data management, and human oversight. An effective security framework must be fluid, resilient, and integrated into legacy systems.
Even industry giants face design challenges, emphasizing the need for stronger security measures. In March 2023, OpenAI discovered a bug in an open-source library. This inadvertently exposed ChatGPT users’ payment information, sending confirmation emails to unintended recipients.
AI security must progress in tandem with the systems it seeks to protect. Effective data management transcends merely securing pipelines and datasets. It requires a definitive strategy that treats data as a competitive advantage while evaluating what data to disclose and leverage.
Operational oversight is equally crucial. Cybersecurity should not remain isolated to specialists. It must be woven into every department and workflow, with real-time monitoring and adaptive feedback loops helping institutions stay ahead of emerging threats and vulnerabilities.
Apart from leveraging cutting-edge technology, nurturing a culture of vigilance is essential in cybersecurity. A 2024 Verizon report highlights that 68% of data breaches involved human elements, such as fallacies to phishing attacks or social engineering. To mitigate these risks, employees must not only identify threats but also learn how to counter them appropriately. Even straightforward initiatives like regular security training and clear reporting mechanisms can make a substantial impact.
Analogous to how aviation secured public trust through rigorous safety protocols, the AI industry must establish strict protections against hallucinations, manipulation, hacking, and latency dilemmas before they result in real-world harms. This requires a comprehensive strategy that weaves together architecture, engineering, data strategy, and responsible AI. Companies embracing security at every layer of their AI strategy will flourish, whereas those persisting with outdated security frameworks will fall behind.
