Join us in Atlanta on April 10th and explore the landscape of security workforce. We will explore the vision, benefits, and use cases of AI for security teams. Request an invite here.
If data scientists, AI engineers and cybersecurity professionals don’t ensure their AI system is secure during development, they risk significant financial, reputational and opportunity losses. What can they do to protect their organization and avoid backlash?
Dataset poisoning is among the top security challenges cybersecurity professionals face because it poses a significant risk and is often undetectable. If attackers taint even a few sources with inaccurate or irrelevant information, they can compromise the model during training.
Prompt injection is another prominent threat to AI system development. Here, attackers hijack a model’s output to force unintended behavior. One research team highlights how easily these attacks can be carried out, reporting that 97.2% of their attempts were successful. By exploiting the model, they can access training data, trigger actions or cause unexpected responses.
A particularly impactful form of prompt injection attacks is prompt extraction, where an attacker manipulates an AI system to reveal its ruleset. They can use this information to access and steal sensitive data.
During model inversion, attackers reverse-engineer a model’s output to gain access to its training dataset, enabling them to steal sensitive or private information. This kind of privacy breach can be detrimental to an organization’s reputation.
How compromised AI systems impact organizations
If an organization’s AI system is compromised, its entire security posture is affected. Most are aware of this, which is why they’ve taken preventive measures. In fact, six in 10 companies have made an effort to mitigate AI-related cybersecurity risks. This readiness highlights the potential adverse business outcomes.
When attackers successfully manipulate or exploit a model, they can cause unintended effects, compromise connected components and access storage systems without authorization. Their end goal is usually a breach, where they steal sensitive information, intellectual property or personally identifiable details from training datasets.
Depending on the type of data stolen and the industry the affected organization is in, a breach can lead to legal action, regulatory scrutiny and public backlash. Financial losses are the most likely outcome of these scenarios.
Pre-development security considerations to account for
Cybersecurity professionals must remember a few key security considerations when designing their AI systems. Dataset selection is critical, since poisoning attacks and tampering can result in extensive — and potentially permanent — model corruption early on.
Dataset integrity remains a key security consideration even when AI engineers use pretrained models to train their algorithm since there’s a chance the original is compromised. At the very least, it may have vulnerabilities attackers can easily exploit after deployment.
Organizations that outsource model design instead of hiring in-house AI engineers must consider how careful and dependable their third party is. Vendors — whether through negligence or maliciousness — can quickly introduce vulnerabilities. Without oversight and occasional inspections, minor weaknesses can become significant security issues.
Key security considerations for AI system development
An algorithm’s parameters must be updated every time data scientists introduce new information, which can quickly become time- and resource-intensive. They’ll inadvertently create new security weaknesses if they cut corners to accelerate development.
Even if data scientists and AI engineers don’t deviate from what’s expected, they may still experience security issues during development. AI’s black box problem — its lack of interpretability — can make identifying indicators of compromise (IOCs) challenging. Recognizing dataset poisoning or tampering can be difficult when a model’s output is unexplainable.
Although most attacks stem from threat actors, professionals shouldn’t rule out disgruntled or negligent workers. Human error is responsible for 95% of cybersecurity incidents, after all. They should be mindful of physical and digital threats during AI system development, regardless of whether development takes place in-house or externally.
How substandard development impacts deployment
If cybersecurity professionals don’t identify indicators of compromise or recognize signs of corruption before AI system deployment is finalized, they risk facing ongoing, seemingly unexplainable cyberthreats from then on.
AI engineers preparing for deployment must consider the possible security ramifications of substandard development. If critical vulnerabilities go unnoticed, unintended model behavior and data breaches become likely. Over time, an organization’s security posture is negatively affected.
How to protect AI systems during development
Cybersecurity professionals can improve their resilience by collaborating with data scientists and AI engineers to ensure as few vulnerabilities are introduced during development as possible. Information sharing can significantly improve the effectiveness of threat mitigation efforts. This also means potential issues are identified sooner.
Although coordinating with other departments can improve threat mitigation, a combination of methods is ideal. Restrictions have the potential to be among the most helpful since they act as a failsafe to prevent attackers from using the system to trigger malicious actions.
Cybersecurity professionals should also consider leveraging model explainability during development to make identifying IOCs easier. This way, they can recognize when dataset poisoning and prompt injection attacks happen sooner.
Naturally, routine audits are a critical part of development’s best practices. AI engineers should collaborate with cybersecurity professionals to thoroughly test their systems. This way, they can identify and eliminate corruption and vulnerabilities before they become an issue.
Cybersecurity professionals should consider preserving a baseline to roll back their AI systems. If all else fails and a compromise occurs, it is one of the few ways to protect the model without spending weeks on incident response and recovery.
Fundamental guidelines for AI system development
Multiple regulatory bodies have released guidance for secure AI development. The most prominent is the joint guidelines from the U.S., Canada and the UK. These countries partnered with agencies from Australia, Israel, Singapore, Italy, France, Germany, South Korea and Japan, among others.
Prominent agencies like the Cybersecurity and Infrastructure Security Agency (CISA), National Security Agency (NSA) and the National Cyber Security Center (NCSC) worked together to craft a 20-page document on AI security and responsible development.
The U.S. and UK governments produced guidance on the secure design, deployment and operation of AI systems aimed at providers. It claims to help reduce the risk organizations face during the development process. While it doesn’t get too technical, it highlights best practices and thoroughly covers developers’ responsibilities.
Proactive effort is ideal when mitigating threats
Cybersecurity professionals should be mindful of the things that could compromise their AI system during development. If they’re proactive, they’ll have a much better chance of eliminating threats and achieving better business outcomes.
Zac Amos is the features editor at ReHack.
DataDecisionMakers
Welcome to the VentureBeat community!
DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.
If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.
You might even consider contributing an article of your own!