USA News Hub

Join us in Atlanta on April 10th and explore the landscape of security workforce. We will explore the vision, benefits, and use cases of AI for security teams. Request an invite here.

---

During model inversion, attackers reverse-engineer a model’s output to gain access to its training dataset, enabling them to steal sensitive or private information. This kind of privacy breach can be detrimental to an organization’s reputation.

How compromised AI systems impact organizations

If an organization’s AI system is compromised, its entire security posture is affected. Most are aware of this, which is why they’ve taken preventive measures. In fact, six in 10 companies have made an effort to mitigate AI-related cybersecurity risks. This readiness highlights the potential adverse business outcomes.

When attackers successfully manipulate or exploit a model, they can cause unintended effects, compromise connected components and access storage systems without authorization. Their end goal is usually a breach, where they steal sensitive information, intellectual property or personally identifiable details from training datasets.

Depending on the type of data stolen and the industry the affected organization is in, a breach can lead to legal action, regulatory scrutiny and public backlash. Financial losses are the most likely outcome of these scenarios.

Pre-development security considerations to account for

Cybersecurity professionals must remember a few key security considerations when designing their AI systems. Dataset selection is critical, since poisoning attacks and tampering can result in extensive — and potentially permanent — model corruption early on. 

Dataset integrity remains a key security consideration even when AI engineers use pretrained models to train their algorithm since there’s a chance the original is compromised. At the very least, it may have vulnerabilities attackers can easily exploit after deployment. 

Organizations that outsource model design instead of hiring in-house AI engineers must consider how careful and dependable their third party is. Vendors — whether through negligence or maliciousness — can quickly introduce vulnerabilities. Without oversight and occasional inspections, minor weaknesses can become significant security issues. 

Key security considerations for AI system development

An algorithm’s parameters must be updated every time data scientists introduce new information, which can quickly become time- and resource-intensive. They’ll inadvertently create new security weaknesses if they cut corners to accelerate development. 

Even if data scientists and AI engineers don’t deviate from what’s expected, they may still experience security issues during development. AI’s black box problem — its lack of interpretability — can make identifying indicators of compromise (IOCs) challenging. Recognizing dataset poisoning or tampering can be difficult when a model’s output is unexplainable. 

Although most attacks stem from threat actors, professionals shouldn’t rule out disgruntled or negligent workers. Human error is responsible for 95% of cybersecurity incidents, after all. They should be mindful of physical and digital threats during AI system development, regardless of whether development takes place in-house or externally. 

How substandard development impacts deployment

If cybersecurity professionals don’t identify indicators of compromise or recognize signs of corruption before AI system deployment is finalized, they risk facing ongoing, seemingly unexplainable cyberthreats from then on. 

AI engineers preparing for deployment must consider the possible security ramifications of substandard development. If critical vulnerabilities go unnoticed, unintended model behavior and data breaches become likely. Over time, an organization’s security posture is negatively affected.

How to protect AI systems during development

Cybersecurity professionals can improve their resilience by collaborating with data scientists and AI engineers to ensure as few vulnerabilities are introduced during development as possible. Information sharing can significantly improve the effectiveness of threat mitigation efforts. This also means potential issues are identified sooner. 

Although coordinating with other departments can improve threat mitigation, a combination of methods is ideal. Restrictions have the potential to be among the most helpful since they act as a failsafe to prevent attackers from using the system to trigger malicious actions.

Cybersecurity professionals should also consider leveraging model explainability during development to make identifying IOCs easier. This way, they can recognize when dataset poisoning and prompt injection attacks happen sooner.

Naturally, routine audits are a critical part of development’s best practices. AI engineers should collaborate with cybersecurity professionals to thoroughly test their systems. This way, they can identify and eliminate corruption and vulnerabilities before they become an issue.

Cybersecurity professionals should consider preserving a baseline to roll back their AI systems. If all else fails and a compromise occurs, it is one of the few ways to protect the model without spending weeks on incident response and recovery. 

Fundamental guidelines for AI system development

Multiple regulatory bodies have released guidance for secure AI development. The most prominent is the joint guidelines from the U.S., Canada and the UK. These countries partnered with agencies from Australia, Israel, Singapore, Italy, France, Germany, South Korea and Japan, among others. 

Prominent agencies like the Cybersecurity and Infrastructure Security Agency (CISA), National Security Agency (NSA) and the National Cyber Security Center (NCSC) worked together to craft a 20-page document on AI security and responsible development. 

The U.S. and UK governments produced guidance on the secure design, deployment and operation of AI systems aimed at providers. It claims to help reduce the risk organizations face during the development process. While it doesn’t get too technical, it highlights best practices and thoroughly covers developers’ responsibilities.

Proactive effort is ideal when mitigating threats

Cybersecurity professionals should be mindful of the things that could compromise their AI system during development. If they’re proactive, they’ll have a much better chance of eliminating threats and achieving better business outcomes.

Zac Amos is the features editor at ReHack.