AI Jailbreaking and Prompt Injection at Enterprise Scale

People interact with Artificial Intelligence (AI) in a number of ways, but notably, written prompts are the main method because basic prompt hacking is understood. Now, let's talk about sophisticated attacks targeting enterprise AI systems. These considerations will explain how an attacker can weaponize AI assistants to extract proprietary data, manipulate business decisions, or pivot through corporate networks. 

Before we dive in, it’s important to familiarize yourself with the following terms: 

• AI Jailbreaking: AI jailbreaking is the act of bypassing the internal controls, or guardrails, within AI to have it perform outside of expected tolerances. In other words, an attacker is capable of having AI perform actions that it was not designed to do. 
• Prompt Injection: Prompt injection is the use of legitimate-looking prompts to disguise malicious inputs 
• Prompt Smuggling: This advanced form of injection hides malicious instructions inside non-obvious carriers, such as emojis, images, or hyperlinks.  

The above concepts are all similar, but not the same. Each are exploits used by attackers to make AI do something malicious. For AI Jailbreaking, this is more akin to getting a car to turn off anti-lock brakes and disable other safety features so that it can be driven recklessly.  

Prompt injection would be similar to manipulating the manual controls (i.e., hand-pulled emergency break for a j-turn) or electronic controls (i.e., turning off traction control) to drive recklessly.  

With prompt smuggling, an emoji string like "🍎🍊🍌" could be interpreted by the model to mean "delete logs," after going through a decoding layer. Additionally, attackers can embed hidden or invisible Unicode characters within emojis. While a user sees a harmless smiley face, the AI reads the invisible Unicode command, which can contain instructions to manipulate its behavior. 

For a deeper overview of the differences between AI jailbreaking and prompt injection, reference Snyk’s article here. For details on prompt smuggling, check out this ISOsecu blog post. 

Who Is Impacted by AI Jailbreaking and Prompt Injection 

Almost everyone is affected by these sophisticated attacks. In 2024,  72% of organizations were using AI tools with 92% planning on investing money in AI over the next three years. This is quite an attack surface area, making users vulnerable as many of these AI systems handle sensitive data, may involve financial details, and may even perform operational controls.  

Given the scope, the compromise of AI systems gives attackers information or access that classical incident response frameworks are inadequate to address. The rate of growth in AI adoption also means a larger unprotected attack surface, due to the lag in both technical countermeasures and awareness among technologists and users. 

Leveling Up: From Basic Prompts to Data Extraction 

AI, built to respond, is going to give you an answer to prompts, like the example below. 

Most AI systems ignore these types of attacks as they are well known, however, there are other methods being attempted, such as: 

• Persistent Context Poisoning: Contrary to when you speak with a friend and they remember what you said last night, most AI tools are set to forget. However, finding a way to get details to persist can alter the perspective of AI, giving attackers excessive information or allowing them to have AI perform operations that it was not intended to do. 

• API-Based Continuing Campaigns: These are large-scale, automated attempts to exploit an AI interface. Think of this as website scanning or brute-forcing a password, but with prompts to see what can get through. 

• Integration with Social Engineering: AI is used to improve reconnaissance about organizations and targets. When an organization’s AI is compromised, the amount of information provided can provide attackers with far more details to deceive people. Imagine how much easier it will be to perform a phishing campaign when these details are available. 

How an AI Assistant Can Be Exploited in Stages 

It may be hard to believe, but these attacks happen regularly in real world practices. Consider an attacker who gained access to a company’s internal AI assistant by way of a compromised employee account. The AI assistant, through authorized access, can begin to interact with the attacker over a series of days in a multi-state attack, which can look like the following: 

Stage 1 - Context Seeding (Day 1):

Stage 2 - Trust Building (Days 7-10):

Stage 3 - Escalation (Days 18-24):

Stage 4 - Exploitation (Days 30+):

The result of these actions provided the attacker with details such as: 

• Database schema, including field names and table relationships 
• Validation patterns that expose the data format(s) of account numbers 
• Whether the provided account numbers exist in the system (information disclosure) 
• Partial or whole data, even if the masking function has exceptions for "audit mode" 

If you are not used to thinking like an attacker, the severity of this may not resonate. Let me give a parallel about someone trying to steal your identity. Getting your phone number is easy enough. Accessing an address is a touch more difficult. By the time an attacker convinces you (or an authorized source) to provide your social security number, they are well on their way to exploiting existing bank accounts in your name, all while impersonating you. 

Attacks in Action 

This attack vector is comparatively new, but the implications are real and significant. The delivery company DPD needed to disable its chatbot after it began swearing. ChatGPT needed to go offline after exposing the chat history of other users. An attack using Microsoft Copilot required no action on the victim, meaning the victim did not need to download or even click on anything– surprisingly, they just received an email which weaponized zero-click exploit. This instance was identified in January 2025 and patched in June of 2025. 

This seems comparatively harmless but now apply that concept to the usage of ChatGPT to generate malware. The ability for a larger count of people to obtain malicious code to launch attacks is certainly not the intent of OpenAI and runs counter to the guardrails in place. 

These details are not helped by the fact that reporting and attributing attacks on AI are difficult individual problems. No organization wants to disclose when their AI does not perform as expected and provides details to an attacker as a result. For the latter, the ability to discern correct usage from malicious usage and the scrutiny of all outputs is a logistical behemoth – because the attack is intended to look like normal usage. 

Actions To Take Against AI Jailbreaking and Prompt Injection 

This is not an easy problem to solve and there is still a lot of progress to be made. According to IBM's 2024 X-Force report, AI-related security incidents increased 238% year-over-year, yet most organizations still lack AI-specific security controls. 

As with many attack vectors in the cat-and-mouse game of hacking, the concepts must be exploited in order for providers to detect them – and detecting the attacks is very difficult. The ability for input validation in traditional concepts does not easily apply to AI models interacting with common speech. 

The good news is that organizations can learn about what makes them vulnerable, and a functional solution is to limit what data the AI system has access to. This concept can be applied in the same way that your doctor or other medical personnel can gain access to your medical records because they have a need to know, yet you would not want that personal information to be disclosed to everyone, or even to other medical professionals not working on your care team. 

If you’re interested in learning more, Schellman is hosting an upcoming educational summit on AI at our headquarters in Tampa. Click here to register today.