What Should Companies Watch Out for When Using AI? A 12 Point Corporate AI Risk and Security Guide

Quick answer: A company that adopts AI gains productivity while taking on a new class of risk. The twelve most critical headings are: shadow AI and data leakage, feeding sensitive data to models, prompt injection, system prompt leakage, the AI supply chain, excessive agency, hallucination, lack of human oversight, copyright and intellectual property, bias, third party risk and, at the top, the governance gap. This guide explains each with a real case and an authoritative source, and gives a concrete countermeasure for each. In one sentence: treat AI not as a capability but as a vendor to be managed and an attack surface to be secured.

In February 2024 a British Columbia tribunal made a landmark ruling. Air Canada's website chatbot had told a passenger he could claim a bereavement discount retroactively; the information was false. The airline argued in court that the chatbot was a separate legal entity responsible for its own words. The tribunal rejected this and held Air Canada liable for the wrong information its own AI gave (Moffatt v. Air Canada, 2024 BCCRT 149). The message is clear: whatever your company's AI says, your company is deemed to have said.

In the same period Samsung engineers pasted confidential source code and notes from an internal meeting into ChatGPT, and Samsung banned generative AI tools on company devices (Bloomberg, May 2023). Two lawyers put six fabricated court decisions invented by ChatGPT into a filing and were fined (Mata v. Avianca, 2023). The picture reveals a paradox: McKinsey reports 78% of organizations use AI in at least one business function, yet ISACA finds only 31% have a comprehensive AI policy. In IBM's 2025 report, 63% of breached organizations had no AI governance policy. This gap is where the most expensive mistakes are born.

This bulletin continues our Roadmap for Companies Using AI: EU AI Act Article 4. The first covered the legal frame of compliance; this second one puts what to watch for in the field in front of you, point by point, with sources.

Data and Privacy

1. Shadow AI and data leakage

Employees paste company data into personal ChatGPT accounts without the organization knowing. This is shadow AI, the most common leakage channel. In Cyberhaven's measurement across 1.6 million workers, 8.6% of employees pasted company data into ChatGPT and 4.7% pasted confidential data at least once; 11% of data pasted into the tools is confidential. IBM's 2025 Cost of a Data Breach Report shows breaches involving shadow AI cost an average of USD 4.63 million, about USD 670,000 more than breaches without it, and one in five breaches involved shadow AI.

What to do: Offer an approved corporate AI tool so employees do not slip to personal accounts. Enable no retention and no training settings in enterprise versions. Monitor confidential content going to AI fields with DLP. Publish a clear usage policy.

2. Which data you may feed a model

Not all data can be given to AI. Customer personal data, health and financial information, source code, contracts and trade secrets must not be entered into an external model uncontrolled. Italy's Garante temporarily banned ChatGPT in 2023 and fined OpenAI EUR 15 million in December 2024 over the same investigation. In Turkey the KVKK follows the topic closely: it published a generative AI guide in 15 questions (November 2025) and a guide specifically on the use of generative AI tools in workplaces (March 2026). The core principles of KVKK and GDPR, above all data minimization and a legal basis, apply to AI too.

What to do: Build an input rules table: which data class may enter which tool and which may never. Apply data minimization, anonymize where possible. Run a data protection impact assessment for high risk uses. Use the table in our EU AI Act roadmap.

3. Legal basis and retention

Every personal data processed with AI needs a legal basis. Where the model stores your input, how long it keeps it and whether it trains on it must be clarified by contract.

What to do: Sign a data processing agreement with the vendor. Audit cross border transfer and retention. Get the no training commitment in writing.

Security: OWASP Top 10 for LLMs

AI applications are open to attacks unlike classic software. OWASP published its 2025 Top 10 for large language model applications. The points below are the corporate translation of that list.

4. Prompt injection

This is the top risk in OWASP's 2025 list (LLM01). An attacker injects commands that change model behavior in unintended ways. In direct prompt injection the user tells the model to ignore previous instructions. The subtler indirect prompt injection has the model read a document, web page or email and execute malicious instructions hidden inside. The EchoLeak attack documented on Microsoft 365 Copilot in 2025 showed that data could be exfiltrated from a corporate AI assistant with only a crafted email and no user click. Unlike SQL injection, prompt injection cannot be reliably blocked with schema validation, because natural language is flexible.

What to do: Limit the data the model can reach and the actions it can take. Treat instructions arriving with external content as untrusted. Validate output before passing it to a system. See autonomous AI agent security.

5. Sensitive information disclosure and system prompt leakage

Models can carry secret information leaking from training data or context into their answers (LLM02). New to the 2025 list, system prompt leakage (LLM07) is the exposure of the application's hidden system instructions and any key, password or business rule inside them.

What to do: Never put secrets, keys or passwords in the system prompt. Manage authorization outside the model with real access control. Filter responses against confidential leakage.

6. The AI supply chain

Open source models, plugins and datasets are a supply chain and can be poisoned (LLM03, LLM04). In 2026 a malicious Hugging Face repository posing as an OpenAI release reached 244,000 downloads before removal and distributed infostealer malware (CSO Online). Earlier, more than a hundred malicious models were found on Hugging Face due to a Pickle file vulnerability. Attackers registering package names that AI coding assistants hallucinate is known as slopsquatting.

What to do: Verify the source of models and libraries with signature and integrity checks. Test third party models in isolation. Scan dependencies. The Guidelines for Secure AI System Development, released by CISA and the UK NCSC with 21 agencies in November 2023, are a good baseline.

7. Excessive agency and improper output handling

Giving an AI agent more power than it needs is OWASP's excessive agency risk (LLM06): if the model can send email, delete files or make payments, a prompt injection hands those powers to an attacker. Passing model output into code, a query or a command without validation is improper output handling (LLM05).

What to do: Apply least privilege; an agent should have only enough power to do its job. Require human approval for high impact actions. Always validate model output as untrusted input.

Accuracy and Liability

8. Hallucination and misinformation

Large language models can confidently invent information that does not exist; OWASP calls this misinformation (LLM09). The Air Canada case showed its corporate cost. So did the lawyers putting fake ChatGPT decisions into a filing; there the real penalty came not from using AI but from using output without verification and failing to withdraw it after being warned.

What to do: Never publish AI output going to a customer or a court without human verification. Force the model to cite sources and check them. We explained why measuring AI agent hallucination is essential in AI forensic agents and hallucination.

9. Human oversight

Article 14 of the EU AI Act legally requires high risk AI systems to be designed so a natural person can effectively oversee them. That person must understand the system's limits, stay alert to automation bias, that is blind trust in AI output, and be able to disregard the output or not use the system at all.

What to do: Keep a human in the loop for high impact decisions. Define approval, override and escalation points in writing. Train staff against automation bias.

Law, Intellectual Property and Fairness

10. Copyright and intellectual property

AI carries copyright risk on both input and output. The New York Times sued OpenAI and Microsoft alleging their models were trained on millions of its articles; in April 2025 the court allowed the core claims to proceed. On the output side, the US Copyright Office reaffirmed that human authorship is required for copyright and that text, images or code generated entirely by AI is not protected (January 2025). So fully AI generated corporate content may not be yours.

What to do: Audit that AI output does not infringe training data copyright. Leave meaningful human contribution in commercially valuable content and code. Add the vendor's copyright indemnity to the contract.

11. Bias and discrimination

AI repeats the bias in its training data at scale. In the US, iTutorGroup programmed its hiring software to automatically reject female applicants over 55 and male applicants over 60, rejected more than 200 qualified candidates, and settled the EEOC's first AI discrimination suit for USD 365,000 (2023).

What to do: Test models for bias in decisions that affect people, such as hiring, credit and insurance. Keep decision rationale explainable. Run discrimination audits regularly.

Governance: The Frame That Holds It All Together

12. Close the governance gap

The single antidote to the eleven points above is a governance framework. In IBM's 2025 report 63% of breached organizations had no AI governance policy, and 97% of those reporting an AI model breach lacked proper AI access controls. Three mature frameworks close the gap: the US NIST AI Risk Management Framework works with four functions, Govern, Map, Measure and Manage; ISO/IEC 42001:2023 is a certifiable AI management system standard; and the EU AI Act sets the legal ground.

What to do: Build an AI usage inventory and classify each use by risk level. Publish an AI policy and assign ownership. Reference NIST AI RMF or ISO 42001. Give staff AI literacy training, already mandatory under EU AI Act Article 4. See AI risk management with NIST AI RMF, ISO 42001 and the EU AI Act.

Quick Checklist

Twelve steps you can start tomorrow morning:

  1. Offer an approved corporate AI tool, close personal account use.
  2. Publish an input rules table: which data goes where.
  3. Sign a data processing agreement, get a no training commitment.
  4. Minimize the model's access and action rights.
  5. Treat instructions from external content as untrusted, test against prompt injection.
  6. Remove all secrets and keys from the system prompt.
  7. Verify third party models and packages by signature, test in isolation.
  8. Require human verification for AI output going to customers.
  9. Keep a human in the loop for high impact decisions.
  10. Audit the copyright and ownership status of AI output.
  11. Run bias tests on decisions that affect people.
  12. Create an AI policy and inventory, reference NIST AI RMF or ISO 42001.

What We Do at DSET

DSET addresses AI security from both defense and offense. Our domestic autonomous engine KAOS scans corporate systems with AI support. Our AI red teaming service tests your models and agents for prompt injection, data exfiltration and excessive agency. The DSET Forensics Benchmark, which measures resistance to hallucination and deception, makes AI agent honesty provable. For a holistic view, see our AI security guide.

Frequently Asked Questions

Should I ban ChatGPT for employees entirely? No, bans usually increase shadow AI. Offering an approved corporate tool, setting clear rules and monitoring leakage is more effective than a ban.

Can my company be held liable for AI output? Yes. The Air Canada case confirmed a company is responsible for the wrong information its own chatbot gives.

Do I own content generated entirely by AI? Usually no. The US Copyright Office requires human authorship; pure AI output is not protected.

I am a small company, is governance still needed? Yes. A policy can start with one page, but inventory, input rules and human verification are needed at every scale.

Where should I start? Start with an AI usage inventory and an input rules table, then plan EU AI Act Article 4 literacy training.

Sources

To make your company's AI use secure and compliant, contact DSET or review our AI security guide.