INCREASING AUTONOMY OF AI

“The increasing autonomy of AI agents creates security risks beyond conventional chatbots.” Discuss the major risks and suggest measures for their safe deployment. (15 marks, 250 words)

Model Answer

Introduction:

AI agents can autonomously plan, decide and interact with external systems, making them more capable—but also more vulnerable to misuse and unintended consequences.

Major Security Risks:

  • Prompt injection: Hidden instructions in webpages/documents can manipulate agent behaviour.
  • Decision-making errors: Faulty reasoning may lead to unintended actions.
  • Excessive permissions: Access to emails, code, finances or databases can magnify the impact of errors.
  • Cascading risks: Interaction with other software/AI agents can spread failures across systems.
  • Hallucinations: Incorrect but convincing outputs can cause misinformation and poor decisions.
  • Unpredictability: Autonomous behaviour is harder to anticipate and control.

Way Forward:

  • Adopt secure-by-design architecture and least-privilege access.
  • Establish strong AI governance and accountability frameworks.
  • Conduct continuous red-teaming and real-world security testing.
  • Maintain audit trails for traceability.
  • Ensure human oversight for high-risk decisions.
  • Strengthen protection against prompt injection, data breaches and model vulnerabilities.

Conclusion:

  • AI autonomy should be matched with proportionate safeguards, transparency and human oversight to ensure that AI agents remain powerful yet controllable and accountable.