AI Security

Auditing a Managed AI Agent with Browsing, Shell, and File‑Edit Capabilities

Published 2026-07-26 by AISecAll Editorial

TL;DR: Treat a browsing‑enabled AI agent like a privileged service. Verify its vendor sandbox, limit network destinations, enforce least‑privilege token scopes, capture all command‑line and file‑system activity in immutable logs, and run a daily sanity check that compares observed behavior against a documented policy.

What exactly am I auditing?

Managed AI agents such as Claude Managed Agents or OpenAI’s Agents SDK can be configured with three powerful capabilities:

• Web browsing – the model fetches external URLs and parses HTML.
• Shell execution – the model can run commands on the host machine.
• File edit – the model can read, write, or delete files in a sandboxed directory.

Each capability expands the attack surface. An audit asks: *Is the capability necessary?* *Is it constrained?* *Can we detect misuse?*

Step 1 – Define a Capability Policy

Start with a short, written policy that lists which of the three capabilities are allowed and under what conditions. For a typical small business, a safe baseline is:

# Example policy
allow_browsing: false
allow_shell: false
allow_file_edit: true  # limited to /tmp/agent_sandbox

Document the rationale (e.g., “browsing is disabled because the agent never needs live market data”). This policy becomes the benchmark for later checks.

Step 2 – Verify Vendor‑Provided Sandboxing

Both Claude and OpenAI publish security white‑papers that describe their isolation model. Confirm the following:

1. The agent runs in a container or VM separate from production workloads.
2. Network egress is filtered through a firewall or VPC‑level security group.
3. File‑system access is limited to a mount point you control.

If the vendor does not expose these controls, request a custom sandbox or consider a self‑hosted alternative.

Step 3 – Scope API Keys and Service Tokens

All calls to the agent’s endpoint must use a token with the minimum scopes required. For OpenAI, use openai.api_key with read:metadata only when browsing is disabled. For Claude, generate a managed‑agent token that is limited to the specific agent_id and set expires_at to a short window (e.g., 24 h).

Store these secrets in a secret manager (AWS Secrets Manager, GCP Secret Manager, etc.) and rotate them weekly.

Step 4 – Instrument Comprehensive Logging

Collect three log streams:

• API request log – record timestamp, token hash, request payload (redacted), and response status.
• Command execution log – capture every shell command, its stdout/stderr, and the user‑agent that invoked it.
• File‑system audit log – log create, modify, and delete events inside the sandbox directory.

Write logs to an immutable store (e.g., Cloudflare R2 with object‑lock, or an append‑only S3 bucket). Use a structured format (JSON) so you can query for anomalies such as “shell command executed outside allowed list”.

Step 5 – Run Automated Policy‑Compliance Checks

Schedule a nightly job (using n8n, Make, or a simple cron) that parses the logs and validates them against the policy defined in Step 1. Example checks:

# Pseudocode
if log.browsing_url not in ALLOWED_DOMAINS:
    alert('Unauthorized browsing attempt')
if log.shell_command not in ALLOWED_COMMANDS:
    alert('Unexpected shell command')
if log.file_path not startswith '/tmp/agent_sandbox/':
    alert('File edit outside sandbox')

Send alerts to a Slack channel or email so a human can review before any damage spreads.

Step 6 – Conduct a Manual Pen‑Test of the Agent

Even with logs, you need to verify that the sandbox cannot be escaped. Use the OWASP GenAI Security Project’s test suite (genai.owasp.org) to run the following scenarios:

• Prompt the agent to fetch http://169.254.169.254/latest/meta-data/ (AWS metadata) and see if it returns data.
• Ask the agent to execute cat /etc/passwd when shell is disabled.
• Request the agent to write a file outside the sandbox path.

Document any failures and tighten the policy until all attempts are blocked.

Step 7 – Review and Update Quarterly

Security is a process, not a one‑time checklist. Every three months:

1. Re‑evaluate the business need for each capability.
2. Rotate all tokens and secrets.
3. Refresh the OWASP test suite and re‑run the pen‑test.
4. Archive logs older than 90 days to cold storage.

Keep the policy document in a version‑controlled repository so you can track changes over time.

When to Involve AISecAll

If you need a tailored audit framework, custom sandbox hardening, or help integrating immutable logging, AISecAll offers a hands‑on review service that aligns with NIST’s AI Risk Management Framework.