Naive approach to create AI agent that fails

Łukasz Kużyński ·

Naive approach to create AI agent that fails

Building an AI agent often starts with a few experiments on your own machine.

You choose a model, set its temperature, and write a prompt. After trying a handful of inputs, you get results that look good enough to ship.

The next day, the agent marks a routine question as urgent and sends a real outage to the wrong team. A feature that was supposed to reduce support work is now creating more of it.

Take a support-ticket triage agent. Its job seems straightforward: read a ticket, choose a team and priority, then call assignTicket to update it.

A first version might look like the SDK-neutral pseudocode below. Here, assignTicket is an application tool that updates the ticket.

await runAgent({
    model,
    temperature: 0.2,
    instructions: `
    Triage the support ticket by calling assignTicket exactly once.
    Send outages and service failures to infrastructure.
    Send invoices and payment questions to billing.
    Send how-to questions to support.
    Use urgent priority for active production outages.
    Use normal priority otherwise.
  `,
    input: ticket,
    tools: {assignTicket},
});

You try “Production is down,” and the agent correctly assigns the ticket to infrastructure with urgent priority. An invoice question goes to billing with normal priority, exactly as you expected.

These examples work, but your users will describe their problems in ways you haven't tested.

A few successful tests leave a lot untested

When testing locally, it's easy to stick to a few straightforward cases. You know them before writing the prompt, and you already have the expected decisions in mind. Real tickets soon expose what those examples missed.

A user writes, “Production is fine. How do I prevent downtime?” The agent picks up on “production” and “downtime” and marks the ticket urgent. Someone interrupts their work to handle a question that could have waited.

Meanwhile, “Nobody can sign in since deployment” gets normal priority. The user described an active outage without using the word “outage,” and the agent missed the urgency. A real incident sits in the queue.

The surrounding system changes too. Teams take on different responsibilities, the provider updates a model, or a tool returns an error. Handling a ticket can take longer than expected, with retries adding to the cost.

Lowering the temperature won't fix these problems. That setting affects output variation, but it doesn't establish whether the agent recognized an incident or followed a business rule. Switching models can change its decisions again.

Every prompt change means another round of testing

After the first failures, you return to the prompt. You collect problem tickets, add instructions, and adjust the wording until the model makes the decisions you want. Then you check whether the earlier examples still work.

This is necessary work, but repeating those checks manually takes more time as the list of examples grows.

Suppose you add “Only mark a ticket urgent when it explicitly reports an active outage.” The downtime question now gets normal priority. The sign-in incident still waits in the queue, though, because the agent expects the user to say “outage” explicitly.

You change the prompt again, then go back through the billing examples to make sure they still reach the right team. Each fix requires checking cases it wasn't intended to affect.

Without a fixed test set, it's hard to tell whether the agent has improved or simply started making different mistakes.

Correct decisions are only part of the requirement. In a running application, you also need to know how long each ticket takes to process and what it costs. Define requirements for all three:

  • Correctness: Does the agent call the right tool with the correct ticket ID, team, and priority?
  • Performance: Does the agent triage the ticket within the time budget?
  • Cost: Does triaging a ticket stay within its cost budget?

Put these requirements into tests so you can check them after each change.

Turning examples into evaluations

An evaluation runs the agent against a prepared set of examples and compares its decisions with the expected results. After changing a prompt, model, or tool, you can run the same cases again and see what changed.

You can use Promptfoo for these evaluations. It runs prompts against different models and automatically checks whether the outputs meet your requirements.

The following promptfooconfig.yaml compares two models on our ticket examples. It checks whether each model selects assignTicket with the correct arguments, without updating any real tickets.

The five-second time limit and $0.02 cost limit per model response are example budgets. Set them to suit your workload.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Support-ticket triage evaluation

prompts:
  - |
    Triage the support ticket by calling assignTicket exactly once.
    Send outages and service failures to infrastructure.
    Send invoices and payment questions to billing.
    Send how-to questions to support.
    Use urgent priority for active production outages.
    Use normal priority otherwise.

    Ticket ID: {{ticketId}}
    Message: {{message}}

providers:
  - id: openai:chat:gpt-5
    config: &triageTools
      tools:
        - type: function
          function:
            name: assignTicket
            description: Assign a support ticket to a team and set its priority.
            strict: true
            parameters:
              type: object
              properties:
                ticketId:
                  type: string
                team:
                  type: string
                  enum: [ infrastructure, billing, support ]
                priority:
                  type: string
                  enum: [ urgent, normal ]
              required: [ ticketId, team, priority ]
              additionalProperties: false
  - id: openai:chat:gpt-5-mini
    config: *triageTools

defaultTest:
  assert:
    - type: is-valid-openai-tools-call
    - type: javascript
      value: |
        const calls = Array.isArray(output) ? output : output?.tool_calls;
        if (!Array.isArray(calls) || calls.length !== 1) return false;
        const call = calls[0].function;
        if (call?.name !== "assignTicket") return false;
        const args = JSON.parse(call.arguments);
        return args.ticketId === context.vars.ticketId &&
          args.team === context.vars.expectedTeam &&
          args.priority === context.vars.expectedPriority;
    - type: latency
      threshold: 5000
    - type: cost
      threshold: 0.02

tests:
  - vars:
      ticketId: T-101
      message: Production is fine. How do I prevent downtime?
      expectedTeam: support
      expectedPriority: normal

  - vars:
      ticketId: T-102
      message: Nobody can sign in since deployment.
      expectedTeam: infrastructure
      expectedPriority: urgent

  - vars:
      ticketId: T-103
      message: Please send a copy of last month's invoice.
      expectedTeam: billing
      expectedPriority: normal

You'll need Node.js and an API key in the OPENAI_API_KEY environment variable. Then run:

npx promptfoo@latest eval -c promptfooconfig.yaml --no-cache

Running these tests makes paid API calls. The --no-cache option ensures each run measures a fresh model response rather than reusing a saved result.

These tests cover the model's tool selection and arguments. You'll need separate sandbox tests for executing assignTicket and saving the ticket changes.

A small set like this gives you a starting point. When a user encounters another failure, add their ticket to the evaluation set. The next time you make a change, you'll be able to check whether that problem has returned.

A cheaper model sends you back to the prompt

Eventually, the agent works well enough, but ticket volume grows along with the API bill. You choose a cheaper model, keep the same prompt, and discover that the new model makes different decisions.

It might miss urgency when users describe a problem indirectly, or send more tickets to the wrong team. Perhaps it needs differently worded instructions. You're now within budget, but some of the correctness tests fail.

Without evaluations, you're back to manually adjusting the prompt and checking a few examples. After another deployment, you wait for users to find the cases you didn't anticipate.

A prepared test set lets you compare both models before deployment. Running them on the same tickets gives you correctness, latency, and cost results to review together. You can then judge whether the savings justify any drop in quality.

providers:
  - id: openai:chat:gpt-5
    config: *triageTools
  - id: openai:chat:gpt-5-mini
    config: *triageTools

Keep the shared triageTools configuration from the full example when comparing providers. The results will show which model routes these tickets correctly within budget. There may still be failures your tests don't cover, but you can catch known problems before they reach users.

The naive approach does not scale

A prompt and a few local tests can get your first agent running. Maintaining it in a live application takes more work, especially as models, requirements, and team responsibilities change.

When prompt tuning is manual, each change brings another round of checking that you haven't broken something else. Written requirements and repeatable evaluations give you a way to compare results, but finding a better prompt can still consume a lot of engineering time.

In Why are you still rewriting prompts by hand?, we hand the prompt-editing loop to GEPA. The follow-up walks through scoring candidates, adding feedback, and loading the optimized program in an application.

Łukasz Kużyński