Why are you still rewriting prompts by hand?

Łukasz Kużyński ·

Why are you still rewriting prompts by hand?

In the previous post, we looked at manual prompt tuning and an evaluation framework that checks whether an agent behaves as expected.

You could quickly notice a pattern:

  1. Modify the prompt
  2. Measure the results
  3. Modify it again
  4. Measure again
  5. Repeat until the behavior meets your requirements

Evaluations automate the checking, but you're still making each change yourself.

You can hand that part to AI too actually.

A quick introduction to GEPA

GEPA (short for Genetic-Pareto) uses evolutionary search to improve prompts. A model examines execution traces and feedback to propose revisions, and evaluations guide which candidates are retained for further search.

We'll use the implementation exposed by AX LLM's optimization API. It can tune instructions and other exposed text components. The top-level optimize(...) helper can also bootstrap successful examples into demonstrations before running GEPA.

Greatly simplified, the process looks like this:

  1. Run the initial program on examples and establish a baseline. Suppose its score is 0.5 on a metric from 0 to 1
  2. A teacher model examines execution traces and any supplied feedback, then proposes revised instructions
  3. Run the candidate with the student model and score its outputs. Keep candidates according to measured performance
  4. Repeat within the configured search budget. In AX LLM, numTrials controls reflection/evolution rounds, while maxMetricCalls limits metric evaluations. Early stopping can end the search sooner

The student is the model that will run your application. The teacher helps improve its instructions during optimization. You should use a stronger teacher while keeping the cheaper student for production requests.

The Pareto part matters when candidates have different strengths. A candidate might perform better on one group of cases, or improve correctness at the expense of brevity. GEPA retains useful alternatives rather than assuming one candidate wins everywhere. AX LLM also supports explicit multi-objective metrics and returns a Pareto frontier.

For this example, we'll track intent, queue, and priority as separate objectives, then save the selected optimization artifact.

Setting up

We'll continue with support triage, using a structured classifier so the optimization example stays small. It returns an intent, a queue, and a priority.

Provide the signature and instructions

A signature describes the inputs and expected output fields. Define the starting program:

import { ax } from '@ax-llm/ax';

export function createProgram() {
  const program = ax(
    'conversation:string, accountContext:string -> intent:class "incident, billing, access, feature-request, account-change", queue:class "support, engineering, security, success", priority:class "normal, urgent"'
  );

  program.setInstruction(`
    Triage the customer case using the conversation and account context.
    Identify the customer's primary intent, send the case to the responsible team,
    and mark cases requiring immediate attention as urgent.
  `);

  return program;
}

These instructions leave room for interpretation. They don't explain which team handles a security issue, or how changing a billing plan differs from asking for an invoice. Our examples need to make those decisions clear.

Provide expected outcomes as training examples

Each training example contains inputs and the outputs we expect. Here are two entries from the training set:

type SupportExample = {
  conversation: string;
  accountContext: string;
  intent: 'incident' | 'billing' | 'access' | 'feature-request' | 'account-change';
  queue: 'support' | 'engineering' | 'security' | 'success';
  priority: 'normal' | 'urgent';
};

export const trainingExamples: SupportExample[] = [
  {
    conversation: 'Every customer sees a 500 error when they submit checkout.',
    accountContext: 'Production account with 120 active users.',
    intent: 'incident',
    queue: 'engineering',
    priority: 'urgent',
  },
  {
    conversation: 'A former employee can still download customer exports.',
    accountContext: 'Enterprise account handling regulated customer data.',
    intent: 'access',
    queue: 'security',
    priority: 'urgent',
  },
];

The full example uses 12 training cases covering incidents, billing, ordinary access problems, security issues, feature requests, and account changes. These two entries alone don't cover all those decisions.

Define the evaluation metrics

The optimizer needs scores it can compare across candidates. Return a separate score for each category instead of combining them into one weighted number:

type SupportOutput = Pick<SupportExample, 'intent' | 'queue' | 'priority'>;
type SupportEvaluation = {
  prediction: SupportOutput;
  example: SupportExample;
};
type SupportScores = Record<keyof SupportOutput, 0 | 1>;

export function metric({ prediction, example }: SupportEvaluation): SupportScores {
  return {
    intent: prediction.intent === example.intent ? 1 : 0,
    queue: prediction.queue === example.queue ? 1 : 0,
    priority: prediction.priority === example.priority ? 1 : 0,
  };
}

Each category scores 1 for a match and 0 for a mismatch. A prediction with the correct intent and queue but the wrong priority returns { intent: 1, queue: 1, priority: 0 }.

Returning an object makes these separate objectives available to GEPA's Pareto search. A candidate that improves intent classification but worsens urgency handling has a visible tradeoff. You can inspect the per-category scores in result.paretoFront before deciding which configuration to deploy.

If missing an urgent security issue is unacceptable, add a separate deployment check for those cases. Separate objectives expose the tradeoff; they don't enforce a safety requirement by themselves.

Add a feedback function

A category score identifies a mismatch. Text feedback tells the teacher what was expected and what the program returned. This feedbackFn checks all three output fields:

export function feedbackFn({ prediction, example }: SupportEvaluation): string[] | undefined {
  const feedback: string[] = [];

  if (prediction.intent !== example.intent) {
    feedback.push(
      `Intent mismatch: expected ${example.intent}, received ${prediction.intent}.`
    );
  }
  if (prediction.queue !== example.queue) {
    feedback.push(
      `Queue mismatch: expected ${example.queue}, received ${prediction.queue}.`
    );
  }
  if (prediction.priority !== example.priority) {
    feedback.push(
      `Priority mismatch: expected ${example.priority}, received ${prediction.priority}.`
    );
  }

  return feedback.length > 0 ? feedback : undefined;
}

For a security case routed to support with normal priority, the feedback includes:

Queue mismatch: expected security, received support.
Priority mismatch: expected urgent, received normal.

The numeric scores remain separate from these messages: GEPA uses scores to compare candidates, while the teacher uses feedback alongside execution traces to propose revisions.

Keep validation examples separate

Use different cases to select candidates. For example, the training set describes a former employee downloading exports, while a validation case says:

export const validationExamples: SupportExample[] = [
  {
    conversation: 'Someone who left the company is still listed as an active admin.',
    accountContext: 'Enterprise account with regulated data.',
    intent: 'access',
    queue: 'security',
    priority: 'urgent',
  },
];

The full example has 10 validation cases; the snippets above are excerpts.

Validation checks whether candidates handle different wording, but the optimizer uses those scores to select its output. For an independent final assessment, reserve a third test set that the optimization process never uses.

Run optimization

Pass the program, training examples, category metrics, and feedback function to optimize(...):

Here, studentAI and teacherAI are configured AX LLM model clients. The example uses GPT-OSS 20B as the student and GPT-OSS 120B as the teacher, both through OpenRouter. Keep the student's model settings consistent between evaluation and production.

AX's optimizer callbacks don't carry these signature-specific types, so keep the type assertions at that boundary:

import { writeFileSync } from 'node:fs';
import { axSerializeOptimizedProgram, optimize } from '@ax-llm/ax';

const result = await optimize(
  createProgram(),
  trainingExamples,
  ({ prediction, example }) => metric({
    prediction: prediction as SupportOutput,
    example: example as SupportExample,
  }),
  {
    studentAI,
    teacherAI,
    numTrials: 8,
    bootstrap: true,
    validationExamples,
    feedbackFn: ({ prediction, example }) => feedbackFn({
      prediction: prediction as SupportOutput,
      example: example as SupportExample,
    }),
    maxMetricCalls: 100,
  }
);

if (!result.optimizedProgram) {
  throw new Error('Optimization completed without an artifact.');
}

writeFileSync(
  new URL('./optimization-artifact.json', import.meta.url),
  JSON.stringify(axSerializeOptimizedProgram(result.optimizedProgram), null, 2)
);

console.log(`Selected candidate score: ${result.bestScore}`);
for (const candidate of result.paretoFront) {
  console.log('Category scores:', candidate.scores);
}

numTrials: 8 bounds the search rounds. It doesn't mean eight model requests: each round can evaluate multiple examples and call the teacher. bootstrap: true enables demonstration bootstrapping before GEPA. maxMetricCalls: 100 makes the metric-evaluation budget explicit.

Apply the saved artifact at initialization

The artifact contains the optimized configuration, such as instructions and demonstrations. Save it to disk, then apply it when creating the program.

Load the saved artifact when creating the program:

import { readFileSync } from 'node:fs';
import { axDeserializeOptimizedProgram } from '@ax-llm/ax';

export function createProgramOptimized() {
  const saved = JSON.parse(
    readFileSync(
      new URL('./optimization-artifact.json', import.meta.url),
      'utf8'
    )
  );
  const artifact = axDeserializeOptimizedProgram(saved);

  const program = createProgram();
  program.applyOptimization(artifact);
  return program;
}

Now run a case through the optimized program:

const program = createProgramOptimized();

const prediction = await program.forward(studentAI, {
  conversation: 'Someone who left the company is still listed as an active admin.',
  accountContext: 'Enterprise account with regulated data.',
});

console.log(prediction);

Use applyOptimization(...) rather than copying only the generated instruction. An artifact can also carry demonstrations and changes to other program components.

When to run optimization

Run it offline, outside application startup and request handling. It makes repeated model calls and can take time and incur substantial cost, especially with a strong teacher or a large evaluation set.

Plan a new optimization run when:

  1. The student model or its settings change
  2. The signature or starting instructions change
  3. The optimization code changes, including metrics, feedback, or search settings
  4. Training examples or business rules change, or production failures reveal missing cases

Re-evaluate after these changes and compare any new artifact with the current one before replacing it. An optimizer can return a candidate that doesn't improve the behavior you care about.

Results

An earlier run using a weighted metric produced:

Before optimization:  83.5%
After optimization:  100.0%
Improvement:         +16.5 percentage points

These scores come from validation cases used during optimization. Test the optimized program on separate cases before shipping.

Let GEPA handle the next prompt revision

GEPA should be near the top of your list for improving prompts automatically. It takes the edit-and-evaluate loop we started with and lets a teacher model propose the changes. Your evaluations decide which candidates to keep.

We used a small support-triage classifier to show how the pieces fit together. AX LLM also supports optimizing larger programs, flows, and tool-using agents. You still define the expected behavior, but you can stop writing every prompt revision yourself.

We're not done with GEPA yet. In the next post, we'll explore its configuration settings and when changing them helps. We'll also look at where GEPA falls short, because automatic prompt optimization isn't a silver bullet.

Łukasz Kużyński