Compare Actions with experiments

An Experiment sends traffic through one of two Actions and records the selected Action on each generation. Use it to compare prompt, model, provider, or configuration changes under the same App.

Experiments complement Evaluate. An Eval runs a defined test set against evaluation criteria; an Experiment collects usage, feedback, latency, token, and cost signals from traffic routed between two Actions.

Before you begin

Create two Actions in the same App. Each Action should accept compatible inputs. If their prompt templates use different variable names, callers must provide the values required by whichever Action handles the request.

The current app sidebar does not display an Experiments item. Open the Experiments page at:

https://app.klu.ai/YOUR_WORKSPACE_SLUG/apps/YOUR_APP_SLUG/experiments

Replace the two slug values with the slugs from your Klu app URL.

Create an Experiment

  1. Open the App's Experiments page.
  2. Select Add Experiment.
  3. Enter an Experiment Name.
  4. Under Select a Primary Action, choose the control.
  5. Under Select a Secondary Action, choose the variant. Klu excludes the primary Action from this list.
  6. Select Create.

The Experiment begins with a CREATED status. Its overview identifies the primary Action as Control and the secondary Action as Variant, including each Action's model and provider.

Test prompt behavior

Open the Experiment and select Prompt. The prompt page initially chooses one of the two Actions at random. You can select either Action explicitly or use the refresh control to choose randomly again.

For Actions with {{variable}} placeholders, Klu renders one field per unique variable found in the Action prompt or system message. Select Show full prompt to inspect the substituted prompt and estimated token count before sending it.

For an Action without template variables, enter a message in Enter your next message to the action.... Select Send or press Enter to run it.

Each prompt creates a logged generation associated with the Experiment and selected Action. The table below the prompt form shows those results. A request with missing or invalid variables fails and displays the returned error; correct the input and send it again.

Route production traffic

Open the Experiment and select Deploy to view its endpoint and request example. Send requests to the Experiment GUID rather than to either Action directly. Unless a caller forces an Action, the experiment endpoint chooses between the control and variant and records the assignment with the generation.

Use forced Action routing only for diagnostics or deliberate traffic control. It bypasses the normal assignment behavior and can bias the comparison if included in your analysis.

Compare results

The Overview tab provides a date-filtered Usage and Feedback chart. It also compares these values for the control and variant:

  • positive and negative completion feedback
  • average completion tokens
  • average completion time
  • total cost

The values appear after the Experiment has logged traffic. A dash or N/A means the corresponding metric has no usable observations yet. Compare feedback rates as well as raw counts when traffic volume differs between the two Actions.

Select Edit to rename the Experiment. Deleting it is irreversible in the UI, so export or record any results you need first.

Connect experiments to Evaluate

Use the Experiment overview to discover promising behavior under real or staged traffic. Then create a Dataset of representative and failure-case examples and run both Action versions through Evaluate. This gives you a repeatable comparison against the same test cases and evaluators.

See Work with datasets for building the test set and Evaluate prompt and model performance for running and comparing Evals.

Experiment API

The public Experiment endpoint supports synchronous, asynchronous, and streaming prompts. It accepts a string or key-value input and can also receive a session GUID, retrieval filters, indices, metadata filters, or a forced Action GUID. Synchronous responses include the generated message and feedback URL; asynchronous responses also provide a result URL; streaming responses provide a token stream.

The Python and TypeScript SDKs also support creating, retrieving, updating, and deleting Experiments. See API actions for request examples and SDK exports for the current clients.