LLM primer

This page is conceptual background for working with large language models. It does not define Klu's current provider list, model availability, pricing, or API behavior. For product-specific details, use key concepts and the API and SDK documentation.

Tokens and context windows

A language model processes text as tokens. A token can represent a word, part of a word, punctuation, or another text fragment. Token counts vary by model and language, so character or word counts are only rough estimates.

The context window is the maximum token budget available to a request. It includes instructions, conversation history, retrieved Context, tool definitions, input, and generated output. If a request exceeds the model's limit, you must shorten, summarize, retrieve selectively, or split the work.

In Klu, monitor the complete prompt sent to the model when you diagnose truncation, unexpected priorities, latency, or cost. The template alone may represent only part of the request.

Prompts and inference

A prompt is the input that guides a model. It can contain system instructions, user input, examples, retrieved passages, and structured output requirements.

Inference is the process of generating a response from a trained model. The response is probabilistic: the same request can produce different outputs depending on the model and its configuration.

Common controls include:

• Temperature and related sampling settings, which influence variability.
• Maximum output tokens, which limits response length.
• Stop sequences, which end generation when a configured sequence appears.
• Response format, which can constrain output to text, JSON, or a JSON schema when the provider supports it.

Provider support and exact behavior differ. Test the configuration with representative inputs before you deploy an Action.

Retrieval and tools

Retrieval-augmented generation adds relevant source material to the request at inference time. In Klu, you store source material in Context and attach that Context to an Action.

Retrieval can make an answer more specific and traceable, but it introduces its own failure modes. The relevant passage may be missing, ranked too low, outdated, or contradicted by another source. Evaluate retrieval and final-answer quality separately when possible.

Tools let a model request an external operation. In Klu, configured Skills provide this capability. Validate tool arguments, enforce authorization outside the model, and handle tool failures in your application flow.

Training and fine-tuning

Pre-training teaches a model broad patterns from a large corpus. Fine-tuning continues training on a narrower Dataset to adapt behavior or output style.

Fine-tuning does not automatically add reliable, current knowledge. Use Context when information changes or must be retrieved from an authoritative source. Use fine-tuning when you have enough high-quality examples and a measurable behavioral goal.

Keep a held-out evaluation set and compare the fine-tuned model against a baseline. A successful training job confirms that training completed; evaluation determines whether the resulting model improved the behavior you care about.

Operational considerations

Model quality is one part of a production system. You should also measure latency, error rate, token usage, cost, retrieval quality, tool success, and user feedback.

Model and provider behavior can change across versions. Pin versions where the provider supports it, record the model used for each Log, and rerun evaluations before changing a production Action.

Continue with LLM operations for production concerns, or run an evaluation in Klu.