LLM
LLM providers are the models that read a question and write the answer — what users pick in the model selector of the chat. This page covers what only exists for LLMs; everything else — activating, defaults, testing, import — is described in Managing LM Providers.
Picking the model from the catalogue
The Model field searches the full catalogue of the service you picked. Every suggestion shows its context window, its price per million tokens and whether it supports tools and images, so you can compare without leaving the form.
Once a model is selected, QAnswer prefills its context window, its prices and its capabilities for you, and shows them as a summary line under the form. Click Customize if you want to set every field by hand.
Capabilities
Capabilities tell QAnswer what the model is able to do, so features that need them are only offered where they work. Tool Calling gates MCP tools and agents, Native Web Search lets the model search on its own, and Prompt Caching lets QAnswer reuse a cached prompt prefix to cut cost.
Switching a capability on that the model does not have does not add the feature — it makes the calls that rely on it fail. When you are unsure, leave it off and use Test connection.
Token prices
The Cost section holds what the provider charges you per million tokens. QAnswer charges every call against these prices, so they drive both the Usage & Costs page and the credit limits set on plans and API keys. Picking a model from the catalogue fills all three for you.
- Input Token Cost — every token of the prompt the model reads.
- Output Token Cost — every token the model writes.
- Cached Input Token Cost — the prompt tokens the provider served from its prompt cache instead of reading them again, which providers bill at a fraction of the input price.
A request hits the cache when it starts with the same text as a recent one — the system prompt, the tool list, the earlier turns of the conversation — so long conversations and assistants with many tools benefit most. OpenAI and QAnswer's own models cache on their own; Claude and Gemini only cache what QAnswer marks, which it does when Prompt Caching is switched on.
Leave the field empty to charge cached tokens at the input price, which is also what providers created before the field existed do. A cached price higher than the input price is never applied: a token read from the cache is never charged more than a fresh one.
Claude models also report the prompt tokens they write into the cache. QAnswer charges those at 1.25 times the input price, Anthropic's own surcharge. No other provider reports cache writes, so this never applies to them.
Reasoning effort
For models that can think before answering, the Reasoning row sets how much effort they spend. The small scale next to it is the default level, and the sliders button chooses which levels this endpoint offers to users at all.
Tick only the levels that make sense for your endpoint, or choose Reset to automatic to let QAnswer decide from the model itself.
Guardrails
QAnswer Guardrails ensure safe and secure AI interactions. Guardrails control AI interactions for two primary purposes: enforcing your organization's safety and ethical guidelines, and preventing sensitive data from being sent to public LLMs.
Guardrails are configured per provider, at the bottom of its edit form, by whoever owns the provider.
When to use Guardrails
- Maintaining Compliance: Enforce data privacy, acceptable use, or content generation policies.
- Protecting Sensitive Information: Prevent data leakage when working with confidential documents by restricting interaction with external LLMs.
- Controlling AI Behavior: Define boundaries for acceptable responses to prevent harmful, biased, or irrelevant content.
Configuring your Guardrails
Define the following parameters when setting up a guardrail:
- Model: Select the LLM that enforces the guardrail. For maximum control and data security, use an on-premise model hosted within your own infrastructure.
- Scope (Input/Output): Select where the guardrail operates:
- Input (in): Monitors and controls prompts submitted to the LLM.
- Output (out): Monitors and controls responses generated by the LLM.
- Both (in-out): Monitors both inputs and outputs.
- Mode: Choose how triggered guardrails are handled:
- Warning Mode: Alerts the user that a guardrail was triggered but allows proceeding with caution.
- Error Mode: Blocks the request entirely when a guardrail is triggered.
- Prompt: Define a prompt that outlines the rules the guardrail enforces. This prompt provides context for assessing and filtering interactions.
Effect of the guardrails
When a guardrail is triggered, the LLM will either warn the user or block the request, depending on the mode you have selected.
Jailbreak Guardrail
The Jailbreak Guardrail protects the AI system's integrity by preventing users from bypassing core instructions and safety mechanisms. It defends against attempts to manipulate the LLM into unauthorized actions or into revealing confidential system information.
A jailbreak attempt circumvents the LLM's intended limitations. Common forms include:
- Access the System Prompt: Discover the instructions initially given to the AI.
- Override Safety Guidelines: Bypass restrictions on harmful or unethical content.
- Manipulate Behavior: Trick the AI into acting outside its intended role.
- Perform Prompt Injection Attacks: Introduce malicious instructions in a prompt to alter AI behavior.
The Jailbreak Guardrail is a specialized output guardrail that analyzes user inputs for jailbreak patterns, using the LLM itself to detect and flag potentially harmful prompts.
Calling the provider from your own code
The API tab gives you a working snippet for the model, in Python, cURL, the OpenAI SDK or LiteLLM. The base URL points at QAnswer, so your quota, logs and cost tracking still apply.
