How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/caffeinelabs/skills --skill extension-inferenceBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
Caffeine Inference
LLM extension for Caffeine AI.
Orchestrator routing notes
Treat “use an LLM / GPT / chatbot / summarise with AI” as a first-class platform feature. The default path is Caffeine Inference: an OpenAI-compatible chat endpoint that Caffeine hosts, authenticates, and bills for the app. The canister gets its credentials from the platform at runtime; nobody pastes an API key, and the app never stores or returns its own key. The frontend-direct path gives the browser short-lived child keys instead.
| User intent | Capability |
|---|---|
| Chat / summarise / classify with an LLM in a Caffeine app | caffeineai-inference-client ChatApi.createChatCompletion via this skill |
| Chat, lookups, or translation where a signed-in user waits for the reply | The frontend-direct path of this skill: the browser calls the API with a key from Ephemeral.Broker |
Call api.openai.com with a user-pasted sk-... | `extension-openai` only |
Do not load extension-openai for a normal Caffeine-app LLM. Do not ask the user for an OpenAI API key. Do not add setApiKey endpoints, a key-settings page, or a model picker.
Choose a path
| Backend path | Frontend-direct path | |
|---|---|---|
| Who calls the API | The canister, as an HTTPS outcall | The browser, with a 3-hour key the canister mints |
| Latency | An update call plus an outcall: seconds, all at once | No update call; the reply streams as the model writes it |
| Cycles per call | One outcall | None. Only key mints are outcalls |
| Prompt | Hidden in the canister, so the backend can trust the result | Visible and editable in the browser |
| Result | The canister has it | The browser has it and saves it in the background |
- Use the frontend-direct path for features only signed-in users use and
wait on: chat, word lookups, translation, suggestions.
- Use the backend path when the backend must trust or keep the result
(grading, moderation, anything other users see), when no user is waiting (timers), for public features, and in apps without sign-in.
- An app can use both, one per feature.
Never mint keys for anonymous callers: a key leaves the canister, so anyone could spend the app's credits with it. Check the caller in the mint method, with the app's own access rule.
- Each backend chat method runs the access rule the spec gives its feature.
A feature for signed-in users checks the caller. A public feature, such as a chat any visitor can use, takes any caller, even in an app with sign-in.
- An app without sign-in uses only the backend path, and its chat methods
take any caller.
Backend
1. Add caffeineai-inference-client to mops.toml
mops add caffeineai-inference-client@0.4.2Requires Mops ≥ 2.13. Minimum version: caffeineai-inference-client ≥ 0.4.0.
2. Config comes from the platform
Config.fromEnv<system>() returns a complete Config — endpoint, bearer, and is_replicated = ?false — from the credentials the platform provisions for the app. There is no key to collect and nothing to configure.
- Call
fromEnv<system>()inside thesharedmethod, or in a
<system>-parameterised helper, on every request. A module-level let config = fromEnv will not compile, and a cached Config can go stale when the platform rotates credentials on a running canister.
- It traps when the app has no inference credentials. Let that error reach
the caller; hiding it makes a failed AI call look like a working feature:
- Never pre-check
CAFFEINE_INFERENCE_API_KEY(or any other environment
variable) and never guard fromEnv with a check that skips the call.
- Never turn a failed call into an empty success, such as an empty string,
an empty list, or placeholder text. Trap or return the error so the frontend can show it and offer a retry.
- Do not add a "configure AI" empty state or a key-input fallback.
- Never log the
Config, never copy itsauthinto actor state, and never
return it (or any part of it) from a query / shared function. The frontend-direct path returns a minted child key, never the app key.
3. is_replicated = ?false is REQUIRED
fromEnv already sets this. Do not override it to ?true or null.
- Security. A replicated outcall sends the bearer from every replica.
- Billing. Replicated outcalls multiply inference spend by subnet size.
- Determinism. LLM bodies are sampled; consensus would fail.
4. Canonical layout
Keep only the endpoints of the path each feature uses.
import Ephemeral "mo:caffeineai-inference-client/Ephemeral";
import Principal "mo:core/Principal";
import Runtime "mo:core/Runtime";
import Inference "lib/inference";
actor {
// A deploy revokes every minted key, so the broker's cache is transient.
transient let inferenceKeys = Ephemeral.Broker(Ephemeral.defaultLimits);
// Replace with the app's own rule, for example an approved-user check.
func requireAiAccess(caller : Principal) {
if (caller.isAnonymous()) Runtime.trap("Sign in to use AI features");
};
// Backend path
public shared func chat(prompt : Text) : async Text {
await* Inference.runChat<system>(prompt);
};
// Frontend-direct path
public shared ({ caller }) func getInferenceCredentials() : async Ephemeral.Credentials {
requireAiAccess(caller);
await* inferenceKeys.grant<system>(caller);
};
};import { fromEnv } "mo:caffeineai-inference-client/Config";
import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import ChatCompletionRequest "mo:caffeineai-inference-client/Models/ChatCompletionRequest";
import ChatCompletionRequestMessageOneOf2 "mo:caffeineai-inference-client/Models/ChatCompletionRequestMessageOneOf2";
import Runtime "mo:core/Runtime";
module {
public func runChat<system>(prompt : Text) : async* Text {
let config = fromEnv<system>();
let userMessage = ChatCompletionRequestMessageOneOf2.JSON.init({
content = #string(prompt);
role = #user;
});
let req = ChatCompletionRequest.JSON.init({
messages = [#user(userMessage)];
model = "router";
});
let resp = await* ChatApi.createChatCompletion(config, req);
if (resp.choices.size() == 0) {
Runtime.trap("Inference returned no choices");
};
resp.choices[0].message.content
?? Runtime.trap("Inference returned no text content");
};
};5. model = "router" — the platform picks the model
"router" is the only public model id, and it is a routing tier rather than a model name. Caffeine Inference sizes each request to the complexity of the query — a small fast model for simple prompts, a stronger one for hard reasoning — and reports "router" back as the response model, so provider names never reach the app.
- Always send
model = "router". - Do not add a model dropdown, a "use GPT-4" toggle, or a
modelparameter on
the backend endpoint. There is nothing for the user to choose.
- Steer quality with the prompt and with the declared sampling fields
(temperature, top_p, max_completion_tokens), not with model selection.
6. Call shapes
- Function form:
ChatApi.createChatCompletion(config, req) : async*— useawait*. - Suite form:
let api = ChatApi(config); api.createChatCompletion(req) : async.
7. Available API surface — chat completions
caffeineai-inference-client@0.4.2 is generated from `public-api-v0.1.0`:
| Module | Entry point | Route |
|---|---|---|
ChatApi | createChatCompletion | POST /v1/chat/completions |
ModelsApi | listModels | GET /v1/models — catalog only; the model is always "router", so an app never needs this |
Ephemeral | Broker.grant, mint | POST /v1/ephemeral-keys — hand-written, not generated |
<!-- motoko-check:skip -->
import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import { fromEnv } "mo:caffeineai-inference-client/Config";Chat completions are the whole product surface. Not available on this host (404, and not in the package): embeddings, images, audio, moderations, files, legacy completions, Assistants, Responses, and raw ic.http_request. If the spec genuinely needs an OpenAI-only API with a pasted sk-..., switch to `extension-openai`.
8. Cycles
Leave cycles unset (null): each call attaches the fee the IC charges for that request, which is zero on subnets where outcalls are free. ?n attaches exactly n cycles: the IC rejects the call when n is below its fee, and the call traps on a subnet where the canister holds fewer than n.
Streaming (stream = ?true) is unsupported in the canister — management-canister HTTP returns the full body. Leave stream = null. The browser can stream; see the frontend-direct path.
9. Things that will bite you
- Call
fromEnv<system>()inside thesharedmethod (or a<system>helper). A module-levellet config = fromEnvwill not compile. - Surface failures. No credential pre-checks, no guard around
fromEnv, and no empty result in place of an error. See §2. model = "router"— not"gpt-4o-mini". See §5.- User turns are
#user(ChatCompletionRequestMessageOneOf2.JSON.init({ content = #string(prompt); role = #user })). JSON.initfor required fields; layer optionals with record update. Do not hand-list everynull.resp.choices[0].message.contentis?Text. Checkchoices.size()first.- One chat call is one HTTP outcall inside an update call: budget seconds, not milliseconds. When a signed-in user waits on it, use the frontend-direct path.
Ephemeral.Brokerreuses a caller's key for 15 minutes and allows 30 mints per caller per hour. Pass your ownLimitsto change that.
Frontend
The app is ready to chat on first load — there is nothing to configure.
- No API-key UI. No settings page, no password input, no "configured?"
indicator, no localStorage. If a spec or mock shows an "AI settings" screen, drop it.
- No model picker. See §5.
- Backend path: call the backend chat method (
chat(prompt)), show a
pending state while it runs (seconds), and offer a retry on trap.
- Frontend-direct path: use the following client. It keeps the key in
memory only, gets a new one when the gateway answers 401 (the key expired, or a deploy revoked it), and streams the reply.
export type InferenceCredentials = { baseUrl: string; apiKey: string; expiresAt: string };
export type ChatMessage = { role: "system" | "user" | "assistant"; content: string };
export function createInferenceChat(getCredentials: () => Promise<InferenceCredentials>) {
let current: Promise<InferenceCredentials> | null = null;
const credentials = (renew: boolean) => {
if (renew || !current) {
current = getCredentials();
current.catch(() => { current = null; });
}
return current;
};
return async function chat(messages: ChatMessage[], onDelta?: (text: string) => void): Promise<string> {
for (let attempt = 0; ; attempt++) {
const { baseUrl, apiKey } = await credentials(attempt > 0);
const response = await fetch(`${baseUrl.replace(/\/+$/, "")}/v1/chat/completions`, {
method: "POST",
headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
body: JSON.stringify({ model: "router", messages, stream: true }),
});
if (response.status === 401 && attempt === 0) continue;
if (!response.ok || !response.body) {
const body = await response.json().catch(() => null);
throw new Error(body?.error?.message ?? `Inference failed: HTTP ${response.status}`);
}
let text = "";
let buffer = "";
const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();
for (;;) {
const { done, value } = await reader.read();
if (done) break;
buffer += value;
const lines = buffer.split("\n");
buffer = lines.pop() ?? "";
for (const line of lines) {
const data = line.startsWith("data:") ? line.slice(5).trim() : "";
if (!data || data === "[DONE]") continue;
const delta = JSON.parse(data).choices?.[0]?.delta?.content;
if (delta) {
text += delta;
onDelta?.(delta);
}
}
}
if (!text) throw new Error("Inference returned no text");
return text;
}
};
}Create one client per signed-in session, for example const chat = createInferenceChat(() => actor.getInferenceCredentials());.
- Show first, save after. Render the reply as it streams. Then save what
the app keeps (a chat exchange, a translation) with an update call that does not block the UI and retries on failure. Say in the UI copy or docs that the last exchange can be lost if the tab closes before the save lands.
- Ask for JSON in the prompt when the feature needs fields (a translation
with an article and a base form), and parse the final text, not the deltas.
- Never put the key in localStorage, sessionStorage, cookies, URLs, or
logs.