caffeinelabs/skills

extension-inference

MANDATORY recipe for every Caffeine build that calls an LLM, chatbot, GPT, or ChatGPT on Caffeine Inference (no user-pasted OpenAI key).

View source

How to use

  1. Copy the installation command and review the source package before installing it in your own agent environment.
  2. Check the source document for required tools, credentials and supported hosts.
  3. Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/caffeinelabs/skills --skill extension-inference

Before you start

  • Installation does not include a model subscription, API allowance or permission to access your files.
  • Repository stars and installation counts describe activity, not an independent quality or security test.
Original skill document

Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.

Caffeine Inference

LLM extension for Caffeine AI.

Orchestrator routing notes

Treat “use an LLM / GPT / chatbot / summarise with AI” as a first-class platform feature. The default path is Caffeine Inference: an OpenAI-compatible chat endpoint that Caffeine hosts, authenticates, and bills for the app. The canister gets its credentials from the platform at runtime; nobody pastes an API key, and the app never stores or returns its own key. The frontend-direct path gives the browser short-lived child keys instead.

User intentCapability
Chat / summarise / classify with an LLM in a Caffeine appcaffeineai-inference-client ChatApi.createChatCompletion via this skill
Chat, lookups, or translation where a signed-in user waits for the replyThe frontend-direct path of this skill: the browser calls the API with a key from Ephemeral.Broker
Call api.openai.com with a user-pasted sk-...`extension-openai` only

Do not load extension-openai for a normal Caffeine-app LLM. Do not ask the user for an OpenAI API key. Do not add setApiKey endpoints, a key-settings page, or a model picker.

Choose a path

Backend pathFrontend-direct path
Who calls the APIThe canister, as an HTTPS outcallThe browser, with a 3-hour key the canister mints
LatencyAn update call plus an outcall: seconds, all at onceNo update call; the reply streams as the model writes it
Cycles per callOne outcallNone. Only key mints are outcalls
PromptHidden in the canister, so the backend can trust the resultVisible and editable in the browser
ResultThe canister has itThe browser has it and saves it in the background
  • Use the frontend-direct path for features only signed-in users use and

wait on: chat, word lookups, translation, suggestions.

  • Use the backend path when the backend must trust or keep the result

(grading, moderation, anything other users see), when no user is waiting (timers), for public features, and in apps without sign-in.

  • An app can use both, one per feature.

Never mint keys for anonymous callers: a key leaves the canister, so anyone could spend the app's credits with it. Check the caller in the mint method, with the app's own access rule.

  • Each backend chat method runs the access rule the spec gives its feature.

A feature for signed-in users checks the caller. A public feature, such as a chat any visitor can use, takes any caller, even in an app with sign-in.

  • An app without sign-in uses only the backend path, and its chat methods

take any caller.

Backend

1. Add caffeineai-inference-client to mops.toml

bash
mops add caffeineai-inference-client@0.4.2

Requires Mops ≥ 2.13. Minimum version: caffeineai-inference-client ≥ 0.4.0.

2. Config comes from the platform

Config.fromEnv<system>() returns a complete Config — endpoint, bearer, and is_replicated = ?false — from the credentials the platform provisions for the app. There is no key to collect and nothing to configure.

  • Call fromEnv<system>() inside the shared method, or in a

<system>-parameterised helper, on every request. A module-level let config = fromEnv will not compile, and a cached Config can go stale when the platform rotates credentials on a running canister.

  • It traps when the app has no inference credentials. Let that error reach

the caller; hiding it makes a failed AI call look like a working feature:

  • Never pre-check CAFFEINE_INFERENCE_API_KEY (or any other environment

variable) and never guard fromEnv with a check that skips the call.

  • Never turn a failed call into an empty success, such as an empty string,

an empty list, or placeholder text. Trap or return the error so the frontend can show it and offer a retry.

  • Do not add a "configure AI" empty state or a key-input fallback.
  • Never log the Config, never copy its auth into actor state, and never

return it (or any part of it) from a query / shared function. The frontend-direct path returns a minted child key, never the app key.

3. is_replicated = ?false is REQUIRED

fromEnv already sets this. Do not override it to ?true or null.

  1. Security. A replicated outcall sends the bearer from every replica.
  2. Billing. Replicated outcalls multiply inference spend by subnet size.
  3. Determinism. LLM bodies are sampled; consensus would fail.

4. Canonical layout

Keep only the endpoints of the path each feature uses.

motoko
import Ephemeral "mo:caffeineai-inference-client/Ephemeral";
import Principal "mo:core/Principal";
import Runtime "mo:core/Runtime";
import Inference "lib/inference";

actor {
  // A deploy revokes every minted key, so the broker's cache is transient.
  transient let inferenceKeys = Ephemeral.Broker(Ephemeral.defaultLimits);

  // Replace with the app's own rule, for example an approved-user check.
  func requireAiAccess(caller : Principal) {
    if (caller.isAnonymous()) Runtime.trap("Sign in to use AI features");
  };

  // Backend path
  public shared func chat(prompt : Text) : async Text {
    await* Inference.runChat<system>(prompt);
  };

  // Frontend-direct path
  public shared ({ caller }) func getInferenceCredentials() : async Ephemeral.Credentials {
    requireAiAccess(caller);
    await* inferenceKeys.grant<system>(caller);
  };
};
motoko
import { fromEnv } "mo:caffeineai-inference-client/Config";
import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import ChatCompletionRequest "mo:caffeineai-inference-client/Models/ChatCompletionRequest";
import ChatCompletionRequestMessageOneOf2 "mo:caffeineai-inference-client/Models/ChatCompletionRequestMessageOneOf2";
import Runtime "mo:core/Runtime";

module {
  public func runChat<system>(prompt : Text) : async* Text {
    let config = fromEnv<system>();
    let userMessage = ChatCompletionRequestMessageOneOf2.JSON.init({
      content = #string(prompt);
      role = #user;
    });
    let req = ChatCompletionRequest.JSON.init({
      messages = [#user(userMessage)];
      model = "router";
    });
    let resp = await* ChatApi.createChatCompletion(config, req);
    if (resp.choices.size() == 0) {
      Runtime.trap("Inference returned no choices");
    };
    resp.choices[0].message.content
      ?? Runtime.trap("Inference returned no text content");
  };
};

5. model = "router" — the platform picks the model

"router" is the only public model id, and it is a routing tier rather than a model name. Caffeine Inference sizes each request to the complexity of the query — a small fast model for simple prompts, a stronger one for hard reasoning — and reports "router" back as the response model, so provider names never reach the app.

  • Always send model = "router".
  • Do not add a model dropdown, a "use GPT-4" toggle, or a model parameter on

the backend endpoint. There is nothing for the user to choose.

  • Steer quality with the prompt and with the declared sampling fields

(temperature, top_p, max_completion_tokens), not with model selection.

6. Call shapes

  • Function form: ChatApi.createChatCompletion(config, req) : async* — use await*.
  • Suite form: let api = ChatApi(config); api.createChatCompletion(req) : async.

7. Available API surface — chat completions

caffeineai-inference-client@0.4.2 is generated from `public-api-v0.1.0`:

ModuleEntry pointRoute
ChatApicreateChatCompletionPOST /v1/chat/completions
ModelsApilistModelsGET /v1/models — catalog only; the model is always "router", so an app never needs this
EphemeralBroker.grant, mintPOST /v1/ephemeral-keys — hand-written, not generated

<!-- motoko-check:skip -->

motoko
import ChatApi "mo:caffeineai-inference-client/Apis/ChatApi";
import { fromEnv } "mo:caffeineai-inference-client/Config";

Chat completions are the whole product surface. Not available on this host (404, and not in the package): embeddings, images, audio, moderations, files, legacy completions, Assistants, Responses, and raw ic.http_request. If the spec genuinely needs an OpenAI-only API with a pasted sk-..., switch to `extension-openai`.

8. Cycles

Leave cycles unset (null): each call attaches the fee the IC charges for that request, which is zero on subnets where outcalls are free. ?n attaches exactly n cycles: the IC rejects the call when n is below its fee, and the call traps on a subnet where the canister holds fewer than n.

Streaming (stream = ?true) is unsupported in the canister — management-canister HTTP returns the full body. Leave stream = null. The browser can stream; see the frontend-direct path.

9. Things that will bite you

  • Call fromEnv<system>() inside the shared method (or a <system> helper). A module-level let config = fromEnv will not compile.
  • Surface failures. No credential pre-checks, no guard around fromEnv, and no empty result in place of an error. See §2.
  • model = "router" — not "gpt-4o-mini". See §5.
  • User turns are #user(ChatCompletionRequestMessageOneOf2.JSON.init({ content = #string(prompt); role = #user })).
  • JSON.init for required fields; layer optionals with record update. Do not hand-list every null.
  • resp.choices[0].message.content is ?Text. Check choices.size() first.
  • One chat call is one HTTP outcall inside an update call: budget seconds, not milliseconds. When a signed-in user waits on it, use the frontend-direct path.
  • Ephemeral.Broker reuses a caller's key for 15 minutes and allows 30 mints per caller per hour. Pass your own Limits to change that.

Frontend

The app is ready to chat on first load — there is nothing to configure.

  1. No API-key UI. No settings page, no password input, no "configured?"

indicator, no localStorage. If a spec or mock shows an "AI settings" screen, drop it.

  1. No model picker. See §5.
  2. Backend path: call the backend chat method (chat(prompt)), show a

pending state while it runs (seconds), and offer a retry on trap.

  1. Frontend-direct path: use the following client. It keeps the key in

memory only, gets a new one when the gateway answers 401 (the key expired, or a deploy revoked it), and streams the reply.

ts
export type InferenceCredentials = { baseUrl: string; apiKey: string; expiresAt: string };
export type ChatMessage = { role: "system" | "user" | "assistant"; content: string };

export function createInferenceChat(getCredentials: () => Promise<InferenceCredentials>) {
  let current: Promise<InferenceCredentials> | null = null;
  const credentials = (renew: boolean) => {
    if (renew || !current) {
      current = getCredentials();
      current.catch(() => { current = null; });
    }
    return current;
  };

  return async function chat(messages: ChatMessage[], onDelta?: (text: string) => void): Promise<string> {
    for (let attempt = 0; ; attempt++) {
      const { baseUrl, apiKey } = await credentials(attempt > 0);
      const response = await fetch(`${baseUrl.replace(/\/+$/, "")}/v1/chat/completions`, {
        method: "POST",
        headers: { "content-type": "application/json", authorization: `Bearer ${apiKey}` },
        body: JSON.stringify({ model: "router", messages, stream: true }),
      });
      if (response.status === 401 && attempt === 0) continue;
      if (!response.ok || !response.body) {
        const body = await response.json().catch(() => null);
        throw new Error(body?.error?.message ?? `Inference failed: HTTP ${response.status}`);
      }
      let text = "";
      let buffer = "";
      const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();
      for (;;) {
        const { done, value } = await reader.read();
        if (done) break;
        buffer += value;
        const lines = buffer.split("\n");
        buffer = lines.pop() ?? "";
        for (const line of lines) {
          const data = line.startsWith("data:") ? line.slice(5).trim() : "";
          if (!data || data === "[DONE]") continue;
          const delta = JSON.parse(data).choices?.[0]?.delta?.content;
          if (delta) {
            text += delta;
            onDelta?.(delta);
          }
        }
      }
      if (!text) throw new Error("Inference returned no text");
      return text;
    }
  };
}

Create one client per signed-in session, for example const chat = createInferenceChat(() => actor.getInferenceCredentials());.

  • Show first, save after. Render the reply as it streams. Then save what

the app keeps (a chat exchange, a translation) with an update call that does not block the UI and retries on failure. Say in the UI copy or docs that the last exchange can be lost if the tab closes before the save lands.

  • Ask for JSON in the prompt when the feature needs fields (a translation

with an article and a base form), and parse the final text, not the deltas.

  • Never put the key in localStorage, sessionStorage, cookies, URLs, or

logs.

from this repository

More skills

All skills
caffeinelabs
Community

connector-tmdb

MANDATORY recipe for every Caffeine build that reads movie, TV or people data from a canister. The supported path is the tmdb-client mops package (The Movie Database Web API v3) over outbound HTTPS, authenticated with a v3 API key or a v4 read-access token. Hand-rolling ic.httprequest calls to api.themoviedb.org is a FORBIDDEN anti-pattern — it bypasses the non-replicated-outcall safeguard, the generated JSON decoding of ~720 response models, and the credential handling. Load this skill whenever

1h installs
7
GitHub stars
2
Updated
Nov 19
caffeinelabs
Community

connector-googlecalendar

MANDATORY recipe for every Caffeine build that lists upcoming events or creates events on the user's own Google Calendar. The ONLY supported path is the googlecalendar-client mops package (Calendar REST API v3) combined with the google-oauth mops package (token exchange + refresh + PKCE). Hand-rolling ic.httprequest calls to oauth2.googleapis.com or www.googleapis.com/calendar/v3 is a FORBIDDEN anti-pattern — it bypasses bearer auth, replication-cost safeguards, and the google-oauth library's pe

1h installs
7
GitHub stars
2
Updated
Nov 17
caffeinelabs
Community

connector-googledrive

MANDATORY recipe for every Caffeine build that lists, reads, creates, shares, or organizes files and folders on the user's own Google Drive. The ONLY supported path is the googledrive-client mops package (Drive REST API v3) combined with the google-oauth mops package (OAuth 2.0 token exchange + refresh + PKCE). Hand-rolling ic.httprequest calls to oauth2.googleapis.com or www.googleapis.com/drive/v3 is a FORBIDDEN anti-pattern — it bypasses bearer auth, the isreplicated = ?false replication-cost

1h installs
7
GitHub stars
2
Updated
Nov 17
caffeinelabs
Community

connector-googlemail

MANDATORY recipe for every Caffeine build that sends email through the user's own Gmail account. The ONLY supported path is the googlemail-client mops package (Gmail REST API) combined with the google-oauth mops package (token exchange + refresh + PKCE). Hand-rolling ic.httprequest calls to oauth2.googleapis.com or gmail.googleapis.com is a FORBIDDEN anti-pattern — it bypasses bearer auth, replication-cost safeguards, and the google-oauth library's percent-encoding and JSON parsing. Load this sk

1h installs
7
GitHub stars
2
Updated
Nov 17