sense-setup

sense: setup and tree design

Querying an existing tree is the sense skill. This one covers making a tree: installing, writing presets, and the design decisions a tree owner faces. Worked configurations for common tree shapes: EXAMPLES.md.

Setup

• npm install -g sensemaking, then sense init at the tree root writes sense.config.json: two presets (default, and large showing what a big tree tunes) and an embed block naming the model. The model fetches once per machine at the first vector search (progress on stderr); sense download prefetches it instead where that timing matters (CI, air-gapped setup). Config discovery walks up from cwd; --config <path> overrides.
• Backing store. The config's store key: sqlite (default, zero-dependency, Node's built-in SQLite), or the experimental duckdb and turso, whose engine package the first command that opens such a tree installs on its own (@duckdb/node-api, a one-time native download of about 110 MB; @tursodatabase/database, much smaller). The same commands and tables run on all three. Two things do not port, and each one decides a tree. FTS5 syntax: under duckdb and turso, search text and raw MATCH reject FTS5's prefix, boolean, NEAR, initial-token and column-filter operators with a named error, and sqlite's FTS5 SQL (MATCH, snippet(), bm25()) does not run, so saved queries written in that syntax are sqlite dialect; a tree whose saved queries or search vocabulary depend on FTS5 operators stays on sqlite. SQL functions: has/basename run on all three (turso rewrites them into portable SQL rather than registering them); segment runs on sqlite and duckdb only, so a tree whose queries call segment stays off turso. sense watch runs on all three; duckdb and turso lock their cache file per connection, so a concurrent command waits out the watcher's current cycle instead of failing. Each store keeps its own cache file (.sense/cache.db, .sense/cache.duckdb, .sense/cache.turso.db); switching stores is a rebuild, not a migration. Speed is the other axis, and it does not follow from any of the above. sqlite is fastest at both building an index and querying one. duckdb and turso both cost noticeably more to build, turso the most on a large tree, since its engine costs more per write and more again to maintain each index; both then answer queries at close to sqlite's speed. Speed is not the only reason to choose, though: the cache is an ordinary database file, so a tree indexed under duckdb is readable by anything in DuckDB's ecosystem, and turso brings concurrent access, non-blocking I/O and encryption. Pick for what you intend to do with the file, then check the per-store figures. Per-store figures: BENCHMARKING.md in the sensemaking repo.
• Globs resolve relative to the config file, never the cwd.
• sense status and sense map show each preset's coverage (files matched, embedded count), so what a config actually indexes is always visible in output. A config edit that changes coverage rebuilds the cache and names the preset that caused it on stderr.

Presets

A preset is a named, self-contained bundle of settings. default (required) is what bare commands use; every other preset is addressed by name (sense search "..." --preset raw, or "preset": "raw" in a saved search). No inheritance: what a preset states is all it does.

Indexing derives from presets. A file is indexed if any preset includes it. Consequences worth designing around:

• Files no preset includes are not indexed at all.
• Presets may overlap; they are views, not partitions.
• Global features (links, sections, rank) still toggle tree-wide; most trees never touch them.

Vectors take two decisions, in two places. The top-level "embed": { "model", "provider": "static"|"openai"|"cohere", "url", "key", "chunkTokens" } block names the model and says whether the tree has vectors at all. chunkTokens is a chunk size ceiling in estimated tokens for small-context models; default 500. A preset's signals says which engines that scope uses: a layer searched for exact wording (ingested sources, archives, generated output) declares "signals": {"words": 1, "links": 1}, costs no embedding, and its searches run on words and links. That is the main scale lever, and it is the llm-wiki split: compiled pages searched by meaning, raw sources searched for the phrasing you are citing. Each named signal's number is its RRF weight, not a toggle. 1 is the default and reproduces equal-weight fusion; a preset can instead raise one signal's number, e.g. {"words": 1, "vectors": 4}, to shift the fused ranking toward that signal without dropping the others. Whether that helps is corpus- and model-contingent, not a fixed rule: benchmark/reports/2026-08-27-embedding-model-selection.md's weight-sweep table measures equal weight against {0.5, 1, 2, 4} and vectors-only on nfcorpus and on MIRACL zh with an HTTP encoder, and the two corpora do not agree on which weight wins. static is the built-in pure-JS Model2Vec loader and handles paraphrase and reworded concepts; tight domain jargon ("heart attack" for "myocardial infarction") is where an openai- or cohere-shaped encoder model tends to do better, measured in the same report's encoder-tier tables. Naming the model in the config is the consent to fetch it: the first vector search downloads it once per machine into ~/.sense/models (or sense download prefetches) (huggingface_hub's cache layout, one snapshot directory per resolved revision), so several models coexist and switching between them rebuilds the index rather than mixing vector spaces. A model holding a path instead of a Hugging Face id points at a local directory, which sense download reports as nothing to fetch. The first search after that embeds the tree (progress on stderr; minutes on tens of thousands of notes, seconds on small trees). A config edit that changes coverage or features rebuilds the cache, vectors included, so settle presets before the first vector search on a large tree or that embedding run is paid twice.

Finding a model. The default, potion-retrieval-32M, is English only. A tree whose text is mostly a language the model does not declare fails loudly at embed time (EMBED_MODEL_MISMATCH, naming the fix), and sense status shows the detected language mix beside the model's declared languages. A model whose card declares no languages leaves that check off, and a mismatched pairing then degrades silently, nearest-neighbour search always returns a neighbour regardless of fit, so check the card's languages yourself when the tree is not English. Picking a model is a lookup against the source, not a name to memorize: for a static model, filter Hugging Face's model2vec library for the target language and read the card for its declared languages, its safetensors shape, and F32 weights (an int8 republish fails the loader's dtype check by design). For an encoder reached over HTTP, Ollama's embedding-model library and LM Studio's catalog list each model's languages and context length; both serve an OpenAI-shaped endpoint, so provider: "openai" with url pointing at the local port reaches either with no other config. Cohere's hosted models are reached with provider: "cohere" instead. One dated measurement stands in for a name table that would go stale: potion-retrieval-32M was the best-measured static English retrieval model as of 2026-08, per benchmark/reports/2026-08-27-embedding-model-selection.md's static-ladder table.

Large vaults: everything except the vector build is measured linear to 100k notes with no tuning (BENCHMARKING.md). The knobs that matter are k (more, smaller results; rows carry lines section ranges, so agents read sections, not files) and "signals": {"words": 1, "links": 1} on the layers that do not earn vectors.

Tree design decisions

These belong to the tree's owner. sense works with any of them and reads no instruction files of its own; each choice only changes what queries can do.

• Frontmatter fields. Columns are discovered per tree: whatever keys notes declare become queryable. Consistent fields across notes make SQL filters and saved queries possible (WHERE status = 'active'). The store's column limit bounds distinct keys per tree: sqlite's compiled 2,000 (sqlite.org/limits.html), turso's 2,000 result-set column limit, duckdb's 10,000 sanity fence (it has no compile-time cap). The crawl stops with an error naming the count and the levers. Reserved keys (dropped with a warning): path, _mtime, _ctime, _size, _rank, content, links, sections. Values keep their YAML type: strings TEXT, whole numbers and booleans INTEGER (true is 1), fractions REAL, lists and maps JSON text; map prints the observed type per field.
• Presets are path-shaped; frontmatter is state-shaped. A preset's coverage must be computable from the path alone (it decides indexing, baked into the cache). Volatile state (status, project, dates) lives in frontmatter and filters at query time (where, has(), datetime()). A state worth different indexing (retired memory, superseded sources) is a state worth moving the file: the archive-folder pattern in EXAMPLES.md.
• What a note omits is also a filter. A layer that deliberately carries none of the fields the saved views filter on is excluded from all of them without any view naming the layer. Sparse fields cut both ways: less of the tree filters when you want breadth, and exactly this separation when layers differ in authority.
• Dates. datetime() comparisons work for dates written as ISO 8601, the only format it parses. A tree that mixes date formats can store them, but can't compare them in SQL.
• Language. No decision needed. A language written without word spaces (Chinese, Japanese, Thai, Khmer, Lao, Burmese) is indexed per grapheme and searched as an ordered grapheme phrase, so sense search "全文" finds what it should. This is substring semantics, what grep gives: a query matches wherever its exact text occurs, including inside a longer run, and needs no minimum length. A language written with spaces is left exactly as it was, storing nothing extra, and its stemming is unaffected. The one place this does not reach is a hand-written content MATCH '...', which cannot be rewritten for its author: pass the terms through segment() there. content.tokenize is a separate lever with a different purpose, substring matching inside a Latin word (trigram) or keeping hyphenated terms whole (unicode61 tokenchars '-_'); naming one turns the grapheme-phrase scheme off, since the tree has then chosen its own scheme. Changing it rebuilds the text index only; vectors, links, and sections are kept.
• Summaries. A one-line summary: is optional and pays twice: it shows in every result row (often answering a question with no file read) and is a weighted search field ranked above body text. The cost is writing and maintaining the line as notes change.
• Folder shape. Globs find the files, paths are queryable text, links resolve by basename at any depth, but presets make folders meaningful: a folder is the natural unit that gets its own coverage and settings.
• Note size. Many small notes: precise search hits, whole-file reads stay cheap, more links to maintain. Fewer large notes: sections, peek, and the lines column carry the cost down to line-range reads. Both work. A long section no longer becomes one oversized vector either: chunking splits at headings and caps each chunk around 500 estimated tokens (CJK counted 1:1, spaced scripts ~4:1), so an oversized note degrades to more, smaller chunks rather than one truncated one; embed.chunkTokens lowers that cap further for a small-context model.
• Recurring questions. Save a scenario an agent will repeat under queries, naming the verb it runs: { "sql": "..." } for filters and reports, or { "search": "...", "preset": "raw", "k": 5 } for a ranked search. Either runs as sense <name>, and running one is how it is validated: a typo'd column or an unknown preset errors and exits nonzero. A parameterised entry validates with any argument, since SQL is prepared before parameters bind.
• Where decisions live. Choices that should outlive one conversation can be recorded in the agent's own instruction or skill files, or in a note in the tree itself; a one-off search over an existing corpus needs none of that.