Skip to content

ADR-0004: External market data — ownership, storage, and access

  • Status: Accepted (Phase 1–2 implemented — kasas market_* read-through cache
  • Alpha Vantage; sillview Benchmark Comparison & Market Series widgets, provider settings, capability gating, and mock coverage)
  • Date: 2026-06-12
  • Deciders: Paul Meier
  • Related: ADR-0001 · ADR-0002 · kasas Integration → The CORS broker · Widget Catalog · kasas ADR 0006 (the backend half of this decision, in kasas's own ADR log — each repo numbers its own; lands with kasas PR #125)

Update (2026-06-12): the provider settings UI moved out of the Settings dialog onto the Sources page. Market data is now a source card alongside SimpleFIN/Plaid/Teller, and every source opens a detail page (/sources/:type) holding its credentials and config — for market, the API key (a single credential, through the same form as a bank token) plus the series manager. The decision below is otherwise unchanged: keys and cache stay server-side via kasas's admin-tier source-credential routes. References to "Settings → Kasas → Market" below are superseded by this.

Context and problem statement

The motivating user story: "How is my mutual fund doing compared to the S&P 500?" Answering it requires data that is not — and never will be — in the user's ledger: benchmark index levels, fund NAVs, FX rates. Call this external market/reference data: time series about the world, as opposed to the ledger's facts about your money.

Today neither side of the system has any of it:

  • kasas stores organizations, accounts, transactions, labels, extensions, and relationships — and nothing else. There is no prices table, no securities/ticker model, no FX, no valuation. Amounts are exact decimal strings; each account has a single currency code and a single latest balance + balance_date (migrations/sqlite/00001_init.sql).
  • Sillview is deliberately thin: a display layer whose renderer can only reach kasas through the main-process CORS broker (kasas Integration). It fetches nothing else and stores only dashboards locally.

Meanwhile kasas already owns a complete ingestion machine: a source.Source / Puller interface with six pull sources (SimpleFIN, Teller, Plaid, Bitcoin, Ethereum, CSV), a gocron-driven poller with cursors and a serialized sync, a sync_log, an event stream, and per-source runtime credentials (internal/source/source.go, internal/poller/poller.go). Its Archetype enum already reserves slots beyond pull — file, webhook, manual, enrichment — with a comment that new archetype interfaces "will be added as those archetypes are built."

Three questions need one coherent answer:

  1. Who owns external data — kasas or Sillview?
  2. Where does it live? (The proposal on the table: a second SQLite database, independent of the ledger.)
  3. How do widgets mesh it with ledger data?

Decision drivers

  • One source of truth, many consumers. kasas is a headless service with other clients than Sillview (API keys, webhooks, rules, plugins). Data only Sillview can see is data the rest of the system can't react to.
  • The CORS broker is the security model. The renderer has no network; everything flows through window.api.kasas.request(). ADR-0001's entire Tier 1–2 safety story rests on user specs being pinned to kasas paths, never a host. Any design that puts market data behind a non-kasas API forces a second datasource type into the spec model and erodes that invariant.
  • Ledger purity. The ledger is exact, sourced facts about the user's money. World data is a rebuildable cache — different provenance, different retention, different blast radius. The design must keep them separable even if co-located.
  • Decimal-string discipline. kasas never floats money; index levels and NAVs get the same treatment (a price is money per unit).
  • Licensing reality. Market data is licensed IP. Personal use of fetched data is one thing; redistribution is almost universally prohibited (see Devil's advocate).
  • "Very vanilla." Prefer extending existing, conventional machinery (sources, poller, migrations, read-tier REST) over inventing parallel infrastructure.
  • Offline dev must keep working. New endpoints need KASAS_MOCK=1 coverage.

Considered options

Option A — Sillview owns it (main-process adapter + local store)

Fetch quotes in the Electron main process and persist them beside dashboards.json (e.g. a market.db in userData). This is the route ADR-0002's kind (b) sketched — it even names "an FX rate, a quote" as examples.

  • Good: fastest to ship — TypeScript only, no cross-repo coordination, no kasas release. The egress-allowlist model is already designed in ADR-0002.
  • Bad: the data is trapped in the display layer. Headless kasas consumers, plugins, webhooks, and rules can never see it. Sillview grows a second ingestion engine (scheduler, retry, cursors, credential storage) duplicating what kasas already has in Go. The "one deliberate hole" in the broker boundary becomes a load-bearing data plane rather than a rare exception. Worst: ADR-0001's query-builder specs are pinned to kasas paths — benchmark data behind a Sillview API means either specs can't use it or the spec model grows a second datasource kind, contaminating the declarative ladder's security story.

Option B — A kasas plugin fetches it (net:fetch)

Use the existing plugin system: a marketplace plugin pulls prices (per-host egress grants already exist, 00016_add_plugin_net_grants.sql) on OnSyncComplete.

  • Good: zero core changes; the kasas-plugins registry is a ready distribution channel; egress is already capability-gated per host.
  • Bad: plugins have nowhere to put a time series. Their write capabilities are labels:write / extensions:write — both attach to transactions. Stuffing daily index closes into transaction extensions is an abuse of the data model, and plugin pages render HTML, not queryable series, so widgets couldn't read the result cleanly. Making this work means adding plugin-owned storage plus custom data endpoints to the SDK — real core work that converges on Option C with extra indirection. Revisit if/when the plugin SDK grows durable storage.

Option C — kasas owns it as a first-class subsystem (chosen)

A new market source (a new source archetype alongside pull), storing series into a dedicated market_* namespace, served by new read-tier endpoints. Sillview consumes it through the existing broker like any other kasas data.

  • Good: reuses the poller, cursors, sync_log, events, and credential machinery that already exist; every consumer (Sillview, specs, plugins, webhooks, API users) sees the same data; the broker boundary stays intact and ADR-0001's "fixed kasas endpoint enumeration" simply grows two entries.
  • Bad: scope creep for a ledger — kasas takes on provider churn, API quotas, and key UX forever. Nothing ships until a kasas release lands and Sillview's bundled binary updates. Slower first pixel than Option A.

Storage sub-decision: separate SQLite file vs separate namespace

The original proposal was a second SQLite database independent of the ledger. Evaluated honestly:

C1 · separate market.db file C2 · same DB, market_* tables (chosen)
Ledger purity physical — strongest logical — by contract (no FKs, wipe command)
Wipe/rebuild delete the file kasas market reset truncates the namespace
sqlite ↔ Postgres parity broken — Postgres deployments need a second database or a sqlite sidecar identical on both dialects (Postgres: same tables, optional schema)
Plumbing second store.go, second migration chain, second connection one migration chain, additive files
Cross-source queries ATTACH (sqlite-only) or app-level app-level joins (the plan anyway)
Data volume a daily series is ~252 rows/year — tiny either way same

The independence the separate file buys is real but achievable by contract at a fraction of the plumbing: market tables carry no foreign keys into ledger tables, are excluded from data exports by default, and are rebuildable from provider + symbol + date range alone. The separate file's one decisive advantage (physical purity) doesn't outweigh breaking the dual-dialect store abstraction. Choose C2, and record C1 as the fallback if the cache ever grows write patterns that measurably hurt the ledger DB (WAL churn, backup time).

Decision outcome

Adopt Option C with storage C2. kasas owns fetching, caching, and serving of external market/reference data — on demand, through a server-side read-through cache with a TTL (amended from this ADR's original scheduled-copy model: copies go stale and accrue backfill obligations; a demand-driven cache self-heals and burns quota only when someone is actually looking); Sillview owns visualization and comparison math.

flowchart LR
    P["market provider<br/>(server-side key)"] -->|"fetch on demand · TTL"| S["kasas market source<br/>(read-through cache)"]
    S --> M[("market_* tables<br/>TTL cache · no FKs")]
    L[("ledger tables")] -." same DB, separate namespace ".- M
    M --> API["GET /api/v1/market/…<br/>read tier"]
    API -->|existing CORS broker| W["Sillview comparison widgets<br/>normalize + chart"]

This narrows ADR-0002's kind (b): market/reference data used for analysis must come through kasas. Kind (b) main-side adapters remain for capabilities that genuinely cannot live in the backend, but "an FX rate, a quote" is no longer the canonical example — it is exactly what this ADR routes through kasas instead.

Because the decision spans two repos, it is recorded twice with a clear split of authority: kasas ADR 0006 is the canonical record for the backend design (the fetch/caching model, source archetype, market_* schema, API routes, provider model); this ADR is canonical for the Sillview-side consequences — broker-only consumption, the kind (b) narrowing, widgets, and mock coverage. The backend sketches below are context, not the contract; if they drift, kasas ADR 0006 wins.

Detailed design

kasas: schema (illustrative)

-- World facts, not ledger facts: a rebuildable cache. No FKs into ledger tables.
CREATE TABLE market_series (
    id        TEXT PRIMARY KEY,   -- stable internal id, e.g. "sp500tr"
    provider  TEXT NOT NULL,      -- e.g. "stooq" — provider choice is ADR-0005
    symbol    TEXT NOT NULL,      -- provider-native symbol
    kind      TEXT NOT NULL,      -- index | fund | equity | fx | crypto
    currency  TEXT NOT NULL,      -- ISO code values are quoted in
    adjusted  INTEGER NOT NULL,   -- 1 = total-return / split-adjusted series
    meta      TEXT                -- JSON: display name, license note, …
);

CREATE TABLE market_points (
    series_id  TEXT NOT NULL,
    date       TEXT NOT NULL,     -- ISO-8601 date; daily close granularity
    value      TEXT NOT NULL,     -- decimal STRING — same discipline as money
    fetched_at INTEGER NOT NULL,
    PRIMARY KEY (series_id, date)
);

Daily closes only. Intraday is an explicit non-goal: it multiplies quota cost, storage, and provider complexity for a personal-finance dashboard that compares months, not minutes.

kasas: the market source (read-through cache)

The fetch model is on demand, not a scheduled copy — recorded canonically in kasas ADR 0006; the shape in brief:

  • The API read path is the trigger: fresh cache → serve; stale → serve immediately and refresh in the background (stale-while-revalidate), then emit a market.updated event that the widgets' existing event subscription picks up; cold → one synchronous fetch under a timeout.
  • Single-flight and per-provider rate limiting live at the server — concurrent requests for the same series collapse to one provider call, and quotas are enforced where they are actually shared.
  • fetched_at in the schema above is load-bearing: freshness is a read-time TTL policy (a daily series is stale once a newer close should exist). Refreshes upsert the viewed window, which also self-heals retroactive restatements of adjusted series — something the scheduled-copy model would have needed detection logic for.
  • Still a registered source (new reference archetype): it inherits source config, runtime credentials (never config.toml), readiness reporting, and the generic POST /api/v1/sources/{type}/sync — which now means warm the cache, an optional convenience rather than the primary path. The provider sits behind a small interface so the first pick (ADR-0005) isn't load-bearing.

kasas: API (read tier)

  • GET /api/v1/market/series → { "series": [ … ] } (named-key wrapping, per convention)
  • GET /api/v1/market/series/{id}/points?since=…&until=… → { "points": [ … ] }

Both readable with the dashboard token, like other read-tier routes.

Sillview: comparison widgets

  • A Benchmark comparison widget: pick an account + a series; render "growth of $10k" — both lines normalized to 100 at the window start. Pure broker traffic (kasas.request({ path: '/api/v1/market/…' })): no new IPC, no new trust surface, no renderer egress.
  • Chart copy must be honest: label the series as price vs total-return, and the account line as balance (which includes deposits) — see Devil's advocate.
  • Settings are the config surface; the engine stays remote-safe. Provider toggles and API-key entry live in the existing Settings dialog and flow through kasas's admin-tier source-credential routes — the same path bank credentials use. This works identically for the bundled backend and a remote kasas: keys and cache live with the server, every connected dashboard shares one cache and one provider quota, and a connection holding only a read-only API key sees provider settings as read-only (it can chart series, not reconfigure them).
  • Capability-gate the widget (ADR-0002). A remote kasas may predate /api/v1/market/*; the widget declares the requirement and degrades to the actionable "requires kasas ≥ vX" tile, never a broken chart.
  • ADR-0001 synergy: the market endpoints join the fixed kasas endpoint enumeration, so Tier-1 query-builder specs gain benchmarks with zero changes to the spec security model.
  • Mock mode: mock.ts grows /api/v1/market/* routes serving a seeded synthetic series generated relative to now (same pattern as existing fixtures).

Devil's advocate

The case against — recorded so we walk in clear-eyed.

  • The comparison the user actually asked for cannot be computed honestly yet. An investment account's value moves with the market between syncs with no transaction; kasas keeps only the latest balance. A historical account-value series cannot be reconstructed from transactions, so "my fund vs the S&P" is, in phase 1, a benchmark overlay for context — not performance attribution. Honest time-weighted comparison needs balance snapshots (follow-up ADR-0006). Shipping a chart that implies more is worse than no chart.
  • Naive overlays mislead by construction, twice. (1) The S&P 500 most people quote (SPX) is a price index; a fund balance includes reinvested dividends. Comparing them flatters the fund by ~2%/yr compounded — prefer total-return series (adjusted exists in the schema for this reason) and label which is shown. (2) Account balances grow with deposits; an account receiving monthly contributions "beats" any index. Without flow-adjusted math (ADR-0007) the widget must say "balance", never "return".
  • Licensing is a minefield. Index levels are licensed IP (S&P DJI); most providers' terms prohibit redistribution even of "free" data; unofficial Yahoo endpoints violate ToS outright. Fetched data is for the user's own display and analysis — never redistributed — and per-provider attribution requirements have to be honored in the chart copy.
  • Provider risk is permanent ops burden. Free tiers are tight (Alpha Vantage: ~25 req/day) and providers die (IEX Cloud, 2024). Users must bring their own key for anything beyond a keyless default; quotas, backoff, and provider migration (symbol remapping across providers) are forever-costs kasas is signing up for. Mitigation: provider-agnostic interface, stable internal series ids, daily granularity.
  • Scope creep. kasas's identity is "exact facts from your institutions." This ADR makes it also a (small) market-data platform. The fence: kasas stores series the user configured, full stop — no securities master, no symbol search, no fundamentals, no intraday. Each of those would need a new ADR on purpose.
  • Cross-repo latency. Option A would demo in a weekend; Option C needs a kasas release plus a Sillview bundled-binary update before the first pixel. Accepted: this is infrastructure, and the self-update path already exists.
  • A second egress class. kasas already calls banks; now it also calls market providers — new hosts in the user's threat model (the only leak is ticker interest, but it should be visible/allowlisted like plugin net:fetch grants, not silent).

Consequences

Positive

  • One copy of the data, visible to every consumer: widgets, query-builder specs, plugins, webhooks, rules, and headless API users.
  • Sillview's security model is untouched — the renderer still reaches exactly one host through one broker, and ADR-0001's spec invariant ("a path, never a host") survives unchanged.
  • Reuses kasas's ingestion machinery nearly wholesale; the ledger stays pure via the no-FK, rebuildable-cache contract.
  • Valuation (BTC→USD, EUR→USD) becomes a natural future client of the same tables (ADR-0008) instead of a separate system.

Negative / risks

  • kasas permanently owns provider churn, quotas, and key UX (mitigations above).
  • The migration chain now carries non-ledger tables; market migrations must stay additive and isolated so a bad one can't threaten ledger data.
  • The first user-visible comparison is honest-but-modest (overlay, not attribution) until ADR-0006/0007 land — expectation management in the widget copy is part of the deliverable.
  • Cold-cache latency: the first view of a never-fetched series blocks on a server-side provider round-trip; subsequent reads are cache-instant (stale-while-revalidate). Widgets must treat it as ordinary loading, not error.
  • Mock mode grows synthetic-series maintenance.

Implementation plan

  • Phase 1 (kasas): market_* cache migrations + store methods; demand-driven read-through fetch (TTL, single-flight, stale-while-revalidate
    • market.updated event) behind a provider interface; optional cache-warm via the generic per-source sync; GET /api/v1/market/* read routes; kasas market reset. (Alpha Vantage provider; verified end-to-end.)
  • Phase 2 (Sillview): Benchmark-comparison widget + Market Series widget through the existing broker; provider settings UI (Settings → Kasas → Market, admin-tier; read-only for API-key connections); ADR-0002 capability gating (useMarketAvailable) for backends without /api/v1/market/*; honest labeling (price vs total-return, "balance ≠ return"); KASAS_MOCK routes for /api/v1/market/*.
  • Phase 3: add the market endpoints to ADR-0001's Tier-1 fixed endpoint enumeration so query-builder specs can chart benchmarks.
  • Phase 4+: governed by the follow-up ADRs below — notably balance snapshots (ADR-0006) before any chart says "return".

Follow-up ADRs

Each is a separable decision deliberately not made here:

  • ADR-0005 · Market-data provider selection, credentials & licensing — which provider(s) ship first (keyless default vs bring-your-own-key), quota/backoff policy, ToS review per provider, symbol-mapping strategy across providers. A candidate matrix (Alpha Vantage, Finnhub, Market Data, Alpaca, MarketStack, plus Tiingo/Stooq/FRED) and two constraining findings — the S&P 500 index level is licensed IP free tiers often won't serve (ETF or FRED proxy), and mutual-fund NAV coverage varies sharply — are recorded in kasas ADR 0006.
  • ADR-0006 · Account balance history (snapshots) — record per-account balance at each sync into a history table; the prerequisite for any honest performance chart. Possibly extends to a holdings/positions/units model for brokerage accounts (units × NAV beats balance snapshots when available).
  • ADR-0007 · Performance & benchmark methodology — time-weighted vs money-weighted return, total-return vs price indices, normalization windows, contribution handling; the math that turns "overlay" into "comparison".
  • ADR-0008 · Valuation & currency conversion — FX/crypto series from the same market_* infrastructure powering a base-currency view of multi-currency net worth; where conversion is computed (kasas vs widget) and how it is labeled.

Open questions

  1. Which provider first, and is there a keyless default so the feature works out-of-the-box? (Decided in ADR-0005; the provider interface here must not prejudge it.)
  2. Should ADR-0006 (balance snapshots) land before Phase 2 so the first shipped widget can show a real value-over-time line instead of overlay-only? The snapshot table is cheap; the argument for sequencing it first is strong.
  3. Series identity: are internal ids (sp500tr) minted by kasas with provider-symbol mapping in meta, or are provider symbols the id? Affects provider migration later; lean internal-id.
  4. Export/backup posture: confirmed that market_* is excluded from data exports by default, or should it be opt-in?

References