Research Dossier · Module 2 of 2

GeoScore: Calibrated LLM Event Extraction for Market Reaction

Author: ZenTrader Core Team  ·  Date: July 31, 2026  ·  Status: Active daily research — draft protocol, not yet peer-reviewed or preregistered

Read the CMA Research Dossier Read the Whitepaper

Executive Summary

GeoScore is the second module of ZenTrader's modular research programme, alongside the Cognitive Memory Architecture (CMA). Where CMA asks how an agent's memory can be temporally valid and evidence-governed, GeoScore asks a narrower, sharper question: can a large language model, used strictly as a perception layer that extracts structured, evidence-quoted facts from news and social events, feed a deterministic scoring formula that produces a calibrated, decayed, directional signal for asset price reactions — measured honestly, with no shortcut around a real prediction problem?

This dossier reports what is verified today, including results that are inconclusive. GeoScore has been in daily, continued development since 2026-07-06; the numbers below are a snapshot, not a final result.

1. Purpose and Status of This Document

LabelMeaning
ImplementedVerified directly against source code, the golden-set evaluation, and the live production database.
PartialA working foundation exists; complete validation is not yet possible (usually: sample too small).
Open questionActively measured, genuinely undecided as of this writing.

All figures in this dossier were checked directly against trading/geoscore/, the git history of golden_events.json/prompt.py/engine.py, and the production geoscore_* tables on 2026-07-31 — not copied from an earlier summary.

2. The Research Problem

Research problem. Can structured, evidence-quoted LLM extraction of geopolitical, macroeconomic, regulatory, corporate, and social events, scored by a deterministic and fully auditable formula, produce a directional market-reaction signal that is measurably better calibrated than (a) an unstructured LLM point-prediction and (b) a simple keyword/sentiment baseline — while keeping every scoring step traceable to a verbatim source quote?

This is deliberately a narrower and more falsifiable question than "can an LLM predict markets." The LLM never does arithmetic and never outputs a price target; it only classifies. A deterministic, unit-tested engine computes the number. This mirrors CMA's invariant that agents propose, deterministic services commit — GeoScore is that principle applied to qualitative narrative interpretation specifically.

3. Current Implementation, Verified Against Code, Golden Set, and Database

The classification prompt (trading/geoscore/prompt.py) forces a structured JSON output: event class, per-asset directional factor scores across six factors (geo, macro, regulatory, direct, narrative, flows), novelty, probability, and confidence — and every non-zero factor requires a verbatim supporting quote from the source text, so an unfounded score cannot silently reach the database.

The deterministic engine (trading/geoscore/engine.py) is fully auditable, five independent multiplicative terms:

AIS = raw_score(active factors, asset-class weights) × confidence × source_reliability × probability × novelty × decay(Δt, half-life)

Golden-set calibration — verified from git history, not restated from memory

StageChangeResult
v1 baseline (frozen)Initial prompt + 10-event golden set0/10
v2 calibrationPrompt redesign (trigger-channel rule, crypto asset-class mapping), any-of factor-sign matcher, widened bands7/10
v3 (current)5 real VIP/influencer events added; 4 of the new events' own expectations corrected against real model output (not the model)14/15, one remaining failure named as boundary noise

Note: an earlier public summary of this history cited "0/10 → 8/10" — that figure does not appear anywhere in the commit history and has been corrected on the whitepaper to the verified 0/10 → 7/10 → 14/15 trajectory above.

A fresh, live re-run of the full golden set was executed for this dossier on 2026-07-31 against the production model (qwen3.6:27b): 15/15 passed, including the previously-flaky musk-memecoin-post event. This was a single run, not the recommended self-consistency mode (--runs 3); given that event's documented history of flickering across runs, a 15/15 single-run result should be read as a good day, not as proof the flakiness is resolved — a self-consistency re-run is on the roadmap below before this is trusted as stable.

Persistence — append-and-chain, verified live

Re-scoring an event never overwrites a row: a new geoscore_event_scores row is inserted and chained via superseded_by, the same discipline as the gate_decisions evidence table described in the CMA dossier. Live volumes as of 2026-07-31: 4,195 events, 20,193 scores, 11,340 narratives ingested.

Outcome capture — live, observe-only, real preliminary results

trading/geoscore/outcome.py runs continuously in production (systemd ... --outcomes). Once a score's absolute AIS crosses the lowest watchlist threshold, an inert geoscore_signal_outputs row is opened (gates_passed={"mode":"observe"} — no size, no direction ever acted on) and back-filled with the realized percentage price move at three checkpoints (1h / 1d / 1w) from the candle store.

CheckpointTracked (n)Directional hit rate
1h975.0% (6/8, 1 flat)
1d1553.3% (8/15) — statistically indistinguishable from a coin flip
1w1464.3% (9/14)

Honest reading: 19 tracked signals is far too small a sample to claim predictive value at any horizon. This table is reported precisely because "we don't know yet, and here is exactly how we are measuring it" is the scientifically correct statement today — not a hedge, but the actual state of an open research question under continuous, dated measurement.

Data-quality finding from this review (2026-07-31). market_candles for at least one thinly-traded symbol (LCID) shows implausible intra-15-minute price oscillations between roughly $2.45 and $6.46 around 2026-07-14, across every stored resolution — almost certainly an upstream feed/ingestion artifact rather than real price action. The outcome percentages computed from that window are unreliable until the ingestion path is audited. Flagged here as an open item rather than silently excluded, and itself an example of the kind of concrete technical uncertainty this research programme exists to resolve.

Signal consumption remains explicitly OFF by design — the same default-OFF discipline as the observe/shadow policy layer described in the whitepaper. GeoScore does not influence any live order today.

4. Research Questions and Hypotheses

QuestionHypothesis
RQ-G1. Does structured, evidence-quoted extraction plus deterministic scoring produce better-calibrated directional forecasts than an unstructured LLM point-prediction or a keyword/sentiment baseline?The structured pipeline shows a narrower calibration gap (stated probability vs. realized frequency) than both baselines at equal sample size.
RQ-G2. Does exponential time-decay and novelty weighting reduce stale-event false signals relative to an undecayed baseline?Removing decay/novelty weighting (ablation) increases the false-signal rate on repeated or already-priced-in events.
RQ-G3. At what sample size does the directional hit rate become statistically distinguishable from 50%, and does this vary by event class (GEO/MACRO/REG/CORP/SOCIAL)?Different event classes reach significance at different sample sizes; SOCIAL/celebrity-driven events are noisier than REG/GEO events.
RQ-G4. Does the mandatory verbatim-quote requirement measurably reduce unsupported (hallucinated) non-zero factor scores versus unconstrained extraction?An audited sample shows a materially lower unsupported-claim rate under the quote-constrained prompt than under a control prompt without the requirement.

5. Design Principles

6. Validation Programme

Planned baselines, none yet run against the same tracked window: B0-Random (coin flip / random walk), B1-NaiveLLM (ask the model to predict direction directly, no structured decomposition), B2-Keyword (simple positive/negative keyword or sentiment-lexicon count), and the current deterministic engine with and without the decay and novelty terms (ablation). Metrics: directional hit rate per horizon and event class with Wilson (small-sample-appropriate) confidence intervals, a calibration curve (stated probability vs. realized frequency), and the evidence-quote audit rate from RQ-G4. No headline predictive claim will be published before reaching a pre-specified minimum sample per checkpoint and running it past the planned baselines — continued daily operation, not new engineering, is what closes that gap.

7. Limitations

  1. Sample size. 19 tracked signals total, most checkpoints below n=15; the hit-rate table above is a measurement snapshot, not a finding.
  2. No baseline comparison run yet. B0–B2 above are specified but not yet executed against the same window as the current engine.
  3. Single model version tested. All golden-set and production scoring to date uses one model (qwen3.6:27b); cross-model robustness is unverified.
  4. Market-data quality. The LCID candle anomaly above; outcome-capture correctness depends on an upstream data pipeline that has not been independently audited end-to-end.
  5. Golden Set size. 15 events catches prompt/engine regressions; it is not large enough to estimate classification accuracy with a tight confidence interval.
  6. No causal claim. Even a confirmed directional edge would show correlation between score and subsequent price movement, not that the extracted narrative caused it.

8. Roadmap

  1. Audit and fix the market-data ingestion path responsible for the LCID-class anomaly — a data-quality gate before any new metric is trusted
  2. Re-run the golden set under self-consistency (--runs 3) to confirm today's 15/15 is stable and not a lucky single run on the historically flaky event
  3. Expand the Golden Set beyond 15 events with deliberate per-event-class coverage
  4. Implement and run the B0–B2 baselines against the same tracked window
  5. Reach a pre-specified minimum tracked-signal count per checkpoint through continued observe-mode operation
  6. Publish calibration curves with Wilson confidence intervals, broken out by event class and horizon
  7. Only after a demonstrated, statistically distinguishable edge: design a gated, deterministic signal-consumption path — never a direct LLM-to-order path, consistent with CMA's "agents propose, deterministic services commit" invariant

Relation to the CMA Memory Architecture

GeoScore is not a separate product — it is the concrete, calibrated implementation of the Qualitative Narratives submodel described in the whitepaper's AAS section, and its append-and-chain evidence discipline (superseded_by, versioned golden set) is the template the CMA event ledger is meant to generalise across every memory class, not narrative scores alone. Both modules share the same Sovereign AI inference infrastructure and the same underlying research posture: state plainly what is measured, what is proposed, and what remains genuinely open.

This dossier accompanies, and does not replace, the ZenTrader Whitepaper and the CMA Research Dossier. It is an internal research protocol shared for technical and funding review; research questions, milestones, and figures are dated measurements from an actively continuing daily research programme, not final results or delivery commitments. Implementation and calibration figures were verified against source code, git history, and the production database on 2026-07-31 — treat any specific number as time-stamped, not evergreen.