Docs

Scorecards and candidates

Two small JSON formats let a model evaluator and Lodestone feed each other without knowing about each other's code. Naija Eval is the first evaluator to use them.

   evaluator (Naija Eval)                         Lodestone (in your tool)
   ──────────────────────                         ────────────────────────
   measures answer quality  ── model-scorecard ──▶  orders models by it, per task
                                                      │
   benchmarks what is in use ◀─ model-candidates ──   lists the models it can call
                                                      │
   (reads live reliability) ◀── model-scorecard ───   records success rate and speed

Nothing here is automatic or hidden. A scorecard changes the order models are tried in, for tasks you name, and nothing else. A model nobody measured is never dropped, added or moved.

model-scorecard/1

Measured results about models.

{
  "schema": "model-scorecard/1",
  "source": "naija-eval",
  "generatedAt": "2026-10-28T09:00:00Z",
  "scores": [
    {
      "model": "anthropic/claude-sonnet-5.5",
      "task": "translate",
      "metric": "chrf++",
      "value": 32.3,
      "higherIsBetter": true,
      "n": 100,
      "measuredAt": "2026-10-08",
      "language": "yo",
      "slice": "en->yo"
    }
  ]
}

parseScorecard() checks all of this and lists every problem. A scorecard with a different schema version is refused, not guessed at.

Using one

import { createLodestone, fetchScorecard } from '@extrafemi-ai/lodestone';

let card; let at = 0;
const ai = createLodestone({
  openrouter: { apiKey: process.env.OPENROUTER_API_KEY },
  // Called per request; keep your own cache. If it fails or takes over 1.5 s, the order is left alone.
  scorecards: async () => {
    if (!card || Date.now() - at > 6 * 3_600_000) { card = await fetchScorecard('https://naija-eval.extrafemi.com/data/scorecard.json'); at = Date.now(); }
    return [card];
  },
  profiles: {
    'translate-yoruba': { tier: 'strong', quality: { task: 'translate', metric: 'chrf++', language: 'yo', slice: 'avg' } },
  },
});

await ai.complete({ task: 'translate-yoruba', prompt });

The rules, so there are no surprises:

• Scores based on fewer than 30 items, or older than a year, are ignored (minN, maxAgeDays to change).
• Measured models are sorted among the slots they already hold. An unmeasured model keeps its slot, so a good unmeasured model is never pushed down by a weaker measured one.
• It applies to the free, paid and fallback model lists. It never reorders an explicit route.
• With two scorecards measuring the same thing, the one based on more items wins.

Live reliability

An evaluator says how good an answer is. It can't say whether you get an answer at all. Lodestone records that itself:

import { createStats } from '@extrafemi-ai/lodestone';

const stats = createStats();                 // last 200 attempts per model and provider
const ai = createLodestone({ /* ... */, stats });

stats.scorecard();   // success_rate, latency_p50_ms, latency_p95_ms per model, as a model-scorecard/1
stats.toJSON();      // save in your store; stats.load(saved) after a restart

The result is an ordinary scorecard (source: "lodestone-live"), so it can be served, shared, or used to order models: quality: { task: 'availability', metric: 'success_rate' } puts the model that answers most often first. Stats never see prompts or answers.

model-candidates/1

The models a tool can actually call, so an evaluator benchmarks those and not a guess.

const candidates = await ai.candidates();

{
  "schema": "model-candidates/1",
  "source": "lodestone",
  "generatedAt": "2026-10-28T09:00:00Z",
  "models": [
    { "id": "google/gemma-4-31b-it:free", "provider": "openrouter", "roles": ["free:strong", "free:vision"] },
    { "id": "qwen/qwen3.7-flash", "provider": "openrouter", "roles": ["paid"] }
  ]
}

Serve it from an endpoint or write it to a file. Naija Eval's eval/candidates.py reads it and prints the exact commands for the models it hasn't measured yet.