New capability
See what every AI call cost and how it did
smrt records each model call as one row: which prompt asked, which model answered, the tokens, an estimated cost and how long it took. The command line turns those rows into reports and side-by-side comparisons.
Verified against smrt 0.62.10
Every call leaves one row
A model call made through is(), do(), describe() or the tracked client is written once to the _smrt_ai_usage table. The row holds the provider and model, token counts, an estimated cost, the duration and the tenant. It also holds the prompt key, when the call came from a registered prompt, and the tier that was asked for and the tier that answered.
- On by default. Tracking and saving to the table are both on unless you turn them off, and rows are saved whenever the app has a database.
- The cost is an estimate from a rate card. A call with no known rate is counted as uncosted. It is never priced at zero.
- A user is recorded only when the app supplies one. A row with no tenant source is a global row; smrt does not guess.
- Usage rows hold numbers and labels, not the text of the prompt or the answer.
import { config } from '@happyvertical/smrt-core';
config({
usage: {
enabled: true, // default
persist: true, // default: save rows when a database exists
estimateCosts: true, // default
// USD per 1K tokens, for a model the rate card does not know
costRates: { 'your-provider:your-model': { input: 0.001, output: 0.002 } }
}
});Runs group the calls that belong together
Some work takes several calls: a chat turn, a workflow, a fusion, or a run type your app defines. smrt lays a run over the usage rows of those calls in _smrt_ai_runs. A run has a status, a budget and a trail of events: calls, actions, observations, checks, approvals and handoffs. A call on its own is not a run.
- A run needs usage recording. With tracking or saving turned off, opening a run throws AiRunTrackingDisabledError before anything is sent.
- Run totals decide whether a budget is spent. Usage rows are the billing record.
The report you see
smrt ai:usage prints one line per group and a total. The columns are calls, blocked calls, tokens, estimated cost, median duration, substitutions (calls that recorded at least one fallback) and evaluation labels. Under the table it states how many calls had no prompt key, what was excluded from the totals and which rate card priced them.
- Group with --by, using any of prompt, model, tier, class, tenant, fusion, source and template. The default is prompt.
- Set the window with --since and --until, as a date or a relative span such as 24h, 7d or 2w.
- --tenant totals one tenant and counts the others as excluded. Nothing is dropped silently.
- --runs adds run totals per run type, by status, with check and approval rates.
- --json prints the same report as data.
- These commands change no usage data. The three that show transcript text (smrt ai:usage --with-transcripts, smrt ai:transcript and smrt ai:compare --transcripts) record who read it, and why, in a log line. They write a persisted audit entry only where your app has set an audit writer.
smrt ai:usage --by prompt,model --since 7d
smrt ai:usage --by tier --tenant <tenant-id>
smrt ai:usage --runsCompare two versions of a prompt, or two models
smrt prompts:history lists each version of a prompt that has run, with first and last use, call counts and the models that answered. smrt ai:compare then prints a table with one column per side, for one prompt key. A side is a prompt version, a model or a tier.
- The rows are calls, blocked calls, tokens, estimated cost, cost per call, latency at the 50th, 90th, 95th and 99th percentile, the fallback and degraded rates, repair, check-pass and approval rates, and the evaluation labels with their mean score.
- No model judges anything. The comparison reads stored rows.
- Both commands need --tenant <id> (use global for rows with no tenant) or --all-tenants.
smrt prompts:history <prompt-key> --tenant <tenant-id>
smrt ai:compare --prompt <prompt-key> --a hash:<h1> --b hash:<h2> --tenant <tenant-id>
smrt ai:compare --prompt <prompt-key> --a model:<provider:model> --b tier:<tier> --tenant <tenant-id>Labels come from your app
An evaluation is a verdict attached to one call or one run: a label, a score, or both. Your app records it when the verdict is known, and names who judged: a person, the app, or a named grader. There are no labels in a report until something records them.
import { recordAiEvaluation } from '@happyvertical/smrt-core';
await recordAiEvaluation(db, {
callId, // the usage row's id
label: 'accepted',
evaluator: 'human'
});Transcripts are separate, and off by default
A transcript is the readable content of a call, kept for prompt tuning. It is opt-in and set for the whole app; a tenant cannot turn it on. When it is on, a sample of calls is kept (one in ten unless you change it), along with calls that failed or fell back.
- Stored text is redacted first: API keys, values of fields marked sensitive, and anything your own redactor removes.
- Every read is audited and scoped to a tenant. smrt ai:transcript <callId> needs --tenant or --all-tenants.
- Transcripts are pruned after 30 days by default, separately from usage rows, so the numbers outlive the content.
config({
inference: {
transcripts: { enabled: true, sample: 0.1 }
}
});