Don’t Trust Documentation, Constants or AI Explanations Until Proven

Because a value that makes a claim about health, money, eligibility, law, time or the physical world can be wrong in a way that looks entirely reasonable on screen — and the only thing separating "correct" from "plausible" is a check somebody can re-run. For every constant, threshold, bound, formula, price or lookup table in a product, the method is the same nine moves.

1. State the exact claim, in language precise enough to be falsified.
2. Identify an authoritative primary source and freeze the version of it you are using.
3. Define the supported input domain, the units and the definitions.
4. Write a throwaway verification script that gets its expected values from an independent oracle.
5. Evaluate the full supported range, including every boundary and transition.
6. Trace the value through the whole pipeline rather than testing the formula in isolation.
7. Compare every platform and every consumer against that same shared oracle.
8. Record discrepancies, tolerances and provenance in an evidence package.
9. Convert the essential part into permanent regression protection.

Moves one to seven are the seven numbered steps below, in that order. Move eight is the evidence package that makes a verdict reproducible six months later, and move nine is the handover to a check a build can run without a human present.

For AI explanations, six more checks join the list: the consent gate, the outbound payload, the correctness of the inputs the payload was built from, the prompt constraints, the output evaluation cases, and the way the result is presented at the point of decision.

The reason this deserves its own discipline is that each failure mode is invisible to the thing most teams rely on. A constant can sit at exactly the right order of magnitude and be wrong only at the edge of its band. A lookup table can cite the correct organisation in a header comment and contain a transcription error in row 240. Documentation can describe behaviour the application stopped performing two releases ago. An AI explanation can interpret a derived value confidently and correctly while the value itself was produced from the wrong reference version.

Worse, these errors get more persuasive as they travel. A wrong percentile becomes a chart, the chart becomes a summary, the summary becomes a sentence of generated language that sounds measured and specific. By the time a user reads it, four layers of polish stand between them and a transcription mistake.

The previous guide in this series argued that you should audit the data rather than the screens. This one is the next step down: once you know which values matter, how do you prove each one is right? Our worked example throughout is a child-growth product built on published reference standards, shipped on iOS and Android, with age calculations, percentile and z-score transformations, unit conversion, subscription pricing and an AI explanation layer. We wrote up what building that app taught us about AI-assisted development separately; this post is only about the verification method, which applies to any product carrying real-world claims.

Trust runs in a fixed order, and almost every bad verification decision comes from reading it upside down. From most authoritative to least:

1. The current authoritative primary reference — the standard, the regulation, the store API, the publisher's own dataset.
2. Versioned source data obtained from it — the actual file you downloaded, with a checksum and a retrieval date.
3. An independently executable comparison — a script that derives the expected answer without asking the product.
4. Application output — what your code actually produces today.
5. Tests and generated fixtures — useful, but they encode a past decision rather than an external truth.
6. Documentation and comments — a claim about the system, written at a moment that has passed.
7. An AI explanation — a fluent interpretation of whatever it was handed.

This is not an argument that documentation is useless. It is an argument that documentation is a claim until it has been reconciled with the source and with execution. The same applies to a test: a passing suite tells you the code still does what it did when the assertions were written, which is a different statement from "the code matches the standard".

AI belongs last for a specific reason rather than a general suspicion. A model can synthesise and explain extremely well. What it cannot know, unless you tell it, is which dataset version your application actually loads, whether the measurement it was handed had already been normalised, whether the age it received used the same convention the reference expects, or whether a cached value upstream is three days stale. Fluent interpretation of a wrong input produces a wrong answer that reads better than the right one.

There is a practical corollary that changes how you write prompts. If you ask an agent "is this percentile correct?", you have invited it to reason from memory. If you ask it "here is the source table, here is the application output, enumerate every row where they differ and show me the differences", you have given it a job with an oracle. That distinction is the subject of the prompts guide earlier in this series, and it is the difference between a verification and a second opinion.

A recurring failure pattern is treating levels 4 to 7 as a chain of confirmations. The output matches the test, the test matches the comment, the comment names the standard, the model agrees. Four things agreeing with each other is not evidence when none of them ever consulted the source.

Search the product for every value that implies an external truth, then write each one down as a row — because the values that escape audits are the ones nobody classifies as a feature. A percentile engine gets reviewed. A postal-code range table does not, and it makes exactly the same kind of claim.

What to look for:

• Formulas and score transformations
• Thresholds, minimums and maximums
• Age bands and date boundaries
• Unit conversions
• Lookup and reference tables
• Prices, billing periods and trial durations
• Tax and financial rules
• Legal retention periods
• Geographic and locale mappings
• Clinical or standards reference versions
• Recommendation and eligibility wording
• AI system prompts and payload field lists

For each one, record enough that a stranger could pick it up. We keep the inventory as a plain table in the repository so it diffs like code:

| Claim ID | Claim                  | Location       | Source / version | Domain              | Consumer      | Verification |
|----------|------------------------|----------------|------------------|---------------------|---------------|--------------|
| R-001    | Age bucket selection   | Growth engine  | Reference vX     | Birth to max age    | Chart, report | Needed       |
| R-002    | kg to lb conversion    | Shared utility | NIST SP 811 B.8  | Supported values    | Forms, export | Tested       |
| R-003    | Annual price display   | Paywall        | Store product    | Locale, storefront  | Offer screen  | Runtime-only |

Three columns do more work than the rest. Source and version forces you to admit when there isn't one. Consumer reveals how far a wrong value travels — a constant used by one screen is a bug, a constant used by a chart, a PDF export and an AI payload is an incident. Verification is deliberately a status rather than a tick: needed, scripted, tested, runtime-only, or blocked.

Build the inventory by search rather than by memory. Grep for numeric literals outside layout code, for every file named with some variant of constants, config, reference or table, for every hard-coded string that appears in user-facing copy alongside a number, and for every field name in an outbound request body. In our portfolio the most common surprise from this exercise is not a wrong value — it is discovering multiple copies of the same value that have drifted apart because only some were updated.

Set the bar for entry deliberately low. A row costs a minute. A missing row is the class of defect that ships.

Precisely enough that the statement itself tells you what to test — which in practice means it should feel uncomfortably pedantic to write down. Ambiguous claims cannot be verified, and most claims start out ambiguous.

A weak claim:

The growth chart follows the published reference data.

A precise claim:

For metric M, sex S and supported age A, the application selects reference
parameters from version V using the stated age convention, converts the
normalised measurement into score Z using formula F, and maps Z to
percentile P within tolerance T.

The second version is not longer for the sake of it. Every clause is a question that had a wrong answer available:

• Which metric, and does the reference define it identically?
• Which sex representation, and what does the product do with values the standard does not cover?
• Age in days, completed weeks or completed months — and measured from which instant?
• Which version of the reference, and is the version pinned anywhere the application can read?
• Which formula branch, above and below the median?
• Which rounding, applied at which stage?
• What tolerance counts as agreement, and is it absolute or relative?

Writing the claim this way has an effect people underestimate: it usually finds the defect before any code runs. Three of the ambiguities above cannot be resolved by reading the implementation, because the implementation made a silent choice. The moment you have to write down "age in completed months, from local midnight of the birth date" you discover whether that is actually what the pipeline does.

Add one more line to every claim: what happens if this is wrong. A rounding difference in a decorative number is a cosmetic issue. The same rounding difference in a value a parent uses to decide whether to call a doctor is not. Consequence is what decides how much verification a claim earns, and recording it stops that decision from being re-argued every quarter.

Keep the wording in one place and reference it by ID from the code, the test and the evidence package. A claim that exists in three slightly different phrasings is three claims.

Go to the publisher, take the version, and freeze it — because a source you cannot name a version for is a source you cannot verify against twice. Prefer primary publishers, official standards bodies, store APIs and governing documentation over anything that summarises them.

Record all of this, next to the data rather than in a ticket:

• Publisher
• Dataset or document title
• Version identifier or publication date
• Direct source location
• Retrieval date
• Licence or redistribution constraints
• Units and definitions as the source states them
• Change notes since the previous version

The licence line matters more than it looks. Reference data is often free to use and not free to redistribute, and that constraint decides whether you can ship the table, must fetch it, or must derive a permitted transformation of it. Settle that before the implementation depends on the answer.

For our growth example the primary source is the WHO Child Growth Standards, published with explicit indicator definitions, age conventions and the technical report describing how the values were derived. For unit conversion the primary source is the definition of the units themselves — and that is a good place to watch a respectable-looking citation fall apart. The SI defines the kilogram; it does not define the pound, and the SI Brochure contains no conversion factor for it. The exact figure, 0.453 592 37 kilograms, comes from the 1959 International Yard and Pound Agreement and is carried in footnote 22 of the NIST Guide to the SI, linked from the pound row of NIST's conversion-factor appendix, which tabulates the rounded 4.535 924 E-01 and reserves boldface for the factors it prints as exact. Writing "SI definition" in the source column of a kilogram-to-pound conversion — which is what row R-002 of our inventory said before someone checked — names a real standards body and still fails to identify where the number came from. That is exactly the failure this post is about, and it is easiest to commit against a source you genuinely respect.

Do not rely on an article summarising a standard when the standard is available. Summaries drop the domain, the conventions and the caveats, which are precisely the parts that produce boundary defects.

Then freeze it. Where the licence permits, store a checksum, a version identifier, or a normalised fixture derived from the source, and treat replacing it as a reviewed change rather than a data update. The rule we apply is simple: an application result must be reproducible against the source version it claims to use. If the upstream publisher revises the dataset and someone quietly swaps the file, every previously approved result silently changes meaning — and every screenshot, export and stored record generated before the swap now describes a version that no longer exists anywhere.

The same discipline applies to sources that are not files. A store price, a policy threshold or a tax rule still has a version: the date you observed it, the storefront you observed it in, and the response payload you captured.

Writing down every input the rule is defined over, and — just as importantly — what the product does with inputs outside it. Every rule operates over a domain, and a rule applied outside its domain fails silently rather than loudly.

For a reference calculation, the domain is metric types, the sex categories the standard supports, the minimum and maximum age, the age interval semantics, the measurement units, the valid numerical range, and the behaviour for missing or unknown fields.

For a subscription, it is the product identifiers, the storefront and currency, the billing period, introductory-offer eligibility, and the full set of subscription states: active, expired, cancelled, in grace, in billing retry and restored.

For a date rule, it is the timezone, inclusive or exclusive boundaries, leap days, daylight-saving transitions where relevant, future dates and historical limits. Date domains are where products most often discover they never had a rule at all. If the answer to "which timezone" is "the device's", then the same record changes meaning when the user flies, and the IANA Time Zone Database updates several times a year with real offset changes that will move your boundaries under you.

Two decisions follow from a well-defined domain.

Write down the out-of-domain behaviour explicitly, because it is a product decision rather than a technical one. Does the app clamp to the nearest supported value, refuse to display a result, show the raw measurement without interpretation, or extrapolate? Three of those are defensible. Extrapolating beyond a standard's supported range and presenting the output with the same confidence as an in-range result is the one that causes harm, and it is also the one that happens by accident when nobody defined the domain.

Because a script that calls the production implementation and asserts against its own output proves only that the code is deterministic. The expected value has to arrive by a path that does not run through the thing under test.

An independent oracle can be any of three things: the authoritative source data parsed directly, a second implementation written from the specification without looking at the first, or a hand-derived vector with recorded workings. What it cannot be is the production function, a snapshot of the production function's previous output, or a model asked to guess.

A good verification script:

• Loads authoritative or derived reference fixtures
• Implements or imports an independent calculation
• Enumerates the supported inputs rather than sampling them
• Calls the production function or reads the exported dataset
• Compares within an explicit, stated tolerance
• Prints every mismatch with full context, not just a count
• Returns a non-zero exit code on failure
• Produces a summary: cases evaluated, mismatches, maximum deviation

for metric in supported_metrics:
  for sex in supported_sexes:
    for age in supported_ages:
      reference = authoritative_table(metric, sex, age)
      actual    = application_table(metric, sex, age)
      compare_fields(reference, actual, tolerance)

print(total_cases, mismatches, maximum_error)
exit(1 if mismatches else 0)

Two properties of that loop matter more than the language it is written in. The first is that it prints mismatches with context — metric, sex, age, expected, actual, difference — because a bare "47 failures" tells you nothing about whether you have one systematic offset or forty-seven unrelated typos. In practice the shape of the mismatch list is the diagnosis: a contiguous block points at a band boundary, an even spread across one sex points at a wrong table, a constant delta points at a units or rounding error.

The second is the non-zero exit. A script that reports failures in prose gets read once by the person who wrote it. A script that fails can be wired into a build.

The script itself can be genuinely throwaway during investigation, and it should be — the goal at this stage is an answer, not an artifact. But the essential protection must survive as a maintained test or build check, which is the subject of the next guide in this series. A verification that lived only in one engineer's terminal has already expired.

Almost all of them — errors cluster at transitions, and the middle of a range is the least informative place to look. If you only have time for a subset of cases, spend it entirely on edges.

The transitions that reliably produce defects:

• The last day of one age band and the first day of the next
• A unit conversion followed by rounding, where the order of the two operations changes the result
• The switch between negative and positive branches of a score formula
• A leap-day birth date, in a non-leap year
• The exact instant a trial expires, in the user's timezone and the store's
• The minimum and maximum supported measurement
• A missing reference row, where the lookup silently returns a neighbour
• Locale-specific decimal parsing, where a comma is a decimal separator rather than a thousands separator

For every boundary, test three points rather than one, and record the comparison in a fixed shape so results are readable months later:

BOUNDARY CASE [ID]
  Input immediately before  :
  Input exactly at boundary :
  Input immediately after   :
  Expected source row/formula:
  Application result        :
  Difference                :
  Verdict                   : match / mismatch / undefined in source

The "undefined in source" verdict earns its place. Standards do not always specify what happens exactly at a band edge, and when they do not, the product has made a choice that is neither right nor wrong until someone writes it down. Recording it as a decision beats discovering later that iOS rounds one way and Android the other because nobody ever decided.

The locale row deserves particular attention if you ship in India or any other market where the product is used in more than one input convention. A weight typed as "8,5" parses to 85 in one locale and 8.5 in another, and 85 kg for an infant is not a value most validation ranges are written to reject with a helpful message. We have seen this class of defect survive an entire test cycle because every tester used the same device language.

Do not accept "representative values" when the risk is boundary selection. Representative values are the specific cases guaranteed to pass.

By tracing the value from the moment it enters the product to every place it leaves, and verifying each transformation separately — because a correct utility handed an incorrect input produces a wrong answer with a green test suite behind it.

User input / import
  → parsing
  → unit normalisation
  → identity and date context
  → reference selection
  → calculation
  → rounding and formatting
  → storage / cache
  → UI, report, export, AI payload

The canonical failure here is subtle enough to survive several reviews. A percentile engine can be provably correct against the source for every row, while the screen that calls it supplies an age rounded to completed months when the reference expects a different convention. Every unit test passes. Every value on screen is wrong by a predictable amount that looks like a small, plausible difference rather than a defect. Testing the formula in isolation cannot see this, because the formula was never the problem.

Verify each hop with its own check:

• Parsing — locale separators, whitespace, unusual but valid inputs, values that overflow the expected precision.
• Normalisation — is the conversion applied once, and only once? Double conversion is common where two layers each think they own units.
• Context — is the correct profile selected, is the date the one the user meant, is the timezone the one the rule specifies?
• Selection — is the reference version the one recorded in the provenance metadata, and does the app fail loudly if it is not?
• Storage and cache — can a stale derived value outlive the input that produced it? Recalculation on read and invalidation on write are different guarantees.
• Consumers — does the export contain the same number as the screen, and does the AI payload contain the same number as both?

The cache hop is the one teams skip. If a derived score is stored rather than recomputed, then editing a birth date, correcting a measurement or upgrading the reference version must all invalidate it. Where they do not, the product will happily display a value that was correct against inputs the user has since changed — and no verification of the formula will ever catch it, because the formula is not being run.

By comparing both against the same external oracle rather than against each other — the moment iOS becomes the reference for Android, you have converted an unverified implementation into a standard. Parity is a useful signal and a terrible target.

The mechanism is a shared golden vector: a compact set of inputs and approved outputs, derived from the independent verifier, run through every implementation and every consumer.

case_id,source_version,raw_input,normalised_input,boundary_category,
expected_internal,expected_display,tolerance,rationale

Run that file through the iOS calculation, the Android calculation, the report generator, the export path and the AI payload builder. Record differences in precision, rounding, missing-data handling and formatting separately, because they have different fixes.

Golden vectors carry one real risk: they can quietly become a second source of truth. Two rules prevent it. Never hand-edit an expected value without updating its provenance, and regenerate the vectors from the independent verifier whenever the source version changes, then review the diff as a change rather than accepting it as an update. A vector file whose expected values were adjusted until the tests went green is worse than no vectors, because it now looks like evidence.

Separate three numbers explicitly and never let them collapse into one:

Display formatting must never alter a stored or calculated value, and comparisons must run on numbers rather than on formatted strings. Record the rounding method too, especially at half values: platform defaults differ, and half-even versus half-up produces exactly the kind of one-digit disagreement that gets dismissed as noise for months before someone traces it. Locale formatting adds its own layer — digit grouping, decimal separators and numbering systems all vary, and the Unicode CLDR is the authority on what each locale actually expects, including the Indian digit-grouping convention that surprises teams the first time a number renders as 1,00,000.

Test the layers separately: reference row selection, internal numeric calculation, band mapping, display rounding, localised formatting, and the input handed to any explanatory content. Localising the difference is what makes the fix safe.

By treating every user-visible documentation statement as a claim with an owner, and pairing it with the runtime or source evidence that confirms it. Documentation drift is not sloppiness; it is the default state of any document that is not executable.

Drift arrives through predictable routes:

• A feature changes and the guide describing it does not
• A privacy policy lists a processor or a payload field that is no longer accurate
• A code comment describes the formula that preceded the current one
• A constant is updated on one platform and not the other
• A marketing screenshot shows a fixed price
• A design board is mistaken for a shipped capability

The last one is specific to how AI-assisted teams work now, and it is worth naming. When design boards, specifications and generated summaries all read as confident descriptions of the product, the distinction between "designed", "implemented" and "verified" stops being visible in the artifacts. Someone reading a board six weeks later has no way to tell which of the three it describes.

The fix is a documentation claim ledger, kept in the same place as the claim inventory:

| Document claim              | Runtime / source evidence      | Status          | Required action              |
|-----------------------------|--------------------------------|-----------------|------------------------------|
| AI features are opt-in      | Consent default + request gate | Verified        | Protect with a test          |
| Exact birth date not sent   | Captured payload inspection    | Verified        | Snapshot / schema guard      |
| Annual price is X           | Store response varies by region| Incorrect claim | Use localised store price    |

Two rules keep the ledger honest. First, evidence must be a thing you can re-run or re-read — a captured payload, a source row, a command — not a person's recollection of having checked. Second, documentation should avoid hard-coding external facts the system can retrieve authoritatively. A help page that states a price will be wrong in most storefronts on most days; a help page that says where to see the current price is correct permanently.

Run the ledger at the same cadence as a release, and give every row an owner. The claims that drift furthest are always the ones nobody owns: the privacy page, the FAQ and the app-store description, all written once at launch and all making specific, checkable, ageing statements.

By checking six stages, only one of which is the text itself — because an explanation that reads sensibly can still be built on the wrong payload, presented without limits, and generated for a user who never consented. Evaluating fluency evaluates the least risky part of the feature.

1. Consent and eligibility. Is the feature off by default? Is disclosure shown before any processing happens? Is revocation actually enforced at the request gate rather than only in the settings screen? Are there age or jurisdiction restrictions that change the answer? Google Play treats this as a policy matter rather than a preference: apps covered by its health content and services policy carry explicit disclosure obligations, including a requirement to remind users to consult a healthcare professional for medical advice, diagnosis or treatment. Whatever the store rules say in your category, read them as a floor rather than a specification — they tell you what you must disclose, not whether your feature is appropriate to ship.
2. Payload minimisation. Capture the actual outbound payload in a safe test environment and compare it field by field with what you have declared. For a growth product, the defensible payload is measurements, age, sex and derived context; the fields that must never appear are the child's name, photographs, exact birth date, contact details and account credentials. The only meaningful proof is payload-level inspection plus a protection that fails the build when a forbidden field appears. A policy document is not a control.
3. Input correctness. Are units normalised, are the derived values current rather than cached, is the selected profile the one on screen, and is the reference version known to the payload? This is where the pipeline work of the previous section pays off twice.
4. Prompt constraints. Does the prompt state the role and its limits, distinguish explanation from diagnosis, require uncertainty and escalation language, prohibit unsupported claims, and require the output to be grounded in the supplied data? Treat the system prompt as a versioned artifact in the claim inventory, because changing it changes product behaviour without changing a line of application code.
5. Output evaluation. Build fixed test cases for typical input, sparse input, contradictory input, boundary values, missing context, out-of-domain values, and prompt-injection-like text where user-supplied content can enter the context window. Score each on factual grounding, unsupported certainty, harmful inference, consistency across repeated runs and actual usefulness.
6. Presentation. Is generated content clearly identified as generated? Are limitations visible at the decision point rather than in a settings page? Can users reach the underlying data? Does the interface give generated language more visual weight than the measured values it describes? Apple's guidance on machine learning in interfaces is worth reading on exactly this point. An accurate model output can still be presented irresponsibly, and the deterministic inputs behind it should never be harder to find than the prose.

Because their source of truth is a live system rather than a file in your repository — so the verification is a captured response in a controlled environment, not a comparison against a constant. A price hard-coded anywhere is a claim that expires without warning.

What to audit for a subscription:

• Product identifiers, exactly as configured
• Storefront and currency
• The localised price string, taken from the store response rather than composed
• Billing period and trial duration
• Introductory-offer eligibility, which is per-account and not per-device
• Renewal, grace, billing-retry, expired and cancelled states
• Restoration on a fresh install
• Server or SDK entitlement mapping
• Cached fallback behaviour when the store is unreachable

Both platforms give you a way to observe the real thing: prices are configured and localised per storefront in App Store Connect, and the equivalent configuration on Google Play drives what the billing client returns. Capture the response, compare the displayed language against the actual configuration, and never verify a user's current offer from a screenshot or a marketing document.

Fallback prices deserve an explicit rule. If a fallback exists, define when it may appear and what uncertainty it introduces — because a polished paywall showing a stale price is worse than an honest "offer unavailable" state. The first takes money against a number the store will not honour; the second loses a conversion. We have written separately about what that trade-off does to conversion in paywall benchmarks; the verification point is narrower, which is that a price is not yours to assert.

India makes this concrete. Storefront pricing, purchasing-power-adjusted tiers and local currency rounding conventions mean the annual price a Mumbai user sees has no relationship to the one in your specification, and a hard-coded "$49.99/year" in a help page or an onboarding screen is simply wrong for the majority of your installs. The same applies to any market where you price locally rather than converting.

Configuration and geographic mappings behave identically. Postal code to region, country to policy, locale to content, OS version thresholds, notification schedules and feature-eligibility lists all look like ordinary data and all make claims. Define overlap and precedence rules, then write a verifier that detects gaps, overlaps and exceptions across the whole range rather than spot-checking entries — a six-digit Indian PIN-code range table is finite and can be walked exhaustively in milliseconds. For a version threshold, test the version immediately before, exactly at, and immediately after the boundary. Store the provenance beside the data so a future maintainer knows what it was derived from:

source_id
source_version
retrieved_at
supported_domain
normalisation_method
checksum
verification_command

Everything a future agent or engineer would need to re-run the verification without reconstructing the original conversation — which is a higher bar than "everything we looked at". The test of a good evidence package is whether someone can reproduce the verdict six months later from the package alone.

Claim ID and exact wording
Authoritative source and version
Normalised reference artifact and checksum
Verification script and the command that runs it
Case count and mismatch summary
Boundary and tolerance policy
Per-platform results
Documentation locations referencing the claim
AI payload and output evaluation, where relevant
Permanent regression protection
Owner and re-verification trigger

The last line is the one most often left off and the one that decides whether the package ages well. Verified is a statement about a versioned system, not a permanent property. Re-verify by trigger rather than by calendar: the source publishes a new version, the supported domain expands, the implementation is rewritten, a new consumer starts reading the value, or the rounding changes. Record which triggers apply to this claim, and put that relationship somewhere durable — the project bible for AI coding agents is the natural home, because an agent making a change six months from now needs to know when earlier evidence stopped applying.

Users need a smaller version of the same thing. Nobody wants a technical report on a chart screen, but they do need enough context to understand what they are looking at. Use progressive provenance: a short source label near the result, a version or date where it is meaningful, a plain-language explanation of what the value means, a path to the detailed method, and an honest message when a value is unsupported or unavailable. Generated text should never conceal the deterministic inputs behind it.

Finally, use the brief below to run a verification end to end. It is the most portable artifact in this post — hand it to an agent or a new engineer with a claim ID and you get back evidence in a comparable shape every time.

VERIFY REAL-WORLD CLAIM [ID]

CLAIM
- Exact statement:
- Product locations and consumers:
- Consequence if wrong:

AUTHORITATIVE SOURCE
- Publisher:
- Document / dataset / API:
- Version or date:
- Direct location:
- Definitions, units, domain:

IMPLEMENTATION
- Constants, tables, formulas:
- Input pipeline:
- Derived outputs:
- Platform implementations:
- Documentation and AI consumers:

EXECUTABLE CHECK
- Independent expected-value source:
- Full supported domain:
- Boundary cases:
- Tolerance and rounding policy:
- Commands to run:
- Machine-readable mismatch output:
- Non-zero failure condition:

AI-SPECIFIC CHECKS
- Consent gate:
- Actual captured payload:
- Forbidden fields:
- Prompt constraints:
- Evaluation cases:
- Presentation and stated limitations:

OUTPUT
- Cases evaluated:
- Mismatches:
- Maximum deviation:
- Missing source coverage:
- Platform differences:
- Documentation drift:
- Verdict: verified / partially verified / failed / blocked
- Permanent regression protection:
- Owner and re-verification trigger:

The expensive mistakes are the ones that produce a feeling of having verified something without producing evidence — because they close the question rather than leaving it open. Eight recur often enough to name:

• Trusting a recognisable source label. Naming a standards body in a comment does not prove the local table matches it, or that it matches the right version of it.
• Comparing a handful of examples. Sampling misses systematic boundary errors by construction. Enumerate bounded domains.
• Testing the formula but not the inputs. Most pipeline defects happen before the calculation runs.
• Using production code as its own oracle. Expected values need an independent derivation, or the test is a tautology.
• Ignoring platform rounding differences. Share vectors and state precision rules explicitly rather than arguing about a trailing digit.
• Treating documentation as self-validating. Reconcile every important claim with runtime or source evidence.
• Evaluating an AI feature by how it reads. Inspect the payload, the constraints, the grounding, the edge cases and the presentation.
• Leaving the verification as a throwaway script. Investigate with the script, then preserve the core protection where a build can run it.

Before approving any real-world claim, ask a short list of questions and require an answer rather than a nod. Can we state the exact source and version? Is the supported domain explicit? Was the expected result derived independently? Were all finite inputs and important boundaries tested? Were pipeline inputs verified, not only the final function? Do platforms and downstream consumers agree? Are internal precision and display rounding separate? Does the documentation match current behaviour? If an AI layer explains the result, was the actual payload inspected? And which permanent check detects drift or corruption from here on?

The method compresses to the same nine moves it opened with. State the claim precisely. Identify the authoritative source and freeze its version. Define the domain. Build an independent executable comparison. Test the full range and every boundary. Trace the value through the whole pipeline. Compare every platform and consumer against the same shared oracle. Record discrepancies, tolerances and provenance in an evidence package. Preserve the essential check behind a regression guard. Real-world claims deserve more than familiar constants, plausible comments and articulate generated prose — and the next guide covers the other half of that last move, which is that a fix is not complete until its protection demonstrably fails when the bug is reintroduced.

If you are shipping a product that carries health, financial or eligibility claims and you are not sure which of them have evidence behind them, we are happy to look at the inventory with you.