Audit the Data, Not the Screens: A Data-First Product Audit

Because the defects that matter most live between screens, and a screen-by-screen review is structurally incapable of looking there. Run the audit in seven stages instead: select the critical data objects, write their invariants, enumerate every mutation path, locate validation authority, map consumers and derived values, exercise failure and lifecycle cases, then add protections that fail loudly when an invariant is violated.

A screen review answers a real question — does this surface work — and answers it well. It cannot answer the question that produces the expensive bugs: is the fact this surface is showing the same fact that was entered, stored, synced, recalculated and exported? That question has no owner on any single screen, so nobody asks it.

This rule changed how we audited a paediatric growth-tracking product built almost entirely with an AI coding agent. The interface contained profiles, measurements, percentile charts, AI explanations, history, reports and exports. A measurement entered on one screen did not stay on that screen. It moved through validation, normalisation, persistence, calculation, charting, insight generation and sharing. A small inconsistency at the point of creation could arrive five surfaces later as a confident, well-formatted, entirely wrong interpretation of a child's growth.

The narrative of that build — what it cost, what we would change — is written up separately. This guide is the transferable method, and it applies to any product with more than one way to change the same fact.

This is part 11 of our AI product development methodology series. Part 10 covered why code review cannot judge visual quality; the same argument runs in reverse here, because a visual review cannot judge data integrity either. Neither discipline substitutes for the other, and most teams have only one of them.

It misses every question whose answer requires two or more surfaces to be compared, which is most of the questions worth asking. The clearest way to see the gap is to run both reviews over the same trivial feature and compare what each one produces.

Take an app that lets a user record a weight. A screen review confirms that the field exists, the numeric keyboard appears, the validation message renders, the save button works and a point appears on the chart. Every item passes. The feature is signed off.

A data audit asks a different set of questions about the same feature:

• Can a weight also arrive through edit, CSV import, backup restore or device sync?
• Are kilograms and pounds normalised identically on every one of those paths?
• Can an edit accidentally reassign the record to a different profile?
• What is the state when persistence succeeds but the chart refresh fails?
• Can a retry after a timeout create a duplicate record?
• Does deleting a measurement remove the derived insight that quoted it, or leave it stranded?
• Does the export use the same canonical value the chart is drawing?
• Can a stale cached record arriving late from sync overwrite a newer local edit?

Not one of those has an owner on the weight-entry screen. They are system questions, and a reviewer working through a screen inventory will never be prompted to ask them — which is exactly why the inventory work in part 3 of this series is a starting point rather than an audit.

Choose the three objects whose corruption, loss or inconsistency would do the most damage to user value or trust — not the three with the most rows. Depth on three beats a shallow pass over thirty, and the constraint is deliberate: three forces a genuine prioritisation argument that a full table list lets you avoid.

Score each candidate against eight criteria:

• Consequence of being wrong — does a bad value mislead a decision, or just look untidy?
• Number of mutation paths — how many pieces of code can change it?
• Number of downstream consumers — how far does it travel once committed?
• Difficulty of correction — can a user fix it themselves, or does it need a support ticket and a migration?
• Privacy sensitivity — is it health, financial, biometric or identity data?
• Real-world interpretation — does the product make a claim about the world from it?
• Sync or offline complexity — does it exist in more than one place at once?
• Commercial or entitlement impact — does it gate access or revenue?

For the growth product, three candidates dominated: the child profile, the growth measurement, and the subscription entitlement state. AI consent was a strong fourth, because it governs whether personal data may be processed off-device at all — the kind of object whose declared handling has to match what you have published in your App Store privacy details, so a defect there is a compliance event rather than a bug.

For a payments product the three would be account balance, order and permission grant. For a marketplace, listing, payout and identity verification state. The objects change; the ranking method does not.

Before moving on, write down what makes the object the same object across every layer of the stack:

Measurement identity:
- stable record ID
- owning profile ID
- metric type
- observation timestamp
- source / provenance
- normalised value and canonical unit
- creation and update metadata

If identity is implicit — if the object is only ever recognised by position in a list, or by a combination of fields that any path is free to change — then edit, merge and sync behaviour cannot be reasoned about at all. In our portfolio, an unstated identity rule is one of the most reliable predictors that duplicate records are already in production.

An invariant is a rule that must hold regardless of which path changed the object or which screen the user was looking at — and it has to be specific enough that you could write a failing test for it today. "Data should be valid" is not an invariant. "Stored value uses the canonical unit" is.

For a growth measurement, the list runs roughly like this:

• It belongs to exactly one valid profile.
• Metric type and unit are compatible with each other.
• The stored value is expressed in the canonical unit, whatever the input unit was.
• The observation date lies inside the supported temporal domain and cannot precede the birth date.
• Derived age uses identical date semantics everywhere it is computed.
• Reference-table lookup receives valid sex, age and metric inputs, or refuses to answer.
• Identical retries do not create unintended duplicates.
• An edit preserves record identity unless a copy was explicitly requested.
• Deletion triggers the defined downstream recalculation rather than leaving orphans.
• Export and chart consume the same canonical record, not two separately derived values.

For AI consent the list is shorter and sharper: off by default; no processing before explicit agreement; revocation affects future processing; only the declared data categories leave the device; excluded identifiers stay excluded; consent status is represented consistently across devices and sessions.

It helps to separate invariants by the layer that must own them, because mixing the categories is how enforcement ends up in the wrong place:

• Domain — a measurement date cannot precede the birth date.
• Storage — the canonical unit is stored consistently for every row.
• Relationship — every measurement references a profile that exists.
• Propagation — the chart and the report both reflect a committed edit.
• Privacy — the AI payload excludes name and exact date of birth.
• Experience — a recoverable failure preserves whatever the user typed.

The interface can enforce the experience rule. It must not be the only thing enforcing the domain rule. Write the invariants before you go looking for bugs, or you will end up writing rules that describe the defects you happened to find, which is a very different and much less useful document.

Enumerate every path that can change the object, then prove the enumeration is complete by searching the source rather than by recalling the feature list. The main form is one writer among many, and it is usually the best-behaved one.

The matrix has one row per path and six columns: trigger, validation, write, failure branch, idempotency or conflict policy, and downstream notification. Nine rows are the standard starting set — create, edit, import, sync, restore, undo, cancel, retry and delete — and each one deserves an explicit answer rather than an assumption:

• Create — add form, UI plus domain validation, local repository write, preserve input on failure, refresh history and chart.
• Edit — history action, domain validation, update by ID, original retained on failure, recalculate derived values.
• Import — file or process trigger, parser plus domain validation, batch write, an explicit partial-success policy, notify all consumers.
• Sync — remote event, contract validation plus conflict resolution, merge write, retry with backoff, notify observers.
• Restore — backup or new account, schema plus domain validation, replace or merge, rollback on failure, reindex.
• Undo — snackbar or menu action, identity and version check, restore the deleted record, expiry behaviour, refresh.
• Cancel — user dismissal or backgrounding, no validation required, no write, discard or preserve the draft according to a stated policy, no downstream notification.
• Retry — after a failed write, idempotency key, reattempt, no duplicate, refresh exactly once.
• Delete — user or system trigger, ownership and policy check, tombstone or hard remove, recovery window, recalculate.

Then hunt for the writers nobody lists: debug and admin tools, migration code, sample-data loaders, background jobs, push and deep-link handlers, widgets and extensions, cloud functions, bulk actions, and tests that construct objects directly rather than through the normal constructor. Restore is the one that catches teams most often, because platform-level backup mechanisms such as Android Auto Backup can reinstate an entire data store without a single line of your validation code running.

To prove completeness, search the source for constructors, repository write methods, DAO and database operations, network mutation calls, serialisation and deserialisation code, event handlers, state reducers and migration logic. Reconcile every writer you locate against the matrix. A function name is not evidence — trace its callers and the conditions under which they fire, or you will classify a bypass as a duplicate of the safe path.

For every invariant, name the single layer that enforces it and prove that every mutation path passes through that layer. Validation duplicated across UI code is fragile by construction: the second writer is written by someone who never saw the first one.

Ask six questions per invariant. Where is it enforced? Is that layer shared by every path in the matrix? Can a caller bypass it? Is the same rule repeated inconsistently elsewhere? Does persistence itself protect the relationship or uniqueness rule? And what happens to historical or imported data that already violates it?

Then draw the enforcement grid — invariants down the side, mutation paths across the top, and one of three marks in each cell: enforced, not applicable, or unknown. A grid for a real product typically comes back looking like this on the first pass: valid-profile enforced on create, edit, sync and delete but unknown on import and restore; canonical unit enforced on create, edit and import but unknown on sync and restore; date domain enforced in the UI only on create and edit; idempotency enforced on sync alone.

The target architecture is central enforcement with contextual feedback. The domain layer — a value object, a repository boundary, a use case, whatever your stack calls it — protects the invariant for every caller. Android's data-layer guidance describes the repository as the single source of truth for exactly this reason. The UI then provides timely, specific, recoverable feedback on top of that guarantee.

The two failure modes are symmetrical. If only the database rejects invalid input, the experience is terrible — the user finds out late, in language written for an engineer. If only the UI rejects it, every alternate writer can corrupt the store silently. You need both, and you need to know which one is load-bearing.

Follow the object from the successful write to every surface and service that reads it, and record how each one transforms, caches and invalidates it. A write can be perfectly correct while half the product shows something else.

For a single growth measurement the consumer list ran to twelve entries: the local database, in-memory state, the history list, the latest-summary card, the chart, the percentile calculation, the trend logic, the AI context payload, the generated report, the export file, reminder notifications, and the sync or backup layer. Each one needs the same seven facts recorded:

Consumer:
Read path:
Transformation:
Cache:
Refresh trigger:
Staleness policy:
Failure behaviour:
Display state when unavailable:

Derived data deserves its own pass, because derived values carry invariants of their own and almost never get their own tests. Percentiles, totals, eligibility flags, streaks, recommendations — for each, establish whether it is stored or computed on demand, which version of the source it was computed from, when it is invalidated, whether stale derived data can survive an edit to its input, whether both platforms implement the same formula, how rounding and boundary conditions are handled, and whether export recomputes or reuses a cached figure.

Divergence between independently implemented platforms is the common finding here, and it is worth stating plainly: two teams implementing the same specification will produce two subtly different answers unless something executable compares them. Rounding at a percentile boundary, an age computed in whole months on one platform and in days on the other, a chart that interpolates where the report does not. None of it is visible until you put the two outputs side by side with the same input.

This is also where analytics quietly diverges from product truth. If your event payload recomputes a value instead of reading the canonical record, your dashboards will disagree with your own app, and nobody will be able to say which of the two is right. Instrument from the canonical record or accept that your funnel numbers are a second, independent implementation of the same formula, with its own rounding and its own bugs.

Exercise the five categories that cross a boundary: partial success, cancellation, retry, concurrency, and restore or migration. These are the scenarios no screen inventory contains, and between them they account for most of the data-integrity defects we have seen reach production.

Partial success. The local write succeeds and sync fails. The payment succeeds and the entitlement refresh fails. The export generates and the share sheet is dismissed. The profile edit commits and a cached header keeps the old name. In each case, name the source of truth and the recovery path. Entitlement state is the one worth extra care because the money is real — StoreKit 2 makes the completed transaction authoritative and expects your app to reconcile against it — the current entitlements sequence at launch and the transaction updates listener while running — rather than against a local flag set at checkout time.

Cancellation. A user dismisses a half-edited form. The app is backgrounded mid-save. A permission prompt is denied after the flow assumed it would be granted. A long-running AI request is cancelled. Cancellation must never leave half-applied state, and it must never silently discard work the user would expect to survive.

Retry. Two rapid taps on save. An automatic retry that races a manual one. A process restart with queued work still pending. This is a solved problem and the solution is an idempotency key generated by the client and honoured by the writer — the pattern Stripe documents for its API is the reference implementation, and the underlying property is the one RFC 9110 defines for idempotent HTTP methods. Verify both the absence of duplicates and the feedback the user sees.

Concurrency and conflict. The same record edited on two devices. A profile switched while a request is in flight. A subscription that changes state during a gated flow. Sync delivering an older record after a newer local edit. State the conflict policy explicitly — last-write-wins is a decision, not a default, and it should be a decision somebody made on purpose.

Restore and migration. An old schema arriving in the current app. Missing fields taking defaults. A reference dataset changing version. Deleted records reappearing after a restore. Migration code is written once, run against test data, and then executes on real user data forever after; Room's documented behaviour for an unhandled schema change is an IllegalStateException at launch, and the escape hatch — fallbackToDestructiveMigration — is documented to permanently delete all data from the tables in the user's database. Core Data's lightweight migration only covers changes it can infer a mapping model for, and anything past that needs a heavyweight migration you write and test yourself. None of these outcomes is an inconvenience. Restore paths are the least tested and the most capable of bypassing every invariant you enforce elsewhere.

Pick the strongest enforceable protection the stack allows, then prove it detects the defect by deliberately reintroducing the defect. An untested guard is a guess with a green tick next to it.

There is no universal strongest-to-weakest ordering because each protection owns a different boundary. The available set includes the type system, a constructor or value object that cannot represent an invalid state, a database constraint, repository-level validation, contract or schema validation at the boundary, an idempotency key, a transaction, unit and integration tests, property-based tests, static checks, runtime assertions and production monitoring. Prefer prevention at the authoritative write boundary, then add detection where invalid data can still enter through imports, migration, concurrency or external systems. A type can protect construction while a database constraint protects every writer; one does not make the other redundant.

Then challenge the protection with a five-step mutation:

1. Add a test asserting that units are normalised identically on the create and import paths.
2. Deliberately mutate the import path to skip normalisation.
3. Confirm the mutated code still compiles and is otherwise valid.
4. Run the test and observe it fail.
5. Restore the correct code and observe it pass.

Only after step four do you know the test detects the regression rather than passing for some unrelated reason. This is not a niche practice: Google's paper on mutation testing at scale describes engineering the technique down to a cost its reviewers can absorb on ordinary code-review diffs, on the grounds that inserting small faults and measuring whether the suite catches them is the strongest criterion available for judging a suite. A suite that has never been mutated is a suite whose sensitivity is unknown.

This matters more when an AI agent is writing the code. An agent asked to fix a defect will very often produce a test alongside the fix, and that test will very often assert the behaviour of the new code rather than the absence of the old defect. It passes before the fix and after it. The mutation step is what separates the two cases, and it is the reason we treat it as mandatory rather than advisable — part 13 of this series is entirely about that loop.

One worksheet per critical object, holding identity, invariants, mutation paths, consumers, derived data, evidence and protections in a form somebody else can pick up six months later. The worksheet is the audit's real deliverable; the findings list is a by-product of filling it in.

CRITICAL DATA OBJECT
- Name:
- Why critical:
- Canonical identity:
- Source of truth:
- Privacy classification:
- Lifecycle states:

INVARIANTS
I-01:
I-02:
I-03:

MUTATION PATHS
For each: create / edit / import / sync / restore / undo / cancel / retry / delete
- Entry point:
- Call chain:
- Validation:
- Transaction / write:
- Failure branch:
- Idempotency / conflict policy:
- Downstream notification:
- Tests:

CONSUMERS
- Surface or service:
- Read path:
- Transformation:
- Cache and invalidation:
- Failure or stale state:

DERIVED DATA
- Formula and reference:
- Version:
- Full-range verification:
- Rounding and boundaries:
- Platform parity:

EVIDENCE
- Source locations:
- Commands and tests run:
- Runtime scenarios exercised:
- Outputs and screenshots:
- Unverified paths:

PROTECTION
- Enforcement layer:
- Regression test or guard:
- Mutation result:

A prompt that forbids the agent from starting at the screens, names the seven stages as required output, and demands evidence rather than assurance at every step. Copy it as-is and change only the product context.

Perform a data-first, invariant-driven audit.

Do not begin with screens. First identify the three data objects whose loss,
corruption or inconsistency would most damage the product, and justify the ranking.

For each object:
1. Define canonical identity, source of truth and lifecycle states.
2. Write domain, storage, relationship, privacy, propagation and experience
   invariants. Number them.
3. Enumerate every create, edit, import, sync, restore, undo, cancel, retry and
   delete path, including migrations, background jobs, debug tools and failure
   branches. Prove completeness by source search, not recollection.
4. Trace each path from entry through validation, persistence, notification and
   downstream consumption.
5. Show exactly where every invariant is enforced. Name any bypass or unknown.
   Do not record an unknown as enforced.
6. Map all consumers, derived values, caches and invalidation triggers.
7. Exercise partial success, cancellation, retry, concurrency, and restore or
   migration scenarios.
8. Run named tests and show their output. Do not infer runtime success from
   reading source.
9. For every proposed fix, specify an enforceable protection and the mutation
   test that proves the protection detects the regression.

Output:
- object ranking with justification;
- invariant matrix;
- mutation-path matrix;
- consumer and propagation map;
- findings with file-level evidence;
- explicit list of unverified paths;
- protections with acceptance criteria.

Two clauses do most of the work. "Do not record an unknown as enforced" removes the single most common way an agent-run audit flatters itself. "Do not infer runtime success from reading source" removes the second. Both belong to the wider evidence discipline covered in part 8 on prompts that produce evidence rather than confidence, and neither is optional here — an agent tracing data paths has enormous scope to sound thorough while having executed nothing at all.

Because the bug lives entirely in the gap between two screens that each work perfectly on their own. This one example crosses identity, concurrency, caching, privacy and presentation, and no screenshot inventory would ever surface it.

An AI explanation is requested for Profile A. While the request is in flight, the user switches to Profile B. The trace looks like this:

Profile A selected
  -> request constructed from A measurements
  -> request sent
  -> user selects Profile B
  -> active UI state changes to B
  -> response for A returns
  -> response stored or displayed  <- the question is: under which profile?

A screen-first audit confirms that the profile switcher works and that the insight screen renders. Both are true. A data-first audit asks which identity travels with the request and which identity is attached to the response, and that question has five invariants attached to it:

• The request carries an immutable profile identifier captured at construction time, not a reference to whatever is currently selected.
• The response is associated only with the profile that originated it.
• A late response cannot overwrite or appear under Profile B's visible insight.
• Cache keys include both identity and the relevant data version, so switching back to A does not serve an insight computed from superseded measurements.
• The privacy payload stays limited to the declared categories for A, even though the UI has moved on to B.

To audit it, inspect request construction, state ownership, the cache key, the response reducer and the UI selection logic — then execute the scenario with an artificially delayed response while switching profiles. The acceptance evidence is not "Profile B remained selected". It is that A's explanation never appeared under B, and that returning to A followed the intended cache or refresh policy rather than firing a second request that quietly costs money.

Health data makes this vivid, but the shape is universal. Substitute account for profile and the same defect shows an insight computed from one customer's data under another customer's name. That is not a bug report; that is a disclosure notification.

After the data is traced, and the review you can then run is far sharper than the one you would have run first. Screens are not exempt from audit — they are simply the last stop rather than the first, and by then you know what each one is carrying.

With the data model in hand, review each screen as both a representation and a mutation point. Six questions:

• Does this screen show the canonical value, or a locally derived one?
• Are the value's age, unit and provenance clear to the person reading it?
• Can the user correct it here, and does correcting it use the same validated path?
• Does a failure preserve the user's work?
• Does the next surface reflect the success this one reported?
• Does the interface communicate stale, partial or offline state, or does it present degraded data as current?

That last one is the most consequential and the most often missed. A chart that renders cheerfully from a cache that stopped updating three days ago is worse than a chart that fails visibly, because failure invites a retry and stale confidence does not. Every offline-capable product needs an explicit answer to "how does the user know this is not current", and most ship without one.

The reviewer who has already traced the data asks these questions naturally. The reviewer who starts at the screens asks whether the spacing is consistent. Both reviews have value; only one of them will catch a report that quietly disagrees with the chart above it.

Rank by how far the bad data travels and how hard it is to detect, not by how obvious the symptom is on screen. Severity models built around visibility systematically under-rate the defects that cost the most.

Score each finding on eight dimensions: number of writers affected, number of consumers affected, how long the bad value persists, whether it propagates across devices or accounts, how hard it is for a user to notice, how hard it is to correct after the fact, the safety, privacy or financial consequence, and whether derived outputs amplify the error.

Compare two findings through that lens. A misaligned label on the history screen is visible, local, harmless and fixable in one commit. A unit-normalisation bypass on the import path is invisible to the user, permanently stored, replicated to every device by sync, and then reproduced in the chart, the percentile calculation, the trend logic, the exported report and the AI explanation that interprets it for a parent. The second one deserves the higher priority even though the form that created it looks completely correct — and it will be ranked lower by any process that asks a reviewer how bad the screenshot looks.

Practically, this means the fix order in your remediation plan should not match the order the findings were discovered in, and somebody has to be willing to argue for a boring storage defect over a visible layout one. If the plan does not reorder, the ranking was decorative.

Walk backwards from the wrong output to the first invariant violation, and do not touch the display until you have found it. The instinct under incident pressure is to correct what the user can see, and that instinct destroys the evidence you need.

The lineage walk has seven steps, in this order:

Observed wrong output
  -> consumer transformation that produced it
  -> source record and its version
  -> last mutation path that wrote it
  -> validation that actually executed on that path
  -> conflict and retry history for the record
  -> originating input as the user supplied it

Each step either exonerates a layer or localises the defect. The point of doing it in this order is that the answer is frequently not where the symptom is: the chart is drawing exactly what it was given, and what it was given was written correctly by a path that received an already-corrupted value from an import three releases ago.

Patching the display first is actively harmful in that scenario. It conceals bad stored data, which continues to sync, continues to feed derived values, and becomes harder to reconstruct with every day that passes — and it removes the pressure to fix the writer, so the same corruption keeps arriving. We have seen a display-layer clamp hide a unit bug for months, by which point separating good historical rows from bad ones required a per-record provenance check that would have been trivial on day one.

Preserve the lineage in the incident record itself. The value of an incident is almost entirely in the invariant it exposes, and the invariant is only visible if somebody wrote down which one broke first.

Seven mistakes account for nearly every data-first audit that produced a clean report and a corrupted database. Each one is a way of narrowing the audit until it stops being able to fail.

• Auditing only the CRUD methods. They are the paths that were designed. Imports, restores, migrations, retries and background jobs are the paths that bypass them, and they are where the unknowns cluster.
• Writing invariants after finding bugs. Rules derived from discovered defects describe your search history, not the system. Define them first, then the audit becomes a systematic search for missing enforcement instead of a tour of what you noticed.
• Trusting shared function names. Two call sites invoking something called save may reach different transformations through different wrappers. Trace actual call paths; a name is a hypothesis.
• Ignoring deletion and undo. Deletion touches derived values, caches, exports and sync, and undo is a full mutation path that usually reconstructs an object without the constructor that validates it.
• Testing storage but not propagation. A passing repository test proves the row is right. It proves nothing about the chart, the report or the export that read it.
• Assuming platform parity. iOS and Android implementing the same domain rule from the same specification will diverge. Compare executable outputs on identical inputs, not the two implementations by eye.
• Calling a path safe because the UI validates. Presentation code is bypassable by definition — every other writer in the matrix bypasses it — so an invariant enforced only there is an invariant enforced nowhere.

Make the matrix an entry condition for design and code review, so that adding a writer or a consumer requires updating it rather than remembering to. Completeness decays silently otherwise, and a stale matrix is more dangerous than no matrix because people trust it.

Two rules are enough to hold it. A new writer must declare its validation authority, its conflict policy and its downstream notification before the change is approved. A new consumer must declare its freshness policy and its behaviour when the data is missing or stale. Both fit in a pull-request template, and both take about a minute for anyone who understands their own change.

Revisit the three-object ranking whenever the product changes materially. A new payment capability, a health feature, a collaboration model or a data-sharing integration can introduce an object more consequential than anything on the original list. Depth should follow current risk, not the age or visibility of a feature — the oldest, quietest table in the schema is frequently the one everything else depends on.

The honest note on cost: the first data-first audit is slower than opening every screen. Considerably slower. What you get in exchange is reusable — once identity, invariants, mutation paths and consumers are mapped, every later feature review starts from a system model instead of rediscovering the architecture. The second audit takes a fraction of the time, and by the third the matrix is doing the remembering for you.

So: choose the critical objects. Define the invariants. Trace every mutation, failure and lifecycle path. Follow committed values into every consumer and derived output. Then protect the rules with checks that fail when the system violates them. Screens are where data becomes visible; they are not where its integrity begins or ends.

Part 12 applies the same evidence standard to the constants, formulas, documentation and AI explanations that make claims about the real world — why you should not trust any of them until they are proven. If you would rather have someone run the first pass over your product with you, talk to us and we will do it against your three objects, not your screen list.