How this works

Investigate a business metric without treating a nearby release or a persuasive narrative as proof of cause. Establish what the metric measures, whether collection changed, which cohorts moved and what independent evidence supports each explanation.

The deliverable is a reviewable diagnostic brief. AI can propose comparisons and summarize evidence; calculations use the supplied data and verified queries. Any fix, rollback or tracking change remains a separate engineering decision.

Before you start

Prepare the inputs

Funnel specification

Required

Ordered events, eligible population, counting unit, attribution window and exclusion rules.

Confirm denominator and completion rules; do not compare a session-based baseline with a user-based current rate.

Baseline and current observations

Required

Event-level or aggregated rows with period, segment, eligible_count and completed_count.

Check time zones, incomplete days, ingestion lag, duplicates and expected arithmetic. Preserve zero separately from missing.

Change timeline

Required

Release, configuration, campaign and incident entries with exact timestamps and affected audiences.

Keep deployment time distinct from first exposure and incident detection; note missing feature-flag history.

Supporting observations

Optional / conditional

Redacted support examples, reproducible errors or permitted session evidence.

Check relevance to the suspected cohort and preserve source references. Do not assume a support anecdote represents all users.

Implementation, step by step

1. 01

   Reproduce the reported change

   • Restate the funnel definition and calculate baseline and current rates from eligible and completed counts. Report both the absolute percentage-point change and the underlying volumes.
   • Compare like periods using the same time zone and conversion window. If the latest period is incomplete, label it rather than extending partial counts into a full-day total.

   Checkpoint: Another analyst can reproduce the rate from the saved counts and filters.

2. 02

   Check collection before explaining behavior

   • Review event name or schema changes, consent changes, instrumentation releases, missing events, ingestion delays and identity stitching around the alert.
   • Compare an independent operational count when available, such as created accounts or completed orders. Explain known differences in definitions instead of forcing totals to agree.

   Checkpoint: Measurement inconsistencies are either resolved or carried as active explanations.

3. 03

   Find the affected segment

   • Start with a small set of business-relevant splits such as device, geography, acquisition channel, customer cohort or authentication method. Record the denominator for each comparison.
   • Separate a change within a segment from a change in traffic mix. Flag small samples and multiple exploratory comparisons; do not invent statistical confidence or claim seasonality is ruled out from one weekday match.

   Checkpoint: The brief identifies where the movement concentrates and where evidence is too sparse.

4. 04

   Build a shared timeline

   • Align the first observed deviation with releases, campaign changes, outages and customer reports. Distinguish when a change shipped from when the affected cohort actually saw it.
   • Link each event to a source and describe conflicting evidence. A release near the drop is a candidate explanation until a mechanism or reproduction supports it.

   Checkpoint: Each candidate cause has an exposure window and supporting or contradicting observations.

5. 05

   Rank hypotheses and propose discriminating checks

   • For each explanation, state the expected observation, current support, contradictions and the smallest next check that could distinguish it from alternatives.
   • Draft a bounded issue or investigation task when warranted. Include affected steps and cohorts, counts, reproduction evidence and open questions; avoid preparing a production patch unless separately requested.

   Checkpoint: The next diagnostic action can change the conclusion, rather than merely repeat the same dashboard.

6. 06

   Publish the reviewed investigation status

   • Ask the metric owner to verify calculations and the relevant technical owner to assess any suspected defect. Mark findings as observed, plausible, reproduced or unresolved.
   • After a separately authorized change, compare the same metric and cohort definitions. Recovery alone does not retrospectively prove the initial explanation; retain the evidence chain.

   Checkpoint: The handoff distinguishes confirmed observations, hypotheses and unperformed checks.

Decisions along the way

What you will produce

Check the result

• Recalculate every displayed rate and percentage-point change from saved counts.
• Check that baseline and current snapshots share the counting unit, exclusions and conversion window.
• Verify each causal statement has stronger support than timestamp proximity.
• Ensure the brief contains a competing explanation and a concrete next check when the cause is unresolved.

When something goes wrong

A worked example

SYNTHETIC EXAMPLE · NOT A REPORTED BUSINESS RESULT

Run a small pilot

Keep the scope clear

• Do not deploy fixes, roll back releases or change analytics definitions through this diagnostic workflow.
• Use the minimum customer-level data needed; prefer aggregates and redacted examples.
• The scenario does not promise a root cause when the available evidence supports only a hypothesis.