Connect metrics, charts, and experiments to a question

A document site wants to know whether its new course directory is easier to use. More clicks may mean users find lessons, or that they repeatedly search because they are lost. Define the decision, events, and comparison before drawing a chart.

The case studies completing a first exercise after entering a lesson. Synthetic outcomes illustrate denominators and missing values, while five latency values illustrate mean and median. None of the data reports a real product result.

Define the metric completely

A conversion metric needs a population, unit, starting event, success event, and observation window. For example, measure assigned users who complete an exercise within a fixed interval after first entry. The numerator counts successful users and the denominator follows the inclusion rule. Repeated clicks do not automatically create additional users.

Separate results outside the window, incomplete observation, and collection failures. Missing data is not failure, but excluding it can also bias a result. If users on poor networks lose more events, known outcomes may not represent everyone. Report total, known, and missing counts together.

Document units, filters, timezone, and definition version. Frontend clicks, backend task completion, and business completion are different events. Shared definitions and query versions prevent disagreements caused by inconsistent measurement.

Describe location and distribution separately

The mean divides the total by the count and is affected by extreme values. The median describes the sorted middle and is less affected by outliers, but does not reveal the tail. Slow-request analysis also needs appropriate quantiles, timeout ratios, and sample sizes.

Python statistics documents these calculations. With an even count, median normally averages the middle pair and need not equal an observed value. Empty data means no calculable value, not zero latency. Define treatment of missing, nonnumeric, and infinite values before calculation.

The example retains a latency of 200. None represents an unknown conversion outcome. It calculates the known subset while preserving that limitation.

from statistics import mean, median
latencies = [10, 12, 14, 16, 200]
assert median(latencies) == 14
assert mean(latencies) == 50.4
outcomes = [True, False, None, True]
known = [x for x in outcomes if x is not None]
rate = sum(known)/len(known)
assert rate == 2/3
print(mean(latencies), median(latencies), len(known), rate)

The actual output is 50.4 14 3 0.6666666666666666. The mean is 50.4 and the median 14; the tail value raises the mean. Three outcomes are known and two are successful, giving two thirds for that subset. This is neither a confirmed overall rate for all four objects nor a statistical test.

Design charts for the comparison

Label axes, units, and grouping. Comparing directory rates needs a consistent scale and visible denominators. Success counts alone depend on traffic volume. Truncated bar axes can exaggerate a small difference. Distribution charts and daily trends answer different questions; appearance does not establish the conclusion.

Vega-Lite expresses data, marks, and encodings through a declarative specification. It can render a chart without deciding metric meaning, filters, or causality. Keep calculation rules and data versions alongside the chart for review.

Inspect groups before aggregation. A change in the mixture of new and returning users can improve an overall rate while each subgroup declines. Segmentation helps investigate mechanisms. Searching afterwards for the best-looking subgroup and declaring victory increases opportunities for error.

Start experiments with assignment units

An A/B test randomly assigns comparable units and keeps their assignment consistent. User and session assignment have different interference risks: a user seeing both directories may carry learning between them. Explain shared devices, collaboration, and cross-device use when choosing the unit.

Specify the primary metric, guardrails, worthwhile effect size, sample size, and analysis plan before starting. Guardrails might include failures or task duration. A two-percentage-point difference alone cannot establish a reliable winner; sampling uncertainty and the experimental design matter.

Repeatedly checking results and stopping as soon as they appear significant changes error probabilities. Define an observation plan and suitable methods. Continuous-monitoring methods also have assumptions. Investigate small samples, allocation imbalance, and missing events before interpreting outcomes.

Separate statistical questions from product decisions

The available NIST page describes Levene's test for equality of variances. It is not a mean or median tutorial, nor a universal test for conversion winners. Choose statistical methods according to data type, assumptions, and the question.

A detectable difference may still be too small to justify implementation and maintenance. A nonsignificant result may reflect inadequate sample size and does not establish exact equality. Report estimates, uncertainty, and limitations, then explain whether to gather more data, repair collection, or stop the proposal.

Correlation charts show joint changes without establishing causation. Holidays, traffic sources, and promotion can affect several metrics together. Randomization reduces some confounding, while execution mistakes, interference, and attrition can still weaken interpretation.

Retain the order of deduplication, filtering, grouping, and aggregation. Calculating an event-level rate and deduplicating users afterwards can differ from a user-level metric. Manually trace several records through the pipeline to verify that the query follows the definition. A correct formula can still answer the wrong question because its unit or filtering order changed.

Exercise: produce a reviewable analysis

Write a plan specifying the population, assignment unit, success event, window, missing-value treatment, and primary metric. Generate two synthetic groups with known successes, known failures, and unknown outcomes. Report denominators and rates separately, alongside latency distributions.

Accept the analysis when units do not appear in both groups, duplicates do not increase counts, missing values do not silently become zeros, and charts and tables share definitions. Check empty samples, all-unknown outcomes, and one extreme latency against the declared rules. Distinguish descriptive differences from experimental inference and identify the conditions still needed for a product decision.