Project: a device-sampling data and trend dashboard

Define what a sample represents

This project builds a trend dashboard from synthetic device data. Generate a local time series to study ingestion, deduplication, aggregation, and display without connecting real devices. The first version answers which valid samples arrived for a period, whether the trend is continuous, and when the last reliable sample was taken.

Each record needs a device ID, event identity, sample time, ingestion time, value, unit, and quality flag. Sample time is the device's claimed measurement time; ingestion time is when the service received it. Network delay and clock error can separate them. Plotting ingestion time as measurement time can make old observations appear to be new changes.

Device plus sequence can identify events, but specify whether a restart reuses sequence values. A session or boot identifier may be needed. Equal sequences from different devices are not duplicates, while retransmitting one event should not increase sample counts. Establish these rules before using records as chart input.

Validate and persist at the ingestion boundary

With Go, use net/http for an ingestion endpoint. Bind the authenticated device to the device ID in its records. Limit text fields, batch size, and request-body size. MaxBytesReader can bound body reads. Header-reading timeouts and body-reading deadlines need separate consideration; ReadHeaderTimeout is not a whole-upload timeout.

Check types, time formats, finite numbers, and units after parsing. A teaching policy may define an accepted range, but passing that range does not establish sensor calibration. Physical validity still depends on the device and measurement procedure. Define rejection or retention rules for bad quality, anomalous time, and malformed records; do not silently turn them into zero.

Persist through operations that enforce unique event identity. A device may retransmit after losing a response. The same identity with different content is a conflict to record and resolve; the later arrival is not automatically correct. Report which events were accepted or rejected so clients can retry within a bounded policy.

Aggregate synthetic samples by event time

This example uses relative seconds. Four inputs include one identical duplicate. A dictionary deduplicates synthetic event IDs, then calculates one-minute window means. Window 0 contains the samples at 0 and 30 seconds; window 1 contains the sample at 60 seconds.

samples = [("S-1", 0, 10), ("S-1", 0, 10), ("S-2", 30, 20), ("S-3", 60, 30)]
unique = {sample_id: (time, value) for sample_id, time, value in samples}
windows = {}
for time, value in unique.values():
    windows.setdefault(time//60, []).append(value)
means = {window: sum(values)/len(values) for window, values in windows.items()}
assert len(unique) == 3
assert means == {0: 15.0, 1: 30.0}
print(means)

The output is {0: 15.0, 1: 30.0}, with three unique events. For repeated IDs, the example retains the last input. It does not compare revision times or resolve conflicting content. Apply the real identity and conflict policy before aggregation.

These are arithmetic sample means with equal weights. Irregular sampling does not make them time-weighted averages. A window without samples is absent from the dictionary; the interface must preserve that missing state instead of filling zero and implying inactivity. The model establishes no calibration, streaming performance, or real observation result.

Distinguish late, missing, and stale data

Specify whether windows use sample time or ingestion time. Sample-time windows may change when late events arrive, requiring a revision period, window version, and refresh policy. Ingestion-time windows describe arrival patterns. Do not silently switch between those meanings in a chart.

Store a consistent time basis and format it in the viewer's timezone, keeping original timestamps available for inspection. Sort before connecting points so out-of-order events do not bend the line backward. Include future timestamps, restarted sequences, and batches uploaded after an outage in acceptance cases.

Freshness compares the latest sample accepted under the policy with the current time. An old point should produce a stale-data state, even if its value remains visible. Show unknown, bad quality, missing, and measured zero separately because each changes whether a reader can rely on the display.

Separate dashboard data from service monitoring

An aggregation response can include window start, count, minimum, maximum, mean, and quality. Vega-Lite declares axes and visual encodings; the application still supplies field types and units. Show the period, unit, refresh time, and gaps, and offer a table or text view for checking values.

Device samples are application data. Prometheus is suitable for operational measures such as ingestion request counts, errors, and latency. It identifies time series by metric names and label sets. Event IDs and large dynamic sets of device IDs can multiply series counts. Avoid sending all raw business events into service monitoring merely to avoid choosing application storage.

Define alerts separately. A teaching rule could require several consecutive valid windows above a threshold and distinguish an unusual value from missing data. The threshold and duration are exercise policies, not recommendations for real safety devices. Specify recovery notices, duplicate notices, and the handling of unacknowledged alerts.

Version the record schema when units or sampling protocols change. Preserve original values and conversion rules rather than changing only the chart label. Isolate or reject incompatible versions, and replay historical records with their documented conversion policy.

Verify replay, interfaces, and display

Start with a fixed sample set, deduplication, window results, and a static chart. Then connect Go ingestion and querying. Keep a data version, and display only responses for the current query range so a slow older request cannot replace a newer chart. Bound result count and choose aggregation granularity for large ranges.

Test exact duplicates, identity conflicts, out-of-order arrivals, window boundaries, zero, empty windows, non-finite or invalid numbers, timezone changes, stale samples, and catch-up batches. Compare raw event count, unique valid count, and window counts. Differences can be expected filtering or a defect, depending on the documented policy.

Check tables and empty states as well as charts. Confirm persisted samples survive service restart, and failed ingestion provides a recoverable status. Evaluating the whole path requires actual endpoint, database, and browser checks; the offline example establishes only its synthetic aggregation result.