ReadThat 7: Observability

Part 7 of the ReadThat case study.

📲 Try it live: Download the ReadThat APK (8.3 MB, Android 8+). Open the file on your phone and allow “install unknown apps” if prompted.

The founding rule of the measurement contract: a number without its boundary and segment defined is not the same metric. Everything below fixes boundaries first, targets second. Full contract: docs/PERFORMANCE_OBSERVABILITY.md.

The pipeline

1
Android features / iOS adapter / browser adapter
2
            |
3
            v
4
vendor-neutral PerformanceEvent (KMP contract, :core:observability)
5
            |
6
      L1 bounded queue  ->  L2 durable Room outbox
7
            |
8
POST /v1/telemetry/performance  (max 50 events / 64 KiB)
9
            |
10
Worker: strict allowlist + per-IP limit
11
            |
12
Analytics Engine distributions
13
  ├── Workers Logs  (accepted-batch audit)
14
  └── Workers Traces (sampled backend dependency latency)

:core:observability compiles for Android, iOS, and browser JS, and deliberately exposes a small OpenTelemetry-shaped contract instead of importing the still-experimental OTel KMP SDK; moving to OTLP later touches the exporter, not the feature modules. PerformanceTelemetry.kt

The Worker enforces the schema as a hard allowlist: every metric name and dimension is enumerated; anything else is rejected at ingest:

1
const metricName = z.enum([
2
  "home_tti", "feed_initial_fetch", "feed-load-success", "feed-load-fail",
3
  "comments_tti", "comments_initial_fetch", "comments_full_fetch",
4
  "media_feed_tti", "mutation_local_commit", "mutation_server_ack",
5
  "interaction_to_next_frame", "screen_frame_summary", "network_request",
6
  "video_time_to_first_frame", "video_rebuffer", "sdui_dropped_cell", ...
7
]);

telemetry.ts#L5-L28

Privacy is structural, not policy: no titles, bodies, URLs, or tokens pass the schema; user and content IDs are HMAC-peppered into irreversible pseudonyms server-side; network events record a route template, never a URL. Product analytics (sessions, impressions with a 600 ms dwell gate, playback time) is a separate dataset with the same posture (docs/PRODUCT_ANALYTICS.md).

Timer boundaries and key SLOs

Boundaries follow Reddit’s published vocabulary so numbers are comparable to their posts. Selected budgets (p50/p90/p99, evaluated per platform × start type × cache tier × network before any rollup):

Frame health is measured per surface via JankStats (screen_frame_summary after 300 frames: target jank < 1%, zero frozen frames). The local-commit / server-ACK split is the most diagnostic pair in the table: a slow local commit is a client storage bug; a slow ACK with a fast commit means the UX held while network/edge/database needs work.

Live telemetry snapshot

The snapshot spans migration-era builds and exposed a contract problem more important than any individual regression. The now-deleted Android-only path started Home TTI at application initialization and labeled cold/warm starts; the shared KMP path starts the same home_tti metric inside its Feed composable and leaves start_type empty. Those historical cohorts landed in one dataset but measured different intervals, so they must not be compared as if they were one experiment. The Android-only path has since been removed, but the remaining shared timer still needs to move to the application boundary (and add a start classification) before Home TTI can serve as a release gate.

In that historical Android-only debug cohort, cold p90 was 8.9 seconds against a 1.2-second budget, while its two-sample warm cohort was 55 seconds. Those values point to a timer crossing authentication or navigation before Home was eventually shown, not an eight- or 55-second render. Fixing the shared timer boundary is the first action; only then is performance optimization meaningful.

Build type matters here. Only 201 performance events and 46 product events in this window came from release builds. The Android release variant enables R8 optimization/minification and resource shrinking; debug does not. These debug-heavy results validate the instrumentation and expose timer or routing defects, but they do not establish release performance. Within the debug cohort, MediaFeed TTI and touch-to-next-frame are inside budget, while comments, initial feed fetch, and frame freezes identify the useful next investigations. The next benchmark should run these same segmented queries over a materially larger release cohort.

Which metrics are levers for DAU and engagement

Perf metrics aren’t goals; they’re levers on top-line metrics. The causal chain this contract is built around:

1
latency/reliability metric  →  behavior metric        →  top line
2
Comments TTI                →  comments viewed/posted →  session length, retention
3
feed-load failure rate      →  posts viewed           →  DAU
4
video first-frame + rebuffer→  watch time             →  time spent
5
touch→frame, jank           →  scroll depth           →  posts viewed

Reddit’s published sensitivities make the chain concrete: their Instant Comment Loading work moved p90 Comments TTI −60% and measured +4% comments viewed; their instrumentation post ties a 0.15% feed-load error rate to roughly 5% fewer posts viewed. Error rates punch far above their percentage; a failed load ends a session in a way a slow load doesn’t. That’s why the release gate blocks on failure ≥ 0.10% but only on a >10% p90 TTI regression.

The levers, ranked by expected top-line torque:

1. Feed-load failure rate: the cheapest DAU lever in the system.
2. Comments TTI: directly gated conversion into the highest-value engagement act.
3. Cold Home TTI: the first-session/notification-open experience; retention lever.
4. Video first frame + rebuffer ratio: time-spent lever; rebuffer > 1% correlates with abandons.
5. Jank/frozen frames: a slow diffuse lever, but frozen frames and ANRs are also a Play-Store-visibility lever (bad-behavior threshold 0.47%).

Diagnostic metrics (payload bytes, edge latency via Server-Timing, cache-tier splits, sdui_dropped_cell) exist to explain lever movement, not to be goaled.

What changes for international markets

The same contract, different weightings; this is the measurement half of part 8:

• Segment or lie. Mid-range Android with 3–4 GB RAM dominates growth markets. A global p90 hides that the p90 in market is the p50 experience there. Every SLO panel splits by device class, network type, and country before rollup.
• Different levers dominate. On congested or metered networks: payload size (feed_query_response_size), rebuffer ratio, and offline correctness (outbox drain success, queued-mutation age) become primary. The local-commit metric matters more than ACK; it’s the metric of “the app worked on the bus.”
• Device capability mix moves frame health and cold TTI from tier-3 to tier-1: decode time for images, jank on low-RAM devices under memory pressure (trimMemory behavior is instrumented for a reason).
• Carrier/network reality: HTTP/3 adoption per origin per country is tracked because QUIC is throttled or broken on some carriers; a silent h3→h2 collapse is a halt-and-inspect trigger in the runbook.
• Cultural/product differences show up in product analytics, not perf: dwell-gated impressions, comments-per-session, and media-vs-text mix differ by market and should re-weight which surfaces get the optimization budget.

Release gates

Rollout blocks on: >10% p90 Home/Comments TTI regression, feed-load failure ≥ 0.10%, any repeatable frozen frame, a lost optimistic rollback, or telemetry schema rejection from a production build. These gates evaluate release-build cohorts only; debug metrics are diagnostic and never decide rollout. Percentiles publish with sample counts; p99 doesn’t alert until the population is meaningful. And a device launch stays in the gate even when all suites are green: an emulator run caught a JankStats initialization-order defect no unit test could.

← Part 6: Backend architecture · Next: Part 8: International strategy →