ReadThat 7: Observability

Part 7 of the ReadThat case study.

📲 Try it live: Download the ReadThat APK (8.8 MB, Android 8+). Open the file on your phone and allow “install unknown apps” if prompted.

The founding rule of the measurement contract: a number without its boundary and segment defined is not the same metric. Everything below fixes boundaries first, targets second. Full contract: docs/PERFORMANCE_OBSERVABILITY.md.

The pipeline

1
Android features / iOS adapter / browser adapter
2
            |
3
            v
4
vendor-neutral PerformanceEvent (KMP contract, :core:observability)
5
            |
6
      L1 bounded queue  ->  L2 durable Room outbox
7
            |
8
POST /v1/telemetry/performance  (max 50 events / 64 KiB)
9
            |
10
Worker: strict allowlist + per-IP limit
11
            |
12
Analytics Engine distributions
13
  ├── Workers Logs  (accepted-batch audit)
14
  └── Workers Traces (sampled backend dependency latency)

:core:observability compiles for Android, iOS, and browser JS, and deliberately exposes a small OpenTelemetry-shaped contract instead of importing the still-experimental OTel KMP SDK; moving to OTLP later touches the exporter, not the feature modules. PerformanceTelemetry.kt

The Worker enforces the schema as a hard allowlist: every metric name and dimension is enumerated; anything else is rejected at ingest:

1
const metricName = z.enum([
2
  "home_tti", "feed_initial_fetch", "feed-load-success", "feed-load-fail",
3
  "comments_tti", "comments_initial_fetch", "comments_full_fetch",
4
  "media_feed_tti", "mutation_local_commit", "mutation_server_ack",
5
  "interaction_to_next_frame", "screen_frame_summary", "network_request",
6
  "video_time_to_first_frame", "video_rebuffer", "sdui_dropped_cell", ...
7
]);

telemetry.ts#L5-L28

Privacy is structural, not policy: no titles, bodies, URLs, or tokens pass the schema; user and content IDs are HMAC-peppered into irreversible pseudonyms server-side; network events record a route template, never a URL. Product analytics (sessions, impressions with a 600 ms dwell gate, playback time) is a separate dataset with the same posture (docs/PRODUCT_ANALYTICS.md).

Timer boundaries and key SLOs

Boundaries follow Reddit’s published vocabulary so numbers are comparable to their posts. Selected budgets (p50/p90/p99, evaluated per platform × start type × cache tier × network before any rollup):

Frame health is measured per surface via JankStats (screen_frame_summary after 300 frames: target jank < 1%, zero frozen frames). The local-commit / server-ACK split is the most diagnostic pair in the table: a slow local commit is a client storage bug; a slow ACK with a fast commit means the UX held while network/edge/database needs work.

Which metrics are levers for DAU and engagement

Perf metrics aren’t goals; they’re levers on top-line metrics. The causal chain this contract is built around:

1
latency/reliability metric  →  behavior metric        →  top line
2
Comments TTI                →  comments viewed/posted →  session length, retention
3
feed-load failure rate      →  posts viewed           →  DAU
4
video first-frame + rebuffer→  watch time             →  time spent
5
touch→frame, jank           →  scroll depth           →  posts viewed

Reddit’s published sensitivities make the chain concrete: their Instant Comment Loading work moved p90 Comments TTI −60% and measured +4% comments viewed; their instrumentation post ties a 0.15% feed-load error rate to roughly 5% fewer posts viewed. Error rates punch far above their percentage; a failed load ends a session in a way a slow load doesn’t. That’s why the release gate blocks on failure ≥ 0.10% but only on a >10% p90 TTI regression.

The levers, ranked by expected top-line torque:

1. Feed-load failure rate: the cheapest DAU lever in the system.
2. Comments TTI: directly gated conversion into the highest-value engagement act.
3. Cold Home TTI: the first-session/notification-open experience; retention lever.
4. Video first frame + rebuffer ratio: time-spent lever; rebuffer > 1% correlates with abandons.
5. Jank/frozen frames: a slow diffuse lever, but frozen frames and ANRs are also a Play-Store-visibility lever (bad-behavior threshold 0.47%).

Diagnostic metrics (payload bytes, edge latency via Server-Timing, cache-tier splits, sdui_dropped_cell) exist to explain lever movement, not to be goaled.

What changes for international markets

The same contract, different weightings; this is the measurement half of part 8:

• Segment or lie. Mid-range Android with 3–4 GB RAM dominates growth markets. A global p90 hides that the p90 in market is the p50 experience there. Every SLO panel splits by device class, network type, and country before rollup.
• Different levers dominate. On congested or metered networks: payload size (feed_query_response_size), rebuffer ratio, and offline correctness (outbox drain success, queued-mutation age) become primary. The local-commit metric matters more than ACK; it’s the metric of “the app worked on the bus.”
• Device capability mix moves frame health and cold TTI from tier-3 to tier-1: decode time for images, jank on low-RAM devices under memory pressure (trimMemory behavior is instrumented for a reason).
• Carrier/network reality: HTTP/3 adoption per origin per country is tracked because QUIC is throttled or broken on some carriers; a silent h3→h2 collapse is a halt-and-inspect trigger in the runbook.
• Cultural/product differences show up in product analytics, not perf: dwell-gated impressions, comments-per-session, and media-vs-text mix differ by market and should re-weight which surfaces get the optimization budget.

Release gates

Rollout blocks on: >10% p90 Home/Comments TTI regression, feed-load failure ≥ 0.10%, any repeatable frozen frame, a lost optimistic rollback, or telemetry schema rejection from a production build. Percentiles publish with sample counts; p99 doesn’t alert until the population is meaningful. And a device launch stays in the gate even when all suites are green: an emulator run caught a JankStats initialization-order defect no unit test could.

📷 Screenshot placeholder: Analytics Engine query results for comments_tti split by cache_tier.

← Part 6: Backend architecture · Next: Part 8: International strategy →