Architecture

Prototype layer vs native streaming engine.

This build is Layer 1. Every screen, control and telemetry value here is real UI driven by simulated data. A web app on iOS or Android cannot capture another app's screen, keep an encoder alive in the background, or draw over a running game. Those capabilities are specified below as a native contract, not faked.

Pipeline

mobile game
  |
  v  native screen capture
     iOS   ReplayKit + Broadcast Upload Extension
     Andr. MediaProjection + foreground service
  |
  v  compositor  (facecam · overlays · alerts · chat · branding)
  |
  v  encode      VideoToolbox / MediaCodec + AAC
  |
  v  egress      RTMPS · SRT · WHIP
  |
  v  streaming server
     ingest · auth · routing · transcode · stats · chat relay
  |
  +--> platform fan-out  (Twitch · YouTube · Kick · custom RTMP)
  |
  +--> low-latency return feed  (WHEP ~300ms / LL-HLS ~2s)
         |
         v  preview window in app, PiP, or floating bubble

Not possible in the web layer

Whole-device screen capture

Impossible in mobile Safari/Chrome — no getDisplayMedia for other apps.

iOS ReplayKit Broadcast Upload Extension · Android MediaProjection service

Streaming while the app is backgrounded

Browsers suspend timers, media, and sockets on background.

iOS broadcast extension process · Android foreground service (mediaProjection)

Preview floating over the game

No cross-app windows from a web view.

iOS AVPictureInPictureController · Android SYSTEM_ALERT_WINDOW bubble

Device / game audio capture

No API for capturing other apps' audio.

iOS ReplayKit appAudio buffers · Android AudioPlaybackCapture (API 29+)

Hardware encode + RTMPS/SRT egress

No raw socket or hardware encoder access.

VideoToolbox / MediaCodec + native RTMPS·SRT·WHIP client

Platform OAuth token storage

Prototype uses mock connections; no real tokens are issued.

Server-side OAuth exchange, tokens in Keychain / EncryptedSharedPreferences

Bridge contract

The web layer talks to native through a single interface (Capacitor plugin or RN TurboModule), defined in src/lib/native/streaming-bridge.ts:

interface StreamingBridge {
  requestPermissions(perms): Promise<...>
  startBroadcast(meta, config): Promise<{ sessionId }>
  stopBroadcast(): Promise<void>
  pauseBroadcast(paused): Promise<void>
  updateEncoder(config): Promise<void>
  getPreviewEndpoint(): { url, source }   // WHEP / LL-HLS
  setFloatingPreview(visible)             // PiP / bubble
  subscribeHealth(cb): () => void
}

Preview source of truth

The preview must be produced from the encoded broadcast, so the streamer sees overlays, alerts, facecam framing and audio state exactly as the audience does. Local composite preview exists only as a labelled fallback when ingest is unreachable. On iOS the only OS-sanctioned way to keep that moving preview visible while the user is inside a game is Picture-in-Picture; a floating window is Android-only. Where neither is available, the app degrades to a Live Activity / notification with LIVE, viewers, timer and connection state.

Streaming server responsibilities

  • · Session auth & per-session stream keys (revocable)
  • · Ingest (RTMPS/SRT/WHIP), transcode ladder when multistreaming
  • · Routing to N destinations through the provider abstraction
  • · WHEP/LL-HLS return feed for the on-device preview
  • · Health telemetry, viewer stats, unified chat relay
  • · Platform OAuth exchange and encrypted token storage
  • providers registered: twitch · youtube · kick · facebook · custom-rtmp