Part 5 | Low-Latency Video Streaming and Telemetry-Driven Adaptive Bitrate Optimizations in Weak Networks

In remote support, even with fast screen capture and rendering pipelines, network jitter and packet loss can introduce display latency and artifacts. In real-world environments, users connect over varying networks (4G, 5G, public Wi-Fi, home broadband) that experience fluctuations in bandwidth.

Attempting to stream high-bitrate video over a congested connection fills the socket buffer, causing packet loss and dropping the connection.

In this final article of our series, we examine how the Easy Connect Suite manages transmission over unstable networks using in-place parameter tuning, network telemetry loops, on-demand keyframe requests, and binary envelope packaging.

---

1. In-Place Encoder Parameter Tuning

To adapt to network congestion, some video streaming setups tear down the active compression session, adjust bitrate or resolution settings, and initialize a new encoder instance.

However, initializing a hardware video encoder is resource-heavy, taking 200ms to 500ms. During this delay, the remote screen freezes or flickers, interrupting the session.

Easy Connect Suite implements in-place parameter tuning across platforms, updating encoder parameters dynamically without rebuilding the session:

• Android: Pass a configuration Bundle to the running MediaCodec instance:

  Bundle param = new Bundle();
  param.putInt(MediaCodec.PARAMETER_KEY_VIDEO_BITRATE, targetBitrateInBytes);
  codec.setParameters(param);

• iOS: Call VideoToolbox properties to update target average and maximum bitrates:

  VTSessionSetProperty(compressionSession, 
                       key: kVTCompressionPropertyKey_AverageBitRate, 
                       value: targetBitrate as CFTypeRef)

• Windows & Linux: Update the media foundation MFT session or VA-API encoding context attributes dynamically.

These updates execute in less than 1ms, adjusting the video bandwidth without interrupting or flickering the display.

---

2. Telemetry-Driven Adaptation Loop

To update bitrates dynamically, the sender needs to monitor the current network capacity. We implement a network telemetry loop:

[ Receiver (Helper Client) ] ──► Collect Telemetry (RTT, loss rate, queue size)
                                         │
                                         ▼ (Shared over Yamux control channel)
[ Sender (Controlled Client) ] ──► Process in Diagnostic Ticker
                                         │
                 ┌───────────────────────┴───────────────────────┐
                 ▼ (Congested: RTT/Loss spikes)                  ▼ (Network recovers)
        [ Scale Down Resolution/Bitrate ]              [ Step Up Resolution/Bitrate ]
        Step down: High ──► Standard ──► Smooth.        Gradually raise target bitrate
        Decrease target bitrate and FPS limits.         and FPS limits.

1. Telemetry Collection: The receiver's diagnostic ticker monitors round-trip time (RTT), packet loss rate, and queue sizes every 100ms.
2. Feedback Channel: Telemetry parameters are sent back to the sender via a dedicated control stream in the Yamux multiplexer.
3. Adaptive Downgrades: If RTT spikes (e.g., > 200ms) or packet loss increases (e.g., > 5%), the sender flags network congestion and adapts:
   • Resolution Scaling: Swaps the stream quality profile (e.g., from High to Standard or Smooth).
   • Bitrate Limits: Lowers the encoder's target bitrate and reduces the frame rate limit (FPS Throttle) to clear network congestion.
4. Gradual Upgrades: When telemetry parameters return to normal, the sender increases resolution and bitrate limits incrementally.

---

3. On-Demand IDR Keyframe Control

In video streaming, keyframes (I-frames/IDR frames) contain the complete image data needed to start decoding. Delta frames (P-frames) carry only difference data and depend on preceding frames. If a P-frame is lost during transit, the decoder cannot render subsequent frames, causing display corruption (smeared screens or green blocks).

Traditional pipelines output keyframes at a fixed interval (e.g., every 2 seconds). This introduces disadvantages:

• Bandwidth Overhead: On static screens, sending large I-frames (which are 5-10x the size of P-frames) wastes bandwidth.
• Network Congestion: Sending a large keyframe over a congested link can saturate socket buffers, increasing packet loss.

Easy Connect Suite implements on-demand IDR keyframe requests:

[ Receiver Client ] ──► Detects missing P-frame or decoding error
                              │
                              ▼ (Sends encoder_sync command)
[ Sender Client ] ──► Receives sync command ──► Forces next frame to be IDR keyframe

1. P-Frame Defaults: During normal operation, the encoder uses a long keyframe interval (e.g., 30 seconds or only the initial frame), sending small P-frames.
2. Loss Detection: If the receiver detects a missing frame or a decoding failure, it immediately transmits an encoder_sync request to the sender.
3. Forced IDR Generation: Upon receiving the request, the sender forces the encoder to render the next frame as an IDR keyframe:
   • On iOS: Set the kVTEncodeFrameOptionKey_ForceKeyFrame option.
   • On Android: Set the PARAMETER_KEY_REQUEST_SYNC_FRAME parameter.
4. Display Recovery: The receiver decodes the new keyframe and restores the display, minimizing screen distortion.

---

4. Packaging Format: compact_binary_v1

To minimize protocol overhead, we use a custom binary wrapper format, compact_binary_v1:

Envelope Layout:

┌───────────┬─────────────┬─────────────┬───────────┬──────────────┐
│  Magic    │ Packet Type │ Frame Index │ Timestamp │ Payload Size │ ... Raw Slice Data
│  (1 Byte) │  (1 Byte)   │  (4 Bytes)  │ (8 Bytes) │  (4 Bytes)   │
└───────────┴─────────────┴─────────────┴───────────┴──────────────┘

• Compact Header: The wrapper header occupies only a few bytes, containing the packet type, frame sequence number, timestamp, and payload length.
• Dynamic Metadata: H.264 configuration metadata (SPS and PPS) is attached only to IDR keyframes. P-frames carry only the raw slice payload, saving bandwidth.

---

Conclusion: End-to-End Latency Control

This adaptive transmission layer completes our real-time video pipeline. Let's summarize the architecture:

1. iOS VideoToolbox: ReplayKit capture and Broadcast Extensions running within Apple's 50MB memory limit.
2. Android Surface Encoding: Zero-copy MediaCodec pipelines with OpenGL ES frame rotation and scaling.
3. Desktop Capture: Native bindings for DXGI (Windows), ScreenCaptureKit (macOS), and PipeWire (Linux) to minimize capture latency.
4. WebCodecs Rendering: Hardware-accelerated decoding inside Web Workers, rendering to OffscreenCanvas with backpressure queue control.
5. Adaptive Transmission: Telemetry-driven bitrate adjustments and on-demand keyframe requests to handle weak network connections.

To review configuration options, check out our Client Usage Guide, or download Easy Connect SSH and Easy Link Assist to experience secure, low-latency remote access.