update skills

This commit is contained in:
2026-03-17 16:53:22 -07:00
parent 0b0783ef8e
commit f9a530667e
389 changed files with 54512 additions and 1 deletions

View File

@@ -0,0 +1,65 @@
# Cloudflare Realtime SFU Reference
Expert guidance for building real-time audio/video/data applications using Cloudflare Realtime SFU (Selective Forwarding Unit).
## Reading Order
| Task | Files | ~Tokens |
|------|-------|---------|
| New project | README → configuration | ~1200 |
| Implement publish/subscribe | README → api | ~1600 |
| Add PartyTracks | patterns (PartyTracks section) | ~800 |
| Build presence system | patterns (DO section) | ~800 |
| Debug connection issues | gotchas | ~700 |
| Scale to millions | patterns (Cascading section) | ~600 |
| Add simulcast | patterns (Advanced section) | ~500 |
| Configure TURN | configuration (TURN section) | ~400 |
## In This Reference
- **[configuration.md](configuration.md)** - Setup, deployment, environment variables, Wrangler config
- **[api.md](api.md)** - Sessions, tracks, endpoints, request/response patterns
- **[patterns.md](patterns.md)** - Architecture patterns, use cases, integration examples
- **[gotchas.md](gotchas.md)** - Common issues, debugging, performance, security
## Quick Start
Cloudflare Realtime SFU: WebRTC infrastructure on global network (310+ cities). Anycast routing, no regional constraints, pub/sub model.
**Core concepts:**
- **Sessions:** WebRTC PeerConnection to Cloudflare edge
- **Tracks:** Audio/video/data channels you publish or subscribe to
- **No rooms:** Build presence layer yourself via track sharing (see patterns.md)
**Mental model:** Your client establishes one WebRTC session, publishes tracks (audio/video), shares track IDs via your backend, others subscribe to your tracks using track IDs + your session ID.
## Choose Your Approach
| Approach | When to Use | Complexity |
|----------|-------------|------------|
| **PartyTracks** | Production apps with device switching, React | Low - Observable-based, handles reconnections |
| **Raw API** | Custom requirements, non-browser, learning | Medium - Full control, manual WebRTC lifecycle |
| **RealtimeKit** | End-to-end SDK with UI components | Lowest - Managed state, React hooks |
**Recommendation:** Start with PartyTracks for most production applications. See patterns.md for PartyTracks examples.
## SFU vs RealtimeKit
- **Realtime SFU:** WebRTC infrastructure (this reference). Build your own signaling, presence, UI.
- **RealtimeKit:** SDK layer on top of SFU. Includes React hooks, state management, UI components. Part of Cloudflare AI platform.
Use SFU directly when you need custom signaling or non-React framework. Use RealtimeKit for faster development with React.
## Setup
Dashboard: https://dash.cloudflare.com/?to=/:account/calls
Get `CALLS_APP_ID` and `CALLS_APP_SECRET` from dashboard, then see configuration.md for deployment.
## See Also
- [Orange Meets Demo](https://demo.orange.cloudflare.dev/)
- [Orange Source](https://github.com/cloudflare/orange)
- [Calls Examples](https://github.com/cloudflare/calls-examples)
- [API Reference](https://developers.cloudflare.com/api/resources/calls/)
- [RealtimeKit Docs](https://developers.cloudflare.com/workers-ai/realtimekit/)

View File

@@ -0,0 +1,158 @@
# API Reference
## Authentication
```bash
curl -X POST 'https://rtc.live/v1/apps/${CALLS_APP_ID}/sessions/new' \
-H "Authorization: Bearer ${CALLS_APP_SECRET}"
```
## Core Concepts
**Sessions:** PeerConnection to Cloudflare edge
**Tracks:** Media/data channels (audio/video/datachannel)
**No rooms:** Build presence via track sharing
## Client Libraries
**PartyTracks (Recommended):** Observable-based client library for production use. Handles device changes, network switches, ICE restarts automatically. Push/pull API with React hooks. See patterns.md for full examples.
```bash
npm install partytracks @cloudflare/calls
```
**Raw API:** Direct HTTP + WebRTC for custom requirements (documented below).
## Endpoints
### Create Session
```http
POST /v1/apps/{appId}/sessions/new
{sessionId, sessionDescription}
```
### Add Track (Publish)
```http
POST /v1/apps/{appId}/sessions/{sessionId}/tracks/new
Body: {
sessionDescription: {sdp, type: "offer"},
tracks: [{location: "local", trackName: "my-video"}]
}
{sessionDescription, tracks: [{trackName}]}
```
### Add Track (Subscribe)
```http
POST /v1/apps/{appId}/sessions/{sessionId}/tracks/new
Body: {
tracks: [{
location: "remote",
trackName: "remote-track-id",
sessionId: "other-session-id"
}]
}
{sessionDescription} (server offer)
```
### Renegotiate
```http
PUT /v1/apps/{appId}/sessions/{sessionId}/renegotiate
Body: {sessionDescription: {sdp, type: "answer"}}
```
### Close Tracks
```http
PUT /v1/apps/{appId}/sessions/{sessionId}/tracks/close
Body: {tracks: [{trackName}]}
{requiresImmediateRenegotiation: boolean}
```
### Get Session
```http
GET /v1/apps/{appId}/sessions/{sessionId}
{sessionId, tracks: TrackMetadata[]}
```
## TypeScript Types
```typescript
interface TrackMetadata {
trackName: string;
location: "local" | "remote";
sessionId?: string; // For remote tracks
mid?: string; // WebRTC mid
}
```
## WebRTC Flow
```typescript
// 1. Create PeerConnection
const pc = new RTCPeerConnection({
iceServers: [{urls: 'stun:stun.cloudflare.com:3478'}]
});
// 2. Add tracks
const stream = await navigator.mediaDevices.getUserMedia({video: true, audio: true});
stream.getTracks().forEach(track => pc.addTrack(track, stream));
// 3. Create offer
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
// 4. Send to backend → Cloudflare API
const response = await fetch('/api/new-session', {
method: 'POST',
body: JSON.stringify({sdp: offer.sdp})
});
// 5. Set remote answer
const {sessionDescription} = await response.json();
await pc.setRemoteDescription(sessionDescription);
```
## Publishing
```typescript
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
const res = await fetch(`/api/sessions/${sessionId}/tracks`, {
method: 'POST',
body: JSON.stringify({
sdp: offer.sdp,
tracks: [{location: 'local', trackName: 'my-video'}]
})
});
const {sessionDescription, tracks} = await res.json();
await pc.setRemoteDescription(sessionDescription);
const publishedTrackId = tracks[0].trackName; // Share with others
```
## Subscribing
```typescript
const res = await fetch(`/api/sessions/${sessionId}/tracks`, {
method: 'POST',
body: JSON.stringify({
tracks: [{location: 'remote', trackName: remoteTrackId, sessionId: remoteSessionId}]
})
});
const {sessionDescription} = await res.json();
await pc.setRemoteDescription(sessionDescription);
const answer = await pc.createAnswer();
await pc.setLocalDescription(answer);
await fetch(`/api/sessions/${sessionId}/renegotiate`, {
method: 'PUT',
body: JSON.stringify({sdp: answer.sdp})
});
pc.ontrack = (event) => {
const [remoteStream] = event.streams;
videoElement.srcObject = remoteStream;
};
```

View File

@@ -0,0 +1,137 @@
# Configuration & Deployment
## Dashboard Setup
1. Navigate to https://dash.cloudflare.com/?to=/:account/calls
2. Click "Create Application" (or use existing app)
3. Copy `CALLS_APP_ID` from dashboard
4. Generate and copy `CALLS_APP_SECRET` (treat as sensitive credential)
5. Use credentials in Wrangler config or environment variables below
## Dependencies
**Backend (Workers):** Built-in fetch API, no additional packages required
**Client (PartyTracks):**
```bash
npm install partytracks @cloudflare/calls
```
**Client (React + PartyTracks):**
```bash
npm install partytracks @cloudflare/calls observable-hooks
# Observable hooks: useObservableAsValue, useValueAsObservable
```
**Client (Raw API):** Native browser WebRTC API only
## Wrangler Setup
```jsonc
{
"name": "my-calls-app",
"main": "src/index.ts",
"compatibility_date": "2025-01-01", // Use current date for new projects
"vars": {
"CALLS_APP_ID": "your-app-id",
"MAX_WEBCAM_BITRATE": "1200000",
"MAX_WEBCAM_FRAMERATE": "24",
"MAX_WEBCAM_QUALITY_LEVEL": "1080"
},
// Set secret: wrangler secret put CALLS_APP_SECRET
"durable_objects": {
"bindings": [
{
"name": "ROOM",
"class_name": "Room"
}
]
}
}
```
## Deploy
```bash
wrangler login
wrangler secret put CALLS_APP_SECRET
wrangler deploy
```
## Environment Variables
**Required:**
- `CALLS_APP_ID`: From dashboard
- `CALLS_APP_SECRET`: From dashboard (secret)
**Optional:**
- `MAX_WEBCAM_BITRATE` (default: 1200000)
- `MAX_WEBCAM_FRAMERATE` (default: 24)
- `MAX_WEBCAM_QUALITY_LEVEL` (default: 1080)
- `TURN_SERVICE_ID`: TURN service
- `TURN_SERVICE_TOKEN`: TURN auth (secret)
## TURN Configuration
```javascript
const pc = new RTCPeerConnection({
iceServers: [
{ urls: 'stun:stun.cloudflare.com:3478' },
{
urls: [
'turn:turn.cloudflare.com:3478?transport=udp',
'turn:turn.cloudflare.com:3478?transport=tcp',
'turns:turn.cloudflare.com:5349?transport=tcp'
],
username: turnUsername,
credential: turnCredential
}
],
bundlePolicy: 'max-bundle', // Recommended: reduces overhead
iceTransportPolicy: 'all' // Use 'relay' to force TURN (testing only)
});
```
**Ports:** 3478 (UDP/TCP), 53 (UDP), 80 (TCP), 443 (TLS), 5349 (TLS)
**When to use TURN:** Required for restrictive corporate firewalls/networks that block UDP. ~5-10% of connections fallback to TURN. STUN works for most users.
**ICE candidate filtering:** Cloudflare handles candidate filtering automatically. No need to manually filter candidates.
## Durable Object Boilerplate
Minimal presence system:
```typescript
export class Room {
private sessions = new Map<string, {userId: string, tracks: string[]}>();
async fetch(req: Request) {
const {pathname} = new URL(req.url);
const body = await req.json();
if (pathname === '/join') {
this.sessions.set(body.sessionId, {userId: body.userId, tracks: []});
return Response.json({participants: this.sessions.size});
}
if (pathname === '/publish') {
this.sessions.get(body.sessionId)?.tracks.push(...body.tracks);
// Broadcast to others via WebSocket (not shown)
return new Response('OK');
}
return new Response('Not found', {status: 404});
}
}
```
## Environment Validation
Check credentials before first API call:
```typescript
if (!env.CALLS_APP_ID || !env.CALLS_APP_SECRET) {
throw new Error('CALLS_APP_ID and CALLS_APP_SECRET required');
}
```

View File

@@ -0,0 +1,133 @@
# Gotchas & Troubleshooting
## Common Errors
### "Slow initial connect (~1.8s)"
**Cause:** First STUN delayed during consensus forming (normal behavior)
**Solution:** Subsequent connections are faster. CF detects DTLS ClientHello early to compensate.
### "No media flow"
**Cause:** SDP exchange incomplete, connection not established, tracks not added before offer, browser permissions missing
**Solution:**
1. Verify SDP exchange complete
2. Check `pc.connectionState === 'connected'`
3. Ensure tracks added before creating offer
4. Confirm browser permissions granted
5. Use `chrome://webrtc-internals` for debugging
### "Track not receiving"
**Cause:** Track not published, track ID not shared, session IDs mismatch, `pc.ontrack` not set, renegotiation needed
**Solution:**
1. Verify track published successfully
2. Confirm track ID shared between peers
3. Check session IDs match
4. Set `pc.ontrack` handler before answer
5. Trigger renegotiation if needed
### "ICE connection failed"
**Cause:** Network changed, firewall blocked UDP, TURN needed, transient network issue
**Solution:**
```typescript
pc.oniceconnectionstatechange = async () => {
if (pc.iceConnectionState === 'failed') {
console.warn('ICE failed, attempting restart');
await pc.restartIce(); // Triggers new ICE gathering
// Create new offer with ICE restart flag
const offer = await pc.createOffer({iceRestart: true});
await pc.setLocalDescription(offer);
// Send to backend → Cloudflare API
await fetch(`/api/sessions/${sessionId}/renegotiate`, {
method: 'PUT',
body: JSON.stringify({sdp: offer.sdp})
});
}
};
```
### "Track stuck/frozen"
**Cause:** Sender paused track, network congestion, codec mismatch, mobile browser backgrounded
**Solution:**
1. Check `track.enabled` and `track.readyState === 'live'`
2. Verify sender active: `pc.getSenders().find(s => s.track === track)`
3. Check stats for packet loss/jitter (see patterns.md)
4. On mobile: Re-acquire tracks when app foregrounded
5. Test with different codecs if persistent
### "Network change disconnects call"
**Cause:** Mobile switching WiFi↔cellular, laptop changing networks
**Solution:**
```typescript
// Listen for network changes
if ('connection' in navigator) {
(navigator as any).connection.addEventListener('change', async () => {
console.log('Network changed');
await pc.restartIce(); // Use ICE restart pattern above
});
}
// Or use PartyTracks (handles automatically)
```
## Retry with Exponential Backoff
```typescript
async function fetchWithRetry(url: string, options: RequestInit, maxRetries = 3) {
for (let i = 0; i < maxRetries; i++) {
try {
const res = await fetch(url, options);
if (res.ok) return res;
if (res.status >= 500) throw new Error('Server error');
return res; // Client error, don't retry
} catch (err) {
if (i === maxRetries - 1) throw err;
const delay = Math.min(1000 * 2 ** i, 10000); // Cap at 10s
await new Promise(resolve => setTimeout(resolve, delay));
}
}
}
```
## Debugging with chrome://webrtc-internals
1. Open `chrome://webrtc-internals` in Chrome/Edge
2. Find your PeerConnection in the list
3. Check **Stats graphs** for packet loss, jitter, bandwidth
4. Check **ICE candidate pairs**: Look for `succeeded` state, relay vs host candidates
5. Check **getStats**: Raw metrics for inbound/outbound RTP
6. Look for errors in **Event log**: `iceConnectionState`, `connectionState` changes
7. Export data with "Download the PeerConnection updates and stats data" button
8. Common issues visible here: ICE failures, high packet loss, bitrate drops
## Limits
| Resource/Limit | Value | Notes |
|----------------|-------|-------|
| Egress (Free) | 1TB/month | Per account |
| Egress (Paid) | $0.05/GB | After free tier |
| Inbound traffic | Free | All plans |
| TURN service | Free | Included with SFU |
| Participants | No hard limit | Client bandwidth/CPU bound (typically 10-50 tracks) |
| Tracks per session | No hard limit | Client resources limited |
| Session duration | No hard limit | Production calls run for hours |
| WebRTC ports | UDP 1024-65535 | Outbound only, required for media |
| API rate limit | 600 req/min | Per app, burst allowed |
## Security Checklist
-**Never expose** `CALLS_APP_SECRET` to client
-**Validate user identity** in backend before creating sessions
-**Implement auth tokens** for session access (JWT in custom header)
-**Rate limit** session creation endpoints
-**Expire sessions** server-side after inactivity
-**Validate track IDs** before subscribing (prevent unauthorized access)
-**Use HTTPS** for all signaling (API calls)
-**Enable DTLS-SRTP** (automatic with Cloudflare, encrypts media)
- ⚠️ **Consider E2EE** for sensitive content (implement client-side with Insertable Streams API)

View File

@@ -0,0 +1,174 @@
# Patterns & Use Cases
## Architecture
```
Client (WebRTC) <---> CF Edge <---> Backend (HTTP)
|
CF Backbone (310+ DCs)
|
Other Edges <---> Other Clients
```
Anycast: Last-mile <50ms (95%), no region select, NACK shield, distributed consensus
Cascading trees auto-scale to millions:
```
Publisher -> Edge A -> Edge B -> Sub1
\-> Edge C -> Sub2,3
```
## Use Cases
**1:1:** A creates session+publishes, B creates+subscribes to A+publishes, A subscribes to B
**N:N:** All create session+publish, backend broadcasts track IDs, all subscribe to others
**1:N:** Publisher creates+publishes, viewers each create+subscribe (no fan-out limit)
**Breakout:** Same PeerConnection! Backend closes/adds tracks, no recreation
## PartyTracks (Recommended)
Observable-based client with automatic device/network handling:
```typescript
import {PartyTracks} from 'partytracks';
// Create client
const pt = new PartyTracks({
apiUrl: '/api/calls',
sessionId: 'my-session',
onTrack: (track, peer) => {
const video = document.getElementById(`video-${peer.id}`) as HTMLVideoElement;
video.srcObject = new MediaStream([track]);
}
});
// Publish camera (push API)
const camera = await pt.getCamera(); // Auto-requests permissions, handles device changes
await pt.publishTrack(camera, {trackName: 'my-camera'});
// Subscribe to remote track (pull API)
await pt.subscribeToTrack({trackName: 'remote-camera', sessionId: 'other-session'});
// React hook example
import {useObservableAsValue} from 'observable-hooks';
function VideoCall() {
const localTracks = useObservableAsValue(pt.localTracks$);
const remoteTracks = useObservableAsValue(pt.remoteTracks$);
return <div>{/* Render tracks */}</div>;
}
// Screenshare
const screen = await pt.getScreenshare();
await pt.publishTrack(screen, {trackName: 'my-screen'});
// Handle device changes (automatic)
// PartyTracks detects device changes (e.g., Bluetooth headset) and renegotiates
```
## Backend
Express:
```js
app.post('/api/new-session', async (req, res) => {
const r = await fetch(`${CALLS_API}/apps/${process.env.CALLS_APP_ID}/sessions/new`,
{method: 'POST', headers: {'Authorization': `Bearer ${process.env.CALLS_APP_SECRET}`}});
res.json(await r.json());
});
```
Workers: Same pattern, use `env.CALLS_APP_ID` and `env.CALLS_APP_SECRET`
DO Presence: See configuration.md for boilerplate
## Audio Level Detection
```typescript
// Attach analyzer to audio track
function attachAudioLevelDetector(track: MediaStreamTrack) {
const ctx = new AudioContext();
const analyzer = ctx.createAnalyser();
const src = ctx.createMediaStreamSource(new MediaStream([track]));
src.connect(analyzer);
const data = new Uint8Array(analyzer.frequencyBinCount);
const checkLevel = () => {
analyzer.getByteFrequencyData(data);
const level = data.reduce((a, b) => a + b) / data.length;
if (level > 30) console.log('Speaking:', level); // Trigger UI update
requestAnimationFrame(checkLevel);
};
checkLevel();
}
```
## Connection Quality Monitoring
```typescript
pc.getStats().then(stats => {
stats.forEach(report => {
if (report.type === 'inbound-rtp' && report.kind === 'video') {
const {packetsLost, packetsReceived, jitter} = report;
const lossRate = packetsLost / (packetsLost + packetsReceived);
if (lossRate > 0.05) console.warn('High packet loss:', lossRate);
if (jitter > 100) console.warn('High jitter:', jitter);
}
});
});
```
## Stage Management (Limit Visible Participants)
```typescript
// Subscribe to top 6 active speakers only
let activeSubscriptions = new Set<string>();
function updateStage(topSpeakers: string[]) {
const toAdd = topSpeakers.filter(id => !activeSubscriptions.has(id)).slice(0, 6);
const toRemove = [...activeSubscriptions].filter(id => !topSpeakers.includes(id));
toRemove.forEach(id => {
pc.getSenders().find(s => s.track?.id === id)?.track?.stop();
activeSubscriptions.delete(id);
});
toAdd.forEach(async id => {
await fetch(`/api/subscribe`, {method: 'POST', body: JSON.stringify({trackId: id})});
activeSubscriptions.add(id);
});
}
```
## Advanced
Bandwidth mgmt:
```ts
const s = pc.getSenders().find(s => s.track?.kind === 'video');
const p = s.getParameters();
if (!p.encodings) p.encodings = [{}];
p.encodings[0].maxBitrate = 1200000; p.encodings[0].maxFramerate = 24;
await s.setParameters(p);
```
Simulcast (CF auto-forwards best layer):
```ts
pc.addTransceiver('video', {direction: 'sendonly', sendEncodings: [
{rid: 'high', maxBitrate: 1200000},
{rid: 'med', maxBitrate: 600000, scaleResolutionDownBy: 2},
{rid: 'low', maxBitrate: 200000, scaleResolutionDownBy: 4}
]});
```
DataChannel:
```ts
const dc = pc.createDataChannel('chat', {ordered: true, maxRetransmits: 3});
dc.onopen = () => dc.send(JSON.stringify({type: 'chat', text: 'Hi'}));
dc.onmessage = (e) => console.log('RX:', JSON.parse(e.data));
```
**WHIP/WHEP:** For streaming interop (OBS → SFU, SFU → video players), use WHIP (ingest) and WHEP (egress) protocols. See Cloudflare Stream integration docs.
Integrations: R2 for recording `env.R2_BUCKET.put(...)`, Queues for analytics
Perf: 100-250ms connect, ~50ms latency (95%), 200-400ms glass-to-glass, no participant limit (client: 10-50 tracks)