Pipelines Gotchas

Critical Issues

Events Silently Dropped

Most common issue. Events accepted (HTTP 200) but never appear in sink.

Causes:

Schema validation fails - structured streams drop invalid events silently
Waiting for roll interval (10-300s) - expected behavior

Solution: Validate client-side with Zod:

const EventSchema = z.object({ user_id: z.string(), amount: z.number() });
try {
  const validated = EventSchema.parse(rawEvent);
  await env.STREAM.send([validated]);
} catch (e) { /* get immediate feedback */ }

Pipelines Are Immutable

Cannot modify SQL after creation. Must delete and recreate.

npx wrangler pipelines delete old-pipeline
npx wrangler pipelines create new-pipeline --sql "..."

Tip: Use version naming (events-pipeline-v1) and keep SQL in version control.

Worker Binding Not Found

env.STREAM is undefined

Use stream ID (not pipeline ID) in wrangler.jsonc
Redeploy after adding binding

npx wrangler pipelines streams list  # Get stream ID
npx wrangler deploy

Common Errors

Error	Cause	Fix
Events not in R2	Roll interval not elapsed	Wait 10-300s, check `roll_interval`
Schema validation failures	Type mismatch, missing fields	Validate client-side
Rate limit (429)	>5 MB/s per stream	Batch events, request increase
Payload too large (413)	>1 MB request	Split into smaller batches
Cannot delete stream	Pipeline references it	Delete pipelines first
Sink credential errors	Token expired	Recreate sink with new credentials

Limits (Open Beta)

Resource	Limit
Streams/Sinks/Pipelines per account	20 each
Payload size	1 MB
Ingest rate per stream	5 MB/s
Event retention	24 hours
Recommended batch size	100 events

SQL Limitations

No JOINs - single stream per pipeline
No window functions - basic SQL only
No subqueries - must use INSERT INTO ... SELECT ... FROM
No schema evolution - cannot modify after creation

Debug Checklist

Stream exists: npx wrangler pipelines streams list
Pipeline healthy: npx wrangler pipelines get <ID>
SQL syntax matches schema
Worker redeployed after binding added
Waited for roll interval
Accepted vs processed count matches (no validation drops)

2.4 KiB Raw Blame History