mirror of
https://github.com/ksyasuda/dotfiles.git
synced 2026-03-20 06:11:27 -07:00
81 lines
2.4 KiB
Markdown
81 lines
2.4 KiB
Markdown
# Pipelines Gotchas
|
|
|
|
## Critical Issues
|
|
|
|
### Events Silently Dropped
|
|
|
|
**Most common issue.** Events accepted (HTTP 200) but never appear in sink.
|
|
|
|
**Causes:**
|
|
1. Schema validation fails - structured streams drop invalid events silently
|
|
2. Waiting for roll interval (10-300s) - expected behavior
|
|
|
|
**Solution:** Validate client-side with Zod:
|
|
```typescript
|
|
const EventSchema = z.object({ user_id: z.string(), amount: z.number() });
|
|
try {
|
|
const validated = EventSchema.parse(rawEvent);
|
|
await env.STREAM.send([validated]);
|
|
} catch (e) { /* get immediate feedback */ }
|
|
```
|
|
|
|
### Pipelines Are Immutable
|
|
|
|
Cannot modify SQL after creation. Must delete and recreate.
|
|
|
|
```bash
|
|
npx wrangler pipelines delete old-pipeline
|
|
npx wrangler pipelines create new-pipeline --sql "..."
|
|
```
|
|
|
|
**Tip:** Use version naming (`events-pipeline-v1`) and keep SQL in version control.
|
|
|
|
### Worker Binding Not Found
|
|
|
|
**`env.STREAM is undefined`**
|
|
|
|
1. Use **stream ID** (not pipeline ID) in `wrangler.jsonc`
|
|
2. Redeploy after adding binding
|
|
|
|
```bash
|
|
npx wrangler pipelines streams list # Get stream ID
|
|
npx wrangler deploy
|
|
```
|
|
|
|
## Common Errors
|
|
|
|
| Error | Cause | Fix |
|
|
|-------|-------|-----|
|
|
| Events not in R2 | Roll interval not elapsed | Wait 10-300s, check `roll_interval` |
|
|
| Schema validation failures | Type mismatch, missing fields | Validate client-side |
|
|
| Rate limit (429) | >5 MB/s per stream | Batch events, request increase |
|
|
| Payload too large (413) | >1 MB request | Split into smaller batches |
|
|
| Cannot delete stream | Pipeline references it | Delete pipelines first |
|
|
| Sink credential errors | Token expired | Recreate sink with new credentials |
|
|
|
|
## Limits (Open Beta)
|
|
|
|
| Resource | Limit |
|
|
|----------|-------|
|
|
| Streams/Sinks/Pipelines per account | 20 each |
|
|
| Payload size | 1 MB |
|
|
| Ingest rate per stream | 5 MB/s |
|
|
| Event retention | 24 hours |
|
|
| Recommended batch size | 100 events |
|
|
|
|
## SQL Limitations
|
|
|
|
- **No JOINs** - single stream per pipeline
|
|
- **No window functions** - basic SQL only
|
|
- **No subqueries** - must use `INSERT INTO ... SELECT ... FROM`
|
|
- **No schema evolution** - cannot modify after creation
|
|
|
|
## Debug Checklist
|
|
|
|
- [ ] Stream exists: `npx wrangler pipelines streams list`
|
|
- [ ] Pipeline healthy: `npx wrangler pipelines get <ID>`
|
|
- [ ] SQL syntax matches schema
|
|
- [ ] Worker redeployed after binding added
|
|
- [ ] Waited for roll interval
|
|
- [ ] Accepted vs processed count matches (no validation drops)
|