update skills

2026-03-21 18:11:27 -07:00 · 2026-03-17 16:53:22 -07:00
parent 0b0783ef8e
commit f9a530667e
389 changed files with 54512 additions and 1 deletions
--- a/.agents/skills/cloudflare-deploy/references/tail-workers/README.md
+++ b/.agents/skills/cloudflare-deploy/references/tail-workers/README.md
@@ -0,0 +1,89 @@
+# Cloudflare Tail Workers
+
+Specialized Workers that consume execution events from producer Workers for logging, debugging, analytics, and observability.
+
+## When to Use This Reference
+
+- Implementing observability/logging for Cloudflare Workers
+- Processing Worker execution events, logs, exceptions
+- Building custom analytics or error tracking
+- Configuring real-time event streaming
+- Working with tail handlers or tail consumers
+
+## Core Concepts
+
+### What Are Tail Workers?
+
+Tail Workers automatically process events from producer Workers (the Workers being monitored). They receive:
+- HTTP request/response info
+- Console logs (`console.log/error/warn/debug`)
+- Uncaught exceptions
+- Execution outcomes (`ok`, `exception`, `exceededCpu`, etc.)
+- Diagnostic channel events
+
+**Key characteristics:**
+- Invoked AFTER producer finishes executing
+- Capture entire request lifecycle including Service Bindings and Dynamic Dispatch sub-requests
+- Billed by CPU time, not request count
+- Available on Workers Paid and Enterprise tiers
+
+### Alternative: OpenTelemetry Export
+
+**Before using Tail Workers, consider OpenTelemetry:**
+
+For batch exports to observability tools (Sentry, Grafana, Honeycomb):
+- OTEL export sends logs/traces in batches (more efficient)
+- Built-in integrations with popular platforms
+- Lower overhead than Tail Workers
+- **Use Tail Workers only for custom real-time processing**
+
+## Decision Tree
+
+```
+Need observability for Workers?
+├─ Batch export to known tools (Sentry/Grafana/Honeycomb)?
+│  └─ Use OpenTelemetry export (not Tail Workers)
+├─ Custom real-time processing needed?
+│  ├─ Aggregated metrics?
+│  │  └─ Use Tail Worker + Analytics Engine
+│  ├─ Error tracking?
+│  │  └─ Use Tail Worker + external service
+│  ├─ Custom logging/debugging?
+│  │  └─ Use Tail Worker + KV/HTTP endpoint
+│  └─ Complex event processing?
+│     └─ Use Tail Worker + Durable Objects
+└─ Quick debugging?
+   └─ Use `wrangler tail` (different from Tail Workers)
+```
+
+## Reading Order
+
+1. **[configuration.md](configuration.md)** - Set up Tail Workers
+2. **[api.md](api.md)** - Handler signature, types, redaction
+3. **[patterns.md](patterns.md)** - Common use cases and integrations
+4. **[gotchas.md](gotchas.md)** - Pitfalls and debugging tips
+
+## Quick Example
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    // Process events from producer Worker
+    ctx.waitUntil(
+      fetch(env.LOG_ENDPOINT, {
+        method: "POST",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify(events),
+      })
+    );
+  }
+};
+```
+
+## Related Skills
+
+- **observability** - General Workers observability patterns, OTEL export
+- **analytics-engine** - Aggregated metrics storage for tail event data
+- **durable-objects** - Stateful event processing, batching tail events
+- **logpush** - Alternative for batch log export (non-real-time)
+- **workers-for-platforms** - Dynamic dispatch with tail consumers
--- a/.agents/skills/cloudflare-deploy/references/tail-workers/api.md
+++ b/.agents/skills/cloudflare-deploy/references/tail-workers/api.md
@@ -0,0 +1,200 @@
+# Tail Workers API Reference
+
+## Handler Signature
+
+```typescript
+export default {
+  async tail(
+    events: TraceItem[],
+    env: Env,
+    ctx: ExecutionContext
+  ): Promise<void> {
+    // Process events
+  }
+} satisfies ExportedHandler<Env>;
+```
+
+**Parameters:**
+- `events`: Array of `TraceItem` objects (one per producer invocation)
+- `env`: Bindings (KV, D1, R2, env vars, etc.)
+- `ctx`: Context with `waitUntil()` for async work
+
+**CRITICAL:** Tail handlers don't return values. Use `ctx.waitUntil()` for async operations.
+
+## TraceItem Type
+
+```typescript
+interface TraceItem {
+  scriptName: string;           // Producer Worker name
+  eventTimestamp: number;        // Epoch milliseconds
+  outcome: 'ok' | 'exception' | 'exceededCpu' | 'exceededMemory' 
+         | 'canceled' | 'scriptNotFound' | 'responseStreamDisconnected' | 'unknown';
+  
+  event?: {
+    request?: {
+      url: string;               // Redacted by default
+      method: string;
+      headers: Record<string, string>;  // Sensitive headers redacted
+      cf?: IncomingRequestCfProperties;
+      getUnredacted(): TraceRequest;    // Bypass redaction (use carefully)
+    };
+    response?: {
+      status: number;
+    };
+  };
+  
+  logs: Array<{
+    timestamp: number;           // Epoch milliseconds
+    level: 'debug' | 'info' | 'log' | 'warn' | 'error';
+    message: unknown[];          // Args passed to console function
+  }>;
+  
+  exceptions: Array<{
+    timestamp: number;           // Epoch milliseconds
+    name: string;                // Error type (Error, TypeError, etc.)
+    message: string;             // Error description
+  }>;
+  
+  diagnosticsChannelEvents: Array<{
+    channel: string;
+    message: unknown;
+    timestamp: number;           // Epoch milliseconds
+  }>;
+}
+```
+
+**Note:** Official SDK uses `TraceItem`, not `TailItem`. Use `@cloudflare/workers-types` for accurate types.
+
+## Timestamp Handling
+
+All timestamps are **epoch milliseconds**, not seconds:
+
+```typescript
+// ✅ CORRECT - use directly with Date
+const date = new Date(event.eventTimestamp);
+
+// ❌ WRONG - don't multiply by 1000
+const date = new Date(event.eventTimestamp * 1000);
+```
+
+## Automatic Redaction
+
+By default, sensitive data is redacted from `TraceRequest`:
+
+### Header Redaction
+
+Headers containing these substrings (case-insensitive):
+- `auth`, `key`, `secret`, `token`, `jwt`
+- `cookie`, `set-cookie`
+
+Redacted values show as `"REDACTED"`.
+
+### URL Redaction
+
+- **Hex IDs:** 32+ hex digits → `"REDACTED"`
+- **Base-64 IDs:** 21+ chars with 2+ upper, 2+ lower, 2+ digits → `"REDACTED"`
+
+## Bypassing Redaction
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    for (const event of events) {
+      // ⚠️ Use with extreme caution
+      const unredacted = event.event?.request?.getUnredacted();
+      // unredacted.url and unredacted.headers contain raw values
+    }
+  }
+};
+```
+
+**Best practices:**
+- Only call `getUnredacted()` when absolutely necessary
+- Never log unredacted sensitive data
+- Implement additional filtering before external transmission
+- Use environment variables for API keys, never hardcode
+
+## Type-Safe Handler
+
+```typescript
+interface Env {
+  LOGS_KV: KVNamespace;
+  ANALYTICS: AnalyticsEngineDataset;
+  LOG_ENDPOINT: string;
+  API_TOKEN: string;
+}
+
+export default {
+  async tail(
+    events: TraceItem[],
+    env: Env,
+    ctx: ExecutionContext
+  ): Promise<void> {
+    const payload = events.map(event => ({
+      script: event.scriptName,
+      timestamp: event.eventTimestamp,
+      outcome: event.outcome,
+      url: event.event?.request?.url,
+      status: event.event?.response?.status,
+    }));
+    
+    ctx.waitUntil(
+      fetch(env.LOG_ENDPOINT, {
+        method: "POST",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify(payload),
+      })
+    );
+  }
+} satisfies ExportedHandler<Env>;
+```
+
+## Outcome vs HTTP Status
+
+**IMPORTANT:** `outcome` is script execution status, NOT HTTP status.
+
+- Worker returns 500 → `outcome='ok'` if script completed successfully
+- Uncaught exception → `outcome='exception'` regardless of HTTP status
+- CPU limit exceeded → `outcome='exceededCpu'`
+
+```typescript
+// ✅ Check outcome for script execution status
+if (event.outcome === 'exception') {
+  // Script threw uncaught exception
+}
+
+// ✅ Check HTTP status separately
+if (event.event?.response?.status === 500) {
+  // HTTP 500 returned (script may have handled error)
+}
+```
+
+## Serialization Considerations
+
+`log.message` is `unknown[]` and may contain non-serializable objects:
+
+```typescript
+// ❌ May fail with circular references or BigInt
+JSON.stringify(events);
+
+// ✅ Safe serialization
+const safePayload = events.map(event => ({
+  ...event,
+  logs: event.logs.map(log => ({
+    ...log,
+    message: log.message.map(m => {
+      try {
+        return JSON.parse(JSON.stringify(m));
+      } catch {
+        return String(m);
+      }
+    })
+  }))
+}));
+```
+
+**Common serialization issues:**
+- Circular references in logged objects
+- `BigInt` values (not JSON-serializable)
+- Functions or symbols in console.log arguments
+- Large objects exceeding body size limits
--- a/.agents/skills/cloudflare-deploy/references/tail-workers/configuration.md
+++ b/.agents/skills/cloudflare-deploy/references/tail-workers/configuration.md
@@ -0,0 +1,176 @@
+# Tail Workers Configuration
+
+## Setup Steps
+
+### 1. Create Tail Worker
+
+Create a Worker with a `tail()` handler:
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    // Process events from producer Worker
+    ctx.waitUntil(
+      fetch(env.LOG_ENDPOINT, {
+        method: "POST",
+        body: JSON.stringify(events),
+      })
+    );
+  }
+};
+```
+
+### 2. Configure Producer Worker
+
+In producer's `wrangler.jsonc`:
+
+```jsonc
+{
+  "name": "my-producer-worker",
+  "tail_consumers": [
+    {
+      "service": "my-tail-worker"
+    }
+  ]
+}
+```
+
+### 3. Deploy Both Workers
+
+```bash
+# Deploy Tail Worker first
+cd tail-worker
+wrangler deploy
+
+# Then deploy producer Worker
+cd ../producer-worker
+wrangler deploy
+```
+
+## Wrangler Configuration
+
+### Single Tail Consumer
+
+```jsonc
+{
+  "name": "producer-worker",
+  "tail_consumers": [
+    {
+      "service": "logging-tail-worker"
+    }
+  ]
+}
+```
+
+### Multiple Tail Consumers
+
+```jsonc
+{
+  "name": "producer-worker",
+  "tail_consumers": [
+    {
+      "service": "logging-tail-worker"
+    },
+    {
+      "service": "metrics-tail-worker"
+    }
+  ]
+}
+```
+
+**Note:** Each consumer receives ALL events independently.
+
+### Remove Tail Consumer
+
+```jsonc
+{
+  "tail_consumers": []
+}
+```
+
+Then redeploy producer Worker.
+
+## Environment Variables
+
+Tail Workers use same binding syntax as regular Workers:
+
+```jsonc
+{
+  "name": "my-tail-worker",
+  "vars": {
+    "LOG_ENDPOINT": "https://logs.example.com/ingest"
+  },
+  "kv_namespaces": [
+    {
+      "binding": "LOGS_KV",
+      "id": "abc123..."
+    }
+  ]
+}
+```
+
+## Testing & Development
+
+### Local Testing
+
+**Tail Workers cannot be fully tested with `wrangler dev`.** Deploy to staging environment for testing.
+
+### Testing Strategy
+
+1. Deploy producer Worker to staging
+2. Deploy Tail Worker to staging
+3. Configure `tail_consumers` in producer
+4. Trigger producer Worker requests
+5. Verify Tail Worker receives events (check destination logs/storage)
+
+### Wrangler Tail Command
+
+```bash
+# Stream logs to terminal (NOT Tail Workers)
+wrangler tail my-producer-worker
+```
+
+**This is different from Tail Workers:**
+- `wrangler tail` streams logs to your terminal
+- Tail Workers are Workers that process events programmatically
+
+## Deployment Checklist
+
+- [ ] Tail Worker has `tail()` handler
+- [ ] Tail Worker deployed before producer
+- [ ] Producer's `wrangler.jsonc` has correct `tail_consumers`
+- [ ] Environment variables configured
+- [ ] Tested with staging environment
+- [ ] Monitoring configured for Tail Worker itself
+
+## Limits
+
+| Limit | Value | Notes |
+|-------|-------|-------|
+| Max tail consumers per producer | 10 | Each receives all events independently |
+| Events batch size | Up to 100 events per invocation | Larger batches split across invocations |
+| Tail Worker CPU time | Same as regular Workers | 10ms (free), 30ms (paid), 50ms (paid bundle) |
+| Pricing tier | Workers Paid or Enterprise | Not available on free plan |
+| Request body size | 100 MB max | When sending to external endpoints |
+| Event retention | None | Events not retried if tail handler fails |
+
+## Workers for Platforms
+
+For dynamic dispatch Workers, both dispatch and user Worker events sent to tail consumer:
+
+```jsonc
+{
+  "name": "dispatch-worker",
+  "tail_consumers": [
+    {
+      "service": "platform-tail-worker"
+    }
+  ]
+}
+```
+
+Tail Worker receives TWO `TraceItem` elements per request:
+1. Dynamic dispatch Worker event
+2. User Worker event
+
+See [patterns.md](patterns.md) for handling.
--- a/.agents/skills/cloudflare-deploy/references/tail-workers/gotchas.md
+++ b/.agents/skills/cloudflare-deploy/references/tail-workers/gotchas.md
@@ -0,0 +1,192 @@
+# Tail Workers Gotchas & Debugging
+
+## Critical Pitfalls
+
+### 1. Not Using `ctx.waitUntil()`
+
+**Problem:** Async work doesn't complete or tail Worker times out  
+**Cause:** Handlers exit immediately; awaiting blocks processing  
+**Solution:**
+
+```typescript
+// ❌ WRONG - fire and forget
+export default {
+  async tail(events) {
+    fetch(endpoint, { body: JSON.stringify(events) });
+  }
+};
+
+// ❌ WRONG - blocking await
+export default {
+  async tail(events, env, ctx) {
+    await fetch(endpoint, { body: JSON.stringify(events) });
+  }
+};
+
+// ✅ CORRECT
+export default {
+  async tail(events, env, ctx) {
+    ctx.waitUntil(
+      (async () => {
+        await fetch(endpoint, { body: JSON.stringify(events) });
+        await processMore();
+      })()
+    );
+  }
+};
+```
+
+### 2. Missing `tail()` Handler
+
+**Problem:** Producer deployment fails  
+**Cause:** Worker in `tail_consumers` doesn't export `tail()` handler  
+**Solution:** Ensure `export default { async tail(events, env, ctx) { ... } }`
+
+### 3. Outcome vs HTTP Status
+
+**Problem:** Filtering by wrong status  
+**Cause:** `outcome` is script execution status, not HTTP status
+
+```typescript
+// ❌ WRONG
+if (event.outcome === 500) { /* never matches */ }
+
+// ✅ CORRECT
+if (event.outcome === 'exception') { /* script threw */ }
+if (event.event?.response?.status === 500) { /* HTTP 500 */ }
+```
+
+### 4. Timestamp Units
+
+**Problem:** Dates off by 1000x  
+**Cause:** Timestamps are epoch milliseconds, not seconds
+
+```typescript
+// ❌ WRONG: const date = new Date(event.eventTimestamp * 1000);
+// ✅ CORRECT: const date = new Date(event.eventTimestamp);
+```
+
+### 5. Type Name Mismatch
+
+**Problem:** Using `TailItem` type  
+**Cause:** Old docs used `TailItem`, SDK uses `TraceItem`
+
+```typescript
+import type { TraceItem } from '@cloudflare/workers-types';
+export default {
+  async tail(events: TraceItem[], env, ctx) { /* ... */ }
+};
+```
+
+### 6. Excessive Logging Volume
+
+**Problem:** Unexpected high costs  
+**Cause:** Invoked on EVERY producer request  
+**Solution:** Sample events
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    if (Math.random() > 0.1) return;  // 10% sample
+    ctx.waitUntil(sendToEndpoint(events));
+  }
+};
+```
+
+### 7. Serialization Issues
+
+**Problem:** `JSON.stringify()` fails  
+**Cause:** `log.message` is `unknown[]` with non-serializable values  
+**Solution:**
+
+```typescript
+const safePayload = events.map(e => ({
+  ...e,
+  logs: e.logs.map(log => ({
+    ...log,
+    message: log.message.map(m => {
+      try { return JSON.parse(JSON.stringify(m)); }
+      catch { return String(m); }
+    })
+  }))
+}));
+```
+
+### 8. Missing Error Handling
+
+**Problem:** Tail Worker silently fails  
+**Cause:** No try/catch  
+**Solution:**
+
+```typescript
+ctx.waitUntil((async () => {
+  try {
+    await fetch(env.ENDPOINT, { body: JSON.stringify(events) });
+  } catch (error) {
+    console.error("Tail error:", error);
+    await env.FALLBACK_KV.put(`failed:${Date.now()}`, JSON.stringify(events));
+  }
+})());
+```
+
+### 9. Deployment Order
+
+**Problem:** Producer deployment fails  
+**Cause:** Tail consumer not deployed yet  
+**Solution:** Deploy tail consumer FIRST
+
+```bash
+cd tail-worker && wrangler deploy
+cd ../producer && wrangler deploy
+```
+
+### 10. No Event Retry
+
+**Problem:** Events lost when handler fails  
+**Cause:** Failed invocations NOT retried  
+**Solution:** Implement fallback storage (see #8)
+
+## Debugging
+
+**View logs:** `wrangler tail my-tail-worker`
+
+**Incremental testing:**
+1. Verify receipt: `console.log('Events:', events.length)`
+2. Inspect structure: `console.log(JSON.stringify(events[0], null, 2))`
+3. Add external call with `ctx.waitUntil()`
+
+**Monitor dashboard:** Check invocation count (matches producer?), error rate, CPU time
+
+## Testing
+
+Add test endpoint to producer:
+
+```typescript
+export default {
+  async fetch(request) {
+    if (request.url.includes('/test')) {
+      console.log('Test log');
+      throw new Error('Test error');
+    }
+    return new Response('OK');
+  }
+};
+```
+
+Trigger: `curl https://producer.example.workers.dev/test`
+
+## Common Errors
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| "Tail consumer not found" | Not deployed | Deploy tail Worker first |
+| "No tail handler" | Missing `tail()` | Add to default export |
+| "waitUntil is not a function" | Missing `ctx` | Add `ctx` parameter |
+| Timeout | Blocking await | Use `ctx.waitUntil()` |
+
+## Performance Notes
+
+- Max 100 events per invocation
+- Each consumer receives all events independently
+- CPU limits same as regular Workers
+- For high volume, use Durable Objects batching
--- a/.agents/skills/cloudflare-deploy/references/tail-workers/patterns.md
+++ b/.agents/skills/cloudflare-deploy/references/tail-workers/patterns.md
@@ -0,0 +1,180 @@
+# Tail Workers Common Patterns
+
+## Community Libraries
+
+While most tail Worker implementations are custom, these libraries may help:
+
+**Logging/Observability:**
+- **Axiom** - `axiom-cloudflare-workers` (npm) - Direct Axiom integration
+- **Baselime** - SDK for Baselime observability platform
+- **LogFlare** - Structured log aggregation
+
+**Type Definitions:**
+- **@cloudflare/workers-types** - Official TypeScript types (use `TraceItem`)
+
+**Note:** Most integrations require custom tail handler implementation. See integration examples below.
+
+## Basic Patterns
+
+### HTTP Endpoint Logging
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    const payload = events.map(event => ({
+      script: event.scriptName,
+      timestamp: event.eventTimestamp,
+      outcome: event.outcome,
+      url: event.event?.request?.url,
+      status: event.event?.response?.status,
+      logs: event.logs,
+      exceptions: event.exceptions,
+    }));
+    
+    ctx.waitUntil(
+      fetch(env.LOG_ENDPOINT, {
+        method: "POST",
+        body: JSON.stringify(payload),
+      })
+    );
+  }
+};
+```
+
+### Error Tracking Only
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    const errors = events.filter(e => 
+      e.outcome === 'exception' || e.exceptions.length > 0
+    );
+    
+    if (errors.length === 0) return;
+    
+    ctx.waitUntil(
+      fetch(env.ERROR_ENDPOINT, {
+        method: "POST",
+        body: JSON.stringify(errors),
+      })
+    );
+  }
+};
+```
+
+## Storage Integration
+
+### KV Storage with TTL
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    ctx.waitUntil(
+      Promise.all(events.map(event =>
+        env.LOGS_KV.put(
+          `log:${event.scriptName}:${event.eventTimestamp}`,
+          JSON.stringify(event),
+          { expirationTtl: 86400 }  // 24 hours
+        )
+      ))
+    );
+  }
+};
+```
+
+### Analytics Engine Metrics
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    ctx.waitUntil(
+      Promise.all(events.map(event =>
+        env.ANALYTICS.writeDataPoint({
+          blobs: [event.scriptName, event.outcome],
+          doubles: [1, event.event?.response?.status ?? 0],
+          indexes: [event.event?.request?.cf?.colo ?? 'unknown'],
+        })
+      ))
+    );
+  }
+};
+```
+
+## Filtering & Routing
+
+Filter by route, outcome, or other criteria:
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    // Route filtering
+    const apiEvents = events.filter(e => 
+      e.event?.request?.url?.includes('/api/')
+    );
+    
+    // Multi-destination routing
+    const errors = events.filter(e => e.outcome === 'exception');
+    const success = events.filter(e => e.outcome === 'ok');
+    
+    const tasks = [];
+    if (errors.length > 0) {
+      tasks.push(fetch(env.ERROR_ENDPOINT, {
+        method: "POST",
+        body: JSON.stringify(errors),
+      }));
+    }
+    if (success.length > 0) {
+      tasks.push(fetch(env.SUCCESS_ENDPOINT, {
+        method: "POST",
+        body: JSON.stringify(success),
+      }));
+    }
+    
+    ctx.waitUntil(Promise.all(tasks));
+  }
+};
+```
+
+## Sampling
+
+Reduce costs by processing only a percentage of events:
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    if (Math.random() > 0.1) return;  // 10% sample rate
+    ctx.waitUntil(fetch(env.LOG_ENDPOINT, {
+      method: "POST",
+      body: JSON.stringify(events),
+    }));
+  }
+};
+```
+
+## Advanced Patterns
+
+### Batching with Durable Objects
+
+Accumulate events before sending:
+
+```typescript
+export default {
+  async tail(events, env, ctx) {
+    const batch = env.BATCH_DO.get(env.BATCH_DO.idFromName("batch"));
+    ctx.waitUntil(batch.fetch("https://batch/add", {
+      method: "POST",
+      body: JSON.stringify(events),
+    }));
+  }
+};
+```
+
+See durable-objects skill for full implementation.
+
+### Workers for Platforms
+
+Dynamic dispatch sends TWO events per request. Filter by `scriptName` to distinguish dispatch vs user Worker events.
+
+### Error Handling
+
+Always wrap external calls. See gotchas.md for fallback storage pattern.