dotfiles/.agents/skills/cloudflare-deploy/references/r2-data-catalog/gotchas.md

Gotchas & Troubleshooting

Common problems → causes → solutions.

Permission Errors

401 Unauthorized

Error: "401 Unauthorized"
Cause: Token missing R2 Data Catalog permissions.
Solution: Use "Admin Read & Write" token (includes catalog + storage permissions). Test with catalog.list_namespaces().

403 Forbidden

Error: "403 Forbidden" on data files
Cause: Token lacks storage permissions.
Solution: Token needs both R2 Data Catalog + R2 Storage Bucket Item permissions.

Token Rotation Issues

Error: New token fails after rotation.
Solution: Create new token → test in staging → update prod → monitor 24h → revoke old.

Catalog URI Issues

404 Not Found

Error: "404 Catalog not found"
Cause: Catalog not enabled or wrong URI.
Solution: Run wrangler r2 bucket catalog enable <bucket>. URI must be HTTPS with /iceberg/ and case-sensitive bucket name.

Wrong Warehouse

Error: Cannot create/load tables.
Cause: Warehouse ≠ bucket name.
Solution: Set warehouse="bucket-name" to match bucket exactly.

Table and Schema Issues

Table/Namespace Already Exists

Error: "TableAlreadyExistsError"
Solution: Use try/except to load existing or check first.

Namespace Not Found

Error: Cannot create table.
Solution: Create namespace first: catalog.create_namespace("ns")

Schema Evolution Errors

Error: "422 Validation" on schema update.
Cause: Incompatible change (required field, type shrink).
Solution: Only add nullable columns, compatible type widening (int→long, float→double).

Data and Query Issues

Empty Scan Results

Error: Scan returns no data.
Cause: Incorrect filter or partition column.
Solution: Test without filter first: table.scan().to_pandas(). Verify partition column names.

Slow Queries

Error: Performance degrades over time.
Cause: Too many small files.
Solution: Check file count, compact if >1000 or avg <10MB. See api.md.

Type Mismatch

Error: "Cannot cast" on append.
Cause: PyArrow types don't match Iceberg schema.
Solution: Cast to int64 (Iceberg default), not int32. Check table.schema().

Compaction Issues

Compaction Issues

Problem: File count unchanged or compaction takes hours.
Cause: Target size too large, or table too big for PyIceberg.
Solution: Only compact if avg <50MB. For >1TB tables, use Spark. Run during low-traffic periods.

Maintenance Issues

Snapshot/Orphan Issues

Problem: Expiration fails or orphan cleanup deletes active data.
Cause: Too aggressive retention or wrong order.
Solution: Always expire snapshots first with retain_last=10, then cleanup orphans with 3+ day threshold.

Concurrency Issues

Concurrent Write Conflicts

Problem: CommitFailedException with multiple writers.
Cause: Optimistic locking - simultaneous commits.
Solution: Add retry with exponential backoff (see patterns.md).

Stale Metadata

Problem: Old schema/data after external update.
Cause: Cached metadata.
Solution: Reload table: table = catalog.load_table(("ns", "table"))

Performance Optimization

Performance Tips

Scans: Use row_filter and selected_fields to reduce data scanned.
Partitions: 100-1000 optimal. Avoid high cardinality (millions) or low (<10).
Files: Keep 100-500MB avg. Compact if <10MB or >10k files.

Limits

Resource	Recommended	Impact if Exceeded
Tables/namespace	<10k	Slow list ops
Files/table	<100k	Slow query planning
Partitions/table	100-1k	Metadata overhead
Snapshots/table	Expire >7d	Metadata bloat

Common Error Messages Reference

Error Message	Likely Cause	Fix
401 Unauthorized	Missing/invalid token	Check token has catalog+storage permissions
403 Forbidden	Token lacks storage permissions	Add R2 Storage Bucket Item permission
404 Not Found	Catalog not enabled or wrong URI	Run wrangler r2 bucket catalog enable
409 Conflict	Table/namespace already exists	Use try/except or load existing
422 Unprocessable Entity	Schema validation failed	Check type compatibility, required fields
CommitFailedException	Concurrent write conflict	Add retry logic with backoff
NamespaceAlreadyExistsError	Namespace exists	Use try/except or load existing
NoSuchTableError	Table doesn't exist	Check namespace+table name, create first
TypeError: Cannot cast	PyArrow type mismatch	Cast data to match Iceberg schema

Debugging Checklist

When things go wrong, check in order:

1. ✅ Catalog enabled: npx wrangler r2 bucket catalog status <bucket>
2. ✅ Token permissions: Both R2 Data Catalog + R2 Storage in dashboard
3. ✅ Connection test: catalog.list_namespaces() succeeds
4. ✅ URI format: HTTPS, includes /iceberg/, correct bucket name
5. ✅ Warehouse name: Matches bucket name exactly
6. ✅ Namespace exists: Create before create_table()
7. ✅ Enable debug logging: logging.basicConfig(level=logging.DEBUG)
8. ✅ PyIceberg version: pip install --upgrade pyiceberg (≥0.5.0)
9. ✅ File health: Compact if >1000 files or avg <10MB
10. ✅ Snapshot count: Expire if >100 snapshots

Enable Debug Logging

import logging
logging.basicConfig(level=logging.DEBUG)
# Now operations show HTTP requests/responses

Resources

• Cloudflare Community
• Cloudflare Discord - #r2 channel
• PyIceberg GitHub
• Apache Iceberg Slack

Next Steps

• patterns.md - Working examples
• api.md - API reference