For request analytics at the edge (using Cloudflare Workers here, but it’s similar for Vercel or AWS Lambda), the cheapest architecture is the one that batches writes at the right level. That level depends on how real-time you need your data to be, how much you care about the occasional dropped event, how much data you have and how fast you need to access that data.

Options

  • Worker -> Database write per request: simple, but usually requires a massive database instance to handle the load

  • Worker -> R2 object per request: simple, but creates many tiny files and one Class A write per event. Good for tiny volumes of traffic, annoying for real logs. Batch or compact later. R2 Standard currently prices storage by GB-month and Class A operations by request volume.

  • Worker -> Queue -> batched R2 JSONL: reliable and non-beta, but the bill is shaped around messages. Cloudflare Queues (pub/sub style) usually count write + read + delete, so a delivered message is roughly three operations. Useful when you need retries and a buffer before R2, but otherwise relatively expensive.

  • Worker -> Pipelines -> R2 Parquet: Pipelines are my new favourite on Cloudflare. They’re shaped around GB processed and GB delivered, not event count, and can write JSON, Parquet, or Iceberg to R2. This is closest to AWS Firehose -> S3 or Google Pub/Sub -> Cloud Storage. You do still need to do file compaction yourself, meaning if you want to go from many small files (e.g. 1 minute batches) to hourly or daily compacted files, you’ll need to run that on a schedule

  • Worker -> Pipelines -> R2 Data Catalog (Iceberg): Same as above, but it takes the awesomeness of pipelines and adds auto-compaction on top. No need to worry about the many small files problem anymore, but you do get a minimum of 60 seconds of latency.

  • Worker -> D1 (distributed SQLite): tempting because simple and SQL is convenient, but it is a poor primary log sink. D1 bills by rows read/written; inserts, deletes, indexes, exports, and cleanup all add write/read amplification. Great for a short-retention “recent bot hits” table, not for a firehose.

  • Worker -> KV: wrong primitive for append logs. KV writes/deletes/list operations are per key, and unique-key-per-event turns into both a pricing and operational mess.

So my current Cloudflare answer is: classify and filter in the Worker, send useful rows to Pipelines, land Parquet or Iceberg in R2, and only query R2 from DuckDB/MotherDuck once the data has “cooled down”.