Parquet

• Querying the entire internet with DuckDB and Common Crawl · Sep 8, 2026

  Common Crawl publishes petabytes of web crawl data on S3. With DuckDB you can query it directly, no cluster and no download, and measure how fast the vibe-coded web is really growing.

  duckdb motherduck common-crawl parquet s3 data ai charts

• Turn Thousands of Messy JSON Files into One Parquet: DuckDB for Fast Data Warehouse Ingestion · Sep 22, 2025

  If you've inherited a bucket full of thousands of tiny JSON files—one per API call, one per event, one per log minute—you know the pain: slow scans, schema anxiety, and rising warehouse bills. This guide shows you how to consolidate them into clean Parquet with DuckDB: handling schema drift, maintaining lineage, optimizing performance, and integrating with dbt. Touch your raw files once, then model against something stable.

  duckdb data-engineering json parquet dbt ingestion performance analytics