Product
For Data Engineers
Build your Bronze layer without building ingest. EdgeMQ gives you a standard HTTP → S3 ingest layer. Every service, device, and partner writes to the same endpoint; you use Snowflake, Databricks, ClickHouse, DuckDB, and Postgres on top. EdgeMQ is the lakehouse ingest layer that keeps your S3 Bronze tables continuously fed for the rest of your stack.
Yet another ingest pipeline
You should be modeling, optimizing, and shipping insights, not wiring up one-off collectors and brittle upload scripts.
- One-off collectors for each team/partner
- curl + cron + Lambda + bash that "just uploads to S3"
- Lossy partial uploads and silent retry failures
- "Near real-time" requests without more infra budget
- Snowpipe / Autoloader / COPY INTO glued to fragile upstreams
EdgeMQ: your standard HTTP → S3 ingest layer
EdgeMQ is a global ingest service for modern data stacks. Producers send NDJSON to a single HTTPS endpoint; EdgeMQ lands it durably in your S3 prefix with clear commit semantics. You keep your warehouses, lakehouses, and tools—EdgeMQ just makes sure S3 is always up to date.
curl -X POST "https://<region>.edge.mq/ingest" \ -H "Authorization: Bearer $EDGEMQ_TOKEN" \ -H "Content-Type: application/x-ndjson" \ --data-binary @events.ndjson
- Append to a WAL on NVMe at the edge
- Bounded queues with honest backpressure (503 + Retry-After)
- Compressed segments uploaded to your S3
- Commit markers define when data is safe to read
From your perspective as a data engineer, every new upstream producer just needs:
- A URL: https://<region>.edge.mq/ingest
- A token
- A shared contract: "send NDJSON; we'll see it in S3"
S3 is your Bronze layer—by design
Most warehouse and lakehouse tooling assumes S3 as the Bronze layer. EdgeMQ is built to feed that layer with compressed NDJSON + commit markers today, and Parquet, CSV, and table-friendly layouts next.
- Your bucket, your prefix, least-privilege IAM role
- Formats roadmap: Parquet, CSV, JSON objects
- Table-friendly layouts (Iceberg-style) on the horizon
Standardize ingest. Unblock your modeling roadmap.
Give every producer the same HTTPS endpoint. Land data durably in S3 with commit semantics—then model at your pace.
Fits right into your existing stack
Use EdgeMQ as your S3 staging area. Snowpipe or COPY INTO ingest from EdgeMQ-managed prefixes into tables.
COPY INTO raw.events FROM 's3://your-bucket/edge-events/prod/' CREDENTIALS=(AWS_ROLE='arn:aws:iam::123:role/edge-snowflake-access') FILE_FORMAT = (TYPE = JSON) PATTERN = '.*\.json\.gz';
Treat EdgeMQ prefixes as your Autoloader or Spark streaming source—continuous ingest into Delta tables.
Use EdgeMQ as a burst buffer: ingest at edge speed → S3; a controlled loader job batches into your databases.
Analysts can query fresh data directly from S3 without spinning new pipelines. As Parquet/table layouts arrive, reads get even faster and cheaper.
SELECT *
FROM read_json_auto('s3://your-bucket/edge-events/prod/*.json.gz');How EdgeMQ helps data engineers specifically
- Provision an EdgeMQ endpoint + S3 prefix per domain
- Hand teams a small client snippet and schema contract
- They publish; you see consistent data in your lake
- Durable WAL at the edge before acknowledgement
- Bounded queues with 503 + Retry-After under pressure
- Commit markers—no guessing which files are safe
- Per-tenant isolation on microVM + WAL volume
- Role assumption via STS; no long-lived access keys
- Least privilege to specific prefixes
- Per-environment isolation (dev/stage/prod)
- Spend time on dbt, table layouts, and performance
- Assume a trustworthy Bronze layer (S3)
- Build reusable Bronze → Silver → Gold patterns
Managed edge infrastructure, clear SLAs, real observability
- Per-tenant microVMs, tuned WAL, shipper, and health checks
- Starter (free), Pro, and Enterprise tiers with guarantees
- Metrics & logs for ingest rates, latencies, failures, and S3 delivery
Turn "getting it into S3" into a solved problem
Keep using the tools you know. Let EdgeMQ be the standard HTTP → S3 ingest layer that feeds them.