For ML Teams

Raw → feature pipeline

• ▸Apps / devices / services send NDJSON to EdgeMQ.
• ▸EdgeMQ lands segments (and, when enabled, Parquet raw or views) in S3 under structured prefixes.
• ▸Your feature jobs (Spark, Flink, dbt, custom Python) transform those segments into feature tables or online stores.

Historical replay

• ▸Rebuild features from historical EdgeMQ segments when you change logic.
• ▸Reproduce past model behavior by training from the exact same raw data.

Multi-consumer

The same EdgeMQ raw data can feed both:

• ▸Offline training / evaluation
• ▸Online feature stores / monitoring pipelines

Once the data is in S3, you're free to wire it into any feature stack you want.

Databricks / Spark

Treat EdgeMQ prefixes as your streaming input:

• ▸EdgeMQ continuously drops compressed segments + commit markers, and optionally Parquet (raw or views), into S3.
• ▸Databricks Autoloader or Spark jobs monitor those prefixes and load data into Delta tables.
• ▸You train models on Delta and build feature tables on top.

EdgeMQ can emit Parquet output as raw/opaque Parquet or schema-aware views. Autoloader and Spark can treat those prefixes like any other Parquet dataset in your lake. Table-style layouts remain on the roadmap.

Snowflake

Use EdgeMQ as the raw staging area for training and analytics tables:

• ▸Producers → EdgeMQ /ingest → S3.
• ▸Snowpipe or COPY INTO pulls from those S3 prefixes into Snowflake tables:
• ▸Use the JSON example below when you're reading compressed NDJSON segments.
• ▸Or, when Parquet output is enabled, treat EdgeMQ's Parquet prefixes (raw or views) as external tables or copy from Parquet directly, driven by the same commit markers.

COPY INTO ml_raw.events
FROM 's3://your-bucket/edge-events/ml/prod/'
CREDENTIALS=(AWS_ROLE='arn:aws:iam::123:role/edge-snowflake-access')
FILE_FORMAT = (TYPE = JSON)
PATTERN = '.*\.json\.gz';

You keep all the power of Snowflake; you just stop worrying about how data got to S3.

ClickHouse / Postgres

Use EdgeMQ as a buffer in front of online and near-real-time stores:

• ▸High-volume events (clicks, metrics, logs) flow to EdgeMQ, not directly to your database.
• ▸EdgeMQ absorbs the spikes and writes to S3.
• ▸A loader job ingests into ClickHouse or Postgres at the rate your cluster can safely take.

This is perfect for near-real-time feature tables in ClickHouse and monitoring/tracking tables in Postgres with controlled load.

DuckDB (and friends)

Give data scientists and ML researchers direct, notebook-friendly access to fresh data:

• ▸EdgeMQ writes NDJSON segments and, when enabled, Parquet (raw or schema-aware views) to S3.
• ▸DuckDB queries those S3 prefixes directly from a laptop or cloud notebook.

-- Raw segments (NDJSON)
SELECT *
FROM read_json_auto('s3://your-bucket/edge-events/ml/prod/*.json.gz')
WHERE event_type = 'prediction'
AND ts >= now() - INTERVAL '7 days';

-- Parquet outputs (raw or views, when enabled)
-- SELECT *
-- FROM read_parquet('s3://your-bucket/edge-events/ml/prod/parquet/tenant=.../dt=.../*.parquet')
-- WHERE event_type = 'prediction'
-- AND ts >= now() - INTERVAL '7 days';

No duplicate pipelines, no extra infrastructure-just query the lake.

Real-time product signals → feature store

• ▸Product backend logs user actions and metadata as NDJSON.
• ▸EdgeMQ ingests those events from multiple regions into S3.
• ▸A feature pipeline in Spark/Databricks/Snowflake converts segments into offline feature tables for training and online features for a feature store or a low-latency DB.

Result: your models see fresh behavioral signals without anyone building a bespoke ingest stack.

Telemetry & sensor data → anomaly detection

• ▸IoT devices POST telemetry to regional EdgeMQ endpoints.
• ▸EdgeMQ handles flaky networks, reconnect storms, and spikes.
• ▸Telemetry accumulates in S3 as a clean, continuous stream of records.
• ▸You train and deploy anomaly detection models (Spark, Snowflake, ClickHouse, or notebooks) using this history.

Result: your detection models ride on top of a robust ingest backbone, not fragile scripts.

Model input/output logging → observability and evaluation

• ▸When your model serves a prediction, your service logs input features, model version, output, and extra context.
• ▸Those logs go to EdgeMQ, not local disk.
• ▸EdgeMQ lands them in S3 under a ml-logs/ prefix.
• ▸You analyze drift, calibration, failures, counterfactuals and "what if?" scenarios.

Because everything is centralized in S3, you can slice, audit, and replay model behavior over time.