Payload transforms

Payload transforms define how incoming HTTP payloads are parsed, transformed, and inserted into a QuestDB table. You define a transform once with a SQL SELECT expression, then POST data directly to QuestDB. No middleware or intermediary service is required.

Use cases include webhook ingestion, IoT device data, and external API responses where you want to skip building a dedicated ingestion service.

For full SQL syntax, see CREATE PAYLOAD TRANSFORM and DROP PAYLOAD TRANSFORM.

Example: Binance order book snapshots​

Store full order book snapshots from the Binance depth API, which returns JSON like:

{
  "bids": [["65000.01","0.5"], ["64999.99","1.2"]],
  "asks": [["65000.02","0.3"], ["65000.05","0.8"]]
}

Create the target table and the transform:

Table and transform definition
CREATE TABLE order_book (
    ts TIMESTAMP,
    symbol SYMBOL,
    bids DOUBLE[][],
    asks DOUBLE[][]
) TIMESTAMP(ts) PARTITION BY DAY WAL;

CREATE PAYLOAD TRANSFORM binance_depth
INTO order_book
DLQ dlq_errors PARTITION BY DAY TTL 7 DAYS
AS DECLARE OVERRIDABLE @symbol := 'BTCUSDT'
SELECT
    now() AS ts,
    @symbol AS symbol,
    json_extract(payload(), '$.bids')::DOUBLE[][] AS bids,
    json_extract(payload(), '$.asks')::DOUBLE[][] AS asks;

Ingest a snapshot:

POST a payload
curl -s "https://api.binance.com/api/v3/depth?symbol=BTCUSDT&limit=5" | \
  curl -X POST "http://localhost:9000/ingest?transform=binance_depth" -d @-

Response:

{"status": "ok", "rows_inserted": 1}

Overriding variables​

The @symbol variable is declared OVERRIDABLE, so you can override it per request via URL query parameters:

Override a variable
curl -s "https://api.binance.com/api/v3/depth?symbol=ETHUSDT&limit=5" | \
  curl -X POST "http://localhost:9000/ingest?transform=binance_depth&symbol=ETHUSDT" -d @-

Any URL query parameter other than transform is matched to a DECLARE OVERRIDABLE variable by name. Variables not marked OVERRIDABLE cannot be overridden - attempting to do so returns an error.

Inspecting failed payloads​

When a payload fails (bad JSON, type mismatch, missing columns), QuestDB writes the original payload, the error stage, and the error message to the DLQ table configured in the transform:

Query the DLQ
SELECT ts, transform_name, stage, error FROM dlq_errors;
tstransform_namestageerror
2026-03-23T14:00:00.000000Zbinance_depthtransformcolumn not found in target table [column=extra]
2026-03-23T14:01:00.000000Zother_transformtransformbad JSON payload

Multiple transforms can share the same DLQ table. See CREATE PAYLOAD TRANSFORM for the full DLQ schema.

HTTP endpoint​

POST /ingest

Query parameters​

ParameterRequiredDescription
transformYesName of the payload transform to execute
Any otherNoOverrides a DECLARE OVERRIDABLE variable by name

The request body is the raw payload, accessible via payload() in the transform SQL.

Responses​

Success:

{"status": "ok", "rows_inserted": 1}

Error:

{"status": "error", "message": "..."}

Permissions​

In QuestDB Open Source, any user with access to the HTTP endpoint can create transforms and invoke /ingest.

Enterprise

In QuestDB Enterprise deployments with RBAC enabled, the following grants are required:

ActionRequired grants
Create a transformCREATE PAYLOAD TRANSFORM and INSERT on the target table (and DLQ table, if configured)
Replace a transform (OR REPLACE)CREATE PAYLOAD TRANSFORM and DROP PAYLOAD TRANSFORM
Drop a transformDROP PAYLOAD TRANSFORM
Invoke /ingestHTTP endpoint grant and INSERT on the target table
Typical Enterprise setup
-- Admin who manages transforms
GRANT CREATE PAYLOAD TRANSFORM, DROP PAYLOAD TRANSFORM TO ingest_admin;
GRANT INSERT ON order_book, dlq_errors TO ingest_admin;

-- Service account that calls /ingest
GRANT HTTP TO ingest_service;
GRANT INSERT ON order_book TO ingest_service;

Request size limit​

The /ingest endpoint rejects request bodies that exceed a configurable maximum size. The default limit is 5 MB. To change it, set the http.ingest.max.request.size property in server.conf:

server.conf
http.ingest.max.request.size=10M

Requests exceeding the limit receive an HTTP 413 (Payload Too Large) response. The entire request body is held in memory during processing, so set this limit based on available memory and expected payload sizes.

Limitations​

  • Single payload per request - Each HTTP request executes the transform once. That execution may produce multiple rows. Sending multiple independent payload documents in a single request is not supported.
  • Per-request SQL compilation - Transform SQL is compiled on every request. This is acceptable for low-rate ingestion workloads. Compiled-plan caching is a planned optimization.
  • No table references - The transform SELECT must not reference existing tables. It can only use functions and expressions, including CTEs.
  • SELECT only - Only SELECT statements are allowed. INSERT, UPDATE, and other statements are rejected at creation time.
  • Schema drift - Column names and types are validated against the target table at creation time. Schema changes to the target table after creating a transform may cause runtime errors.
  • Concurrent DDL - CREATE, DROP, and OR REPLACE for the same transform name are not serialized. If two sessions operate on the same transform name concurrently, the outcome is last-writer-wins.
Related documentation