Skip to main content
Version: 0.17

Sink writer

Except for the IN_MEMORY_QUEUE sink writer, a sink writer conducts the

  1. keeping "connection" (network connection, file handle) to a foreign sink.
  2. fetching rows from a sink stream.
  3. converting the rows into arbitrary data that the foreign sink can accept.
  4. writing the data to the foreign sink.

Let's look at an example sink writer:

Sink writer definition
CREATE SOURCE READER sink_writer1
FOR sink_stream1
TYPE NET_CLIENT OPTIONS (
PROTOCOL 'TCP',
REMOTE_HOST 'localhost',
REMOTE_PORT '12345'
);

This statement defines the following source reader (the edge in the diagram).

The sink_writer1 does the following:

  1. Connect to the TCP port 12345 on the localhost.
  2. Fetch a row from the sink_stream1.
  3. Convert the into a JSON object line like: {"ts": "2020-01-01 00:00:00.000000000", "c1": 1}
  4. Write the JSON object line to the connection, and repeat from 2.

Sink writer types and options

A sink writer has a name, a type, and options. A type of a sink writer determines the available options.

Here lists the currently available sink writers.

IN_MEMORY_QUEUE

Writes data into an in-memory queue allocated in the application's memory space linked to SpringQL. The application pops rows from the queue using *pop() API.

See: Client API.

  • Options
    • (required) NAME: queue's name.

NET_CLIENT

Connects to a foreign sink via TCP.

  • Options
    • PROTOCOL: must be TCP.
    • REMOTE_HOST: host name or IP address of the remote host.
    • REMOTE_PORT: port number of the remote host.

HTTP1_CLIENT

Requests to a foreign sink via HTTP/1.1.

Currently only a single BLOB is supported for a HTTP body.

  • Options
    • REMOTE_HOST: host name or IP address of the remote host.
    • REMOTE_PORT: port number of the remote host.
    • METHOD: HTTP method.
    • URL,
    • HEADER_{HTTP header name}: HTTP header.
    • BLOB_BODY_COLUMN: sink stream's column name to put as a HTTP body.
HTTP1_CLIENT example
CREATE SINK WRITER http_sink_1 FOR sink_1
TYPE HTTP1_CLIENT OPTIONS (
REMOTE_HOST '{remote_host}',
REMOTE_PORT '{remote_port}',
METHOD 'POST',
URL 'http://{remote_host}:{remote_port}/test',
HEADER_Content-Type 'application/octet-stream',
HEADER_Connection 'keep-alive',
BLOB_BODY_COLUMN 'http_body'
);

Sink data format

Currently, only 1-line JSON objects are supported as an output datum from a source writer. Foreign sinks must accept such data format.

Say, you have the following sink stream:

Sink stream
CREATE SINK STREAM ss (
ts TIMESTAMP NOT NULL ROWTIME,
c1 INTEGER NOT NULL
);

A sink writer can output the following JSON.

JSON output example
{ "ts": "2020-01-01 00:00:00.000000000", "c1": 1 }