Kafka Sink

The Kafka Sink command (Command Type: 1201) sends the job result to the sink service, which publishes it to a Kafka topic.

Command Type

Command Type ID: 1201

Parameters

Parameter	Type	Description	Required
topic	string	The Kafka topic name to publish to	Yes
dataSegmentToSend	string	Which part of the data to send. Possible values: `"fullData"`, `"data"`. Defaults to `"fullData"`	No
headers	object	Optional Kafka message headers (values must be strings)	No
key	string	The message key	No

Usage Examples

Basic Kafka Publish

{
  "command": "kafka-sink",
  "params": {
    "topic": "sports-events"
  }
}

Publish Only the Data Segment

{
  "command": "kafka-sink",
  "params": {
    "topic": "live-odds",
    "dataSegmentToSend": "data"
  }
}

Publish with Headers and Key

{
  "command": "kafka-sink",
  "params": {
    "topic": "events",
    "key": "event-123",
    "headers": {
      "source": "scraper",
      "version": "2"
    }
  }
}

RabbitMQ Sink - Publish to RabbitMQ
S3 Sink - Upload to S3
MongoDB Sink - Write to MongoDB

Quick Start

Commands Glossary

Command Type

Parameters

Usage Examples

Basic Kafka Publish

Publish Only the Data Segment

Publish with Headers and Key

​Command Type

​Parameters

​Usage Examples

​Basic Kafka Publish

​Publish Only the Data Segment

​Publish with Headers and Key

​Related Commands

Command Type

Parameters

Usage Examples

Basic Kafka Publish

Publish Only the Data Segment

Publish with Headers and Key

Related Commands