Kafka Publisher

The Kafka Publisher connector allows a tuple to be published to a Kafka topic.

Using Kafka Publisher

1. When creating a new data table, select Kafka Publisher from the Connect to data dialog.

The Kafka Publisher Connection dialog is displayed:

 

2. Enter the connection details including:

1. • Bootstrap Servers

          List of host/port pairs of Kafka servers used to bootstrap connections to a Kafka cluster.

By default, the value is localhost:9092. However, this can be overridden by specifying another bootstrap server in the External Settings text box (as specified in step 3).

• Schema Registry Host

         Where the Schema Registry is located. This can be in a different location from the Kafka cluster.

1. • Schema Registry Port

          The port number of the schema registry which provides the serving layer for the metadata. Default is 8081.

3. To support authentication (i.e., username and password), expand External Settings to specify the system settings for the Designer and Server.

NOTES:

• If there is no need to connect to the Panopticon Visualization Server and publish workbooks, enter the External Settings in the Designer text box only.         

1. • If the bootstrap server is not secure, then there is no need to authenticate and you may leave the External Settings blank.

Below is an example of system settings for an SASL authentication in the Designer text box:

bootstrap.servers=localhost:9093

security.protocol=SASL_PLAINTEXT

sasl.mechanism=PLAIN

sasl.username=dwchuser

sasl.password=dwchpwd

 

Below is an example of system settings for an SASL authentication in the Server text box:

bootstrap.servers=localhost:9093

sasl.jaas.config=\

  org.apache.kafka.common.security.plain.PlainLoginModule required \

    username="dwchuser" \

    password="dwchpwd";

 

4. Click Fetch Topics to populate the drop-down list and select a Topic.

NOTES:

1. • Ensure that the ability to ping is enabled in the ZooKeeper Host. Otherwise, if ping is disabled, the Fetch Topics button will not be able to populate the list of topics and you need to manually enter the topic names.          

1. • For Avro format messages, make sure to select an output topic. This populates the list of columns, with the data type found from inspecting the first ‘n’ rows of the file.

          This also automatically enables the Use Schema Registry box, as well as display Avro in the Message Composer drop-down list and the Update Schema button.

 

Clicking Update Schema ensures that the latest schema of the topic is being applied.

 

5. For non-Avro format messages, select Json in the Message Composer drop-down list box.

6. Click Add Column to add columns to the Kafka connection that represent sections of the message.

 

7. Enter or select:

1. • Name

   • Type (Numeric, Text, or Date/Time)

   • Value (can either be a parameter or data entry that can be used as a publish value)

You can also opt to click to delete a column in the list.

8. Click OK to confirm the selection. The source table is returned in the Edit Data Table view, with Kafka Topic as the title and all fields displayed in the Data Source Preview.

Initially, there is no data displayed in the Data Source Preview section of the Edit Data Table window.

Click the Start Preview button to display the data and refresh the values depending on the defined Refresh Period.