Apache Kafka

The Kafka connector allows connection to Apache’s Kafka message bus on a real-time streaming basis. Specifically the connector allows Panopticon Designer (Desktop) to subscribe to XML, JSON, Text or FIX based messages that are published on topics. The data format itself is arbitrary, and consequently, the connection includes the message definition.

Using Kafka

1. When creating a new data table, select Kafka from the Connect to data dialog.

The Kafka Connection dialog is displayed:

 

2. Enter the connection details including:

1. • Bootstrap Servers

          List of host/port pairs of Kafka servers used to bootstrap connections to a Kafka cluster.

By default, the value is localhost:9092. However, this can be overridden by specifying another bootstrap server in the External Settings text box (as specified in step 3).

• Schema Registry Host

         Where the Schema Registry is located. This can be in a different location from the Kafka cluster.

1. • Schema Registry Port

          The port number of the schema registry which provides the serving layer for the metadata. Default is 8081.

3. To support authentication (i.e., username and password), expand External Settings to specify the system settings for the Designer and Server.

NOTES:

• If there is no need to connect to the Panopticon Visualization Servers  and publish workbooks, enter the External Settings in the Designer text box only.         

1. • If the bootstrap server is not secure, then there is no need to authenticate and you may leave the External Settings blank.

Below is an example of system settings for an SASL authentication in the Designer text box:

bootstrap.servers=localhost:9093

security.protocol=SASL_PLAINTEXT

sasl.mechanism=PLAIN

sasl.username=dwchuser

sasl.password=dwchpwd

 

Below is an example of system settings for an SASL authentication in the Server text box:

bootstrap.servers=localhost:9093

sasl.jaas.config=\

  org.apache.kafka.common.security.plain.PlainLoginModule required \

    username="dwchuser" \

    password="dwchpwd";

 

4. Click Fetch Topics to populate the Topic drop-down list. Initially, the first topic in the list is displayed in the Topic drop-down box.

NOTE: Ensure that the ability to ping is enabled in the ZooKeeper Host. Otherwise, if ping is disabled, the Fetch Topics button will not be able to populate the list of topics and you need to manually enter the topic names.

 

Click the drop-down list to search and select the desired topic.

For the unprotected broker endpoint:

 

For the secure broker endpoint:

 

If the selected topic is available or currently running in the Schema Registry Host and Port, the schema is displayed. This populates the list of columns, with the data type found from inspecting the first ‘n’ rows of the files.

The dialog may change, depending on the type of selected topic.

For Avro topics:

 

 

Note that the Update Schema button becomes available.

 

For non-Avro topics:

 

 

Select the corresponding steps to proceed in retrieving the data table using the Apache Kafka connector:

1. • Avro topics

• Non-Avro topics