Apache QPID

The QPID connector allows connection to Apache’s QPID message bus on a real-time streaming basis. Specifically the connector allows Panopticon Designer (Desktop) to subscribe to XML, JSON, Text or FIX based messages that are published on particular topics. The data format itself is arbitrary, and as a consequence the connection includes the message definition.

Using Apache QPID

1. When creating a new data table, select QPID from the Connect to data dialog. The QPID Connection dialog displays.

   

2. Enter or set the following:

1. • Broker

   • Username

   • Password

   • Exchange

   • Routing Key

3. Modify the Real-time Limit to vary the data throttling. This defaults to 1000 milliseconds.

4. Select the Message Type:

1. • FIX

   • JSON

   • Text

   • XML

   If Text has been selected, confirm the Decimal Separator, Text Qualifier, Column Delimiter, and if the first row of the message is included in column headings.

2. 

5. Select either the dot (.) or comma (,) as the Decimal Separator.

6. The schema of the source file is then defined by clicking the Generate Column… button.

This populates the list of columns, with the data type found from inspecting the first ‘n’ rows of the file.

7. Click Add Column to add columns to the QPID connection that represent sections of the message.

8. Enter or specify:

1. • Name

1. • XPath/JsonPath/Fix Tag/Text Column Index (for example, price/@stock)

1. • Type (Numeric, Text or Date/Time)

2. • Date/Time format when the Data Type is Time

1. NOTE:

2. To parse and format times with higher than millisecond precision, the format string needs to end with a period followed by sequence of lower case Fs. There can be no additional characters following them.

3. For example: yyyy-MM-dd HH:mm:ss.ffffff

3. • Enabled (determines whether the message field should be processed)

   • Filter (defined parameters that can be used as Filter)

NOTE: The Filter option is only available for JSON, Text, and XML Message types

Fix

JSON

Text

XML

You can also opt to click to delete a column in the list.

9. You can opt to save the columns definition into an XML format by clicking the Save button. The saved XML files are placed in …\Public\Public Documents\Datawatch Desktop\Column Schemas.

10. Click Load to load the column schema.

11. If the Type is selected as Text, it will be listed in the Id Column drop list box, and can be used to select a key column to manage data updates and inserts.

 

You may opt to check the Add Last Update Time and Age box. NOTE: This is only available when No Time ID has been set.

 

This allows the latest update time and its age to be highlighted by the defined color range in the output dashboard. Refer to Highlighting the Latest Data in Real Time Streaming Connectors for more information.

Furthermore, a streaming time series window can be generated by creating a compound key with the Id Column, plus a separately specified Time ID Column.

This Time Id column can be from the source dataset, or alternatively automatically generated.

If the Time Id column is selected, then a scrolling time window can be specified. As new data arrives from the subscription new time slices will automatically be added, and old ones will be deleted.

 

Define or select the following information:

1. • Time Id Column Name (for Automatic Time Id)

   • Time Id Barring

Select the barring period. This conflates the data set to a defined granularity or any of the following time intervals:

• Time Window (seconds). Default is 0.

• Real-time Limit (ms). Default is 1000.

As new data arrives from the subscription new time slices will automatically be added, and old ones will be deleted.

Once the ID Column has been selected from the drop-down list box, the OK button is enabled.

The ID column is used to ensure correct processing of both inserts, and updates.

If a new ID is received, a new row is added to the in-memory data set representing the Qpid topic subscription. While if an existing ID is received, an existing row is updated.

12. Check the Persistent Server Subscription box which is the default behavior. This means that it will not be purged.

If not checked, the Panopticon Server can purge or cancel the subscription if it is orphan and is running out of memory. Note though that it can be purged for other reasons as well, depending on how the user has set it up.

13. Click OK to confirm the selection and retrieve the record set into Panopticon Designer (Desktop).

Initially, there is no data displayed in the Data Source Preview section of the Data Prep window.

Click the Start Preview button to display the data and refresh the values depending on the defined Refresh Period.