Apache ActiveMQ

The ActiveMQ connector allows connection to Apache’s ActiveMQ message bus on a real-time streaming basis. Specifically the connector allows Panopticon Designer (Desktop) to subscribe to XML, JSON or FIX based messages that are published on particular topics. The data format itself is arbitrary, and as a consequence the connection includes the message definition.

Using Apache ActiveMQ

1. When creating a new data table, select ActiveMQ from the Connect to data dialog. The ActiveMQ Connection dialog displays.

   

2. Enter or set the following:

3. • Connection Details including the Broker, Username, and Password

   • Topic

3. Check/uncheck the Use durable subscription box.

IMPORTANT: When connecting to a message bus, it is recommended to disable durable messaging. When it is enabled, this puts a heavier load to the server, and slows down the start and stop of subscriptions.

4. Check/uncheck Messages can contain partial data box.

5. Modify the Real-time Limit to vary the data throttling. This defaults to 1000 milliseconds. (When data should be pushed through to the desktop). Refer to the Important section below for more information.

6. Select any from the following Message Types:

7. • Fix

   • JSON

   • Text

   1. If Text is selected, confirm the Decimal Separator, Text Qualifier, Column Delimiter, and if the first row of the message include column headings.

1. • XML

7. The schema of the source file is then defined by clicking the Generate Columns… button.

This populates the list of columns, with the data type found from inspecting the first ‘n’ rows of the files.

8. Click Add Column to add columns to the MQ connection that represent sections of the XML/JSON/FIX message.

9. Enter or select the following:

1. • Name

1. • XPath/JsonPath/Fix Tag (for example, price/@stock)

1. • Type (Numeric, Text or Date/Time)

1. • Date/Time format when the Data Type is Time

NOTE:

To parse and format times with higher than millisecond precision, the format string needs to end with a period followed by sequence of lower case Fs. There can be no additional characters following them.

For example: yyyy-MM-dd HH:mm:ss.ffffff

1. • Enabled (determines whether the message field should be processed)

   • Filter (defined parameters that can be used as filter)

NOTE: The Filter option is only available for JSON, Text, and XML Message types

Fix

JSON

Text

XML

You can also opt to click to delete a column in the list.

10. Save the columns definition into an XML format by clicking the Save button. The saved XML files are placed in …\Public\Public Documents\Datawatch Desktop\Column Schemas.

11. Click Load to load the column schema.

12. Expand Real-Time Settings.

 

Refer to Defining Real-Time Settings for more information.

13. Click OK to confirm the selection and retrieve the record set into Panopticon Designer (Desktop).

Initially, there is no data displayed in the Data Source Preview section of the Edit Data Table window.

Click the Start Preview button to display the data and refresh the values depending on the defined Refresh Period.

 

IMPORTANT:

The default values in subscribing to ActiveMQ are:

• Real Time limit = 1000 ms

• Refresh Period = 900 s

The first client request initializes the subscription. Initially there is no data, so an empty data table is returned.

This empty dataset is cached on the server with a TTL set to 900 seconds.

The subscription being initialized updates the server in-memory table with the latest snapshot of the data (as defined in the data table definition).

Consequently, the end client will show no data.

Manually clicking refresh in the end client will cause a new data request, which will be checked against the TTL of the server, and return the cached empty dataset.

Only after the cache has expired, is a subsequent request passed down into the in-memory snapshot of the latest streaming data, which is again cached, and delivered to the end client.

It is therefore recommended to set the Refresh Period to 1 second for streaming sources.

However, if the source does not change frequently, the Refresh Period may be increased to a few seconds. This will reduce load on the server and more users can retrieve data from the cache.