• Contents
  • Index

 


 

Kx kdb+ Discovery On Demand Queries

The default behavior when using data connectors is to retrieve data into memory for visual analysis to then occur, where the data is aggregated and filtered in memory. This retrieval may be the consumption of a whole dataset, or through the use of parameters the retrieval of a dynamically selected subset of the data. This approach is however limited by the memory of the machine, and the overhead of retrieving and processing large volumes of data on the desktop.

The Kx kdb+ Discovery connector supports ‘on-demand’ queries, through the checkbox under the Timeout field.  ‘On-demand’ queries provide ROLAP functionality to the Altair Visual Data Discovery products, where the aggregation and filtering tasks are largely offloaded to the underlying data repository.

 

 

The software will dynamically generate q query for:

  • Filter domains (Categorical Listing & Min/Max for Numeric Fields)

  • Aggregated & Filtered Data returned in the visualizations

Each filter and visualization is driven by a separately generated q query, ensuing that each query is simplified, and returns the minimum amount of data.

This on-demand capability dramatically reduces the amount of data to be transferred across the network and onto the desktop, and ensures that the heavy data intensive tasks occur in Kx kdb+ instances.  However when using this mode the following functionality is disabled:

  • Percentile Filtering

  • Copy Raw Data

  • Pivot & Un-pivot Data Transforms

  • Aggregated & Filtered Data returned in the visualizations

  • Non-Additive Data support

  • Shared selection across visualizations

  • Numeric Bucketing

  • Time Part Specific Options (Decade, Quarter, Weekday, Millisecond, Nanosecond)

  • Ranking

  • R Tranform

  • Python Transform

 

NOTE: Every q query that is generated, will be logged if the logging level is set to Info, under Tools > Options.

IMPORTANT:

kdb SELECT queries cannot be issued to tables that include fields or column names with special characters (e.g., Mcap(USD), Close(local), etc.)

Therefore, for on-demand queries, column names should not include special characters.