Data Access and Caching

Panopticon Designer (Desktop) assumes in general data is never at rest.

Data is assumed to be too big to simply load all into memory, and is either:

• Subscribed Against

• Polled (Periodically refreshed)

This means either:

• Load Subset of Data in Memory

• Load Summary & Parameterized Detail Views

• ROLAP (Dynamically explore datasets)

Consequently for direct access Panopticon Designer (Desktop) is only as fast as the underlying data platform, or the refreshing of resultset caches.   

When data is not changing on a timely basis, such as a daily updated data warehouse, there is the additional option of retrieving data into the queryable cache.

 

Consequently:

• Only required data is retrieved.  The majority of data stays in the underlying data sources.

• Typically aggregated, conflated, filtered data is retrieved.

• Behind each Dashboard Part (Visualization) is a micro-cube.

• Each cube is designed for streaming real time updates.

• Behind each cube is a real time data table. (also powering filters)

• Behind each data table is a resultset cache.

• Behind the cache is the underlying data repository.

• Caches can be loaded on the fly, or pre-loaded on a periodic basis.

• All caching is optional.

As a result data access is either:

• Work Directly against underlying Sources. (Either Exploratory Analysis (ROLAP), Or Pre-Defined Parameterised Views)

• Extract & Cache Data from slower underlying sources.  And query this data extract locally.  (Similar to competitor products).

In reality usage is typically Hybrid. Based on the characteristics of the underlying data, you choose whether to extract and load, or query directly.

This is to cater for real world data landscapes, where different data has different data retrieval latency characteristics, and different timeliness; and where there is too much data to simply load all into memory.