undefined | Better HN

0 pointscrudbug9y ago0 comments

I am using a time-series db where each metric measurement requires some static context information to be included, e.g. Event 1 happened in Device A at Node B, Location C. These static entities (A, B, C) are perfect for graph vertices. But, when I am persisting the graph on disk - I store all the data statically, No links just columns - the static entities become dimensional data.

Is there a better way to this ?

Ideally, we should persist the Object Graph directly without any data conversion. I see a difference in - How we compute and How we store data.

0 comments

rodionos9y ago

Not specific to GraphEngine, but you could store all meta-data about the device separate from the time-series itself. The composite metric name (key?) would then include the measurement name and the entity name (device name in this case).

crudbugOP9y ago

That is one possible solution. But, for queries we will need some kind of JOIN at runtime.

I do store the entity data separately in a transactional row store.

For analytical queries, we use a column store. For this, I have to include the entity data (context) for each fired event.

The whole Compute & Storage semantics are different.

If only, we had a persistent data-aware compute & storage graph engine that supported transactional & analytical workloads.

rodionos9y ago

Storing time-series in a columnar format (blobs o time-values) and entity data in row format sounds good to me.

If you look at data historians (PI et al), that's how they lay out the data: metrics storage is separate from the asset store, or Asset Framework as it's called.

j / k navigate · click thread line to collapse

0 pointscrudbug9y ago0 comments

Is there a better way to this ?

Ideally, we should persist the Object Graph directly without any data conversion. I see a difference in - How we compute and How we store data.

0 comments

rodionos9y ago

crudbugOP9y ago

That is one possible solution. But, for queries we will need some kind of JOIN at runtime.

I do store the entity data separately in a transactional row store.

For analytical queries, we use a column store. For this, I have to include the entity data (context) for each fired event.

The whole Compute & Storage semantics are different.

If only, we had a persistent data-aware compute & storage graph engine that supported transactional & analytical workloads.

rodionos9y ago

Storing time-series in a columnar format (blobs o time-values) and entity data in row format sounds good to me.

If you look at data historians (PI et al), that's how they lay out the data: metrics storage is separate from the asset store, or Asset Framework as it's called.

j / k navigate · click thread line to collapse