undefined | Better HN

0 pointsALLTaken0y ago0 comments

Hi,

I'm building a distributed application based on Hypergraphs, because the data being processed is mostly re-executable in different ways.

It's so refreshing to read this, I was also sitting down many nights and was thinking up about the same problem that you guys solved. I'm so glad about this!

Would it be possible to plug other storage engines into Restate? The data-structure that needs to be persisted allows multiple-path execution and instant re-ordering without indexing requirements.

I'm mostly programming in Julia and would love to see some little support for it too =)

Great work guys!

0 comments

sewen0y ago

Thank you for the kind words!

The storage engine is pretty tightly integrated with the log, but the programming model allows you to attach quasi arbitrary state to keys.

So see whether this fits your use case, would be great to better understand the data and structure you are working with. Do you have a link where we could look at this?

ALLTakenOP0y ago

> Do you have a link where we could look at this? Hi, thank you for your reply, highly appreciated.

Happy to explain in more detail =) But it's not public yet.

I'm working on the consensus module optimised for hypergraphs with a succinct data-structure. The edges serve as an order-free index (FIT). Achieving max-flow, flow-matching, graph-reduction via circuits is amongst the goals.

Targeting low-latency/hig-performance distributed inference enabling layer-combination of distinct models and resumable multi-use computations as a sort of distributed compute cache.

A data-structure follows a data-format for persistence, resumeability and achieving service resilience. But although I've learned quite a bit about state management, it's still a topic I have much respect for and think using restate.dev maybe better than re-inventing the wheel. I didn't have in mind to also build a cellular automaton for state management, it maybe trivial, but I currently don't feel like having the capacity for it. Restate looks like a great production ready solution before delaying a release.

I intend to open-source it once it's mature. (But I believe binary distribution will be the more popular choice.)

j / k navigate · click thread line to collapse