Show HN: dstack – an open-source tool to build data applications easily

``` You hereby grant to Company an irreversible, nonexclusive, royalty-free and fully paid, worldwide license to reproduce, distribute, publicly display and perform, prepare derivative works of, incorporate into other works, and otherwise use and exploit your User Content, and to grant sublicenses of the foregoing rights, solely for the purposes of including your User Content in the Site. You hereby irreversibly waive any claims and assertions of moral rights or attribution with respect to your User Content. ```

Are these texts common ?

Seems like it could be good for making data-driven dashboard graphs. Although [2] the react library looks like it needs a bit more work.

Congrats on shipping something though!

[1] https://dstack.ai/

[1] https://github.com/dstackai/dstack-react

Can you elaborate on “What’s next”?

> User callbacks- so that application shows not just pre-calculated visualizations but also can fetch data from a store and process it in real-time.

How are you envisioning this working? Will dstack be like a database? How will “user callbacks” be triggered?

It looks like you are more of a wholistic platform, including a workflow scheduler etc.

For instance, suppose I have a notebook that takes 2 hours to generate a model. From the viewpoint of explaining it I'd like to make a notebook where I start from the beginning, train the model, then use it.

If I want to show it to people I want to save all the results and re-render them, not rerun the calculation, certainly if I want to show off the results in a 1 hour talk!

From the viewpoint of reproducibility, however, you have to be able to run the notebook from top to bottom and get a 'correct' result. I'm not going to say the 'same' result because many calculations are stochastic in nature (e.g. random numbers) or because often the data changes. (Let's say I have somebody make a notebook that does April's sales reports -- shouldn't I just be able to point it to the may data to make May's sales reports?)

Between the long time delays (longer than people can hold a context in their mind, longer than they want to wait) for the system to settle down and the total complexity I find that many people involved with data science violently resist confronting the above issues. The effects are much like the visual "blind spot" -- you might get a series of projects that were 98% completed but didn't quite deliver business value although everybody feels like they did their part.

Like other vendors in this crowded space, dstack leads with technology as the key problematic "e.g. supports Python and R", "matlib, Tensfolow, plotly, ..."

It's certainly true that people don't want to face up to reality in that area. Maybe 50% or 90% of the "waste" in the area involves setting your dependencies up, begging your boss to get you access to "the cloud of your choice if that's what's needed". The trouble with is that investment in particular technologies are of temporary value (maybe people will still be using R in 2030, maybe they won't be using Tensorflow, almost certainly plotly gets bought by Google and shut down by then)

Years back I researched the problem of running Tensorflow models that we got off the pavement, building a database that says TF version X depends on CUDA version Y, CNN version Z, and being able to have multiple copies of the userspace GPU drivers installed simultaneously (e.g. just put 'em in a directory and set the library path to point at 'em -- don't even need containers!)

I could have sworn Google looked at my source because they did the one thing that could have broke that strategy. Also the company I was working for lost interest in that particular shiny thing. That's a basic problem with maintaining a distribution of other people's software -- like treading water it takes effort just to stay in one place.

The more fundamental problems that turn up in going from data to decision and products are eternal and not tied to a particular technology. If you solve those problems rather than chase the shiny you might break out of the pack.