Most of what we see on HN is about building applications, servers, websites etc. Big, monolithic things that take weeks to years and are deployed to somewhere else and used by lots of people. Most programming tools are built for this kind of work, where the time between writing the code and actually using it is days or months.
But the people we want to make programming accessible to are mostly knowledge workers. Their work is characterised by a mixture of manual work and automation, throw-away code and tools rather than applications. It's better supported by Excel, SQL, shell scripting etc than by the big languages and IDEs.
We realised that we can do much more good focusing on that kind of programming.
As an analyst the tools I use are excel, access, SAS Enterprise guide and Oracle SQL developer. One of the big problems I face is that we have no good way to abstract away a process and really make it reusable.
My general work flow is using SAS to pull data from multiple sources, combine and run the data through some series of logic/calculations. Then take the resulting data, copy to excel for some additional analysis or report. This might be for a monthly/quarterly report or an analysis that needs to be update with the additional runout of data.
But these steps are all tightly coupled together. If I want to rerun the same logic on a different data set, or an updated data set I will copy and paste all of the files, update the queries. I have no way to bundle them together so that I can easily reuse with different data sources, or refreshed data.
Really want I want is someway to encapsulate different sets of data transformations/calculations into to functions to reuse them in different contexts and among different people.
It's a graphical tool - you drag-n-drop modules, then configure and connect them, though it can also run scripts (it has JavaScript, Java, Bash and Ruby support, besides SQL, of course) - but after configuring the transformation/job, you can also run it on the terminal, which is useful for periodically re-running it.
It's more about making computers into personal tools. If you look at the tools the average person uses - email, excel, google etc - they all work really well individually but they are really hard to extend or compose. Each application is a world unto itself and doesn't play with the outside world. What would really help people work is not the ability to build their own applications but the ability to move data around and glue tools together. It's kind of like applying the unix philosophy to office suites.
Chiusano has a pretty interesting take on this - http://pchiusano.github.io/2013-05-22/future-of-software.htm...
There are definitely some differences between building large apps and these more communication/analysis tasks, but we think the foundation itself applies to both. The language is an adaptation of the Dedalus[1] semantics, which the BOOM lab did some amazing things in distributed systems with [2]. If it can build a clone of hadoop, chances are it can build most things. We've built a number of our compilers in Eve, several of our editors were bootstrapped, we've built numerous examples, most recently a complete clone of Foursquare. Before we'd want others to try and do that though, we need our tooling to get a bit better. We expect that the Eve editor will get there eventually kind of out of necessity - we're going to bootstrap a lot of it this time too, starting with the compiler.
[1]: http://db.cs.berkeley.edu/papers/datalog2011-dedalus.pdf
Eve will always tend to be missunderstood at Hackernews or Reddit. It simply is not aimed at professional developers and intended to build production systems.
The rationalization is that anyone using an easier tool to program computers without as much code must not be able to use code. Therefore us programmers are better than them.
Its very similar to the earlier era of punchcard programmers scoffing at assembly language programmers. Or hand-tool craftsmen sneering at mass-produced component-based manufacturing.
That infantile belief system will persist until super-intelligent AIs revise it or (maybe) the next generation wises up.
I think the larger implication is: Programming tools are actually pretty good, and the larger process of programming is fairly solid, and while improvements are possible, they're probably going to be evolutionary improvements that build on what we have, rather than throwing everything away in favor of a brand-new approach.
The question is: Is it hard essentially, or is it hard accidentally? That is, can you remove unnecessary complexity from programming and suddenly it'll become easy?
That was the proposition behind first LightTable, and then Eve as originally conceived. But neither of them really found a satisfying answer, a way to say "hey, for making your web app or whatever, if you throw away your existing stuff and use this tool/process, now it's super-easy."
That implies heavily that a lot of the complexity and difficulty is essential. Not all of it -- things will get easier and better over time, as they have over the last ten years -- but enough so that blowing it all up and starting from scratch isn't likely to lead to wins.
I find it really useful to use Ruby's built-in CSV library to process data for spreadsheets then visualize it with a graph in Numbers. I pull data from copy-paste tables, extract from Sqlite databases embedded in applications, etc. But for something like scanning Facebook friends as they suggested I'd have to first figure out the API or how to scrape the data. Once I figured that out it probably wouldn't be too hard to write a script for it. I could probably even put it in a crontab (LaunchAgent plist on a Mac).
Anyway, I think there's a huge opportunity to empower more people who are inquisitive and maybe somewhat technical but are not programmers. Make it easy for these people to solve problems rather than trying to teach them to "program". The end goal in my mind is the StarTrek programming model where you discuss with the computer what you want to do to solve a problem. Those crew members are scientists and engineers but only a few of them are "programmers". (StarTrek is a useful yardstick because we seem to like copying technology out of it. E.g. StarTrek communicator which is the late 90s cell phone and StarTrek PADD which is the SmartPhone/tablet.)
We can come up with better and better formal languages and ways for editing these languages (and probably should), but I would guess even a visual formal language, will still pose many of the same difficulties for users that existing formal languages do.
2. Why denigrate group ware?
While this may not succeed, I have little reason to believe future improvements will all be evolutionary (I know you hedged there, saying "probably"). A lot can be learned from shedding assumptions. If the result of the project is inspiration for building tools on top of traditional programming models, the originators may be disappointed, but I'd call that success in basic research.
Probably, yes -- but working on a problem starting from the beginning is something more people should do.
It is riskier, for sure. And I am glad their team is doing it.
There is nothing technically difficult about these problems; and it's work that's really more suited to a business analyst anyway. While developers should obviously check their logic to make sure it's sound; gathering the requirements is 90% of the work in these situations.
Anyway, that market seems a lot easier to compete in than the straight up dev tools market. Dev tools are so personalized, with every person/team/project/company having different needs and requirements that it seems like the only way to succeed would be with a niche product (which naturally limits the scale of your success).
I do not believe it is reasonable to draw that broad conclusion based on one example
The example given here by the OP strikes me as a good example of how and why programming is complicated, and what people generally want their programs to do is unlikely to be doable without knowing how to program.
Case in point: why can't a layperson just make a little app "to scan through your facebook friends and check to see if someone who usually isn't in your area currently is"? The ease, glib answer is: well, Facebook's developer API requires several hoops to jump through, including OAuth of clients and so forth. So that's why there's no drag-and-drop-plug-and-play module system for such a feature.
The bigger answer is the answer to the question of why does Facebook's API have to be so complicated? Well, besides business reasons...FB's API is a public-facing abstraction over a system in which a billion people have agreed to (semi-)authenticate themselves and communicate a variety of real-time things about themselves. As annoying as it is to program your own little FB apps...it's complicated because the system it interfaces with is overwhelmingly and amazingly complicated.
I don't see much room for improvement in making programming easier in this regard. It'd be like making Shakespeare more digestible to people who don't want to learn to read (OK, ignoring oral storytelling, for this limited analogy)
I read the tutorial and I am sort of struggling to see what benefit this would give someone who was already an Excel power-user.
The task was to send alerts whenever a friend was nearby. Excel is amazing for non-professional programmers, but it doesn't deal well with changing data, unless you want to click the "export my friend list to Excel" button every five minutes. There are tools that will act as real-time data sources in Excel but it's not a natural fit. If we did nothing but make "real-time Excel" it would still be incredibly useful for a lot of people.
There is absolutely no inherent complexity to this problem.
So many people (myself included) were/are incredibly empowered by that program, and I still have a fond place in my heart for Access, as it was my bridge from Excel macros to "real programming".
Hopefully Eve doesn't get DabbleDB'd... the world really needs a modern MS Access!
It's early days both for Eve and for us, but it kinda feels like we're approaching the same problem from different angles. Eve's more focused on the programming experience while we're starting by focusing on the data.
Is there anything here that addresses this problem?
The short answer is we have lots of tools in mind that will help with this, but programming this way just creates a very different kind of system. We've built some complex things and they've remained fairly flat and we made sure that it is both readily apparent what is contributing to your current query and easy to navigate into it if you want to see more. For the most part these problems boil down to navigation and debugging issues, both of which we have really powerful ideas for. For example, we want our debugging story to be what's called a "why? debugger" where you can click on any value in the system and Eve will show all the data that went into calculating it and every query it went through to get to here.
There's assuredly going to be lots more experimentation needed here, but we have every intention of making this handle more complex things. We're bootstrapping the compiler and eventually the editor, we've built clones of websites, and we'll continue to push the edges of what we can do with it :)
[1]: https://github.com/witheve/Eve/blob/master/design/visualProg...
"Along the way to version 0, we tried everything from a Mathematica-like notebook built on functional programming to a purely spreadsheet-like model. We built dataflow languages and madlib based editors and read through papers from the foundations of computing. One thing that ran through all of this, however, was to make sure we never drank our own kool-aid too much. We dug through research and created postmortems and landscape summaries of all the projects that have come before us. We tested our ideas with actual people and against real projects. That meant that we "threw away" most of what we did to get here. It was the best way to keep ourselves honest."
... that is the minimum effort required. I have long advocated that people picking this problem up hit up what's already been done and hit the research to make sure they're not going down a known failure path.
If anyone's going to get to stick my skepticism about visual programming back in my face, it's someone who's doing the stuff in that paragraph, not someone who goes (basically) "Programming sucks, by extension you all suck, visual programming is obviously the solution because, visual! And look, guys, here's my 2-week solution that proves it out!".
I won't lie to you, I remain skeptical, but, well, I'm just generally skeptical about things that don't exist yet. I wish you all the best, and I promise you that if you do succeed I won't do that thing where I pick nits to claim it's a failure anyhow. And I also promise you that I'm happy to say you've been successful in some niche, not set the bar at "REPLACES ALL PROGRAMMING, EVERYWHERE!" or something equally silly.
These tools are often ridiculed and their use by non-programmers for creating business tools is often frowned upon but they allow business users to quickly create flexible, makeshift solutions to their problems. Not every business problem needs to be solved by a complex, cumbersome JEE application and an expensive application server.
While Lotus Notes apps certainly look awful and feel clunky most of the times there is a certain elegance to being able to quickly whip up a solution to a business problem or an urgent information need without having to go through a lengthy collection of requirements and approval process first. The same applies to Excel spreadsheets: They're a great tool for iterating quickly and getting a certain class of jobs done. Something like a REPL for non-programmers.
They do, which is both great and terrible. They're powerful and easy to get started with, but therein lies the danger.
When used to quickly whip something up they're great, but when those things grow or end up being relied upon they're no better than the hack that the CEOs kids friend who is 'good with computers' produces.
So, since they're both useful and dangerous, are there things we can do?
Perhaps a spreadsheet that allows some form of testing? Are there simple tests we could start to encourage people to use? When I've used spreadsheets for some financial things, I know that if I increase one cell, I expect another to increase (for example). I know certain combinations of inputs that should result in certain outputs.
Also, perhaps a clear path from spreadsheet -> application? Often intermediate values are displayed somewhere, so could a spreadsheet app lead someone to naming them all (typically they'll have a 'variable name' just to the left of them).
edit - I should really have read the article first, but I think a focus on making things testable is important.
We have a bunch of ideas queued up. One of the simplest is generating fake data as you write code so that you have a better chance of noticing edge cases. We are also planning to proactively hint about integrity constraints (types, unique/foreign keys etc) eg if a column only contains integers, show a button that fixes the type to integer. There is an optional typing system in the wings too.
There is also lots of similar research aimed at spreadsheets (eg http://eusesconsortium.org/wysiwyt.php) which we can borrow from.
Ronald Reagan allegedly once said "Nothing lasts longer than a temporary government program.". The same applies to throw-away code and 'prototypes'.
> Perhaps a spreadsheet that allows some form of testing?
For complex, interrelated calculations this definitely makes a lot of sense. From my experience the most common problems with Excel sheets are collaborative editing and version control, though. I'm not sure if more recent versions of Excel and Office 365 in particular solve these issues. Google Spreadsheets sort of does (the UX is lacking, though) but then again hardly any company will even consider putting sensitive data in the cloud and rightfully so.
Even convincing users to use complementary tools that improve Excel-based processes might be difficult because due to prolonged, habitual (ab)use many users don't even see the problems that arise anymore ("Why change this? It's always been done this way."). Implementing such tools in a way they're accepted by users is hard. You can't just add Git or some acceptance testing framework to the flow and expect users to be happy with that. It'd have to be something that seems very intuitive and natural to use for the average Excel users.
A clear path from spreadsheet to application would be an interesting approach, too. Spreadsheets are a conglomeration of model, view and controller logic. If you could somehow separate those semi-automatically and generate a boilerplate application from that this might by a viable approach.
Seems to be built on a rather interesting Rust + TypeScript stack? Can't say I've seen that one before - anyone have experience with such a stack?
I would like to hear their rationale for Rust/TypeScript over Clojure/ClojureScript (or Clojure/Javascript).
EDIT: typo
> While running in the browser is a requirement for Eve, it's always been clear that using javascript directly was not a long-term option. So many of our implementation problems come down to lack of control over data layout. For Eve we need to implement:
> New types (like intervals) - but there is a space overhead of 24 extra bytes per object Polymorphic comparisons - but dispatching on typeof is slow Cache-friendly indexes - but it's hard to store multiple js objects sequentially in memory Radix tries - but converting strings to bytes is slow
> We also want to be able to distribute native code for mobile devices and use real threads on servers. Lastly, there is some benefit to using reference-counting for the indexes so that we can avoid copying nodes when we know we have sole access.
> We ruled out C++ and D on aesthetic grounds - we have a preference for small, simple languages that we can understand completely. Rust wins points for safety and abstraction but the toolchain is not nearly as mature and there are issues that currently prevent compiling with Emscripten. C gives us less support in the language but is much more future-proof at the moment.
I also wrote about our experiences with Rust later at http://scattered-thoughts.net/blog/2015/06/04/three-months-o...
@mej10: Turn on showdead if you want to see jamii's response to your question.
The thing is... you can. Not a presentation / PP, but if you want to create a self-updating dashboard, you can do it in Excel. It's not going to be super easy, but you can do pretty much all of it by clicking.
Here's excel with a sheet that comes from sql database. It's trivial to add another sheet with just graphs on it. https://support.office.com/en-ie/article/Connect-to-a-SQL-Se...
But if you want to do a "dashboard proper", you can use the Power BI tool: https://support.powerbi.com/knowledgebase/articles/471664 (which looks super amazing by the way for an office product, is free, and I want to have a reason to actually use it...)
It's strange that they didn't even mention those possibilities in the post. I see how they could try to improve a lot in those approaches however.
Dabble is the main reason I don't have much hope for Eve.
Dabble was amazing, it did everything in the Eve Tutorial much better than Eve does, it didn't require installing anything on your computer, it looked beautiful, it was very easy to use, it had great collaboration, and it had a TON more features than Eve, and it had excellent customer support. Dabble is one of my all-time favorite apps and I think it's one of the best webapps ever. And it didn't succeed.
I guess it's hard to know what might have happened with Dabble had they not created a separate analytics tool that was attractive to Twitter. Would someone else have bought Dabble and continued to support it? What if it had been built on a more mainstream architecture? I always thought Dabble would have been sustainable as a small business. Something about recent history (i.e. LightTable) tells me these guys aren't likely to toil away supporting this for the next 10-20 years to earn a modest income.
I can think of at least three limitations Dabble had that some future version of Eve might not have:
1) There were only so many primitive operations (conversions, calculations, filters, etc) and so there were some "programs" that you couldn't create.
2) For creating a UI, you basically had forms and reports. They had a great implementation of both of those, but they were the only tools you had.
3) You couldn't see the code. Just as Excel has the code hidden in a cell somewhere, in Dabble if you had a derived field (some calculation) you had to click on the field to see how it was calculated. It appears Eve has or will have some way of graphing these calculations so a program can be "read" without clicking on fields to see their formulas.
http://www.selflanguage.org/ (esp. the papers on the UI)
http://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.h...
I like to try to experiment with this stuff as game development tools, because games are highly realtime / graphical / interactive and that's hard. It's easy to write an A -> B transform (like a compiler is a lang1 -> Either error lang2 transform for example) functionally, because ultimately that is a function. Doing interactive compute this way is hard, and that's where FRP, FRelP (functional relational programming) and a bunch of stuff could be used.
I was trying some interactive game dev stuff with this: http://ludumdare.com/compo/2014/08/27/reminisce-post-mortem/ (allows live coding and live edit) but some issues popped up as highlighted in the paper about it. I think a prototype-based approach like Self could be a good way to go. Also doing it in lispy languages to abstract the language upward while abstracting the problem downward.
The Out of the Tar Pit paper is really a good one.
Doing things this way allows more immediate connection to the creative spirit, as on the other side of the more 'logic'/'rational'-based one, which is sort of like static typechecking in human thought -- it prevents error, but to move forward you some times have to make leaps of faith/intuition. Like between two paradigms (check out Kuhn on scientific revolutions, or Science, Order and Creativity by Bohm). Need to be in and about the artwork. Sorry, been reading a bunch of Nietzsche / Psycho Cybernetics / Prometheus Rising type stuff and this is on my mind (http://www.paulgraham.com/top.html) right now haha.
Games also have an aesthetically-minded end point, rather than a solution-minded one, which keeps the focus on aesthetics/human values in focus.
For example, no non-programmer I've ever talked to can correctly explain the difference between:
A and B or C
A and C or B
And to be fair, it's only because of arbitrary precedence rule choices that those are different at all.
I've personally found that dealing with groups and instead of having "AND, OR" you have "ALL, ANY" and always group rules (even if they're groups of one rule).
But even when you have that, you then have to deal with nesting rules, and nests of nests.
The actual implementation of the backend of such systems is easy, the composite pattern / delegates pretty much deals with the implementation.
But the front-end side? They tend to then be forgotten and universally suck, to the point that either it gets handed off to a developer or query-tool expert to use, or some horrific mistake such as accidentally mail-shotting everyone[1] which causes them to never try to have automatic query rules again.
Graphical query building for the end-user is a really difficult area which hasn't seen enough research.
[1] I want to mail Visited Yesterday And are either Men Or Under 30. Instead of "Visit > Yesterday AND (Men OR Under 30). They forget the brackets. Whoops, that's half their clients hit.
I find it interesting that BOOM emphasized scalability, but Eve seems completely uninterested in that aspect.
For me, I want to try it out because there are a million little things I would like to do just for myself, not at web scale...
Although, people have called making a generalised CRUD software, nearly impossible task, but I bet that, in that domain lies a room for an innovative conceptualisation of the problem. Maybe, in the next decade, we can see software with which people don't have to look for freelancers just to build a Data-Entry App. Good luck, Chris!
These graphical abstractions can be excellent when tailored to a domain. We are shipping a product right now that abstracts the database away through a little graphical graph editor like this one to transform it into the domain language of the people (non programmers) that are consuming the data.
It is quite excellent.
It is actually quite simple. There is a graphical designer that connects to a data warehouse. There someone defines a graph like structure with nodes that contain a collection of data rows and relations to other nodes of data rows. This results in a library that you can interop with in existing software tools of the customer to query that graph model naturally (Stuff like Location('airport')=>Car('sedane')=>RentalHistory). It's basically a glorified ORM mapper with a graphical programming language for the codegen of the objects to produce a kind of data DSL.
I'll definitely keep an eye on this, I feel it could be very useful.
$ git clone --depth=1 https://github.com/chilicuil/eve-vagrant && cd eve-vagrant
$ vagrant up #this may take a while
$ xdg-open http://localhost:8080/editor
The above uses a plain precise 32 box and install eve and its dependencies in the provisioning phase, I've also created a modified box (583MB) with eve dependencies hard-coded, which could serve better those who don't have precise32.box anyway.
$ git clone --depth=1 https://github.com/chilicuil/eve-vagrant && cd eve-vagrant/partial
$ vagrant up #this may take a while but not as much as the above
$ xdg-open http://localhost:8080/editor ```
"even many of them expressed wanting to automate processes or bring a bunch of different types of information together"
...makes me wonder: are we just talking about a better ITTT type of thing?
I think, perhaps, it's because there's a fair amount of praxis already out there in the enterprise and small business sectors about this, and I didn't really see reference to any of that, Lotus Notes notwithstanding. I know it's nitpicky, but it was a strong reaction so I thought I would share it. I'll try to unpack it more:
Basically, creating data processing tools for humans is such a fundamental application of computer science that we even have a name for it: Information Technology. And, ever since the Mother of All Demos we have been trying to make a kind of "omni-tool" for data processing, and pretty much falling on our faces.
This is perhaps because a "general purpose tool" usually turns out to be a particular kind of "special purpose tool". The question is whether a large enough segment benefits from general purpose tooling, which entails taking on the overhead of learning how to use this "tool that makes tools" in order to accomplish their many tasks. In other words, there's a layer of indirection. Or, are most people's problems disjoint and specific, so that they benefit more from using a few special purpose tools that can then be loosely coupled together.
For example, let's say for my job I have to manage the generation of reports, etc, and post them on a company website that I maintain. I can use a document editor to edit documents, a communication service to send links to the documents, and a web-based CMS tool to post the final reports to the website. It's not clear that I would be better served, or even could be served (due to the network effect), by an all in one tool-builder tool. Three special purpose tools, which can guide you effectively in each task, might be easier to deal with than one general-purpose tool, all user grousing aside.
This particular type of omni-tool could be described as a"Distributed Filemaker." Filemaker-like tools are definitely popular, but tend not to unseat other special purpose tooling. And they have a problem shared by all powerful data modeling tools: they provide you quite a bit of rope to hang yourself with. If you want to really improve in this space, I suggest you focus on providing a data modeling and coherency paradigm that is appealing to non-technical users, yet successful for managing long term data that changes meaning over time. That would be profound, as poor data modeling is pretty much how all of these projects eventually crash on the rocks. Your users will not ask you for such a thing though, as they don't understand it.
In spite of the eye roll, I wish you luck!
Sorry, but this project is way to ambitious and run by people who have way to weak of a track history to inspire confidence. Go back and finish what you started on, and what you took money for, instead of taking more funding for an even bigger project.
Reminds me of half the Kickstarter games these days: "whelp that didn't work, good thing we don't have to give you your money back!".
I sponsored the kickstarter but never really followed up on what happened to it.
> Never heard of Eve? [Read more here](http://witheve.com/)
...
> Eve is our way of bringing the power of computation to everyone, not by making everyone a programmer but by finding a better way for us to interact with computers. On the surface, Eve is an environment a little like Excel that allows you to "program" simply by moving columns and rows around in tables. Under the covers it's a powerful database, a temporal logic language, and a flexible IDE that allows you to build anything from a simple website to complex algorithms.
Take a look at Excel, for instance. Excel is the most widely used programming language by non-programmers simply because of its dead-simple programming abstraction: a grid of cells than can hold data and reference each other with formulae. History has shown anybody can get this.
People have taken this surprisingly far, but there are still drastic limitations to what you can build this way (how do you manage state, UI, external access to APIs and data?). Trying to make programming in its current form simpler is a dead-end; as a task, programming has built up too much incidental complexity over the years. Instead, we are approaching it from the Excel angle of trying a completely different abstraction, which (we hope) will attract the same kind of attention from non-programmers as Excel has.
I'd love to see a web-native supported version of Access or File maker that was fully open source and had modern programming languages fully baked in.
Every time something like this happens, "real developers" feel threatened, while business users love them for their side projects when the "real developers" are too busy to care.
Maybe these types of tools are always destined to come up again (since we reinvent the platforms all the time), and then a few users use them, while real programmers for the most part simply yawn once more.
Are we ever going to push programming to a level of maturity where we can use building blocks and be real productive, yet have the flexibility to create real, sophisticated applications?
Seems we never quite get there.
But it is positive to see that new generations of developers don't stop trying.
Oh boy, here we go again. A more direct admission of not knowing what to build is hardly imaginable.
