So instead of having asymmetrical services (database, web server, workers, logging, messaging) spread across clusters of symmetrical nodes, one would have all of these services within a single process on a single core machine.
Each node would be responsible for itself: from the time it boots to the time it dies, continuously integrating from the master trunk, accepting data partitions from other nodes, checking up on the health of other nodes etc.
One advantage would be that data and workload are evenly distributed, for example each node would have a few workers running which would only operate on local data. So all services would automatically be sharded, not just the database. If each node's workers only operate on local data this would simplify locking across workers throughout the system.
There would be no master nodes, rather replica sets, all multi-master similar to Dynamo. Expanding capacity across all services would mean booting a new instance in EC2.
Also, having the services vertically integrated within a single evented process would eliminate network overhead, deserialization/serialization overhead, parsing overhead, queue contention, locking etc.
Everything could be written in the same language, to make for easier extensibility, more in-house understanding of core components, and more opportunities to reduce waste.
You have 3 database nodes. Your application requests write access. The DB driver opens an event loop and tries to connect to all three at once. As soon as it has two live connections, it starts a distributed transaction (I know, these are unreliable, but bear with me, and assume that they work for now. This is the same thing as used on the server anyways). The application completes the work, and it is committed to two servers, which is the majority. The third server will replicate the changes as soon as it is able, verifying that the majority of the servers agree that this is in fact a valid changeset.
I think that this approach, combined with a good locking system (e.g.: the client would declare all locks upfront), could result in a robust system. It also makes it easy to change where your system lies on the CAP triangle: just tune your driver to require a connection to all servers, not just the majority to make the system lean more towards consistency, and make downed nodes that are recovering refuse any connections until they are caught up.
Does anyone have any thoughts on this?
But if the client crashes after acquiring the locks, wouldn't your system be deadlocked?
