Persisting state between AWS EC2 spot instances (opens in new tab)

This looks very interesting, good competition to Avere based on info so far. Is there any native kubernetes integration in the works?

And even if that wasn't true, network-attached storage (unlike local storage) has no semantics for communicating a "partially completed" write of a block. Your server either manages to send an iSCSI packet to the SAN with a completed checksum, or it doesn't. Which means that—for the problems that would arise from a sudden power-cut to a VM (let's say from unexpected hypervisor failure)—using a journalling filesystem on your network disks would perfectly compensate for those problems.

> If your application can't handle a normal system shutdown without losing data, your application is at fault, not your system setup.

Unless something in the system shutdown fails to give the application what it needs (for instance, time) to shutdown cleanly. Which is entirely possible considering that Amazon is selling you the spot instance on the given assumption that it can give the hardware at any time to somebody who is willing to pay more. Amazon does not guarantee the time needed for a clean shutdown (only that a two-minute warning will be available via their proprietary mechanism, if you architect your application to monitor for it) for a spot instance anywhere in their documentation, and you would be ill-advised to not architect for that.

> Storing your important spot instance data on EBS is recommended by AWS

Because EBS itself is reasonably reliable. If you have configuration data (i.e. in /etc) for a legacy application that isn't managed, it's reasonable to mount that data on EBS since it's rarely written to and writes are generally human-initiated and human-monitored (with operations policy possibly mandating a snapshot even before any changes are made).

That's still very different from daemon writes to /var. Take for instance, the PostgreSQL documentation which warns that snapshots must include WAL logs in order for the snapshot to be recoverable, and that it is quite difficult to restore from a snapshot if you stored your WAL logs on a different mount: https://www.postgresql.org/docs/10/static/backup-file.html

You need to understand precisely how your application is treating your storage and act accordingly. Thinking that all applications interact with storage the same way is dangerous and liable to cause data corruption and loss. That's all.

> Assuming your database software

You're assuming that people are saving their state in databases to begin with. If you're saving state to a database in production, typically you're communicating with that database over a network connection, and not running the database on the same machine as your application. Containerizing databases is a whole separate issue.

OP's specific example is saving /var/opt/gitlab to an EBS volume and expecting to be able to move it from one spot instance to another without corruption. That strikes me as insane.

- EBS, being block storage, doesn't recognize the filesystem format on top of it, and therefore doesn't recognize if you formatted the block storage as ZFS and therefore will not use ZFS snapshots when using Amazon's native EBS snapshotting. If you wish to use ZFS snapshots, you have to build that on top of what Amazon gives you, along with all the other aspects of ZFS storage, i.e. building a ZFS storage pool from separate EBS volumes. I mean, it would be nice if Amazon had a hosted ZFS solution, but so far, doesn't seem like it.

- Yes, you get a notification, but it's a proprietary notification scheme that your application must be designed to poll for. Why can't Amazon use standard signals like SIGPWR to indicate imminent shutdown?

- Just because it isn't smart for non-spot instances doesn't suddenly make it smart for spot instances ;)

EBS is not the root volume.

AWS Lightsail is AWS’s option there.

Digital Ocean is still a premium provider.

I would look at providers like OVH and even cheaper (Treudler, TransIP, RamNode, etc.) For example, an SSD with 2 vCPUs, 8GB RAM and 40GB SSD is 13.49$ per month from OVH.

Here’s a list of providers by cost:

https://git.io/vps

(PS: Don’t use DigitalOcean, they tend to steal your credit if they feel like it. Lost 100 bucks "promotional credit" that way with only a few days notice)

Not quite $6, but close: https://www.scaleway.com/pricing/

Hostus.us

The deal was found on LowEndBox, not sure if it's still available, but there are many other ones.

It is 3X more expensive ($0.30/gb vs $0.10/gb for us-east), but it's replicated across AZ's (so is more durable than EBS which is only replicated within an AZ), and you only pay for what you use, you don't need to overprovision the EBS volume to account for peak dataset size.

And since it's shared, you don't need to replicate data across multiple nodes... so if 10 compute nodes needs access to the data set, they can all just read it from the same EFS filesystem, no need to download it 10 times to each compute node.

So EFS can still be very cost effective compared to EBS.

And while we are talking about costs, make sure you check for unused WBA volumes frequently, as you still pay for them if they aren't attached/used - and sometimes a dev will create a provisioned iops drive and forget to delete it and you pay a lot for those volumes..