Interview with CEO of rsync.net: “no firewalls and no routers” (opens in new tab)

Yes, certainly.

We frequently fill out very detailed checklists and questionnaires related to our quality policy, standards, internal policies, etc.

We're also very honest about how we approach these issues:

https://www.rsync.net/resources/regulatory/pci.html

... and they generally appreciate the honesty.

We initially had some troubles navigating these waters in the financial sector, but once we were able to convince 1 big customer to try our system on a trial basis, everyone else started to play along really nicely. No one wants to be the first one to try a new thing and get burned by it.

In 2021, you can sometimes leverage things like technological FOMO to make a business owner believe that they are going to lose out on future business value relative to competition, who you might frame as be willing to take on a bigger technological risk. And indeed, smaller clients in our industry are willing to overlook certain audit points (at least temporarily) in order to compete with bigger players.

Some might not like it, but being able to engage in the sales process and bend some rules occasionally is absolutely required to play in the big leagues. Once you are in, it's a lot easier to move around. No one has a perfect solution and everyone knows it. It's just a matter of who is the better sales person at a certain point.

Could we have argued with them during the sales process? Only if we wanted to lose the sale. The Fortinet was cheap compared to the value of the contract.

And in many cases on the vendor side its some dude from sales filling it out... so pretty noisey on both ends.

This is a little disingenuous because their product is a modern firewall. It drops packets and conditionally allows sessions to your backend.

https://www.cloudflare.com/waf/

They may be even aware, they are just bound by their companys ruleset...

You just described my workplace. We have some rules that nobody understands and nobody remembers where they come from, but we have to follow them blindly. For example, they require that any access to the web services should go through a VPN, which would be fine if:

- The VPN actually worked, but it doesn't.

- The servers already uses TLSv1.3, all the services require user authentication, and there are 3 layers of firewalls and an integrated virus scanner in front of the services.

- We are an international project with people from 10 different organizations in 6 countries on 2 continents, and it's really difficult to impose these kind of rules.

So for example, I'm managing a GitLab instance that I can't use myself. I can only SSH login from a very specific computer to manage it, but I can't upload my own code from my office computer.

And I don't want to go into their blind devotion to the firewall and their concept of one way connections...

So I'm just letting time go by, until everybody is so angry they are finally forced to change. Doesn't help that this is Japan, the epitome of rigidness and "even it is broken, don't fix it".

Security is also about depth. You should assume breaches can happen and have another level of defense.

That only increases the attack surface but it's a much better approach for imperfect beings.

On the other hand, a firewall is an explicit declaration of the ports you want open and who you want them open to, which seems like, at the very least, a useful thing to do. If nothing else it seems like defense in depth. I'm not sure I buy that a system designed around "default deny" is an increase in secrity complexity, certainly it's complexity that would hurt availability, but complexity that would hurt security?

Either way, the real security comes from monitoring the reality of what ports are actually open/listening and verifying a person's assumptions about their systems.

I've been using them in a small but important-to-me way continuously since 2008, and I have occasionally forgotten the service needed maintaining at all - at one point I forgot to pay them for an embarrassingly long time after a credit card expired, and they kept my storage going for me until I finally got myself in order. Please don't try that.

(My first contact with them was in 2007, to ask whether they supported pushing directly from git - the answer was no, though they added the feature a few years later - a bit ironically, I've never used it)

So instead I have to use restic, which re-implements many features of ZFS and this also feels wrong.

I did a usesthis a little while ago. https://usesthis.com/interviews/matt.lee/

    javascript:(function(){w=window;d=w.document;b=d.querySelector('body');s=d.createElement('style');s.innerHTML='body{margin:0 auto!important;width:100%;max-width:120ch;font:normal 18px/1.5 Helvetica,Calibri,Arial,sans-serif;color:#333;background:#f9f9f9;}main,#main{margin:0 auto!important;}';b.setAttribute('style','');b.append(s);})();

I was going over the lines with my finger to see if I could feel them. Thought it was internal. Wasn’t until it switched to another app that I saw them move.

Scared the shit out of me.

Yes, one minor thing ...

Although you are correct that Apple is not officially supporting the latest versions of OSX on that hardware, there is a trivially easy hack of the system that will allow you to load newer versions of OSX.

So, like many of you, I am not running Catalina but I am running an updated, patched version of OSX.

You might be paranoid. I've been browsing on a few 2008/2009 obsolete Macs for a while, on the highest OS that they will run.

Eventually they'll be a pain to use because of browser incompatibility, pages will get even more bloated and these machines will run them even slower.

The benefits are tremendous, however, and go beyond day to day operations. A dumb switch has no credentials to protect and there is almost zero attack surface.

Further, if our switch dies we can immediately replace it with any other dumb switch that just happens to be lying around.

If you read failure studies - like those in the excellent Charles Perrow book _Normal Accidents_[1] - you see that in many cases there is a very special component that fails and everything goes to hell when they can't find a replacement for it.

So, while I can't encourage everyone to use dumb, unmanaged switches (because not everyone can) I can encourage everyone to remove as many very special components as they can.

[1] https://en.wikipedia.org/wiki/Normal_Accidents

In the right situation it's doable and potentially highly desirable due to the simplicity, but requires a lot of discipline by everyone involved, and the right conditions to make it work.

It was a design I supported and thought it was a great idea for the right situation, but I also was hesitant to introduce it to anyone but the 'right customer'.... who probably already knew what they needed to know about it.

In the past, the benefits of a firewall were more clear-cut, but these days I think that it’s reasonable to have “defense in depth” without using a firewall as part of your solution.

I came to the same conclusion. I was aware of rsync.net and tarsnap, and have checked their prices in the past, but for raw storage it's simply not competitive. Some of the other features they offer might make it worth it though if you need those.

Personally i just need a place to dump a backup of my family photo albums and documents. A full backup is around 1TB (deuplicated, somewhat larger raw), and for that there are much cheaper solutions.

Self-hosting at home (or in the office) is a great option for some if you’re not worried about needing an offsite backup. For those that do care about this sort of thing, though, the extra you pay to have someone else manage the thing is well worth it.

DIY is often cheaper if you ignore the cost of “doing it yourself” - and securely storing an offsite server is more than just the cost of a disk.

Native ZFS is also a feature for those who can use it.

We give you an empty UNIX filesystem. So, if you push up files over rsync or sftp, they will sit here unencrypted.

However, there are now excellent "tools like rsync that encrypt the remote result with a key rsync.net never sees" - chief among them being 'borg'[1]. Other options include duplicity and restic - all of which transport over SFTP.

So it's up to you and you have total control. If you want ease of use and you want to browse into your account (or one of your immutable daily snapshots[2]) and grab a file over SFTP you probably don't want to encrypt everything on this end.

On the other hand, if you want a totally secure remote filesystem that is nothing but encrypted gibberish from our standpoint, you should use 'borg'.

"Are you able to access this data using a shared key?"

We are running stock, standard OpenSSH and you can, indeed, use an SSH keypair to authenticate with. In fact, you have a .ssh/authorized_keys file in your account so you can specify IP restrictions and command restrictions as well ...

" ... how will you be able to recover from physical failure if you only appear to be serving from a single location?"

A standard rsync.net account has no replication. We are the backup and your account lives in, and only in, the specific location you choose when you sign up. However, for 1.75x the price (ie., not quite double) we will replicate your account, nightly, to our Fremont, CA location.[3]

"As a sidenote, what keyboard are you using?"

It is a Keytronic E03600U2.

[1] https://www.borgbackup.org/

[2] We create and rotate/maintain snapshots of your entire account that are immutable/readonly - so you have protection against ransomware/mallory.

[3] ... which happens to be the core he.net datacenter - one of the nicest and most operationally secure datacenters I have ever been in.

Notably, their website only claims transfer encryption, not encryption at rest. You can of course encrypt your files yourself with your own keys.

Personally, I feel like if you're going to encrypt your data, you should be encrypting it on your end, before sending it to some backup provider who may or may not be keeping your data secure.

So we have a dumb switch in our rack, but they have routers.

In 2021 that's a weird bandwidth product and a weird setup but in 2001 it was "normal" and we just stay with that setup out of inertia (and the fact that we can't connect to he.net in San Diego).

A similar setup exists for us in Zurich with init7.

However, you are correct and we need to edit that FAQ language: our geo-redundant site in Fremont does not work that way.

(I will note that it has been 11 years since we put that location in place (he.net in Fremont) and it has zero minutes of downtime)

A tremendous amount of complexity and attack surface are eschewed by living with that setup and we're always looking for new ways to make that tradeoff.

[1] Castle Acess datacenter on Aero drive. Is now a KIO managed datacenter.

Not lack of physical memory, but lack of ability to address it as the UFS2 tools, like fsck, were not written to handle billions of inodes ...

We really can't thank Kirk M.[1] enough - he wrote custom patches to ufs and fsck just for our (dirty) filesystems and, as I mention in the article, eventually gave us the push to migrate to ZFS.

[1] https://en.wikipedia.org/wiki/Marshall_Kirk_McKusick

In terms of product / tech-stack I don't think I would change anything.

In terms of marketing and word of mouth I think we should have given away hundreds of free accounts in the early years (2006-2010) rather than trying to chase them down as paying customers. I believe we had a lot of decent word of mouth but I don't think I appreciated the power of influencers and their ability to amplify a message.

As for business decisions, I continue to wonder how much business we miss due to not having a Canadian location and we have considered deploying in Montreal for years now but have not pulled the trigger. I don't know if a Canadian location (but still a US company) solves the regulatory requirements of Canadian customers.

Having said that, I do think the site would really benefit from a new paint job. A good UXer can make it so much more aesthetically pleasing, while still retaining its simplicity and quick load time. It doesn't have to be fancy. Just static HTML with elegant styling and a few minor tweaks.

For example, I was really surprised to see big name clients such as Disney, 3M, ARM & ESPN hidden a few clicks away (behind a button which wasn't very informative, from what I remember). Same for being in business for 20 years. A good UX/product person will tell you to put this front and center in your landing page, and rightfully so.

@rsync: I love what you're doing, but please get a UX person involved :)

This doesn’t make sense given that the specific invocation of “rclone serve restic --stdio” doesn’t open any network sockets, it’s no less safe than e.g. “tar”

However, in 2005 or 2006 when we spun out of JohnCompanies[1] and incorporated under the name "rsync.net" I requested, and was given, explicit permission to use the name and domain by the maintainers of rsync.

[1]:https://www.rsync.net/products/borg.html [2]:https://medium.com/@mormesher/building-your-own-linux-cloud-...

Test your backups.

https://messengergeek.wordpress.com/2018/03/09/backblaze-rev...

If a disk break, who changes it?

However, if you want a backup process then you will, indeed, need to find some way to run 'borg' or 'restic' or 'rclone' on Windows.

I've never used WSL so I can't comment, unfortunately ...

If anyone has a recommendation for backing up Windows servers I'd love to hear it.

Though if the descriptions of the service on their web page does not make you salivate, perhaps it's not for you.

But rsync.net is a backup product and OneDrive is a large file sharing drive.

Attempt to use that 6TB in backup situations and you may experience issues.

I have a copy running in my NAS to always have a copy available, one in my laptop, one in my desktop, and I was thinking about having one in my phone to run only when I'm charging (so I don't kill my battery).

My setup to do this is that I run my own nextcloud server, which handles the computer and phone etc. syncing, then nightly that's backed up to a small computer in my house (I just use rsnapshot for that), which then backs itself up to rsync.net (using plain old rsync.)

Especially since they're running the boxes that it's connected to.

They can do resiliency, network segmentation, and monitoring on their platform.

What's a Cisco box going to do for them?

> Disadvantage #1 – Open ports on unmanaged switches are a security risk

Why? Is there something that would prevent an attacker with physical access from unplugging an existing cable? Does the average managed switch config have mac limits and auto shutdown if a link is lost for just a few seconds? Mac limits are easilly bypassed, even without (permanently) disconnecting the legimate device by inlining an active device, maybe some mac spoofing.

I don't include 802.1x or automatically shutting down a port that loses an uplink as a "simple and effective security precaution", it would be a right pain for many situations. Is the latter even a feature? I certainly haven't come across it (unlike normal portsecurity like limiting number of mac addresses, which just adds to overhead with limited effective security).

> Disadvantage #2 – No resiliency = higher downtime

If my device has one ethernet cable into one switch, how does that help? If my unmanaged switch goes pop, I have a spare that I can put in and be back running in a minute. My managed cisco edge switches take 10+ minutes just to reboot.

If my device has two ethernet cables, one into one unmanaged switch, one into another, losing that switch isn't a problem.

> Disadvantage #3 – Unmanaged switches cannot prioritize traffic

Correct they can't. Managed switches without qos set up can't prioritise traffic either. If your switch is dropping packets, you don't have enough bandwidth. I've seen packet loss when sending 500mbit down a 1G uplink on managed switches, even on QOSed traffic. Indeed I've seen higher priority traffic drop and lower priority not drop. QOS isn't trivial. Ultimately it comes down to how big your buffers are whether your packet gets through or not, so your application should cope with some loss, and if you get too much loss you need more bandwidth. If you have 48 devices connected at 1Gbit each, each firing 100mbit of traffic every second, all bang on the second, with a 10gbit uplink, on paper you only need 4.8gbit of uplink. You'll also need a 600MB packet buffer and expect a lot of delay on your packets, whether you have managed or unmanaged, QOS or no QOS.

> Disadvantage #4 – Unmanaged switches cannot segment network traffic

Correct, but then if I have 8 desktops in a cluster why wouldn't I pop in a desktop switch with 8 1G ports? I want them all on the same vlan anyway.

> Disadvantage #5 – Unmanaged switches have limited or no tools for monitoring network activity or performance

They don't, but again do I want that for a specific use case?

If I want a managed switch (which I usually do), then I'll spec a managed switch. It's unlikely it will be cisco. If my requirements don't need features of a managed switch then I won't bother.

I find it interesting that there's no mention of preventing broadcast storms, or IGMP snooping - both of which are far more useful for a typical edge switch than qos.

Personally, I tend to use managed switches - indeed I just bought a couple of 24 port TP Link POE switches for an event I'm planning. I'm not 100% sure I'd go for an unmanaged switch in rsync's case, but from your list

1) Doesn't apply -- servers are in a secure location

2) Doesn't apply -- servers are either single connected (so need a physical visit, and replacing an unmanaged switch is far quicker and easier than a managed switch), or they're dual connected to two different switches

3) If they're doing inline management then you might want to carve out a small part of your uplink to prevent yourself from being dossed by a dodgy server (if your server is saturating your uplink bandwidth and you ssh session can't establish that could be an issue. If you've got OOB access on a separate link though, not a problem, and clearly they don't have that problem)

4) Doesn't matter -- they don't want different vlans

5) They presumably measure the bandwidth use of each of their servers. The question thus is "does the ISP give me logs I can rely on for the wan". Personally I wouldn't, but I can see the idea

Spanning tree: Secure network, they aren't going to connect one port to another to cause a storm

IGMP: They presumably aren't using multicast for anything major so bitrates would be very low even if they were there

Reasons to use a firewall or a switch with an ACL in this specific case that I can think of:

1) 2 points of control -- a zero-day on freebsd's firewall could open a port to an unintended source which was listening but blocked by iptables (or bsd's version). If you had a non-bsd firewall it's unlikely the same zero-day would work

2) Port 22 is only open to a specific IP range, again there's a zero-day, and TTL of outbound packets is high enough to establish a session

Reasons to use a managed switch even ignoring firewalling:

1) Reliable traffic stats -- you could guess at these by summing the uplinks of all the connected devices although some packets will be dropped and some may be going to other devices on the network

Reasons to use QOS on a managed switch:

To allow inband managment if something goes wrong. A separate ilo/ipmi/kvm connection would be better for that though.

I don't think they'd need features like span ports (I personally use them all the time, and fibre taps, but I have a different use case which is UDP heavy and loss-intollerent)

https://www.twilio.com/docs/sms/send-messages

Hetzner is throttling bandwidth after traffic exceeds ~5x the storage capacity while rsync.net doesn't seem to. Hetzner also only supports a very small number of snapshots in total while rsync.net supports more per day.

I don't think Hetzner and rsync.net are really competing with each other. rsync.net's focus is more on business customers, while Hetzner targets private customers.