This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises.
rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies.
Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most file metadata and they usually need many non-default flags to make complete file copies.
Nevertheless, rsync is the best of all alternatives, because when invoked correctly it accurately copies all file metadata even between different file systems and different operating systems, in cases when other copying utilities may lose some file metadata without giving any errors or warnings for the users.
https://serverfault.com/questions/842531/how-to-perform-incr...
Btrfs and apfs also support auto-snapshotting and replication, but I don't have experience with them.
However many old versions of file systems lacked support for things like high-resolution timestamps, extended attributes or access-control lists. Because such support has been introduced much later, there are a lot of programs for archiving, backup and copying which lose such metadata, at least with their default options.
While equivalent metadata exists on Linux XFS and EXT4, Windows NTFS, FreeBSD UFS and ZFS etc. each platform has different APIs for accessing file metadata (and on Linux the API may vary even between the many available file systems), so only few programs, like rsync, are able to copy everything while taking care to use the correct API to replicate the metadata without losing information.
tl;dr try this if rsync fails in unexpected ways:
echo "Syncing ~/.ssh directory"
rsync --archive --delete --xattrs ~/.ssh/ laptop:.ssh/- Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely on the user to figure out if something didn't work". Even for home backups, this is just wrong.
- As an example of the above: This script notes that it fails if a backup takes more than 24 hours.
- The "look for other rsyncs running" part of the code is an odd way of approaching locking, but for a single personal "push" backup I guess it is ok.
I've got an rsync wrapper that has been battle tested over a couple decades and hundreds of servers here: https://github.com/linsomniac/tummy-backup/blob/master/sbin/...
Features of it are:
- As the filename implies, the goal is to rsync to a zfs destination, and it will take a zfs snapshot as part of this. It is easy to customize to another backup destination, I've had people report they have customized it for their own laptop backups, for example to an rsync.net destination.
- It goes out of its way to detect when rsync has failed and log that.
- It does do "inplace" rsyncs, which dramatically save space if you have large files that get appended to (logfiles, ZODB databases).
- This is part of a larger system that manages the rsyncs of multiple systems, both local and remote. Things like alerting are done if a backup of a system has failed consistently for many days.
- In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails.
The hardlink trick only works for fairly small data sets. Issues include: Managing hard links takes a lot of overhead, especially on spinning discs. Large files being appended to use a ton of space (a 4GB file with 1K appended every day uses 128GB to store 14 dailies, 6 weeklies, and 12 monthlies). ZFS is a pretty good destination for rsync, as similar snapshots will use 4GB to store.
--inplace update destination files in-place
--whole-file, -W copy files whole (w/o delta-xfer algorithm)
https://superuser.com/questions/576035/does-rsync-inplace-wr...
Also, when transferring from different systems, make sure that both rsyncs are of a high enough version, again preferably the same.
MacOS ships with a really old version of rsync that doesn't support extended attributes at `/usr/bin/rsync`
rsync version 2.6.9 protocol version 29
I've always thought this isn't the right approach.
How do you know if the email server is borked or you commented out the script in cron to debug it and forgot to put it back in?
Either there's a weekly status report to tell things have been green or you could place cron checks like healthchecks.io (You can self host it.)
Also it's much better to use 'zfs send' with large volumes to backup if both ends have zfs as zfs knows which files have changed and it doesn't have to scan for what has changed on each go as any other tools do like rsync.
https://arstechnica.com/information-technology/2015/12/rsync...
Things I have done to ensure reliability (again, this core script has been running for a dozen or more years):
- Nagios monitoring of backups: An active check from a monitoring server that alerts if no recent successful backups.
- "paper path" monitoring of e-mail: Send an e-mail to an external mailbox and have an active check in Nagios that reports if it has not seen an e-mail recently.
- With hundreds of machines, we were in the management interface enough (not daily, but at least monthly) that we would tend to notice before TOO long if something was out of whack.
- Regular backup audits: We would perform quarterly backup audits of the important machines, we had a whole workflow for those, which would also give us confidence that the backups were running as expected and that if something got out of whack it didn't go too long. Many of these depend on your definition of "too long".
As far as "zfs send", I totally agree. However, even today I have very few machines other than my backup machines that are running ZFS, so that's not really an option for these backups.
#!/bin/bash
#!/usr/bin/env bash
set -euo pipefail
It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.
# Make sure no one else is using rsync
pro_on=$(ps aux | grep -c rsync)
(
flock -n 9 || exit 1 # Critical section-allow only one process.
...single thread shell script
) 9> ~/.empty_lock_file
"${SOURCES[@]}"
minutes=$(($minutes - 1))
ECHO="/bin/echo"
Perhaps use this instead, in a subshell to avoid stomping on variables:
myecho () ( z=''; for x; do printf "$z%s" "$x"; z=' '; done; )There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures.
Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted.
When your script does fail unexpectedly, you'll want to re-run it with at least tracing enabled, so the third line should be something like
[ "${DEBUG:-0}" = "1" ] && set -xReally? In this script's "ps aux | grep -c rsync" for example, if "ps" fails, you'll just get 0 without the grep failing.
(Speaking of that line: chasil's completely right that it's much better to use "flock" than "ps" for locking...)
You don't let go of -e for that.
dont_mind_failure || true
important_process
Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.
It doesn't work at all on any BSD OS, which does not store bash in /bin - instead it is in /usr/local/bin
Specifying "env bash" makes it work on any UNIX, since the location of env is a constant, unlike bash.
Find me a single Linux distro where bash, if installed, is not available in /bin
However I've seen it happen quite frequently in systems that were designed with container like 'chroot-lite' prod setups where the system bash and the deployed environment may contain different bash.
The different bash is the one that was tested in the test env with the automation. There may even be multiple different versions of an interpreter on the system with multiple app environments running.
This was a pretty common way to package apps before easy access to containers and container managers.
This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field.
[1] https://github.com/rsnapshot/rsnapshot [2] https://www.cyberciti.biz/faq/howto-install-rsnapshot-filesy...
Very reliable.
I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair"
Course following the 3-2-1 rule you're probably good, but aye I'm treating borg repos as delicately as I would a striped raid now. I am also monitoring that `borg check` actually comes back successfully before considering the backup complete too :)
Also if you're in a small company, start up, etc.. Set yourself a 3-2-1+ backup system. + is the miracle backup that makes you look good.
A backup that isn't tested is not a true backup, it is a disappointment waiting to be found! This can happy with any backup tool, my hand-crafted¹ rsync based scripts included.
Testing isn't hard to setup if you don't mind the final step being manual. Snapshots have a checksum file, and daily one is picked and rechecked, any difference is an indication of bit-rot on the storage medium or something accidentally getting deleted/modified otherwise. After the newest copy is created by the main backup script a list of files not touched since the backup started is made and pushed up, the backup site checks those files and sends the result back so it can be compared. Any difference in these checksums results in an email to an account that makes my phone shout in a distressed manner. The manual part here is occasionally checking the results manually because not getting an email could either mean all is well or that something has broken to the point that the checks aren't running at all².
For specific systems like my mail server I have a replica VM, not visible to the Internet at large, that wipes itself and restores from the latest off-site backup. I look at that occasionally to see that it is running and has the messages I've recently sent and received. As a bonus this VM could quickly be made to be publicly available and take over with a few firewall DNS changes, should the main mail server or its host physically die, and even if it doesn't take over its existence proves that the restore method is reliable should I need to restore the one in the primary location. Some extra automated checks could be added to this too, but again there comes a point where writing the checks takes more time than just doing that manually³ and I'd still do it manually out of paranoia anyway.
[1] If I'm honest, “string together” would be much more accurate than “crafted”
[2] I could automate that a bit too, but then that automation still needs to be verified occasionally, it quickly gets to the point where it is double-checks all the way down and making sure you check manually occasionally is far far more maintainable a system.
[3] If these were a business thing rather than personal services, then the automated procedure vs manual checks desirability balance might change somewhat.
[1]https://opensource.com/article/17/10/backing-your-machines-b...
Then I moved to Borg and haven't looked back. It does the same end goal in a better way, is way faster, and easier to work with overall [2].
[1]: https://nuxx.net/blog/2009/12/06/time-machine-for-freebsd/ [2]: https://nuxx.net/blog/2019/11/10/using-borg-for-backing-up-n...
I dump my backups and also run a calibre server.
Now all I need is better client side software for these things, something I could use on my moms computer. Vorta is not that good, although I commend the effort.
Until the UI side is sorted, my moms laptop stays on Backblaze.
The script here really only makes sense for servers you physically own. You wouldn't accept SSH access from a key located on a plain VPS, right? Also this script doesn't seem to encrypt the data at all!! Very dangerous on a VPS.
With Borg, I use ssh -A from my laptop to start backups going. I can think of some other schemes like using multiple user accounts on a single Storage Box (Hetzner supports that and I believe offers an API) so that different VPS can't clobber each other's backups, that old backups become read-only, etc. It might be interesting to add some finer access controls on the server side. Borg supports an append-only mode but right now, that's only a config option rather than a security setting, I believe.
I've only recently started using Borg so I'm not really familiar with its intricacies yet. There are some things I would change but it is mostly well thought out, imho.
I'm still not clear - does the Hetzner storage box have the borg binary installed on their end ? As in, one could run:
ssh user@hetzner borg --version
Asking for a friend ...
Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455
https://news.ycombinator.com/item?id=29210222
Using zfs snapshot is the best way to go but having a cloud destination wouldn't be easy for these tools unfortunately but there are services that accept borg and zfs send.
Eagerly awaiting a configurable chunk size, if they decide to do that.
Wrong ps grep usage. MacOS specific. No shell quoting.
At least he doesn't advocate rm -rf / by an incorrect usage of --delete.
The cynic in me believes this was intentional to drive engagement and discussion.
It's that old adage of "if you want to learn something just say something wrong about it on the Internet" at work.
I don't entirely know what the motivation is, but as a grumpy old GenXer, I wish it would stop.
If your prod system(s) is/are logging into the backup system, then you have a big problem because if any of them are compromised, they can wipe out / corrupt / etc all of their own backups, and possibly the other server's backups as well (if the same user account is used for all of them.) This problem goes away if your (isolated, hardened) backup server logs into the prod systems. Of course, the inverse problem is that if an attacker manages to get into your backup system, then they also have (at least read) access to all of your prod systems!
There was no email notification if the backup completed or failed, or even started to run. There was no notification of how long it took, how much data was backed up, etc.
There were plenty of incidents of data loss.
https://wiki.archlinux.org/title/Systemd/Timers
EDIT: I'd love to hear if anyone knows of any downsides, edge cases for running rsync under systemd timers.
Use https://github.com/instacart/ohmycron
> MONTHROTATE=monthrotate # use DD instead of YYMMDD
Use https://rotate-backups.readthedocs.io/en/latest/readme.html
> $RSYNC -avR "$SOURCE" "${RSYNCCONF[@]}"
Create a one-command script with the hardcoded rsync command you want to use and replace the directory to sync as a command-line argument, e.g.
#!/bin/sh
rsync -avR \
--delete \
--exclude=/Volumes/Raid/.DocumentRevisions-V100 \
--exclude=/Volumes/Raid/.TemporaryItems --exclude=/Volumes/Raid/.Trashes \
--exclude=/Volumes/Raid/.apdisc \
"$@"
Use logrotate
> mail a report
Use Cron's built-in mail feature
I use restic for most of my backups but hand-written rsync on an external drive with ntfs for the stuff that would be important to my family.
The backup script is much simpler, the repository is properly encrypted, takes snapshots, dedup, mounts the remote, integrates with backends, has clean output, and various useful features for working with repositories.
Rsync over SSH is not even encrypted at rest.
For encrypted backups to e.g. a NAS, Duplicity[1] is my go-to choice (full backup every month or so, with incremental backup every day inbetween).
pro_on=$(ps aux | grep -c rsync)
# if someone is using rsync
# grep is also producing one entry so -gt 1
pro_on=$(ps aux | grep -c rsyn[c])
minutes=$(($minutes - 1))
((minutes--))That's a great trick, thanks! I'm so used to adding '| grep -v grep' at the end of a 'ps | grep' command, your way is much nicer.
https://news.ycombinator.com/item?id=29210222
If you have extra space, you could rclone the result of locally run borg to a cloud storage or find a service that accepts borg directly.
The lesson I learned in that class was to not use rsync, sadly.
This is a poor lesson! If your TA tripped on the stairs and broke their nose, would you swear off stairs forever?
rsync is an essential tool for anyone who works with files in the UNIX world. There are other options, sometimes, but rsync is almost-always present, and does almost-everything you usually want.
Any tool can be used incorrectly, but rsync is easy and follows the same pattern as most other UNIX file copy tools. If you can use cp, you can use rsync safely.
I'd like to have a backup system that I didn't just kludge together myself over a weekend that accomplishes both delta change detection and rotation as well as allows me to keep my data encrypted with a key only I control.
"Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there.
If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)
tl;dw: many of these adapters are so poorly made they might catch on fire. They put the (insulated) wires in and then pour in something kind of like hot-melt glue around them (to hold them in place, and/or for extra insulation, not sure). Sometimes the wires are well separated within that medium, but sometimes they're touching, and possibly the wire's own insulation is melted by the glue. There also seems to be some contaminants in the glue, and possibly corrosion from the way it was soldered.
At the end, he shows a better kind where crimped wires are put into hard plastic channels, which reliably keeps them in place without the same risk of compromising their insulation.
did your guys use shitty injection molded connectors that melted or is there another reason?
I think this is the only plausible explanation - someone had a bad experience with a really crappy connector and it stuck.