Nowadays I build things so that they move and I have moved things about a bit so I know they work.
Quite. I'm old enough to remember machine uptime being a badge of honour.
However, being older and not really wiser, I look for service uptime these days. Yes we did have similar back in the day, that's why MX and the like DNS records exist.
Old school clusters were pretty esoteric but the lessons were learned (split brain n that) and that's why we still argue the toss with kiddies about why a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
I don't care that VMware glossed over the whole two node HA cluster thing years ago with a massive bodge. They were wrong then and they are probably still wrong because that nonsense is probably still baked in.
Sorry, slight digression.
High uptime implies no patching. We all love patching.
The more you know!
>a Proxmox cluster with two nodes is fucked and why we recommend an additional "witness".
Reminds me of the three Magi from Evangelion: https://magi.kinta.ma/
For a physical machine which you can't easily copy, it's a different story.
Mostly ~/apps/appname, where each appname has a docker compose file, and the data directories mounted under appname... I can compose down and (s)ftp the data out for hard archives or to move a site/service. I had been running a few VMs under a dedicated server, but switched to separate VPSes on OVH. Only gotcha with OVH is if you want to run mail, you want to avoid the local zone VMs that don't allow mail hosting.
YMMV
I think FreeBSD has some interesting technical merits, but like it or not Docker is the default for a lot of open source software, and I have neither the time nor the inclination to translate everything to freebsd jails.
I was a bit rougher though: I just took a backup, then let my Hermes agent (Gemini 3.1 Pro) loose on it. It upgraded everything that needed to be upgraded, patched what needed to be patched, then proceeded to migrate everything to it's most recent equivalents. After that, a fair bit of server hardening was carried out followed by debloating of unused services. Likely would have continued to procrastinate doing this if it wasn't for AI support.
- PM2 was buggy on FreeBSD, which I used to manage my processes
- An alternative, using `rc.d` to run daemons was just so hard to get logs working.
- The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
FreeBSD does include this! It's implemented using IPFW instead of PF. Check out `firewall_type` key in `rc.conf`: https://cgit.freebsd.org/src/tree/libexec/rc/rc.conf?id=8e08...
For a very easy single-machine firewall, one could set `firewall_type=client` or `firewall_type=workstation` if you want to host anything. For the latter, `firewall_myservices` and `firewall_allowservices` control what ports are enabled and who (other networks/IPs) have access to them.
For a very simple NAT gateway, one could set `firewall_type=simple` and then `firewall_simple_(iif|inet|oif|onet)(_ipv6)?` to configure the ISP-side and internal-side interface names and IPv4 and IPv6 network ranges for each.
For more details and to see exactly what each option actually does, check out `/etc/rc.firewall` where this is all implemented: https://cgit.freebsd.org/src/tree/libexec/rc/rc.firewall?id=...
For supervision?
> - An alternative, using `rc.d` to run daemons was just so hard to get logs working.
The unix way is to use logger(1) If you only want some simple message, or redirect to files using newsyslog(8) for managing the sizes of the files.
> The firewall required too much self configuration to get it right with all the best security practices (ie. What does one do with ICMP.) I was missing something like a template with the defaults that come with UFW, for instance.
I would recommend The Book of PF[0]. While FreeBSD has syntax difference with OpenBSD's pf, this should give you enough insight on how a firewall operates to get a sense of what rules to write.
I actually just did this for a web application on FreeBSD 15. It wasn't too bad, but I used `daemon` to send stdout & stderr into syslog and then I just added a syslog config to send that to it's own file. Roughly, I did this:
# /usr/local/etc/rc.d/app:
# PROVIDE: app
# KEYWORD: shutdown
. /etc/rc.subr
name="app"
rcvar="app_enable"
load_rc_config $name
: ${app_enable:="NO"}
: ${app_user:="www"}
: ${app_database_url:=""}
app_command="/usr/local/bin/app"
cpidfile="/var/run/${name}/${name}.pid"
pidfile="/var/run/${name}/${name}d.pid"
logfile="/var/log/${name}.log"
command=/usr/sbin/daemon
command_args="-P ${pidfile} -p ${cpidfile} -S -t ${name}-super -T ${name} ${app_command} start"
start_precmd="${name}_prestart"
app_prestart()
{
# ... Set environment variables here with EXPORT, using values from rc.conf
export DATABASE_URL=${app_database_url}
# This is also a good place to use install(1) to create
# config files and log directories if they don't exist
}
run_rc_command "$1"
After sending the log messages to syslog, it's a simple matter of routing them to the desired destination. That's easy enough to do by creating a file as follows:
# /usr/local/etc/syslog.d/app.conf:
!app
*.* /var/log/app.log
You can additionally set up log rotation with:
# /usr/local/etc/newsyslog.conf.d/app.conf:
/var/log/blog.log 640 7 * @T00 Z
For a while I used CentOS 7 on all of those small VMs, because it got security updates for a really long time. With minimal risk of breaking things on updates.
PS: after a bit of research Alma/Rocky Linux are probably the best choices for now. 10 years of support. But are they maintained well?
Then I figured that version upgrades were miserable, not just because I had painted myself in a weird corner with ungodly packages mix-ups, but because the upgrade path was always best-effort. I think I gave up during the 6 to 7 transition, as I realised that all I needed was fedora: with yearly or half-yearly updates I have no need to fight the distro's packages: stuff stays current and in working order, major distro upgrades go smoothly, downtime is minimal. I'm not considering going back to any "server distribution" ever.
LTS distributions are great if your applications don't change (much), or are explicitly compatible with the specific distribution you run them on.
Alma has a few affordances as it's no longer RHEL source compatible, which means it could ship priviledge escalation fixes with new kernel updates faster.
Rocky responded with an extra, optional to enable, security repo to provide mitigations to the exploits while waiting for RHEL to downstream.
Look pretty well maintained to me. If only judging by recent events.
The manuals, indeed are good, though for more esoteric issues I land too often on a gated answer page.
I don't care much about being fully RHEL compatible, or no ABI changes at all. I just want a system that gets security fixes quickly with as little chances of breaking things as possible.
Fedora CoreOS in particular has had a good track record delivering patches quickly. Like for CopyFail was pushed to the stable channel in about a day, IIRC, but the patch was already available within a few hours of disclosure in the "next" / testing channel.
Talos and Flatcar are also worth considering if you want an even smaller attack surface, from what I heard they weren't even affected by CopyFail.
Went with openSUSE MicroOS myself, it updates and reboots almost daily so I can be pretty confident my server is healthy and it's atomic so if something does break and I don't feel like dealing with it, I can just click rollback button from cockpit and deal with it whenever I have time.
you do not belong in IT
There are things that need 9^5 and there are things that don't. If someone backs up their application configs and data properly, then the only thing that really matters is a proper backup strategy.
All my critical files are backed up periodically (manually) via rclone to S3 glacier, and all my services are documented in dokuwiki. If you use ansible or want to store configs and installation scripts, a private git repo would do well.
After that, I don't see a problem running rolling or short-support OS like Fedora Server for application hosting.
My own service images already have a script that runs daily that pulls latest git updates and builds fresh images.
I've upgraded Debian stable (both pure and with some cherry-picked backports) and Ubuntu (non-LTS and LTS) systems in place and rarely broken anything, for years and years. When stuff has broken it's been a quick google and then slapping myself for not having read the upgrade guide.
I do generally wait about 2-3 weeks before upgrading, giving time for them to catch stuff that was missed until the great masses were set loose on it.
Not the OP, but I support Ubuntu as desktop and server OS for an engineering collage and have for 10ish years. Some LTS upgrades don't require many changes (mostly minor package name changes) and some take months of work to get rolled out (mostly for workstations, the server upgrades are usually quick.). Not everything gets upgraded every new OS release. If we had to upgrade everything every 6-12 months it would eat up a significant amount of time for our small team.
I've had issues with Ubuntu/Debian upgrades more than once. Some third party binaries breaking with the update. Or some specific config tweaks that break, because the structure of /etc changed too much.
For some small VM with a specific purpose I prefer a distribution that changes as little as possible for as long as possible. Less work, more uptime.
The naming conventions drive me crazy as well. When you deal with 2 things that have dumbshit naming conventions, like ubuntu and ROS, its really obnoxious to pretend to case enough to keep track of.
For updates, Debian and Ubuntu are great. For upgrades… not so much for me.
RHEL is definitely the most stable major distribution. Alma and Rocky are essentially downstream clones of RHEL.
I have been running NixOS on several servers for more than a decade. No reinstalling, upgrading, or any breaks whatsoever.
I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden.
NixOS is not rolling release. This is a common misconception. You can use the unstable channel, which is a rolling release, or the regular channels which get released twice a year. These are really stable and move very slowly. You can also mix and match, running software from different channels.
> I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden
I have done this recently as I kept an airgapped machine, which I decommissioned, connected to the Internet and updated to the latest channel. Everything worked just fine. I just had to change a couple of options in my configuration which had become outdated. Nix is functional, so it's much less prone to all stateful issues that plague other package managers.
It's very rare to find something that prevents you from directly updating. Nixpkgs tries very hard to no require new Nix features, so it evaluates with even Nix versions from a decade ago. Also, NixOS options and packages are frequently changed, but the automatic migrations (mkChangedOptionModule, mkRenamedOptionModule, alias, etc.) are never removed in practice.
Since the binary cache has never been cleared since its creation (2002?), it should actually be easy to install a super old NixOS release and upgrading it to the latest to see what happens.
By the way, there are LTS versions of NixOS, just not officially supported. See https://docs.ctrl-os.com/.
Keep in mind: this was just a simple rest server. But for some reason it needed to (nondeterministically) build the word from scratch to send that single request.
I’ll take a docker system thank you.
It has been the most headache-free Linux I've used, simply because I'm less scared to play with and fix stuff. The fact that rollbacks are trivial and snapshots are automatic, and since everything is declarative in a text file anyway, I am way braver. If I do something like screw up the video driver, or the wifi driver or make it so the system doesn't boot anymore, all I need to do is reboot and choose a previous generation.
The main reason of a LTS distribution is not having to play around and fix stuff. Install something once, and it keeps running without any changes, but still gets security updates.
However, I have had one machine become unbootable as it could no longer mount its encrypted disks after an upgrade, forcing me to mount a rescue image remotely, mount the disks manually, lift the data out, and do a complete reinstall (migrated the box to OpenBSD at that time). Similarly, NixOS once messed up systemd (or vice versa) so badly that I could not even reboot without forcing a power cycle. Lastly, I have had a package break for my use cases by maintainers enabling so many custom flags by default for a package that they enabled one I have never seen enabled by any other packaging team and that then broke RTSP in "funny" ways. Ubuntu did tend to break things like graphics between releases at times back when I used it, but I have never had any other distribution or operating system throw curve balls like the three things I mentioned here.
My general impression of NixOS is that the core is solid, but that nixpkgs just has such a large number of things that it supports that the maintainers struggle to test them all and can not anticipate the interactions between all the packages and options. The default Julia package being so broken that it produced incorrect mathematics due to nixpkgs' insistence on allowing you to swap out the Blas library and also having turned off the unit tests for example springs to mind. This was shipped to end users for a long time before I noticed it by accident by enabling the unit tests and stepped in to clean it up. It all feels very "Gentoo", which was indeed an inspiration for NixOS by the way.
Now, return to that last sentence in the first paragraph that I wrote about feeling empowered to tinker, ultimately, I feel like you should try to resist that urge as it is what pushes you into the untested fractal of possible configurations that NixOS allows you to explore. My other main operating system is OpenBSD, where the mentality is "Stick to the defaults or suffer the consequences"; with NixOS, I feel like everyone's box is more or less a tailored suit, which comes with both its ups and downs.
But I have had, uh, non-trivial breakages happen also when I upgrade the system itself to the next yearly release. Non-bootable kernel kind of breakages.
But I will give you that I can just boot from the generation before the upgrade, and it works again. So there's that :)
ELTS is 10 years and paid. It's great that it exists, but not relevant for my toy projects.
I haven't used a lot of different distros, but for me, Debian has been a good balance of those factors. You may need to do more upgrades per decade, but the ones that you do are more liable to go smoothly.
Just my 2¢ on the topic (:
It seems to me in the 2020s that 5-7 years is plenty of support for a single OS release, and that OS support teams should be nimble enough to roll out new instances and migrate data at that cadence.
Either the 1-2 hours is a drop in the bucket compared to what you spend on it anyway (like a blog you still regularly update), or you don't actively update the project but still care enough about it to spend half an evening every few years, or you should just admit you don't care about it enough anymore to do even that. In the last case just delete the project.
For a box that sits in a corner doing its joband you don't want to pay attention to it's not a good choice IMO. On a desktop you want the latest of everything on and you have time to keep up it's the best.
Note, I did try Artix, but when it broke last week after a restart (in which evidently something had gone wrong with an earlier kernel update), and I had to pull out a rescue ISO, I decided I didn't want to mess with that. I switched that machine to Devuan, but the jury is still out for me. I don't have any major complaints, but I'm still in the burn-in phase. :) I'm running Arch on a laptop, but they have been a bit hostile in the community with censorship, so I'm just waiting for a free weekend to blast it and put something else on. I don't want political drama in my software.
This all comes at an interesting time, though. This is the first time that I purchased a new laptop and didn't even let it boot into Windows, but instantly installed Linux. And everything "just worked". And now that I'm excited to try Linux, so many of the big players are embracing the steps to erode privacy (AI everywhere... age attestation/verification... telemetry on by default...). It's sad, and I'm just going to "nope" out of any interactions with them.
I use Debian now much more. With all the supply chain attacks, Debian Stable feels like an absolute jewel, even if there are always a few packages I need to handle separately because I want or need a more updated version. But I love the old school no-nonsense engineering ethos.
You've been misled.
That's mostly problem of Arch/Artix, they're the bleeding edge, which is not always the best for stability. But no one said that rolling distro is supposed to always ship latest versions of everything. I've been using Void Linux past months - and while it's a rolling distro, it runs LTS kernel (mainline is also available) and maintainers are more focused on stable versions of apps than on faster updates.
I've also got a couple of test systems running Chimera - going to wait until it hits stable before relying on it too much though. Experimenting a bit with AerynOS too.
It used to be hard to find dedicated servers or VPSs with any of the BSDs, I think I settled on Panix.com or something?
Before that I remember some company called 15MinuteServers (NAC?) out of NJ I think that offered them. Just kind of rambling down memory lane at this point though.
I have FreeBSD with Hetzner and OVH. I’ve also used Vultr in the past.
Probably, it's because of ZFS ARC (Adaptive Replacement Cache). It's similar to Linux's page cache, can be claimed back any moment and different tools name it differently: https://www.linuxatemyram.com/
kqueue was a huge win too.
A huge thank you to the FreeBSD developers. I ran my first company for 15 years on FreeBSD with incredible uptime and resilience.
Be careful if you have anything that autostarts that reaches out
Also, debian/ubuntu systems can easily be setup to auto update and reboot on a regular basis, leaving you manual maintenance only for the larger version upgrades.
A reboot doesn't break anything. Bugs do.
Any time I had a regression after a kernel update on a linux distro I could boot it on a prior version from the grub menu. Any time I had a regression with a software package I could rollback to a prior version. Rolling back updates is a problem that has been solved for decades, at least on linux systems.
The key with unattended upgrade is you want to have decent monitoring to make sure you never run out of disc space and do not figure it out weeks later if you have had an issue.
In the end it is easier to schedule a weekly reboot window if packages have been updated. You aren't running a single server if you are interested in 99.99999990% of uptime anyway.
Imho a regular reboot is good practice: you are more likely to remember what you did a week earlier if an app/service fail to restart after you tweaked a config file than if it happens months later.
There is no reason to be afraid of reboot when they happen on a regular basis.
If you're a university student and want to learn OpenBSD administration or how to host your own blog, or just how to use VSCode, these are all extracurriculars.
But that’s ok. The author isn’t claiming FreeBSD is way better than Linux. It’s just a comparison of what he had vs what he has now.
For example, look at these benchmarks from 2003[1]. The newest benchmarks I could find[2], [3] point in the same direction.
[1] http://bulk.fefe.de/scalability/
[2] https://matteocroce.it/blog/freebsd_linux_networking/
[3] https://www.phoronix.com/review/freebsd-15-amd-epyc-linux
Both 1 and 2 are old and irrelevant, the 3rd has no clear winner?
I've been using ZFS for about a decade on several systems and can't say enough good things about it: rock solid, feature rich and easy to use are the top benefits.
It really needs more love!
Knowing you way around *BSD how check things, mount a USD drive, collect data & or evidence when need arises is well worth having beyond just surviving bare Linux skills. BSD's are alive and kicking on commercial appliances and devices.
But I personally don't really use OpenBSD for security. Sure, good security is important, but for a simple person, I think any updated OS, with good passwords/pubkey auth, good config, being careful etc etc... Is good enough.
OpenBSD is a coherent OS. It's simple (for geeks), and you can use it, by just using the documentation. There's no need for looking up tutorials really, because you don't have to read a 500 page book to understand certain tools, just basic man pages and some computer science knowledge.
With OpenBSD, you go back to a simpler time. Without all the hectic bullshit and an ever-faster pace of constant changes that makes our lives worse, rather than better. The only useful thing it can't do is gaming - with some exceptions, for that I use Windows.
Talking about ports again: OpenBSD comes with batteries included. Not everything though, but you don't really need the ports that much for just a server, if you aren't doing anything complex.
I also use it on desktop/laptop systems, booting it up (yes, it's relatively slow...) always gets me to a state of tranquility. The good ol' days. Maybe that's just my type of brain, but life needs to become simpler again.
Really, what post-2010 information technology has really improved our well-being? Can't think of much.
OpenBSD may have to many rough edges for a desktop system though, even for most geeks. But for those, there is FreeBSD (have it on one laptop). Just get a well-supported machine for that.
I just setup Semaphore the other night which adds a web UI to manage Ansible playbooks, it works like this:
1. I host my own Forgejo git repos
2. Semaphore is granted access to the Ansible repo
3. FreshRSS notifies me when a service I am running has new release
4. Check the release note, then run Semaphore to run the ansible-playbook
I could fully automate it all but I have the need to read release notes.
As for the OS, they are Debian 13 Netinst and fully local only, I could run them until the services can no longer run, which the ansible-playbook can spin up another LXC container running Debian 14 or whatever.
The goal is to automate everything as much as possible.
My guess would be that fastfetch probably reports actual memory usage while btop probably reports the total usage of all processes. The former is probably higher because of things like filesystem caching
The other Hail Mary reference is on top of HN today.
Well done Andy Weir.
Sleek like apple
Discoverable like android
Worksuitable like windows
Our usp is:?
The applications run just fine, but I don't even know where to start. Apparently I coded them directly into the server, no dev machine!
[0]: https://cdn.idiallo.com/images/assets/daily/98/old_servers.j...
It's an incredible journey to take--whether you stick with it or not. Migrating to FreeBSD gives you new eyes into what Linux was, is, and the awesomeness of FreeBSD that is so hard to articulate; like describing the color blue. It must be taken as a whole to appreciate it; and I'm not just saying the OS, it's commands, kernel features, but, the end-to-end compute experience, over time.
If I could draw an equivelent, it would be like when Djistrka savagely destroyed the GOTO statement with a single, short, paper. It took a brilliant mind to articulate that and there has yet to be such a mind to describe the beauty of FreeBSD. So, the best I can do, is just to challenge you to try it.
If someone is willing to use something like Hugo instead of garbage sites like Medium why not use a VPS? For many people working in tech $10/month and free are the same thing.
And yes you can have preferences to keep things simple while others can make something unnecessarily complex. For personal projects this is fine and part of learning. If you had said “I much prefer… because…” it would have been fine but you said “objectively inferior in every meaningful way” which ignores people’s subjective preference for over engineering hobby stuff for learning.
On the other hand, dedicated servers are more fun, even if the cpus I get for $35/month are ancient. Is it $30/month fun? Probably not, but it's near zero given my situation.
At least I'm sensible and don't have an actual colo space to visit.
In the unlikely event I go viral, I'm pretty sure my server can manage serving https at 1gbps, and that's plenty. Maybe TLS is too much though, the cpus are too old for AESNI.
There's additional concern with tying your work to something like github it makes it more of a pain to pull it off and put it somewhere else.
I'm not really sure what you mean by objectively inferior. It's trade offs like everything in this field.
As far as harder, I don't really think the lift for a personal VPS is that high. Again it's a fun hobby project for most. It's fun to run your own stack.
If you want to opt into the github cloudflare goodness that's fine they're good services but I wouldn't say it's better or degnegrate others for not doing that.
That's great if that's what you want, but you are commenting in a thread full of people gleefully spouting off about decades-old installations that they self-admittedly have “no idea” how to upgrade. Most people in here would be better off if they admitted to themselves that they are not actually taking advantage of the opportunity to learn, and are instead undertaking a liability.
You will never win this crusade, because there are too many people here who know from experience a VPS is neither expensive, nor under-performing up to millions of users a day, nor hard.
For 5 EUR you get 20 TB traffic on Hetzner.
You can put it behind cloudflare for free.
Many of these small VPSs can be had for less than a couple bucks a month. Tons of popular influencers run their own machines for their blog.
insinuating that it's unsafe to run your own machine is insanity. I don't understand this mindset of being scared to run your own stuff. Especially if you're doing doing it at such a large scale there's nothing wrong with doing it with nginx and a linux box on a vps. You'll learn a hell of a lot more and be fine. At the end of the day it's a computer. We've been hosting websites since the 70's. With the advant of cloud compute is easier than every to run your own.
(edited to be less mean)
