Actually, it was timing out.
So I hooked up a monitor and a keyboard to this old old P1/133 with 32MB of ram that had been taken out of a dumpster to be our router to see what was going on.
A few months prior to this I had been playing around with rrd and started generating traffic graphs so that I could get pretty pretty pictures of our traffic (and to figure out when one of my roommates was streaming his porn).
I had set the cron jobs to run every 10 minutes. Initially the data sets were so small that this wasn't an issue, but over time the database got larger and suddenly the cron job was taking 10 minutes and 10 seconds, and then 10 minutes and 20 seconds... on and on until finally cron jobs were running over cron jobs.
So after about 2 minutes waiting to get logged in, I finally get the load average for the system and it was somewhere around 146.20. After another couple of minutes, I was able to disable the cron jobs and kill off rrd graphs. In another 10 minutes, the load average was back to < .70.
All this time it was passing traffic without a hiccup and I believe it had another 100+ days of uptime after this before I moved out. What a box. I miss that router.
Now, admittedly, I've never worked on big networks. But, I know what I like, and I know working on a Cisco feels like 1996, and that working on a full-blown modern UNIX/Linux system feels like home. And, fast network gear is available for Linux and the BSDs. 10Gb interfaces that work with Linux/BSD are not difficult to find, and 1Gb interfaces (sometimes several of them) have been standard on every server and barebones rackmount on the market for a decade.
I understand there are special purpose boxes that do things no off-the-shelf commodity box can do (I don't know enough about high end networking to know where to draw that line), but I honestly doubt 95% of business networks need them. And, yet, a huge percentage of business networks that have real routers (rather than little wifi routers from LinkSys/NetGear or whatever) have a Cisco of some sort, even when they don't have anyone on staff who knows how to work the thing. I've walked into several businesses that had expertise in house for everything except their router, which is also weird...why pick a Cisco router for your little ~100-1000 node network, if you've got Linux or BSD guys hanging out in the server room every day? They'd call someone and sit around waiting when their network was acting up.
I learned how to use Cisco routers just so I'd be able to replace them easily (enough knowledge to read the routing tables, firewall rules, etc., but not enough to do much useful), when I found situations like that.
Or with a generous budget, you can do what BT have done with their 21CN project: For each macro-block (access, aggregation, core), have two redundant networks each one built by a different set of vendors.
The answers is the same as in software industry: Configuration management.
Plus it makes then harder to develop corporate cultures of deploying well known patterns that makes routers best used.
Plus it increases the surface of vulnerability for government backdoors.
In real life, there is a trend for going BSD box for cost/formation/stability/security reasons in the ISP world ... especially in non US countries that are getting kind of thinking the USA is becoming the new USSR in terms of systematic spying on every one.
These Europeans really don't get it, isn't it? They never were in the middle of cold war spying insanity where both power would puppet so called terrorists groups to make sure citizen would vote the right way under the fear.
What a bunch of stupid persons not believing that business companies are incorruptible and think they have incentive to betray them in exchange for gvt subsidies/regulations.
Why on earth were you running a hosting business with Cisco 2950s? Those are wiring closet desktop switches with tiny buffers. That alone probably caused a lot of pain. You didn't know about it, but they did.
Second, the Juniper J4350 is a software router based on BSD. There's no special hardware in there like ASICs. It's just a PC.
JunOS (historically) is based on BSD. They're moving to Linux now.
I love OpenBSD for small projects too, but let's admit that there was some bad logic in this decision making process. This could be called "how the mx10 saved my ass because I went with undocumented, unproven open source project that has been on github for 3 days" if the technologies were reversed.
Do you have details on this? Given how active they are in the FreeBSD community, I would have thought to have heard about that already.
QFX5100 runs a Linux KVM hypervisor with the traditional FreeBSD based Junos running as a guest. Same with the new x86 based SRX (SRX1500). There are more moving to this model.
The x86 based SRX is the most interesting with the forwarding daemons running on the hypervisor OS and the Junos based VM is just for management and protocols. It's not that hard to see that management and protocols can be easily ported.
There are more details that are locked behind an NDA.
Later on, I was working a network overhaul at a private college dedicated to video game creation (DigiPen). Each student was provided a desktop with their own roaming profile. These profiles got huge because there were things like 3D Studio Max projects being transferred to whatever computer the student happened to be at during that period. So we're talking about hundreds of students signing on at about the same time & massive amounts of data being transferred in a very short time. The old network topology was melting down on a regular basis causing the entire campus to come to a halt.
We fixed that by putting in a pair of -- you guessed it -- OpenBSD/PF routers. Now this wasn't some "oh crap" kind of bandaid fix; we had planned this all out & implemented over the summer months when there were no students. It was all planned out (the only problem was that the original plan called for FreeBSD & PF, which crashed -- once we had OpenBSD running, it was 'set it & forget it.')
Maybe I'm just too dumb to know the difference, but those 2950s and OpenBSD routers helped me sleep a lot better at night... they were perfect for what they were intended to do.
And as more than a decade of experience since has proven, the right tool for the job is often the one that gets scoffed at as a "toy" or good "for small projects."
One of my first bosses thought his Dell switches were awesome until I showed him all of the drops.
Also, roaming profiles were a great idea in principle and a horrible idea in practice. There's a good reason those things went away circa Windows 2000. ;)
Do you have a source?
[0] http://www.enterprisenetworkingplanet.com/nethub/juniper-emb...
That said, you can buy a ton of good gear cheap on the secondhand market. This is extremely popular in the lower end of the hosting market. It's cheap. It works. It's a known quantity. You can get secondary support from companies like NHR. You don't have to buy gear new.
Used gear was probably a much better decision than new gear here.
1: https://en.wikipedia.org/wiki/Comparison_of_BSD_operating_sy...
That being said, both OSes are free, and so is VirtualBox, so why not spin up a couple virtual boxes and give them both a try when you have the time?
The only time it failed -- the ONLY time -- was due to a full hard disk. Why was the hard disk full? Silly me, I forgot to implement some sort of logging discipline and a couple years' worth of logs of attempted exploits (mainly Windows NT exploits) glancing harmlessly off OpenBSD's hull had filled the disk.
Doing Enterprise IT Arch work, I turned down more than one hot new system and put others through significant lab work, to avoid this fun. It's hard, it's expensive and it's why I wish more hardware was open.
> With the new CPUs tending to increase the core count while the CPU frequency goes down, we'll get less and less "CPU cycles per packet". For example E5-2699v3 45M is 18 cores@2.3GHz. 10Gbps with 256 byte packets is achieved with 4.53Mpps, this makes ~500 clocks per packet on this CPU. Now make this 100Gbps and you get 50 clocks. You get this budget to receive, process and send the packet. 50 sounds pretty low to me.
Numbers refer to using only a single core. That particular CPU has 18 cores and 36 hardware threads.
https://groups.google.com/forum/m/#!topic/snabb-devel/pDd_uB...
(On a hardware note, increasing cores and decreasing frequency makes the CPU look more like an ASIC :) )
> The only reason I use “real” network gear anymore is ... for core routing 10gbps links.
There's some progress, though (like PF_RING in Linux and netmap on FreeBSD).
still, you need some tweaking or things like dpdk/openonload/vma/etc...
Honestly I wonder, who the hell buys these overpriced networking appliances? Similarly with expensive proprietary databases, or ESB systems. Are they just leftovers of a different time (note article is from 2007)? Is it a case of management buying from the salesman who bought them a nice dinner? Some place where these things generate actual value that I'm missing?
Appliances promise all sort of fancy features that might be hard to exactly replicate yourself (if you actually need them often is a different question, but often networking isn't exactly in a position to say "no" to such requests). Although there is more and more a trend of manufacturers also offering the software for virtual environments, many appliances are x86-servers anyways.
+ as the article mentions all those cases where you need specialised hardware, e.g. routing/switching at high speeds, or working in special environments.
If you buy Cisco and a Cisco specialist, you'll be buying Cisco for as long as the specialist is employed, regardless of any leaps or bounds Juniper et al. make.
Same deal with databases. Oracle is often a smart choice, but so is Postgres, SQL Server, Cassandra, Redis. If I've hired a DBA with 10 years of Oracle experience, the chances are that any move away from Oracle will rub against "Why hire an oracle specialist if you're not going to use them?".
My hope is always "Because they're smart, genuinely understand storage and retrieval and will learn the other systems faster than a smart generalist". But that isn't how it plays out.
I would hope that hiring a specialist means that, at some point, they come to you and say "So, I think I can save us money and increase uptime if we move to X". It happens in software all the time. Its hard to keep devs on any one particular platform before they get twitchy (also a problem).
In my mind buying Cisco is like using Heroku, Rails, Ember. All fine tools, that help you get a job done quickly whilst you figure out what your real problems are but shouldn't be seen as the final solution.
people that need to switch more packets than what you buy off of newegg can handle.
edit: oh that and when you have a network of thousands of routers and switches that you want to centrally manage in a way that "mass ssh" doesn't scale to and you just want to buy the management software from someone that has written it before and not spend a while getting the software wrong before you get it right.
if you can count the number of routers you need in your head, you probably should just be using beige boxes with a commodity OS...
Running Linux/*BSD/Whatever on off-the-shelf boxen which you can pickup everywhere (and have cheap spares) and fix yourself will let you get back online quickly.
Google, for example, is able to use commodity hardware because what they are routing works with a non-traditional setup...specifically a Clos network. That works for them only because their app is designed in a way that it can served up like that. It would not work for the typical kinds of traffic that you see at a normal company.
At a certain scale, barring unusual exceptions, you really do have to go with someone like Juniper/Cisco, as the ASIC based acceleration and other features aren't there in the commodity world.
There's certainly some things in progress that may change all of that, like Intel's DPDK, QuickAssist, etc.
https://code.facebook.com/posts/843620439027582/facebook-ope...
I've seen this many times and it will continue to happen.
What's latency like on commodity hardware vs dedicated equipment? I've no experience but would expect the dedicated hardware to be better. BSD is a monolithic kernel, so you'd have the overhead of passing through the for each of the routed packets, right?
