LOL that paragraph hides all of it. On prem means that someone is administering your application that does not understand it. Which means YOU are administering it but via slacking with your customers infra engineers rather than through your own control plane.
The article is right that there are customers who require this and if you want them you'll have to figure this out, but it is a huge pain in the ass and should be avoided if at all possible.
I think B2B software should be able to create manageable installations for technically competent IT professionals..
If your install has that long of a manual task list, your cloud SaaS infra is probably junk too.
Back then, learning how to manage your own PC and install software was tantamount to earning an IT certificate. Doing so for a few years in high school would be enough to qualify you for a career in IT.
So I politely disagree. The industry was never able to ship software that a non-technical person could install. Especially not B2B, but barely B2C as well.
There has to be a name for this argument-type: A leads to B, and those who seem to know only B start arguing for A because they don't remember why A lead to B in the first place. I see this all over the place.
At some companies, this is clearly demarcated from the SWE role - at others, stereotypically the "IBM mainframes, SAP and Oracle DBs" category the OP mentions, they kinda shade into each other. Which is, of course, why those technologies have such negative reputations. (There's also the Palantir route, where you have a clear demarcation but lie about it to new grads who don't fully understand the conflict between "forward deployed" and "software engineer".)
Basically, if you want to build an on-premise offering, look at the way Nginx, Postgres, Redis and similar projects are packaged and do the same.
Nope. You should look at how an app that uses all of those is published. That’s what on-premise software truly is. A combination of dependencies and your companies secret sauce.
Your own app code is just one cog in that engine.
These are orthogonal. Resourcing is no guarantee of ops competency, or a culture where ops is enabled to deliver, hence the concept of shadow IT. On-prem software is bought by orgs with low risk appetite or other compliance objectives that skew procurement towards running on prem. I have seen exceptional ops teams on a shoestring ramen budget, and I have seen dumpster fire ops teams in companies with thousands of workers and billions of dollars a year in revenue who could not get you a single virtual machine in less than 90 days even if preventing the end of the world depended on it.
There's no reason other products couldn't do this as well, even those with a complicated stack that requires provisioning cloud resources (or equivalently, SSH'ing into servers on the local network or calling private cloud API endpoints). If you can deploy a commit from a pipeline, then you can make a self-hosted bootstrappable installer for your users.
The installer could be a tiny shim running on the user's laptop that's responsible for bootstrapping the rest of the infrastructure and then deleting itself after redirecting the user to the cluster admin panel. The trick is to make it smooth, so even non-technical users can provision and eventually administer a secure, self-updating cluster as easily as they can keep their iPhone up to date. Walk them through the process of getting AWS credentials. Heck, scrape and script the login flows if that makes it smoother (it's their computer running the code so who cares?). Ask the user the minimal questions needed to fill the config. Once all that's in place, trigger your deployment scripts on the cluster, to pull images from a container registry or whatever. Wait for it to be up and running.
When the cluster's up, redirect them to its admin panel that's more sophisticated and controlled by you. Let them check for updates, install plugins, run common operations (like "restart," etc).
Then just don't ship broken code, make sure your scaling policies work (or at least that they work well enough to avoid phone calls from customers that don't make you any money). And boom you've got a nice product.
I'm not saying it's easy. But if you already have automated deployments (you do have automated deployments, right?), then it's not that many extra steps. Productizing it is hard but can be done.
I got a first-hand view into how this model falls apart - and it wasn't the fault of the customer's infra engineers; it was entirely our fault and we walked away from piles of money because we couldn't deal with the issue.
Here's my rather salty take on the issue.
The killer issue with SaaS software that attempts to migrate to on-prem offerings is that SaaS companies tend to build up cultures that are locked into the "move fast and break things" model. Simply creating an on-prem offering from a SaaS product does not generate the cultural shift needed to successfully ship on-prem software.
If it's extremely easy to ship then the development cycle can be fast and sloppy. You can ship buggy software, bypass meaningful testing, lean on your ops team/on call rotation to limp the software along, skip documentation and use humans to glue the thing together. SaaS is popular because of the accelerated iteration cycle because shipping a bug is much less difficult.
Things fall apart extremely fast when moving from a SaaS offering to an on-prem offering because _all of the developers are still conditioned for the fast iteration cycle_. Developers are still going to be targeting SaaS first because they can interact with it, debug it, and evaluate telemetry from that running system. They won't be developing in the model where they're truly freezing features, polishing implementations, and shaking bugs out of the system. This ultimately means that regular releases are more of snapshots of main rather than something that can be shipped to a customer and run unattended.
In addition, having an ops team or on-call rotation for a SaaS offering means that troubleshooting/operational guides can be largely informal or loosely written runbooks. For an on-call/ops person, if given the option between scratching together a loose note or using Slack for documentation, or taking the time to write well polished documentation that can be consumed by outsiders, operational velocity is going to drive people to follow the easier, less labor intensive. The end result of this is that on-prem offerings are not going to have a lot of critical operational documentation.
I was on the team responsible for building, shipping, and supporting an on-prem version of our software; it was one of the most grindingly miserable work experiences I've had to date. In addition to dealing with the on-prem version my team was also on the on-call rotation and I can say from first hand experience that the SaaS model was only held together with a great deal of firefighting; shipping an on-prem version exposed every single weakness we had on the SaaS version in painful detail.
The project ultimately failed; millions of dollars of deals were left on the table and my entire team was laid off because the company was not capable of shipping a reliable, documented project that could be operated without constant babysitting. The product itself had a great number of incredible features and if people slowed down and focused on improved quality rather than sheer velocity then we'd be drowning in enterprise contracts.
> Which means YOU are administering it but via slacking with your customers infra engineers rather than through your own control plane.
If your product requires that you're remotely administering your software because your customers can't operate it successfully, then please invest in documentation, validation and testing, and troubleshooting. Your customers will need it as much as your on-call rotation.
You can also sell a managed on-prem setup where you set it up and maintain it in the client environment.
It’s not a huge pain in the ass as long as the software is reasonably designed to be multi-tenancy as well as multi server. For serverless architectures creating an internal service provider to route to your serverless (or theirs).
A lot of non prem can be made easier by ring azure friendly which in turn has some ways to get on prem easier.
SaaS is just specialization of a vendor in operational software delivery, freeing end-user organizations from having to maintain and support the competency in house, and properly supporting, securing, auditing, certifying, and updating software is hard and complex at scale — an expertise that is, as you say, a huge pain in the ass.
The main driver at scale (excluding Enterprise, security sensitive and real-time apps that really should be on prem) of the abandonment of this efficiency is excessive profit taking by SaaS vendors and the overall expense being extraordinarily more expensive that the in-house expertise, or the considerable misappreciation by end customers of how hard developing and supporting a competent dc/Noc operation can be by end user customers.
I’ve seen both.
SaaS vendors move to value added pricing and nose bleed out of contention frequently because the most profitable top-market customers are who the category market leaders build their valuations and VC pleasing returns on. So there is this massive pressure to push prices and disqualify less profitable customers, leading to surges in prices for market leading solutions as SaaS companies mature.
And then there is the “don’t stack the coffee machine on the network rack” crowd, which can get pretty sophisticated. I’ve seen some impressive network operations inside of non-descript office buildings even recently. But then everyone is confident until it’s time for SOC 2 certification and realize their IT staff doesn’t even know what they should be getting audited, or they get their first successful hacker infiltration event that gets the attention of the CEO and their emergency PR group realizes they have nobody to fire but their own VP, or somebody finds the unsecured Postgres database with nmap and hits up their journalist friend who writes an article about it… suddenly it’s back to AWS or vendor soup.
And beyond those two there is also an optics problem.
IME in a lot of big companies you sell an on-prem solution and install it only to have a management change 3 years later and their entire board pitch is about outsourcing to the cloud to gain efficiency and moving off “legacy” systems and you realize you screwed up because on-prem doesn’t have the right optics to the ocean of management consultants who see your category as an easy win… unless you’ve got hardcore integration with the mainframe in the basement that runs the bean counter machines that nobody wants to touch because integration is done with an rj-11 jack and serial port adapters from 1993 and only get their support from Susan or Steve who are half retired but keep the mainframe up because they are the last people in the county who have the expertise.
It’s hard to counter these trends with on-prem. Regime changes are killers even when the on-prem solution is justified or even a no brainer — it’s tough to pitch and support on-prem for a lot of use cases and hard to keep broad support.
I remember when supermicro would drop entire rack assemblies off preconfigured with software and burned in ready to go, and they couldn’t keep it going because the cloud and SaaS trend was just a deal killer. Maybe that’s changed, but I don’t see much evidence beyond organizations at scale looking to recapture the millions AWS takes in profits. Maybe companies like backblaze and Dropbox are exceptions but they are among the companies who should run dc ops because it’s core to their business — it’s what they do. That’s not true of most companies.
Just my 2c.
I am not against on-prem offerings, in fact quite the opposite. However, a significant swath of our customers are small-to-medium size businesses, and many don’t have access to qualified IT to run sophisticated software systems. Our contractual stipulations are very lax in terms of requiring customers to keep their hardware, software, and environment in a conducive, working order; we recommend ____ but if you fall out of spec, you assume the risk. Fine, except our contracts still put us on the hook to make sure, with reasonable effort, that the software is running. That means we spend our sparing and critical resources trying to cut pared down or specially tailored copies of our deliverables and trying to log into countless boxes to troubleshoot every banal and tedious tech support issue as to why a package isn't running or won’t install (spoiler: it’s almost always environmental: GPO restriction, firewall rule we told them we need an exception for, antivirus problems, dependencies not installed, excruciatingly out of date system, etc).
These two things are not compatible especially as software evolves and becomes more reliant on other services. The problem is that early on the organization decided to bend over backwards to make sales, now we have to choose between letting this problem spiral out of control and drag us down with it, or renegotiate the contract with better/clearer requirements or ditching on-prem which will be a mess and probably result in losing customers. That especially sucks for many of those customers, because they live in areas with poor connectivity.
Moral of the story: if you give your customers enough rope to hang themselves with, you’re also measuring out a length for yourself.
Path A: Run a windows server VM and SQL Server somewhere in our customer's infrastructure. Our customer is 100% responsible for the health of these items and provides them as a service to us. Our customer has to notify us of any infra or app issues reactively.
Path B: We operate as custodians in our customer's Azure tenant. We deploy our software as a Function app and use Azure SQL. We are 100% responsible for the health of everything. Our customer is notified proactively of any issues.
To support both paths, we have two different executable project types (console app and azure function app) that consume a common library type. Only the barest boilerplate code actually changes between on-prem and cloud. 99% of the difference is HTTP Trigger function vs AspNetCore hosting syntax. Everything else is identical.
We work with small US banks and we haven't found one that is total stonewall against moving to cloud. Many are cautious but open. The #1 concern is the PII/data. We've been working with a few angles to cast that in a more realistic light. Most of our customers have had some degree of an incident with on-prem infrastructure that could have been made impossible if business was being conducted via a cloud provider.
I don't know why this went out of fashion but it works just fine. Your customers clearly don't want you to manage the infrastructure, based on your comment. Sell them what they want.
If the issue is that your system is so complex that a reasonably trained admin cannot manage it, you should probably fix that.
What our customers want is a technology partner who can adapt to their changing business needs over time. A one time cdrom+pdf distribution unfortunately will not get the job done in this case.
Our customers are paying for a business outcome. Our software is just a fraction of that puzzle.
Largely because "control your workflow" (a-la the toyota method) requires that a vendor is unable to change things on your behalf without your knowledge.
"On-prem" is about being permitted to invest your own resources into ensuring that your $thing will work during the hours it needs to work and you can pour as much or as little into the problem as you deem necessary.
If you use a SaaS you must somehow figure out how that factors into their pricing model and it's still largely their whim, and worse: they will still change things without notice.
It's also true that a lot of companies do not assume that the internet is always available.
Getting 100Gb/s to a server room is easy for 500 people in an office building.
But getting 100Gb/s to your SaaS platform might not even be possible without rearchitecting. (IE: Perforce in game dev, usually on-prem.)
* It’s much easier to do on-prem if you do a monolith, or at least a small number of services.
* You have to build and test your application against a range of OS and hardware combinations. You’ll almost certainly need to support RHEL, and probably Windows Server 20XX. Many companies won’t let you use Docker.
* Your software need to be tolerant of infrequent updates. Customers expect some level of stability in practice.
* You need to integrate licensing checks.
* You need to build an installer (that works out of the box…)
* You’ll need to agnostically support the common parts of SAML/OpenID for Auth only, and not rely on any bespoke features from a commercial provider like Auth0 or AzureAD.
Productisation, infrastructure expertise?
Why do anything?
I worked until recently at a traditional engineering industry focused startup company which decided to deploy a Python application on-premise to a big customer. From the use case it was very clear to me that an on-prem deployment of the software was likely to be on the cards, but there was nobody there in the non-technical leadership to say at the beginning "wait a second, maybe we shouldn't write this in Python for this use case".
It is.
> Why do anything?
Don't, unless it's 100% critical for success. You have to run super lean as a startup. You don't have time for ANYTHING except validating your business.
In particular if you deploy in Cloudflare’s style you can deploy workerd as a single binary and load it up with your functions.
It’s a little immature right now and persistent storage can be a problem though.
The company I'm with supports our releases for two years, most of our products release quarterly, and many of our customers are kind of annoyed because we release too often (in their eyes). We're looking at getting out of supporting the on-prem game, but personally I think it's unlikely because many of our clients are in government and defense.
When I went to work at a University as a research software engineer, a particularly memorable incident was a research group getting very angry that we were upgrading the cluster to CentOS 7 thus meaning the ancient version of COMSOL they were contractually obliged to use in a contract with Rolls Royce would no longer run at all. We ended up getting something working using Singularity but it was very painful.
I'm honestly shocked that Sentry, for example, doesn't have a self-hosted enterprise option. We're all terrified of PHI being caught in a stack trace and sneaking through any PHI-scrubbing feature. We had to move off of Confluent because CFK's pricing is outrageous compared to Confluent Cloud. It's clearly designed to nudge customers towards Cloud, but we can't use it due to contractual requirements with customers. Everyone is so anxious about breaches that even a BAA doesn't cut it anymore.
So there's absolutely a market for this kind of stuff.
Operationally speaking we built the products from the ground up to be available either way (at the time, it was a less contrarian take). The resulting cloud architecture is not multitenant (just high efficiency, orchestrated shared hosting). "Multi single tenant" I think they call it these days.
I think its an extremely pragmatic albeit unpopular choice for a B2B SaaS company.
Needless to say we don't have a Fortune 500 budget. When I see a nice, affordable app that says "and here's a Docker image!" next to the SaaS options, I am like ( ˘ ³ ˘ ) ♥
This comes back to nontechnical leadership, as business process issues tend to do.
Leadership has something cool, they think "what's cool for us is cool for everyone", and then they try to sell into a market that maybe doesn't do cool in exactly the same ways. Take something like SAP B1P or JIRA. Both established or very powerful cloud apps. You'd be tempted to think that anyone in manufacturing would want some of this . . but then you have to build everything so that it's able to be portable-ized, which adds a ton of development time, and an ongoing maintenance cost that also involves headcount/travel/US persons/security clearance.
How do you know if that cost could be justified? Well, that's where "nontechnical" comes in. The right way is matching up costs of dev and maintenance with what the industry will minimally pay. The wrong way - which seems dominant - is to use your Leadership Spidey Sense to determine whether or not to take the plunge. Then you get stuck in, you short the dev or the maintenance, and end up with a platform not optimized for either cloud OR on prem.
I was an engineer on a product that had an On-Premise (OP) offering. Some of our biggest moneymaker accounts were On-Premise and liked the product.
The downside was that we had to have a dedicated team just for On-Prem stuff. I was one of the engineers on that team. We had to develop custom management interfaces for the software that were only deployed in an On-Premise environment. It wasn't crazy hard or anything but it was time and effort.
Upgrades and installation were also a big track of work unto themselves. When you deploy into an On-Premise environment you are completely at the mercy of the clients infrastructure. One support call I did was clocked at 14 hours - there are so many things that can come up. Plus there's the whole hassle of actually getting the calls scheduled and making sure clients aren't too many versions behind.
Sometimes we had to spend a lot of time waiting on support calls because the customer didn't have their stuff together. Like sitting and waiting for a network team to open up the firewall so that bits could be downloaded from our distribution server. And of course the firewall had to be closed up right afterwards, so this was a common thing.
Another time we spent weeks troubleshooting a client installation. In the end it turned out that the client had our software talking to two different internal NTP servers which were slightly out of sync, which caused our SSO/SAML integration to break.
Database migrations were also challenging - in a cloud environment you often have flexibility to migrate things live while the system is running. But often with the on-premise upgrades, you had to take the whole system down while the migration was running (back to the 14 hour support call). This could mean scheduling the support calls at really weird hours.
On-premise also has challenges if you start selling to the US government, especially defense. There are super strict requirements that support personnel must be US citizens on US soil - so hopefully you have more than one person on the team meeting this requirement!
All in all, I am not a detractor of offering an On-Prem solution. It can be a real differentiator and it will open a lot of doors. There are some companies that simply won't buy SaaS, either due to internal or regulatory reasons. Once you land these clients it can be very lucrative and long-term. But, it almost always requires dedicated staff to support and manage.
That's not unique to either on-prem or the government, alas. Get involved with HIPAA, SOC2 or a few other funny acronyms and the restrictions around who may access production data or servers gets very tight indeed, even if you are a SAAS company.
Enterprise customers tend to have more edge cases and scaling requirements, along with a huge workforce with more difficult lines of communication. Anecdotally, they seem to need more follow-ups and hand-holding than smaller nimble companies.
An ideal customer is often one that subscribes, pays their bill, and rarely asks you for help. Your product already solves their problem out of the box without needing to go down any rabbit holes.
This is why consumer electronics products have those stickers that warn you you void the warranty by opening up the device. It's because nobody wants to deal with those customers who open up their device, break something, and then demand a replacement because the product is defective.
“Premise” means something very different, so I recommend you use “on-premises” or “on prem” if you’re trying to demonstrate competence to the majority of your audience who will know the difference.
But honestly, how harmful is it to your business unless your addressable market is mostly English majors? And even then, it probably prompts interaction from the English majors, who can finally show off a little with a correction.
At my current employer I chose to use their cloud offering. I thought, hey, it's just me here, and I don't have time to sysadmin yet another thing. A few weeks ago I needed to enable the REST API, and for them to create a KVstore. On-prem these are 2-minute processes (set a value in a config file and restart the process). In the cloud offering, you need to make a support ticket. It took 17 days, and that's only because I ended up doing half of it myself -- they enabled the REST API first, and I found I could use the REST API to make the KVstore.
I have some regrets.
On premise goes away and you can also setup private cloud to be filtered on firewall to give access to customer only or connect it via vpn or whatever else.
Biggest upside is that you still control deployment and are responsible for uptime but you have best knowledge in-house to deal with it.
Off-prem feels that way to me. It's "the mainframe" run by the off-prem vendor. I have a dumb terminal.
On-prem seems so much easier in today's corporate IT environments. Ship the Customer a VM. Use SAML for auth. Interact w/ clients over HTTPS.
It's worked out fine for us (of course there are tradeoffs but for us they're manageable). Definitely requires an investment by people who know what they're doing as the article alludes to.
You're putting the cart before the horse. The only places still making money off of on-prem installations are charging a premium for it. If a big government account comes up and offers to pay you extra for a special service contract, you're going to do it.
I get the appeal to developers of just throwing out a self-service experience with minimum commitments. But it's not a good, sustainable business model on its own.
If you aren't confident enough in your software to hand me that code in an install file, I guarantee I don't want to rely on your janky startup's hacked-together spaghetti.
Feel free to reach out at forrest@sentineldevices.com if you’d like to chat. We’re building and trying to take people on part-time if anyone’s interested in this space! Also if anyone just wants to talk about challenges in the space I’m happy to, HN usually gives really great conversation partners.
I love the idea of on-premise for small startups (or solo startups) because it shifts the operational/oncall burden to the customer.
Also "on-premise" in 2023 can easily mean something deployable to a customer's AWS/Azure/GCP account.
not premise
