Ask HN: How to avoid over-engineering software design for future use cases?

> engineers go extreme in designing things/code for future cases which are not yet known

They're afraid.

Fear: If I don't plan for all these use cases, they will be impossible! I will look foolish for not anticipating them. So let's give into that fear and over-architect just to be safe. A bit of the 'condom' argument applies: better to have it and not need it than to need it and not have it.

But the reality is that if your design doesn't match the future needs really well, you're going to have to refactor anyway. Hint: there will always be a future need you didn't anticipate! Software is a living organism that we shape and evolve over time. Shopify was a snowboard store, Youtube was a dating website, and Slack was a video game.

So my answer: relentlessly cut design features you don't need. Then relentlessly refactor your code when you discover you do need them. And don't be afraid of doing either of those things because it turns out they're both fun challenges. The best you can do is to try to ensure your design doesn't make it really hard to do anything you know or suspect you'll need in the future. Just don't start building what no one has asked for yet.

Something I've recently realised after having listened to Kevlin Henney talk about software engineering is how much of the existing knowledge we ignore. The early software engineers in the '60s and '70s were discovering pattern after pattern of useful design activities to make software more reliable and modular. Some of this work is really rigorous and well-reasoned.

This is knowledge most engineers I've met completely ignore in favour of the superstitions, personal opinions, and catchy slogans that came out of the '90s and '00s. It's common to dismiss the early software engineering approaches with "waterfall does not work" – as if the people in the '60s didn't already know that?! Rest assured, the published software engineers of the '60s were as strong proponents of agile as anyone is today.

Read this early stuff.

Read the reports on the NATO software engineering conferences.

Read the papers by David Parnas on software modularity and designing for extension and contraction.

Read more written by Ward Cunningham, Alan Perlis, Edsger Dijkstra, Douglas McIlroy, Brian Randell, Peter Naur.

To some extent, we already know how to write software well. There's just nobody teaching this knowledge – you have to seek it yourself.

The junior developer sees a pattern and thinks why not make this a bit more generic so it handles similar cases. Oh but these cases fit a larger pattern.. and eventually you'll have written yourself an entire framework. This is very valuable, for the experience. Not for the framework, which will never be used. But you should go through it.

The mid-career developer knows this from experience and sticks to the minimum necessary to meet the requirements. Fast and efficient in the short term.

The senior developer mostly agrees with that but also knows that there is nothing new under the sun and all software ideas repeat. So from experience they can selectively pick the additional implementation work to be done ahead of time because it'll save a lot later even if it's not needed yet.

As an aside, this is one of the many reasons why I interview based on discussing past projects and don't care for algorithm puzzles. Unless I specifically need an entry-level developer, I'd prefer to have the person who has written a few silly (in hindsight) complex frameworks and has painted themselves into a corner a few times with overly simplistic initial implementations. That's the person I know I can leave alone and they'll make sane decisions without any supervision. The algorithm puzzle jockey, not so much.

It really depends on what you're doing.

If you're writing code for a device that's going to hang out in the forest for 10 years with no updates, write just enough code to solve the problem and make it easy to test. Then test the fsck out of it.

If you're writing code for a CRUD web service that you know will get rewritten in 10 months, write just enough code to solve the problem and make it easy to test. Then, test the fsck out of it.

If you're writing an Enterprise app that will be expanded over the next half decade, write just enough code to solve the problem and make it easy to test. Then, test the fsck out of it. You simply cannot know how your code will have to change so you cannot design an architecture to make any arbitrary change easy. Accept that. The best you can do is write clean, testable code. Anyone... anyone who thinks they can design a system that can handle any change is flat wrong. And yet... time and again, Architecture Astronauts try it.

When I was young and fresh, I wanted to cater for all sorts of future possibilities at every turn. But what if things change this way? What if things change that way? I came to realise that when things change, unless the design change is trivial (allow use of database Y as well as Z, etc) then you probably haven't anticipated the way in which it is going to change anyway. Better to have clean, straightforward code than layers and layers of abstraction and indirection, to the point where it's hard to tell what's actually going on. You'll probably find that when change does come, your abstractions are an annoyance and a straight-jacket.

A junior engineer recently presented me with some completed work. There was a data schema change on one interface and an API version change on another. He had created a version of the microservice he was working on which could be configured to work on either the old or new version of each of these, with feature flags, switching functionality and maintaining the old and new model hierarchies for both. He viewed this as an achievement. While technically it was, the result was code bloat and unnecessary complexity. The API upgrade and schema change were happening at the same time, and would never be rolled back, his customisability was a net negative.

Code is a liability. Be sparse. Build with some flexibility but overall just build what is required and when the future comes, change with it. Don't build to requirements you can imagine, because the ones you don't will kick your ass.

The short answer is: don't design for future use-cases.

Period.

Instead, only build what you need to solve the problems you have in-hand, but do so in such a way that your software and your systems are as easy to change as possible in the future.

Because you very, very rarely know what your future use-cases really are. Markets change over time, and your understanding of the market will also shift, both because some of your assumptions were wrong, and because you can never really have a total understanding of such a complex phenomenon.

Your business needs will change as well; your top priority from 2019 is likely very different than it is today.

That is why you build to embrace change, rather than barricade against it.

As to how, I'd start with Sandi Metz's 99 Bottles of OOP: https://www.sandimetz.com/99bottles

Learning to write readable code is also pretty important; Clean Code is a good starting point (https://amzn.to/3168z3A), but I'd be keen to know of any shorter books that cover the same materials.

Growing Object-Oriented Software Guided By Tests (GOOSGT) is a good read as well: https://amzn.to/3du1sEL

32 comments so far and no mention of the word budget. There's a great analogy between software engineering and construction. Does your organization build skyscrapers and gorge-spanning bridges? Or does it build driveways and swimming pools? Commercially developed software consumes capital to get something in return. Are the people spending capital budgeting for a driveway or for a skyscraper? Must it be done this month or in two years? Sure, those are false dichotomies, but they illustrate the point: it is desirable for the bosses to clearly define engineering spend.

Over-engineering can be avoided by carefully sticking to a budget.

For a lot of developers, there's a trade-off between rationality (in the sense of ROI) and feeling good. It doesn't feel good to make every engineering decision against a budget. My dopamine levels [0] skyrocket when I visualize making some component generic, or future proof. I come crashing back to earth when I realize the budget is for a driveway. Budgeting earns the bosses / customer / capital a better return. Engineering for future use cases or making things generic (or the mere anticipation thereof) is an easy way to get a massive hit of neurotransmitter that makes you feel good.

[0] Not a neuroscientist. But I think it's useful to label that spike of feel good and motivation. It may have nothing to do with dopamine.

Don't design for future use cases unless they are known in detail (in which case they're not really future use cases anymore). Things you don't know, will change. Your extra effort may actually hurt you in the future.

Better is to design for change. Keep everything modular. Keep your concerns separated. When something needs to change, you can just change that thing. When the whole basis of the system needs to change, you can still keep all the components that don't need to change. The only future use case you should design for is change, because that's the only thing you can be certain of.

So don't make things more generic than you need today, but make sure it can be made more generic in the future. And in that future, don't just add new features all over the place, but reassess your design, and look if there's a more generic way to do the thing you're adding.

I do this sort of stuff all the time on my current project, and it works quite well. There's no part of the system we didn't rewrite at some point, but all of them were easy to rewrite.

The chapter The Second-System Effect form The Mythical Man-Month book (Brooks, Jr. Frederick P.) talks about this problem:

“An architect's first work is apt to be spare and clean. He knows he doesn't know what he's doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used "next time." Sooner or later the first system is finished, and the architect, with firm, confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.”

The overall advice is to practice self-discipline:

“How does the architect avoid the second-system effect? Well, obviously he can't skip his second system. But he can be conscious of the peculiar hazards of that system, and exert extra self-discipline to avoid functional ornamentation and to avoid extrapolation of functions that are obviated by changes in assumptions and purposes.

A discipline that will open an architect's eyes is to assign each little function a value: capability x is worth not more than m bytes of memory and n microseconds per invocation. These values will guide initial decisions and serve during implementation as a guide and warning to all.

How does the project manager avoid the second-system effect? By insisting on a senior architect who has at least two systems under his belt. Too, by staying aware of the special temptations, he can ask the right questions to ensure that the philosophical concepts and objectives are fully reflected in the detailed design.”

I am a firm believer that the most important superpower a software engineer and their team should have is refactoring.

By refactoring what I mean is continually revisiting the architecture of your code, identifying common functionality, better organisation of code, better abstractions. The right decisions for your codebase change as it evolves and so you need to keep reexamining these (implicit) decisions.

Continuously incrementally refactoring your code enables so many things to work better. Your example here is a great one - don't implement something you don't have a direct need for. Don't design for it. If you have a culture of refactoring then you can be confident that in the future if that need crops up, you can implement it with appropriate refactoring that the result will look at least as good as if you designed it in up front. If you don't refactor as a team, then you have to do it now, or putting it in later will result in a mess.

Data is the only thing that I personally care about "making future proof". Migrating data is a pain. Most everything else can easily be feature flagged or coded around.

Even then, I don't dig too deeply into things. I look for two things:

* What a migration path to realistically expected features _might_ look like. If I'm debating between two column types or table setups, go with the more flexible option.

* What scenarios will be extremely hard to get out of. Avoid those when reasonable.

----

In other words, instead of proactively planning for some feature to evolve into the unknown in some way. I'm looking to make sure I'm keeping flexibility and upgradability.

Remember that design is about making choices that actually limit the possibilities in design space, rather than extending them.

Building something “generic” to anticipate “future use cases” is not design; it’s the postponement of design.

If you anticipate some unpredictable future feature, then either you do not understand the design space yet, or you are following the directive of a business which does not understand it.

Startups (prior to product/market fit) have a legit reason for not understanding their design space; they’re still exploring it. That makes design pretty hard. Instead of creating The One Generic System To Rule Them All, I’d recommend small, less-generic, low-risk prototypes, that can be easily replaced or refactored. Keep them uncoupled. Meanwhile, use those prototypes to build up your understanding, so that you will be able to make informed design choices at a more mature stage.

I’m speaking in broad strokes; reality may apply.

> How far should we go in making things generic?

My rule of thumb for this is to repeat yourself at least 3 times before trying to build an abstraction to DRY them up.

3 use cases is obviously not always sufficient inform the design of a good abstraction, but IME:

- abstracting at 2 use cases is very often premature and results in leaky/brittle abstractions where the harm from added coupling between fundamentally incompatible usages far outweighs whatever little harm can be introduced by keeping the 2 usages as repeated code, and

- with any more than 3 use cases, the maintenance cost of keeping the usages in sync starts to scale non-linearly, so at the very least thinking about a design for a potential abstraction at that point becomes a worthwhile exercise, even if we end up deciding it's not yet appropriate (hence the "_at least_ 3 times").

This is an interesting question. Intuitively, there should be some sweet spot, some golden middle between going 100% ad-hoc and writing for the future with no real use cases. I'm not sure if anyone in this world knows where is this sweet spot though.

In 2017 I started https://wordsandbuttons.online/ as an experiment in unchitecture. There are no generic things there whatsoever. There are no dependencies, no libraries, no shared code. Every page is self-sustaining, it contains its own JS and CSS, and when I want to reuse some interactive widget or some piece of visualization somewhere else, I copy-paste it or rewrite it from scratch. In rare cases when I want multiple changes, I write a script that does repetitive work for me.

This feels wrong, and I'm still waiting when it'll blow up. But so far it doesn't.

Sure, it's a relatively small project, it has about 100 K lines along with the drafts and experiments. It is although inherently modular since it consists of independent pages. But still, this kind of design (or un-design) brings more benefits I could ever hope for.

Since I only code that I want on a page, every one of my pages is smaller than 64KB. Even with animations and interactive widgets.

Since all my pages are small, I have no trouble renovating my old code. Since there is no consistent design, I forget my own code all the time, but 64 KB is small enough to reread anew.

And since there are no dependencies, even within the project, I feel very confident experimenting with visuals. Worst case scenario - I'm breaking a page I'm currently working on. Happens all the time, takes minutes to fix.

I still believe in the golden middle, of course, I'll never choose the same approach in my contract work; but my faith is slowly drifting away from "design for the future" in general and "making things generic" specifically. So far it seems that keeping things simple and adoptable for the future is slightly more effective than designing them future-ready from the start.

I think a common conflation is seeing "making something future proof" as "making it more generic".

IMO, good future-proof design is about putting in place good components and system boundaries.

Those components and boundaries can be highly specialised and have as few options as possible - it's much easier to make a system boundary more complex than to make it simpler. So start as simple as possible!

Now, with those boundaries, you can easily write tests, and iterate on the different parts of the system. Bad code in one component doesn't "infect" bad code in another part of the system.

Most "balls of mess" systems that I have seen came down to not having clear boundaries between components of the system, rather than being too generic or not generic enough.

1. Embrace refactoring/rewriting as inevitable;

2. Understand it's very often easier, faster and better to write something trivially but twice than writing it well once;

3. Realize no matter how hard you try, unless this is something you've already done once (which brings you back to #2 above), then your information of the problem is incomplete and therefore any all-encompassing solution you may have will essentially be partial and based on extrapolation of your knowledge and guesswork.

The above also deals with a significant cause of procrastination - an internal subconscious feeling we can't do what we're about to do well, therefore we'd rather not even start it. Just sit down and tell yourself, "today I'm going to write a bad module X", and with enough time you'll be able to sit down again and write a good X.

The one thing you should spend a bit more time on is interfaces and decoupling, but as for internal implementations of things, or even entire system blocks that you can swap later on - don't bother too much. It'll be OK at the end, and you'll get there faster. Even if you need to rewrite the whole god damn thing.

Two things in combination help me fairly well in that regard.

1. Keep It Super Simple (KISS).

Implement the solution which works the easiest first. Copy-pasting code is ok at this point. So if an "if" will do it, use an "if". (Don't start with a BaseClass with an empty default method and a specialized class which overrides it).

Once you have something working (hopefully with some automated tests?) you are allowed to refactor and abstract. But see next point.

2. The Power of Three!

You are only allowed to abstract code once you have copy-pasted it in at least three places. If you only have two, you must resist the urge and move on. Maybe add a comment saying "this is very similar to this other part, consider abstracting if a new case appears".

After abstracting stuff, run tests again (even if they are manual) to make sure you have not broken anything.

Be warned that this method is not guaranteed to produce the "best possible code for future you". If you keep doing this long enough, you might get stuck at "local maxima" in design. Future you might need to do big refactors. Or not. That is the nature of programming IMHO.

Knowing the point where you pass good design and start over-engineering is an art developed through wisdom and experience. Sort of like the art of making good LOE estimates given incomplete requirements and an unknown team.

I would look to senior people who have boots-on-the-ground experience delivering and maintaining projects/products, ask them if they think you're overdoing it.

edit: I want to add, get input from non-engineers too. Ask them "in your experience, how far have projects diverged from the initial requirements/purpose? I'm trying to plan for the future but not over-complicate things"

The stupidest over engineering pattern I've seen is having an interface for every single class in a Java project. This may make sense if you are building an API within a framework or library that people other than you are allowed to implement but when the interface is only used within the project it was created in then it is absolutely meaningless to add the interface before you need it. You can always add a necessary interface later by editing the code. Doing a search for "ClassName" and replacing it by "InterfaceName" isn't exactly difficult.

Some of this is the experience to know what parts of a system are likely matter in the long run and some of it is the discipline to hold yourself and your team accountable.

If I had to boil the first part down, I'd say you need to focus your engineering on the interface points - network protocols, schemas, API sets, and persistence formats immediately come to mind. It's a truism of the profession that those are the most expensive aspects of a system to change, and therefore they're the least likely to be mutable down the road.

That's really the easy part... the harder part is maintaining the team and self discipline to keep things simple. For better or for worse, engineering job placement (and oftentimes personal satisfaction) is highly resume-keyword driven. Organizations and people all tend to chase the next resume keyword on the assumption that it'll help them deliver more efficiently or get the next job placement or write the next blog post or whatever else. The net of this is that there's a very strong built in tendency for projects to veer in the direction of adding components, even before considering whether or not they're appropriate for use. So keeping your eye on actual, real project requirements over all else is both important, and can require convincing and political work throughout a team and organization.

Disclaimer: I have never worked in large teams but this problem arises everywhere with the caveat that in smaller teams and solo development the coordination requirements are much simpler.

That said, I think there are two steps in designing abstractions:

The first is to split up and isolate both code and data into small, simple pieces. These can easily be non-DRY (structurally) and often are. You'll have cases where you often (always?) pass things or do things in sequence in the same way.

The second is to merge the pieces with parametrization (or DI in OO), defining common interfaces, polymorphism etc.

The first part is very often beneficial and makes code more malleable, which makes future features/changes less cumbersome.

The second part however is the dangerous but also powerful one. It can lead to code that is much more declarative, high-level and adding new features becomes faster. But the danger is that a project isn't at the stage where these abstractions are understood well enough, so you end up trying to break them up too often.

I try to default to writing dumb, simple small pieces and deferring abstractions until they become provably apparent. Fighting the urge to refactor into DRY code.

Now there is also another issue with writing "future proof" code, which doesn't involve abstractions but rather defensive programming, which is an entirely different issue.

One heuristic I've adopted: when faced with a design decision, I ask the question "what's the simplest possible way to do this (that meets the requirements)?" So, do you need a fully event driven Kafka based system, or would a cron job running a python script be sufficient?

The followup is, if we have to change/replace said system, how difficult would that be? If scaling the simplest solution would be difficult/painful, then one can start look to higher complexity solutions. The idea isn't necessarily to always choose the simplest solution, but having it in mind can be helpful. It crystalizes one of the endpoints of the simple/complex spectrum and helps weigh the pros/cons of various approaches.

As a side note, it's sort of amusing that a lot of design focused interview questions are around things like "design a twitter/instagram like system that'll scale to billions of requests per day." I've never had to do anything like that IRL, but no interviewer has ever asked me to design, say, an invoicing system that gets called rarely. So perhaps one of the reasons the arc of software engineering bends towards complexity is that we're continuously rewarding a "build massive scalable systems" mindset?

If you ask an ice skater how not to fall, they may likely tell you that they have falling tens of thousands of times and continue to fall all the time. But they will know, they know how not to fall because they are intimately aware of what it feels like to fall and know how to avoid it as a result.

I don't know that there is any readable wisdom that will teach you how not to overengineer or underengineer, such that with that knowledge, you automatically know how to achieve the right balance. It is likely a necessary part of the process to build out software projects that are in fact overengineered or underengineered and to intimately learn from that process as well as the aftermath how to tune one's own process to strike an artful balance between the two extremes.

put another way, the MS and Google teams you worked with were screwing up, but screwing up in a way that is necessary for people to learn, if they do in fact learn (they might not learn).

all of that said, starting out by intentionally underengineering, if you know that you are one to start overbuilding, might be a good strategy. but you might have to do some really huge refactorings subsequent to that when you learn your architecture is insufficient.

This by far the main reason to have people with decades of experience in engineering teams.

Make your code modular. Use dependency injections. Write short, clear functions.

When everything is DRY and functions follow SRP, it's hard for your code not to be "future-proof."

Tests only when they make sense, not when you are mocking half the app.

> At what point we should stop designing for future use cases?

Guessing the future and engineering to cope with it is a risky, error-prone business. Good engineering practice should always seek to minimise reliance on unreliable (prediction) data to create future-proof designs.

So I'd go with stopping as soon as it meets current use cases AND then shift gears to make it easy for someone else to pick-up in the future, e.g. write whatever tests are necessary and document thinking.

If it's easy to extend now, it will still be so in the future. Plus, there will be whatever learning has occurred since to make an even better job of it.

> How far should we go in making things generic?

Only if: 0) It's not only about guessing future needs. 1) The maintainers are over 90% certain it will make it easier for them to maintain and test now.

> Are there good resources for this?

There's a mountain of media on "Agile" software development, and it's different flavours. It's not particularly Software Engineering focused, but I enjoyed the "The Lean Startup" by Eric Ries.

Good luck and have fun.

This tendency has been noted since at least the Manhattan Project, described by none other than Richard Feynman. The prescription for oneself is pretty simple - just say no! It is relatively easy to catch yourself whenever you think about some improvement that's irrelevant to delivering so long as you have convinced yourself that you need to wholeheartedly focus on delivering.

It is more difficult to persuade others not to over-engineer. I have tried and failed many times to do so. In fact, if you try too hard you may just make them hate their job. Folks get into software development for a whole host of reasons and only one of them is shipping. They may not be satisfied with a job where they funnel requests from a PM directly into code with little creative input. I'm not sure I can give good advice on this front, other than to look out for certain red flags during hiring.

Summed up as YAGNI

You aren’t gonna need it.

As with anything tho, ruthlessly cutting stuff or rounding corners is in tension with supporting future capabilities, sometimes you do need it it turns out.

Over-engineering is the curse of smart guys without enough experience.

I believe this is well documented today, but maybe not that much in academia.

What they need to avoid this trap is to talk with people who got burned earlier.

As a rule of thumb, don't design for future use case, ever.

Future-proofing is a fool errand.

It depends on the impact of the change to add flexibility in the future. A schema change to a table with many rows that could involve re-indexing usually makes me think harder about the initial data model and schema design. Refactoring code usually involves less cost and risk, especially with modern CI/CD practices, so I'm less likely to add a layer of abstraction if it's not required.

I've also found that designing for unknowns can sometimes be resolved by asking questions about the unknowns until they're known. Sometimes, business stakeholders have an answer, they just weren't asked.

Designing for the future rarely works out very well. What does work, however, is encapsulation.

For example, in the 80's the linked list was my go-to data structure. It worked very well. But with the advent of cached memory, arrays were better. But my linked list abstractions were "leaky", meaning the fact that they were linked lists leaked into every use of them.

To upgrade to arrays meant changing a great deal of code. Now I encapsulate/abstract the interface to my data structures, so they can be redesigned without affecting the caller.

In my mind it's surprisingly simple: you VERY honestly ask yourself the question of you know exactly what you are building. If you do then don't be afraid to plan ahead. If you don't, then ship the smallest thing that works :)

If you are doing a rewrite of an existing system or have many years of experience with a similar product then thinking ahead can save time. Otherwise you are probably better of not trying to be too clever.

The difficult part for most people is actually being honest in the process :)

Also Kent Becks timeless advice is good to keep in mind

1) make it work

2) make it right

3) make it fast

In that order. You might not need all three :)

I have been in this trap myself. I tried writing only perfect code. I ended up thinking more about the design than actually progressing on the issues. The solution I found was to write explicitly hacky solutions first, then improve them as you go forward. Either hacky by being slow, or by not having all the features you want it to have eventually. Not hacky by having huge bugs, don't do that, as finding them is a huge cost the later you find them! Put TODOs about the aspects that need improvement, this allows you to grep for them. Also, as you write the first implementation, you'll gain more knowledge about the problem domain than any non-coding research could give you. Maybe you don't need that one feature after all, or maybe your approach was totally wrong and you should do a different one. It's better to find that out when you only have invested time into a prototype instead of a full generic solution!

And this is not an obligation. If you are really sure about the design of some component, you can also do it right the first time. But in most places, you usually don't know the design well enough.

Also, the imperfect solution can help as a validator. Have a problem that has a unique solution? Make a slow algorithm for it which you are sure is correct, then build a faster version and use the slow algo to compare for correctness. You can also use the slow algorithm to tell you about non-output values that both algorithms compute implicitly or explicitly.

It's a difficult battle in many teams where some people will just go overboard for multiple reasons, usually in this order: fun, fear of being seen as short-sighted and/or promotions. It mainly happens in big orgs and your examples definitely fit the bill although I haven't worked there specifically. I'm not that old (very late 20s) but I've seen overengineer code (or infra) ending in the trash so many times... many people could see it coming and warned about it, yet it still happened.

I usually try to support the opposite idea by bringing agile to the table. Agile doesn't say "please don't overengineer" but "this can change a lot in a couple sprints, requirements may be different, there are more (or none anymore) use cases now" and hence why to spend so many resources doing something that not only doesn't cover all potential unknown use cases but also adds overhead when reading the code for no benefit. For some teams this has clicked very well. Appropriate agile training can help a lot.

However it doesn't always work and if the person(s) doing this have higher ups backing it up (even if unintentionally) you're fighting the wrong battle. If you can't change it, move teams or to a company where money is a bit tighter and/or where these behaviours aren't tolerated - in other words, where things needs to get done and there's no time for experimenting with things that most likely won't yield returns.

Solve the problem directly in front of you.

Then understand a new problem.

Repeat.

If you then start doing things like externalizing inputs, writing tests then when your past solutions become future problems.. You will have created the guide rails on how to solve both the past problem and the future problem together. This is the futureproofing that you should be aiming for. Make sure you write at least a few unit tests that mock out the important logic parts, and write up a short 1 page document about the intent of the program, maybe with a picture and drop that in the README.

One of the most challenging things in growing a software product is managing complexity. If you are designing a product for future use cases, you add complexity upfront. To keep a healthy balance, I try to follow simple guidelines:

* Focus on the current assignment. Implement it using clean code principles, don't overthink the problem.

* Rather than spending time on design decisions, allocate time on handling edge-cases. These will save you from PageDuty alerts.

* Plan excessive for future use-cases only around data models that insert/read into the database. Data migration is super-painful. A more generic design around your database is almost always preferable.

* Have a feature toggling[1] service in your codebase. It will provide you with a better understanding of how you can implement new features alongside existing codebase in the same branch. Releasing features from long-running separate branches is almost always a wrong decision.

* Always keep in mind time-constraints and how urgent is the requested functionality. Don't let perfection be the enemy of productivity.

* Have a process in place that allows for the technical debt tasks to be tackled independently. It helps fix some bad design decisions, which become apparent in light of new requirements.

[1] https://martinfowler.com/articles/feature-toggles.html

You should immediately stop designing for future use cases.

Use TDD and only write only just enough code for the currently known required functionality. When you get to know more required functionality the tests protect you from breaking existing functionality and you can extend your code to support the new use cases as well. At this point you should make your code just generic enough to support all known use cases without code duplication or too much boilerplate code. If you can support functionality in more than one way you can decide what way to choose based on what you expect in the future. But choosing the most simple solution trumps attempting to future proof your code. It turns out that predicting the future is quite hard and there will be new feature requests that nobody had foreseen and code that has been made as generic as possible will not handle this well.

Spiderman says that with great power comes great responsibility. The converse also holds true. With great responsibility comes great power. You cannot just pick the easy part of TDD, be irresponsible and expect to have any power. The less easy parts of writing a test for every use case and of refactoring all the time make the practice of not attempting to guess the future possible. If you leave out the prerequisites the end result will not be so very pleasant.

Follow YAGNI: https://www.martinfowler.com/bliki/Yagni.html

There are a few first principles from which we can design code.

The first is: Requirements change.

For many reasons. User needs change. Business processes change. Technology evolves.

If you're going to keep up, you must design code that is easy to change.

Code that is easy to change tends to be code that can work with lots of different code.

Code that can work with lots of different code tends to lean towards the general end of the spectrum, rather than being too specific.

Designing code that is modular does not mean that you need to over-engineer, and it doesn't even mean that it needs to be used more than once.

It should mean that the code has locality: The ability to understand the full effect of the code without also understanding the full context of the code around it or the full history and future life of every external variable it uses.

It may sound to new developers like I'm talking about something complicated, but it's the opposite:

Code that can be easily adapted to future use-cases tends to be more simple. It tends to know the least about it's environment. It tends to do only one thing, but do it so well as to be perhaps the optimal solution.

What follows from the first principle is perhaps the most important principle in software development. Remember this and you'll find yourself needing to do a small fraction of the work you once did to produce the same value:

A small change in requirements should lead to only a small change in implementation.

I personally like the simplicity of designing for change. Simple rules like TDD help you to think about the design up front.

I was talking to a new guy that just joined my team yesterday on this subject. You really cannot predict the future.

You could also look at it from one other angle. If you are only building the bare minimum to satisfy the requirements, that is a lot less code you are writing. If you need to replace the system, that is a lot less work to go back and rework.

When you have good refactoring tools, a lot of this is significantly less issue. So, design for easy refactorabilitym lack of repetition and readability.

I dont like "used the simplest solution possible" advice, because people who I claimed doing that in real life tended to do unmaintainable spaghetti mess. It was "simple" in the sense of not having abstractions or nested function calls, but hard to read and understand big picture. Sometimes generic is a way to untangle such previous spaghetti mess. As in, it is second step on the road that requires 2 steps till it is really good.

Understand politics. A lot of those "future cases" are things that analysis or management requires initially or indirectly. They are also often result of trying to hit vague requirements - you dont know what customer really needs in enough detail, so you do it configurable in the hope of hitting the right place too. They are also situations in which people burned out in the past or special cases of special cases.

Otherwise said, the complicated thing is often requirement, initially. Someone had reason to ask for it or thought to have reason. It gets forgotten and ignored after a while in which case it is ok to cut it off.

Invest in very good refactoring tools.

Invest in very good testing, especially higher level integration / system testing.

Invest in a good dev/staging setup for your production environment, and also try to make rollouts and rollbacks automated and as painless as possible.

There will always be the need to change stuff, so get the pieces in place to make changes easier code, easier to test, easier to deploy, and easier to back the fuck up when you inevitably cause something to burn.

I find the best designs are simple ones that reflect the underlaying concepts. Most designs that lead to complexity are based on the wrong abstraction. I have found this to be true nearly in all cases.

Of course the issue you run into is that the problem software is trying to solve changes over time and then the original abstraction which were correct become wrong over time. Then you should advocate refactoring to the new abstractions if possible.

By being aware of the true costs: It's not the cost of making the code more generic (typically relatively cheap) -- but refactoring costs when it turns out that the code actually needs to be more generic, but in a different way. It's easy to refactor something simple into something more complex/generic, but it's hard to refactor something complex into something that's still complex, but in a different way.

> At what point we should stop designing for future use cases?

Immediately. Never design for a future use case until it's a present use case and you're implementing it right then.

> How far should we go in making things generic ?

It depends on what it is. By not designing a thing to be generic up front, you have to figure out what n=2 looks like. Is that a function? A class? Copy a little bit of code? Then n=3. Once n=10, I feel like I have a good idea of the problem and how to make it generic, and it's rarely what I would have thought at the beginning.

Sometimes n never reaches 2. Then you've saved a lot of time. Also, you realize when you have to change things - maybe it's once a release, or very frequently. Things that are touched frequently probably need a refactor.

My rule of thumb is: never make tomorrow's possible problem today's complexity. If you design for future use cases, not only will it take you longer, but your code will inevitably be more complex than it needs to be, and therefore have more bugs at the very least due to that complexity.

If there is an actual customer, try to get the scale of users/data the system is expecting. A system handling 1/1k/10M things every day will be quite different and need different solutions.

Problems arise when people reach for the Cool Tools and start building Webscale things that can handle 100M operations every second. ...but the customer only needs the system to handle 4 users who type in everything crap by hand. But hey, it has a clustered database and Kafka and Kubernetes and looks REALLY good on your Resumé.

When the scale is determined, I personally like the mantra of "Make it work first, then make it pretty". First build an MVP (or an ugly Viable Product), that proves your plan actually works, then you can iterate over it to make the implementation cleaner or faster or have better UX.

If you get stuck making everything super-generic and able to handle all possible cases, you'll spend time bikeshedding and never get anything deployed. Just Repeat Yourself with wild abandon and copy/paste stuff all over until your Gizmo actually works. You can spend time figuring out which parts are actually possible to make generic later.

Disclaimer: the following method may not work if your library is public facing, as breaking changes to public API is usually a terrible idea.

Ask yourself the following questions: * Are you an experienced developer on this particular problem domain? * Do you have good understanding of the future use cases, and the data structure when the new feature is added? * Will the new feature make business or technological sense?

Unless you have a concrete “yes” on all questions above, your design should go minimalistic. Without good understand of the new feature, the code base will likely need refactoring when the feature is implemented. Your effort of going the extra mile (which definitely should be applauded!) is better spent on making the code easier to read and refactor - better tests, better documentation, code review to propagate knowledge and catch bugs.

Another thing, extensibility is sometimes in conflict with readability! Here’s an hilarious example: https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...

This is a typical case I am struggling with during my whole career in software. It is very hard to fight against because it is counter-intuitive. Saying that "let's put aside all those designs and let's do a minimalistic version to satisfy current needs and then we move from there" is always less attractive than "let's do a great architecture which can support our 'exponential growth'".

In my experience following categories of people tend to fall into the trap easily (and it is really hard to convince them):

- Corporate people.

- Business school people.

The following can amplify it:

- Fund raising... you were successful in raising money so that means there is a market and people love your product ... well guess what? you are wrong.

If you are in such a situation, debating is not useful. Changing this mindset is only possible when you hit the wall at least once in your life. BUT ... you can accelerate the process of realizing it (before it is too late). Try to find a way to make people realize that all those bells and whistles and designs were not necessary cause nobody actually asked for and we are changing it anyway now.

There can be no definitive answer to your question but of course some "thumb rules" are applicable.

* The nature of your "system" defines how much attention you should pay towards futureproofing. If it is an end-user app/feature only do what is required and nothing more. You are solving one instance of a (maybe) small class of problems. The future will dictate what needs to be added/modified/refactored when you reach that point and not before i.e. no predictions unless you know the domain well.

* If you are building an "architecture" component like OS/Framework/Library etc. then you need to pay attention to generality and extensibility and design with futureproofing in mind. Use standard best practices like data/api versioning, narrow module interfaces, information hiding etc.

* Always focus first on Readability and Maintainability to the exclusion of everything else. Only when you hit a wall with respect to aspects like Performance etc. do you go back and redo/refactor the code for the newly prioritized requirement.

Don't design for future use cases, unless it's a library that may be extended. Instead, write concise code.

Modify the code when the use cases change. In most of the times, open-closed principle is a trap.

However, over-engineering is not only a technical problem. It's more of a problem on project / people / finance, here are some examples but it would definitely not limited to these:

1. The margin is big and the team is big, so we need to keep these people busy.

2. The current stakeholder will cover the budget these iterations until we finish these features. after that, the maintenance costs maybe partially falls on us, or there's will be no budget at all. The less problems in the future, the better.

3. The current budget is for functionality A and someone pays it. And we plan to implement another feature B that is similar to these, and the budget comes from our own. Better make the solution generic so we can use it almost for free.

4. The list goes on...

Better fix those root problems first.

Demo driven software development oddly enough

* A demo is a thing you can show. The first demo is usually

   - the program will compile, run and print "hello world"

* A demo is a contract between stake holders

* Demos happen frequently. For a developer each day, for a team each week, etc.

Why does this help? A demo, actually showing something, is the only enforceable token that commits both part to the contract. In that way it is almost a currency. No stakeholder, whether manager, sales or developer can argue - either the demo was met, or not or was not clear.

This is much like sprints, test-driven, but a demo has the contract aspect.

Demo-driven reduces many of the causes/motives for over-engineering. * Is the over-engineering bit in the contract? Why not? * New function requires a new contract so no more last minute changes. (Been there done that). * Stakeholders gain experience designing demos. * Demos are adaptive. They provide tactile feedback.

My 2 cents

Historically, this made much more sense. RAM was incredibly scarce as were CPU cycles and the hardware was often tied intrinsically with the software so a modification later down the line was a really big deal.

With modern higher-level languages and scalable cheap hardware, this motivation should have gone away and we should be writing code that is relatively easy to re-factor.

If I don't know that we will need this thing in the future, it doesn't go in. Simple. If 6 months down the line, we now need to add the new feature, I like to think that my code is largely maintainable enough to refactor it to add the feature.

The only exception I can think of is where something is designed to be extendible like e.g. Instagram filters where you might have 10 when you launch but you know that you will have more in the future so you write your code to allow additional filters to be plugged in relatively easily.

I think most of the time this is just Parkinson's Law playing out: "work expands so as to fill the time available for its completion"

Things get over-engineered because too much time is allowed for the task.

The forcing function that's helped me is simply giving me and my teams small but meaningful time boxes to get a feature done. Sort of like "sprints", but with more teeth. E.g. An important feature is going to get announced in a newsletter ever 2 weeks. Sprints too often don't seem to focus on the shipping to customers part.

So we focus on shipping the smallest thing that could possibly work in an arbitrary time box. You know users need X. So you make X or X' or X'' - some version or whittled down version of X that'll relieve user pain. The time box does works wonders from keeping over complexifying from getting out of control.

By having a time limit. When I was new and over excited young guy I started with the assumption that we'll be working on it for a really long time but it rarely proved true.

Truth is most projects will be shelved before even they are complete.

You'll be taken off some projects because of financial or political BS.

So start with the assumption, there is limited time to deliver - now ask yourself - how can I maximize the effectiveness of that time? How can I do what truly matters without going into the rabbit hole of optimizations and using the best thing possible everywhere.

And once you've done that, you can always come back and improve the thing if the project is still around but most likely it will not be. Either you'd have switched the company or company would have switched you or the project has switched the company.

We overengineer when we overestimate how hard it is to modify details. As juniors, we wrongly learn that changing code is hard and must be avoided at all costs. As mid-level and seniors, these refactors are much simpler, but the painful memories stick.

Architectural boundaries are hard to change later. Drawing the dependency graph and isolating nodes that can change independently is where the majority of our effort should go (imo). Even still, simple is less risky than complicated. Anytime there are more moving pieces than necessary, there is a risk of an unexpected requirement blowing a hole in a design. So identifying those dangling pieces and spending a lot of thought and energy in removing them is where I've found it to be rewarding to "overengineer".

I like to build a forward looking document for a team/groups software which extends out to cover the companies goals + N months/years. The goal of the document is to help define the core components, their role, and integration points such that everyone can reason about future looking tradeoffs. Teams can then use the document to see how their specific roadmap fits into the broader picture. This also helps frame use cases terms of current, goals, and dreams. If you're building for a use case beyond the team's dream features then you're probably over engineering the solution.

In practice goals start to move after horizon/2. And a new document needs to be created to capture where the business is doubling down and where it is trimming.

Break the problem you're trying to solve up into simple steps. Each step should be nothing more than a single task.

Ex. if you have a problem to solve that goes like: Must create store with products that are blue only.

Then you'd break it up like:

Create store Filter products (Blue only) Create products in store

Then when you start coding you solve nothing more than what you put down as each task.

Ex. you don't do anything more than what's required. You might think you need to create a store and stores are businesses so you need to create some business wrapper etc. but nope. You don't need that until a client comes one day and requests it. Right now all you need is a store with blue products and that is all you're going to solve.

Often when you over-engineer something then you never need the extra abstractions you created.

It's a balancing act.

You need to always ask yourself "What are the reasonable future use cases? How much will it cost to add infrastructure now to enable each of them, and how much will completing the feature in the future cost? How much will it cost to refactor my code later to add it if I don't make preparations for it now?" and only prepare now for things where not preparing would cost a lot more.

I'm finding the game of Go (Weiqi, Baduk) to be a great way to train in this skill, because it's all about seeing potential moves and deciding not to play them yet, and judging if a move should be played sooner or later and how much of a shape should be built now to enable it to be built later without wasting resources on building all of it now.

I wrote about this a few years ago.[0] The gist to avoiding premature over-architecting is to keep sticking with as static/hardcoded behavior as possible to meet the requirements. Just make sure it’s well organized and cleanly written in your language of choice. Then over time add configurability to that behavior as specs change. Architecture is easy to change not when it correctly predicts future change (almost impossible), but when it is straightforward enough to follow and reshape.

In my experience, keeping behavior static/hardcoded is the architectural equivalent of avoiding premature optimization.

[0]: https://max.engineer/cms-trap

Imo it is an organizational thing. Big is good, powerful and important. The more subordinates the more powerful a manager is. Allocated resources (e.g. dev hours) must be spent on something and more often than not must deliver just a little bit short in order to allocate more the next time. Overengineering is rewarded, it creates more tickets to work on and keep the people busy. Then on the other end there are the resume driven folks, who are mutually interested in piling up complexity. Still, invoking Hanlon, it can not be ruled out that some people just do not know better. Sometimes I think KISS is directly at odds with enterprise development.

Intelligence signaling. Programmers love to show off to their colleagues how smart they are, so they try to anticipate all possible upgrade paths. "What if the user doesn't have an email address, betcha didn't think of that. That's why you hired me, the tech master, with over 9000 confirmed code commits". This leads to code that is overly generic, but still a big ball of mud. What you want, instead, is for people to be humble and accept that future changes won't be something anyone can anticipate now, and instead adhere to principles that, in general, lead to modular codebases which can respond to changes flexibly.

Management overvalues complex systems and undervalues more elegant approaches. Architects are often more than willing to oblige mostly to live out their fantasies.

Thus, we have every company it seems these days running around trying to implement microservices because “our architecture is like Netflix’s”. No, that LOB CRUD application isn’t like Netflix and you don’t need polyglot storage, Kubernetes, and microservices to capture input from a web form. However, one of the managers between the architect and the CEO (see: Peter Principle) is extremely impressed by this idea and can use it to advance their personal career.

I usually take YAGNI further and avoid using database (sql or nosql) at all (at least in the beginning)

My typical approach is to log all the commands to the file(s), and keep the system state in memory. When the server restarts, it simply replay most of the commands (skipping those intended to cause side effect)

This is like simplified event sourcing, or command sourcing.

I'm not relaying on any framework to build the backend, simple library handing rest and websocket API is enough.

I'll open source it soon but it's mostly as simple as it sounds, with some optimization to reduce disk space consumption after they becomes a problem in practice.

On a module or service scale I recommend using TDD [0]. It takes some time before writing actual code, so I can use that time to think about clean and effective architecture. After that, I have less time to write actual code, so most of my efforts are going into just painting red tests green, and not overengineer or thinking about phantom future use cases. After all, I have code AND tests, and it's always nice to have.

[0] https://en.m.wikipedia.org/wiki/Test-driven_development

No one has added my favourite quote yet so hear goes: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.", Antoine de Saint-Exupéry

Make code easy to read and extend without breaking it. By now, many of us are aware of the importance of writing tests. However, there is a problem in making them readable.

Here's one Kevlin Henney's lecture [1] that crystallizes it. It took me a long time to find it, so you are welcome.

Once you start naming things like this, adding future use cases becomes far less risky and thus you don't need to waste time on them.

https://www.youtube.com/watch?v=tWn8RA_DEic

My observation is that as I’m getting older as an SDE the number of classes I create in a project goes down! Like a lot of people in this thread point out: almost most of your code has to be changed anyway in a refactor no matter how clever and abstract the design is, what’s the point in over engineering and thinking too far ahead of success? I would rather focus on my data structures and their flows. I know if I screw up the foundational of the data, I could be in a lot of trouble comparing to just a bad code organization.

There's a fine balance between over-engineering and under-engineering, right now I'm working on an old piece of software that has been designed without any significant engineering process, while things remain easy to understand and fix, there's a deep sense among us that simple dump code repeating code has brought duplication, inconsistency and bugs that are easy to fix in one place but impossible to fix everywhere. I think most software outside of silicon valley may well be under engineered.

Simple: don't. Do the simplest possibly thing that can work. Only code for today's requirements. You can't predict the future. Creating extension points without a past pattern of data of how the system is being extended is just guesswork, not engineering.

Once you have a established a pattern of how the system is regularly extended, then you can use that to make predictions about the future. Keeping your codebase well-tested, small, and light will do far more to help you respond to change than guessing.

Sometimes the source of the problem is political: one trying to make a more general framework to deprecate another team, or to avoid being deprecated by generalizing in a slightly different direction.

How do people “defend” that in big Corp?

Boat the technical roadmap and requirements with fantasy future features, so they can say “well that framework doesn’t support X, and our framework will support X”. It doesn’t matter whether X is useful in practice or not, since the managers don’t always have the power to make that call.

1: Simple code is easier to refactor than complicated code. Try to keep your design as simple as possible.

2: It's easier to refactor code with good automated tests, like unit tests, because you can push a button and know that it works.

3: Make sure that your startup order is well-defined. It's easier to debug a well-defined startup order than a startup order where everything is implicit.

4: Know the difference between a design pattern and a framework. Frameworks don't replace knowing how to use a design pattern correctly.

Don't design your software for future features you might need. Design it to be easy to change. This means it has to be decoupled and simple. The trick is to make the software decoupled without making it complex. Adding layers and interfaces and abstractions is the easy way to achieve loose coupling, but it also adds complexity. Making software that's simple while still being easy to change is much harder than making some layer lasagna.

Best practice is to follow the example of the subway system and build short spurs at the end of each line in the directions you might want to continue expanding. That way you're not incurring any real additional cost upfront, but you're saving a ton of money in the future if you decide to keep expanding.

Don't do work before you need it, but also don't give up optionality unless you're getting commensurate benefits from doing so.

There’s a lot of folks in here shouting “don’t develop until people ask for it!”

But there is a certain joy that comes when people ask “but what about XYZ” and you can respond “Yup, we thought of that!”

Granted, I work mostly on developer facing tools and services, which makes it much easier to anticipate the needs of my customer since I too am a developer.

And even with that caveat it’s not always a slam dunk... but it certainly is possible to anticipate requirements in a useful way.

It is different if you are working on a team or as an individual. On a team it's important to build ways for people to work autonomously because once that's done the team as a whole will have more throughput. For an individual those same separations might be a drag on productivity. Regardless, careful consideration for what is _needed_ at every step of the way is important. Extreme Programming is worth a look.

Although software development tends to come across as deterministic with rules on what to do when, a lot of it is up to developing good judgement.

The more time you spend thinking about the business and how what you're building will support it the better judgement you'll develop. You might not be able to articulate or argue it but you'll have an instinct on when an abstraction is going to be useful vs brittle.

See https://www.codesimplicity.com/post/the-accuracy-of-future-p... for some thoughts on the accuracy of future predictions. I've found Max Kanat-Alexander's writings on software to be very workable; I've had many successes applying them.

Never design for future use cases. Design only for the use case in front of you by developing a point solution. Accept the tech debt and move on.

Understand and implement hexagonal architecture. https://en.wikipedia.org/wiki/Hexagonal_architecture_(softwa...

If you build systems with these principles in mind, you can create systems that are extensible, without creating technical debt and YAGNIs.

Read this post a month ago about future welcome architecture and found it quite practical (with some philosophical musings though), hope it helps - https://www.twowaystolearn.com/posts/future-welcome-architec...

Just another vote that the right answer is do NOT add any code/plans/schema for “future” functionality.

It’s actually not that hard to refactor things when you do need a new feature... and who doesn’t love refactoring? It’s fun!

It’s also easier to refactor when your code is smaller and simpler because it’s only been coded to do the things you actually wanted it to do now.

Unless mandated, don't design for unknown future use cases. Keep the code as simple and dumb as possible. Cater only to those use cases that are known. Strong-type everything. Lot's of unit tests.

If new requirements come in it's anyway lots of typing. You might as well spend the effort only when the full situation is known.

Big companies like to brag about operating in 'agile' teams like startups. And in startups you can't afford to over-engineer. It kills lots of startups. So, pretend you're running a startup that has a solid amount of cash? Not sure if there's an analogy for that with what you're doing...

I don't mean to be controversial but I think planning like that shows lack of experience and more than likely a problem with how work gets done. The developers might feel they have no chance to develop the software further after the first release. That means you have to capture all scenarios straight away.

Always complete each use case from scratch in the simplest and most efficient non-abstract form. Then try to merge use cases into an abstract framework.

Accept the abstractions if: - the stack trace depth is never greater than double - the total code size is decreased - performance/memory hit is less than 10%

Thank you for bringing this up.

As Kent Beck says [0]: "for each desired change, make the change easy (warning: this may be hard), then make the easy change"

[0] https://twitter.com/KentBeck/status/250733358307500032

Seeing the forest here, if you learn to ask better questions, you will get a better idea of what the business side is trying to accomplish.

It is really hard to know what other people want or what they mean. If you can really understand what someone wants, perhaps you can avoid write 50% of a system you initially imagined.

If possible I try to apply the rule "it's not a problem, if it's not a problem". Its much easier to write new code than to get rid of old one. Its really helpful if you have a strong feedback cycle that challenges the amount of time you want to spend on something (budget or time scope).

Trying to design for many unknown futures is expensive.

Changing in the future costs something.

Designing for future changes up front makes sense if cost(future change) > cost(future proofing)

SAAS? Do virtually no future proofing.

IOT, do some.

Space probe? Do lots.

Also, if you haven't built quite a few relatively similar systems, don't do future proofing without talking to people who have.

You should never design for the future use case. You don't know what the future will be. You should only design for the current use case you have today and deploy. When the future comes, you return to the code that you wrote and update it to fit the new needs.

It’s hard to flip your brain, but abstractions are bad. Copying code for different business purposes is good. Simple patterns are vastly better than complex frameworks, even if you think it improves unit or integration testing.

1 - Define an executable specification of the next smallest thing you can do to move towards your goal;

2 - Write the simplest code to fulfill this specification;

3 - Improve on what you wrote so it will express better exactly the specification you have so far.

IMO if designing for future use cases is hard, you probably don't understand the problem well enough to be designing for future use cases. Any design you make will be wrong. Write what works and budget in the rewrite.

Sometimes designing for hypothetical cases makes software simpler. The general case is often easier to understand.

Other thought: good engineers can guess future cases more accurately, that's why they're good engineers.

YAGNI https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it

I have been learning Haskell and Gluon. 100% worth the time investment. In functional programming, the over engineering is one or two lines of code compared to hundreds.

I lean towards building it to the exact known specification and don't worry about how it's going to change. If you write tests you can refactor anything.

Best way, IMO?

A well-maintained, well-pruned test suite.

Writing tests forces you to clarify -- to yourself and to others -- precisely what this piece of software is and is not going to handle.

I was brought into to help at an early stage startup a few years ago. The company was building an e-commerce platform and the product owner had this idea of 'attributes' that could be attached to any kind of entity in the system (e.g. product, category, order, customer). If they needed a new attribute they would be able to simply set it up through the admin UI without any developer intervention (because developers are expensive!).

When I joined the attribute system had been build with a beautiful UI, and the backend was mostly working for managing attributes, but that was pretty much it. The first feature I was working on was showing products on the store, and for this the idea of attributes made sense. If you are selling a product in the "OLED TV" category you probably want a "Screen Size" attribute, and to be able to use it to compare against different products in that category. Through the platform we had maybe 500 product level attributes, with more being added all the time, so having them hard-coded wouldn't have been manageable. That was pretty much as well as it worked though.

Sellers needed to be able to manage their stock through the system, so on the warehouse entity there were attributes describing the number of products in stock, lead time, how often they restock, etc. The attributes didn't really have validations, but they had types which described what UI element should be displayed when they are entered. However all of the validations around that were at the whim of the front-end, and in some cases it would send what you would think should be a numerical type as a string (and then if you try to change it something would break as that expected it to be a string), so doing any kind of calculations or logic on the attributes was basically impossible. In the end I just added db-level fields to the stock entities, with validations in the backend to make sure these were as expected. The backend was a Rails app, so this took 10 minutes vs days trying to coerce the attribute system into doing what I needed.

As it was a Rails app we couldn't actually name the model of these Attribute, so had to give it another name, and whenever someone new joined (it was an early stage startup, so had high turnover) we had a 30 minute discussion explaining this. I never got the explanation of how the product owner expected logic to be attached to these attributes without a developer doing any work, but I'm sure they had an 'ingenious plan' for that too.

Needless to say, the startup burned through all it's funding without even launching, then managed to convince the investors to give them a little bit more, launched a half working product, and it turned out nobody wanted it.

Minimize LOC, that points true north, everything else is fake and results in absurdities like AbstractProxyFactorySingleton

are they doing something besides making software compasable and modular? because if so the point is those patterns expand the possibility space of what you can do down the line by making the software easy to change. as long as type software is following principles of comparability it is not over engineered it is just good design.

If you have too many engineers on a problem, they will overengineer stuff. This is a management problem.

> How far should we go in making things generic ?

Never make things generic apriori.

my rule of thumb is:

requirements -> tests -> specific code

if i reach the same code more than once, refactor and bother to generalize....

today i think it has become a disease. now simplicity is disliked. people see complexity more useful.

Preface:

Once upon a time, I worked for a company who rents movies on DVD via kiosks. When I joined the team, pricing was hard coded everywhere as a one (1), because YAGNI. The code was not well factored, because iterations and velocity. The UI was poorly constructed via WinForms and the driver code for the custom robotics were housed inside of a black box with a Visual Basic 6 COM component fronting it. It was a TDD shop, and the tests had ossified the code base to the extent that even simple changes were slow and painful.

As always happens, the business, wanted more. Different price points! (OMG, you mean it won't always be a one (1)!!?) New products (OMG, you mean it won't always just be movies on DVD??!) And there were field operational challenges. The folks who stocked and maintained the machines sometimes had to wait for the hardware if it was performing certain kinds of maintenance tasks (customers too). Ideally, the machine would be able to switch between tasks at a hardware level "on the fly". Oh, and they wanted everything produced faster.

I managed to transform this mess. Technically, I would say it was (mostly) a success. Culturally and politically it was a nightmare. I suffered severe burnout afterwards. The lesson I learned is that doing things "right" often has an extremely high price to be paid, which is why it almost never happens.

On "over-engineering":

I find this trend fascinating, because I do not believe it to be an inherent issue. Rather, what has happened, is that "engineering" has moved ever closer to "the business", to the point of being embedded within it. What I mean by "embedding" here is structurally and culturally. [Aa]gile was the spark that started this madness.

Why does this matter? Engineering culture is distinct and there are lessons learned within we ought not ignore. However, when a group of engineers is subsumed into a business unit, their ability to operate as engineers with an engineering culture becomes vastly more difficult.

The primary lesson I feel we're losing in this madness is the distinction between capability enablement and the application of said abilities.

Think about hardware engineering: I do not necessarily know all of the ways you -- as the software engineer -- will apply the abilities I expose via my hardware. Look at the amazing things people have discovered about the Commodore 64 years after the hardware ceased production. Now, as Bob Ross would say, "Those are Happy Accidents." However, if I'm designing an IC, I need to think in terms of the abilities I expose as fundamental building blocks for the next layer up. Some of those abilities may never be used or rarely used, but it would be short sighted to not include them at all. I'm going to miss things, that's a given. My goal is to cover enough of the operational space of my component so it has a meaningful lifespan; not just one week. (N.B. This in no way implies I believe hardware engineers always produce good components. However, the mindset in play is the important take away.)

Obviously, the velocity of change of an IC is low because physics and economics. This leads everyone to assume that all software should be the opposite, but that's a flawed understanding. What happens today is we take C#, Java, Python, Ruby, etc. and start implementing business functionality at that level. To stretch my above hardware analogy, this is like we're taking a stock CPU/MCU off the shelf and writing the business functionality in assembly -- each and every time. Wait! What happened to all that stuff you learned in your CS undergrad!? Why not apply it?

The first thing to notice is that the "business requirements" are extremely volatile. Therefore, there must be a part of the system designed around the nature of that change delta. That part of the system will be at the highest, most abstract, level. Between, say the Java code, and that highest level, will be the "enablement layers" in service of that high velocity layer.

Next, notice how a hardware vendor doesn't care what you've built on top of their IC component? Your code, your problem. Those high-delta business requirements should be decoupled from software engineers. Give the business the tools they need to solve their own problems. This is going to be different for each business problem, but the pattern is always the same. The outcome of this design is that the Java/C#/whatever code now has a much lower change velocity and the requirements of it are future enablement in service of the tools and abstraction layer you've built for the business. Now they can have one week death march iterations all they want: changing colors, A/B testing, moving UI components around for no reason...whatever.

There are real-life examples of this pattern: Unity, Unreal Engine Blueprints, SAP, Salesforce. The point here isn't about the specifics of any one of these. Yes, a system like Blueprints has limits, but it's still impressive. We can argue that Unity is a crappy tool (poor implementation) but that doesn't invalidate the pattern. SAP suffers from age but the pattern is solid. The realization here is that the tool(s) for your business can be tailored and optimized for their specific use case.

Final thoughts

Never underestimate that the C3 project (where Extreme Programming was born) was written in Smalltalk, with a Gemstone database (persistent Smalltalk). One of the amazing traits of Smalltalk is that the entire environment itself is written in Smalltalk. Producing a system like I describe above, in Smalltalk, is so trivial one would not notice it. Unfortunately, most business applications are not written in environments nearly as flexible so the pattern is obscured. I've held the opinion for a long time that XP "worked" because of the skills of the individual team members and the unique development environment in use.

As I stated at the beginning, this path is fraught with heartache and dragons for human reasons.