The experience that lead me to this was a particular team I was on where a previous developer had said "aha! All of these pieces of code relate to the same domain model, so we should build a common package of domain model objects and have all the modules share those objects! That way, we don't have to write them all over and over again!"
These weren't just model objects for data passing, but actual business logic inside these models. Usually very simple stuff, but occasionally a particular module would need some special logic on those objects, so in it went.
The problem was that over time, there modules started drifting. Their domains became different ever so slightly and the owners of those modules became different teams. Now if you wanted to modify the domain object, you had to check it wouldn't break code the other teams were developing.
It became a real mess. And all to avoid writing the same simple pojo classes more than once.
A lot of young data scientists and analysts (and IT types in general) code in a way that solves the immediate problem.
But with a bit of time you start to realise that the initial brief is anyways part 1, an executive/customer will change their mind 15 times before the end of the project. What seems like the same problem now will not be the same problem in two weeks, let alone two years. Doubly so if you're interacting with entities or sources that aren't software engineers.
Over the long run, excessive modularisation creates what I'll call Frankenstein programs. You've been tasked with making a man, and what you end up with is a shambling golem made from rotting, pulsating, mutating parts all stitched and held together. If you're really unlucky, it will evolve further into the akira/tetsuo program, where you begin to lose control of the mutations until it self destructs under its own complexity.
The interesting part is that the answer to this can also be partly found in nature: you modularise and specialise, but you also make strategic choices where you're deliberately redundant.
Too much redundancy is spaghetti code. Modularisation and structure save you there.
Not enough redundancy leaves you vulnerable to changes in your environment and mutation as the project ages and evolves.
As I've gotten older, I'm placing more and more value on the later. Your mileage may vary...
The problem was that over time, there modules started drifting. Their domains became different ever so slightly and the owners of those modules became different teams. Now if you wanted to modify the domain object, you had to check it wouldn't break code the other teams were developing.
Isn't the real issue here that the architecture didn't keep pace with reality? It sounds like the dev who made the package had the right ideal. The real issue was subsequent devs introducing their domain specific stuff into a common package, instead of extending it or composing it with their own domain specific code.
Basically, if the semantics of your domain model are specified by an RFC, you can probably get away with turning them into a shared library dependency. Because, even if they weren't shared, everyone would just end up implementing exactly the same semantics anyway.
If someone's implementation was "off" from how the RFC did it, that implementation wouldn't just be different—it'd be wrong. There are no two ways of e.g. calculating the difference between two datetimes given a calendar. There's one correct way, and you can make a library that does things that way and be "done."
---
On the other hand, there is a good reason that Rails et al don't automatically create a migration that creates a User model for you. Every app actually has slightly different things it cares about related to the people using it, and it calls these things a "User", but these aren't semantically the same.
In a microservice architecture, service A's conception of a User won't necessarily have much to do with service B's conception of a User. Even if they both rely on service C to define some sort of "core User" for them (e.g. the IAM service on AWS), both services A and B will likely have other things they want to keep track of related to a User, that is core to how they model users. It might be neatly separated in the database, but it'll be hella inconvenient if it can't be brought together (each in its own way) in the respective domain models of A and B.
Once you ignore frameworks like Django, which provide you with user model, then yes, you'll be correct.
Ill-formed or too many dependencies seriously constrain speed of development and much worse, they suck the fun and flexibility out of development. "Build tools" that manage dependencies are usually opaque about whats gone wrong. Not fun. Spent too many hours trying to find "which version of this library does this function need?"
Too much redundancy is a bug magnet and I cannot emphasize how much I loathe it. There really is no good answer to 'Why is this function duplicated with one extra parameter ?'
But the buried assumption there is that you actually want the change to happen everywhere. A hard thing to decide without hindsight.
Redundancy vs dependencies is a constant battle, even internally or inside your own head.
Many times it depends on the project, team size and ability of libraries to be maintained whether budget, time or goal of the project and its lifeline.
For instance in gaming, you might have common libs for product purchasing, profile systems, networking, data, serialization, and platform libs to abstract calls. But systems like game objects, physics, gameplay and ui aren't as lib-able unless it is a series of the same type of game or system.
It is better sometimes to just get it out, prototype it and iterate to ship, then harvest and integrate useful/obvious items into dependencies/libraries for the next one. If you are always organizing to dependencies first, you will end up doing lots of unnecessary optimization and sometimes impact shipping speed.
The balance on this is a tough one to learn and changes as you grow more experienced.
When I start developing I encourage redundancy. Premature abstraction is a very dangerous thing that can pin you into a tight corner.
Dependency is also fine, I keep things coupled together a bit close when they are still in development and malleable and decouple once they start to solidify.
It's my personal style of development when working alone. When working in a team, I follow whatever convention that works best for the team.
Unfortunately many applications, or their programmers, never reach maturity. Also there is greater security and dependability in managing certain critical functionality more directly instead of relying upon support from a third party that may not exist with the same level of compatibility in the future.
Do you find that many teams have the discipline to do this? The cynic in me says that software far more often gets slower, more complicated, fat and dependency ridden over time.
More constructively I could ask: what is it that gives a team or project the discipline to do this work of improvement. How could I encourage such a virtue in my own environment?
It depends on the motivation. Ultimately it comes down to discipline.
As applications get popular over time their code base starts to age and get filled with chaotic cobwebs as new features are added or requirements shift. If you don't really care you will do the minimum required to complete your assigned task and await the next assigned task. This is what it is like when the people who do the actual work have no skin in the game. There is no sense of failure so long as you ship unbroken code, and so everything is very mediocre. If you completely suck as a developer then mediocre is your personal resounded success, and so what is just another layer of abstraction if it makes your job easier.
In startups, personal projects, or ambitious open source initiatives failure and pressure are constant reminders of reality. Everything is highly visible and it is always your fault until you ship something amazingly awesome. You need to ship quality code and make changes in the shortest time possible without dicking up the application. You don't want to get caught with your pants down updating a very simple bug only to discover a larger series of bugs in code you don't own that prevents you from shipping your little tiny code fix.
Also keep in mind that nobody (well... very rarely) is going to look into your code to see if you have a million dependencies, but they know when applications are slow heavy pieces of crap. If an application constantly breaks because the dependencies don't play well together and your developers lack the confidence to solve elementary problems your brittle house of cards pisses people off. I am a web guy and many major companies ignore these problems by doubling down on their branding, instead of training their people to write better code, which really pisses me off.
> what is it that gives a team or project the discipline to do this work of improvement.
Ownership of success/failure. When your job or equity are directly impacted (the more directly the better) the emotional connection to quality drastically changes. I apply this level of thinking to my big open source project even though I lose money on it, because I know people depend upon me and I own it. I can be more successful than I am now if I continue to improve what I own or I can be a disappointing failure.
This is harder than it sounds because metrics of product quality the confidence to attain such a level of quality differ by experience. Not everybody has years and years of experience. In that case you need somebody who can supply some technical leadership and not be afraid to hurt people's feelings when things get tough.
There is something to be said about libraries quality and trivial pieces of code. But I feel that this is just a rant about some environment that lacks a good command line parsing library.
Ok I forgot to update some code in one place because I was doing duplicated code. If it is often used place it will be found out quick by QA or users. I kind of have more control over what can break.
On the other hand I made something that is abstract and used in many places in code. I might not be able to tell what will break maybe it will be 5 places maybe none.
In the end you should be able to find similar code in all project files with the tooling. The same with references to abstract code, but abstractions are behind interfaces or are subtypes and it is harder to find in my opinion than doing CTRL-F on all files. Specially with OOP code until you run the program you might not know which method is going to be called.
Trying to DRY would turn this code into a monstrosity (templates - even more interfaces - more ifs than actual logic?). I often find myself forced to let the repetition stand because it is the more obvious code. But it never feels right.
