10-20 years ago that wasn't really the case, and such practices were much rarer even in more tech savvy teams and organisations.
Same with web development related stuff. People aren't using tables for layout anymore, nor are things like accessibility some sort of completely ignored concept that no one takes seriously. And people do actually use CSS for layout purposes rather than font tags and spacer images and whatever else the days of Geocities style web development had in store.
So it's definitely advanced in many areas.
In the 1990s, We used FTP to send files to web servers because we didn't have WebDAV or any more secure protocols at the time. We used tables because that is what we had.
We had fewer tools than today, it really doesn't amount to any learning, just better tool availability. Knowledge itself seems to have stayed about the same.
To state the obvious, 10 years ago was 2012. Absolutely all of the things you list were standard practice. 20 years ago was 2002, also same thing (no, git didn't invent version control).
30 years ago I was using version control at my first job, it was expected practice. As was having test coverage. None of these things are new.
Deployment was different, granted. We didn't "deploy to production", we shipped a box of floppies to users.
But writing software is really like creating plans: plans for what the machine should do. Of course there is much room for reusing old plans as components of your new plan. But still every plan must be about what is needed at the moment, not about reusing existing plan-components.
In human communications we don't "reuse" old communications much do we? Well maybe we do a bit, using common utterances. But those are more like idioms of the language, not "subroutines".
There's also a weird opposite myth that old software is bad, and urgently needs to be rewritten with newer paradigms.
I'm currently hired to replace a piece of extremely stable '90s software with hundreds of janky eventually consistent microservices because for some reason. That's how software must be written nowadays, and doing it that way makes it better in some unspecified fashion.
It's not at all clear what is broken and what we're trying to improve with this work. But it's top priority.
Well, AngularJS was such a trending thing that we HAD to rewrite everything in it. For no reason. No new feature. Bonus time : i was the only dev on this project (because i was replacing the only front-end dev who was leaving) and i had not a single idea of how AngularJS worked at the time.
No supervision, no prior experience, no business goal. I'm not proud of the result. But at least i learnt a lot of things (including CoffeeScript, i'm sorry for who took the codebase after me).
Some of the challenges with old software is the ecosystem around it. Is the language it's written in still supported on modern OS? (Visual FoxPro for example). Is the database it's using still around? What about the libraries? Are they tied to a certain runtime version with no update path? Do you even have access to the source code for these dependencies? Or the application itself? Can it build on a recent machine?
I've seen projects where some infrastructure around an old piece of code was re-written to make it run on modern platforms, but that takes good engineering (some assembly required) that companies often simply can't afford.
One of the curses of modern software development is that so many of the resources we rely on are highly opinionated. We have frameworks and toolkits and scaffolding scripts and proprietary formats and convention-over-configuration to do things that could have been done with flexible, self-contained libraries and open standards.
These opinionated resources are often easier to create but they can cause great harm over time as a small application that got a boost from using them initially grows into a large application with changing requirements. Maybe those requirements can't easily be met if nothing composes and nothing can be adapted and each dependency becomes a fight between the direction its developers choose to take it that might not be right for the application and the need to keep updating because the developers don't want to support older releases of their code with essential bug fixes and security updates any more.
In this kind of environment, it is hardly surprising that some teams prefer to build things in-house even if they are reinventing a few wheels. It is sometimes the only way to retain control of your own code or to regain that control after the technical debt becomes overwhelming.
Or the time. When you've been given a hard one-week deadline to produce something, you can commit to writing something that works with a lot more confidence than you can commit to figuring out somebody else's piece of garbage.
If a company doesn't have a stock-based comp and sticks to prevailing wages/CoL there's no incentive to help the company succeed. Especially with non-technical management, the best thing to do for ICs is often to aggressively rewrite (in the language that's trendy right now and that better companies use) and get promoted by showing off a significant output. Then jump to a better company having spent a year training on their stack at the previous company's expenses.
Or that old C library written for old versions of OS/other libs.
Sort of.
People who write individual plans don't reuse them or believe in reuse. They're useless once the plan needs to change.
People who write metaplans aren't solving problems, they're trying to to solve general classes of problems, like "Manage booking at any business that has things that are booked"
Actually a lot of mechanical manual work we do are subroutines.
Examples
1. Make an omlette (params like: which kitchen or ingredients are derived from context)
2. Change engine oil
3. Take blood sample
> 14. In the long run every program becomes rococo - then rubble.
And;
> 15. Everything should be built top-down, except the first time.
I think these are helping to understand the lifecycle of a project.
A first top down design should be understood as a draft, a direction. With experience these drafts get better, but the act of bottom up programming is necessary to discover details, problems and requirements that inform a more robust and flexible design.
A rewrite can be partial or total. Maybe it is enough to refactor some components, maybe a new design was discovered during the initial iterations, which yields a system that composes and evolves more robustly and works better as a whole.
As a web developer I program user facing applications, so there is always a kind of pressure to have highly domain and feature specific interfaces. Just get it done, move to the next.
But by learning from the Lisp world [1], Out of the Tar Pit [2], software design books [3][4] and the pain of experience, I've come to the conclusion that there is a better way. There is often a general layer that wants to be discovered that we can use to build the needed features. A workflow, framework, DSL or maybe just a bunch of functions or a data model.
This layer is often very hard to get right the first time around, especially if we don't have strong analogies already. It is often more natural to discover it by writing or trying to write the specific code first. Premature abstraction is the root of all evil.
But also [0]:
> 7. It is easier to write an incorrect program than understand a correct one.
We should be brave enough to move forward, while assuming that we're probably wrong. Scientists often practice this principle and try to figure out in what ways they _are_ wrong. I think this is a good way to approach programming too.
[0] http://www.cs.yale.edu/homes/perlis-alan/quotes.html
[1] https://dspace.mit.edu/handle/1721.1/6064
[2] http://curtclifton.net/papers/MoseleyMarks06a.pdf
[3] https://web.stanford.edu/~ouster/cgi-bin/book.php
[4] https://mitpress.mit.edu/books/software-design-flexibility
It's almost like the issue is not that code is available, but how people use the code that's available, and no one seemingly likes funding open source code.
It’s almost like software is extremely complex and security is very hard in general. You’re always going to make some kind of trade off.
The problem is that we as humans don’t know how to correctly estimate risks like security risks. That means it’s not priced in when you go and ask “should I incorporate software package X into my build??”. “Should I automatically take updates from my upstream?”. There’s no good answers here either. Ultimately you need to be careful about which dependencies you take on and which ones need to be kept up with and which ones should be pinned (but even in the best case scenario issues will occur)
I’m not sure how this meme got started but it’s toxic. Why does free software need funding? Free software needs contributions. Big corporations make contributions by paying engineers. Everyone benefits in this ecosystem.
But we lack a decent way to express that sort of data flow constraint when deploying software.
But maybe that's the part that this industry is incapable of learning.
And why are moving to apps that only work online? Networks are slow compared to desktops. Programming seems to be the art of doing the worst thing possible. Our computers are getting faster and faster, but we're relying on communications that are vastly slower.
And another thing: Windows updates, or even Firefox for that matter. Software shouldn't be so full of problems that you need to constantly update it. Just update it every 2 years. Sure, you get the supposed latest and greatest, but updates are a messy process.
Perhaps the difference between programming and "real" engineering is this: parsimony. In engineering, you have to do more with less. In programming, the attitude seems to be to shovel more spaghetti onto the plate.
Update: I'll add a further point. It's not just parsimony, it's also cost of errors. In physical products, a flawed design that makes it into production is costly. So you have to get it right. With software, you can afford a slap-dash approach. And that's what we see.
But there are upsides as well: the devices are usually attached to services for which the device acts as a conduit. If the service is valuable, you can sell devices and keep collecting money after first sale, driving huge margins. Didn't Hackernews post a lot of Fs in the chat for the original BlackBerry service -- one of the first devices of this kind to reach a mass audience -- once it was shut down?
You could argue that we should adopt the same testing strategy of mission critical pieces of logics where 1 LOC has 1+ LOCs of test code, aim for 100% coverage, ... . But then your windows license would cost 1000s of $.
And about your final remark. What you call real engineering has to cope with well defined requirements and the context is well known. You know beforehand the maximum weight a bridge should be able to support. You know the configuration and properties of the soil where the bridge will be built, and you have good estimates of 1 in 100 year extreme events magnitude.
Software doesn't have that luxury. If we build a web server to handle 1k qps it still is somewhat likely that said server might face spikes of 10k qps. Try to do that with the bridge mentioned above. Forget failing gracefully. Additionally users have few ways to use the bridge. You either cross it by car / bus / on foot. The same web server might face someone trying to send a payload of GBs where you would expect few KBs. That's at least partly why code is messy. The space of possibilities is much greater and we somehow need to write software that still works.
"order for learning to most effectively improve our state of operation"
Things don't move "most effectively" in anything. I guarantee that other engineering disciplines do not react to new tech / innovation in any sort of optimal way. People are driven to work in certain ways by all kinds of things, and mostly has to do with incentives and social reasons.
"But I very rarely see articles looking back at past languages or technological fads and looking at current trends through that lens."
There are historical surveys all over the place, even on this site.
I don't want to attack this article too hard, and the motivation to ask whether or not we are are really learning from our experiences is good, I just feel like people are more flustered by the fact that there aren't magical best practices you can memorize and apply yet, and that most human knowledge comes from experience and intuition.
The problem is not with developers, it's in crappy foundational computing layers.
NPM is somewhat the result of JavaScript itself being so bare bones. The cost of this community-based pseudo standard library is enormous: fragile, insecure, unstable. This doesn't mean we have to drop it, it means perhaps we do need a JavaScript standard library. Possibly mixed with NPM vendor packages that take responsibility of large clusters of functionality as a single dependency that is maintained over time.
Almost all software now requires very frequent security updates and we even dispose physical products like smartphones because they no longer get security updates. Instead of accepting this reality, why is nobody working on a foundational software stack that isn't this damn insecure to begin with?
Almost all websites on the web do not meet accessibility standards. And then we beat developers over the head in needing to learn proper accessibility best practices. Which doesn't work. Progress is near zero. So we might as well ask: why are our tools so primitive and bad, why do they not lead to accessible UX by default?
We complain about performance, or the lack thereof. But the problem is mostly in the stack itself, crappy and slow abstraction layers.
I could go on, but I hope you catch my drift. In development, you need to take the lowest common denominator. An average "bread programmer". We expect this person to personally dodge the many gaps in our crappy computing layers and it's far too easy to go wrong. This single individual is expected to be a top engineer, architect, security expert, performance expert, accessibility expert, and so on. And they also need to ship tomorrow.
We basically expect 28 million software developers to all be super heroes. They're not, they keep making the same mistakes not because they suck, instead because we have computing layers that suck. No amount of "awareness" will fix that.
A problem the article doesn't mention, is that the percentage of still-programming developers who are young (<30 years) is higher than in other professions, for several reasons, so the lessons learned 15 years ago are often lost.
I wish the meme of "software engineering is young" would die. This profession is not younger than any other technical profession, in the modern sense. It was born out of prior practice and is still taught using the same techniques and philosophy. We are not special.
Furthermore, modern engineering is a mid to late 20th century practice - across all domains - not just ECE. We inherited some practices of the first industrial revolution, but by and large the techniques and tradition of education of engineers is consistent across the disciplines and evolved out of the post-WW2 marketplace for new technical products.
It would serve the author to remember this well coupled with the following :
Engineering is the discipline of systematic problem solving. Computer science is the mathematical study of abstraction. Software development is the practice of applying computer science to the domain of engineering.
This results in a force amplifier - much like a lever - but rather than creating torque what we're doing is amplifying the ability of creative problem solving to be applied and re-applied without re-engineering complexity. That abstraction and reuse propagates throughout entire industries in ways that are now invisible because they have become ubiquitous.
"Does the software industry learn?"
Well of course it does. We encode our learning of the problems we face everyday. The chips powering the computer I'm typing this on was designed with software that prototyped and ultimately yielded their design; packaged on circuit boards designed with software that automated hardware engineering checks to validate the design; encased in an enclosure designed with software that encompasses decades of 3D design and visualization technology; all fabricated using hardware and software tools with ambiguity resolved through digital communications that can be seen as a recursion of this paragraph.
It's turtles all the way down.
If you think we aren't learning everyday, propagating that learning, and preserving it for generations to come you do not understand the practices we undertake everyday. There is so much learned technique, practice, and theory encoded into the basis of our everyday lives it now seems so invisible you can blog about it.
And because of this, software has poor standardization practices.
Engineers in other sectors could prove their qualifications by showing certification diplomas, even education and years of experience would count. In software your qualification would be determined by some random dude and how well you fit his CS fetish of the month and that is it. Experience and education is only important while filtering CVs. If you designed a chip, an aircraft or skyscraper it is immediately impressive. If you built an FX trading platform it means almost nothing if you are a bit skeptical about pair programming.
It also applies the other way round. If project uses microservice architecture, what exactly does that mean? The result would be a wildly different if you compare such projects at FANG and some investment bank. How software is tested and released will be different in any team. And how that affects the poor developer will vary too. What is a good software? We can have an opinion, but there is no formal definition.
And because of this fragmentation and fluidity software as and industry is doing very different things. It will learn in some cases in others it will be incredibly ignorant.
Other crafts define their professionalism according to a standard baseline defined by governance and credentials. Examples of credentials are licenses and certifications. Examples of other crafts that require credentials: medicine, law, engineering, flying, driving, real estate, information security, networking, food preparation, education, public securities, and many more. When this is brought up in software the result is excuses and whining, hardly an indication of maturity.
The problems with the industries exaggerated immaturity result in profuse subjectivity. There is always a perception, at hiring, that software developers are rare. Perhaps poorly trained or developed less than desired expectations but software developers are not rare. Nonetheless the problem is exaggerated in a cycle of attempts to increase availability among an undesirable candidate pool by abstracting away many challenges inherent to the practice. This is cyclical because instead of bridging a training or development gap it instead breeds entitlement and insecurity that is paid forward. Since there are no industry standards of acceptable practice this cycle becomes a race to the bottom.
This is the very essence of young.
Software has a deliverable that can be tested prior to delivery. And there are domains where software has to be developed within a QMS, those domains have quietly built elaborate process and systems dedicated to quality of their software.
Using licensing to ensure quality results is an easier sell in scenarios where such a system can't exist, like when the practitioner doesn't get (m)any retries.
Are we worried something is falling through the cracks of the quality system approach? Are there properties of a deliverable that those systems can't ensure?
Some product domains are so inconsequential to life that it would be a laughable non-starter to write bars for their quality into law. Is a blanket licensing requirement a way to get at practitioners operating in those fields anyway? Because the individuals will be more of a pushover and easier to wage a public campaign against than the company they hypothetically work for?
For my part, I try to contextualise the way I/my peers work now with the other things we tried over my career, and maybe the decade or two before that which contextualised _my_ junior experience. For example, I run a monthly magazine with a colleague, and in our devops issue we discuss how ideas in DevOps come from manufacturing, from Rapid Application Development, and other inspiration (https://deprogrammaticaipsum.com/play-it-again-sam/). I just recorded a podcast episode where I look at _what_ documentation was considered “comprehensive” at the time of the agile manifesto, and what docs are still useful despite the industry-wide tendency to eschew all prose (https://www.sicpers.info/podcast/episode-47-comprehensive-do...).
So, disclaimer, I am new to Typescript-NodeJS and the npm world but have a significant background in the JVM / Spring backend world with Kotlin and Java. I am also too young to be this cynic but I am honestly just baffled.
To me the entire NodeJS backend movement seems like they are "Java sucks!" hipsters jumping into the trendy JavaScript ecosystem of the modern web era just to show how cool and progressive they are (???) - all while rediscovering and reinventing backend features that have been invented and in widespread and battle-hardened use for 2-3 decades now.
Wow we are using NestJS, Jest, annotations, a dependency injection module and Typescript... congrats, you just rediscovered type systems, mocking, Maven and Spring Boot as they are very handy for backend development.
All while bloating your fresh hip ecosystem to an even uglier Frankenstein monster than you can find anywhere in the Java enterprise, transpiling back and forth with 10,000 config files to make framework XYZ interact with each other.
And then, JavaScript is running underneath with all its limitations and quirks that will reach feature parity with Kotlin with ES2048. But yea, I guess now you can send JSON natively between systems because it's first-class.
Yes, we have things like NestJS, which I refuse to use precisely for the reasons that you point out: it looks like Java Spring and adds a lot of complexity where it is not needed.
I am also painfully aware that TS can be abused to write Java-like code but that's on the developer. However, the abuse notwithstanding, the fact that JS is a simpler language protects it from Java's key problem:
"Because the problem with object-oriented languages is they’ve got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle."
My (limited) exposure to Java was quite painful because I had to deal with layers and layer of inheritance and abstractions that at some point made it very difficult to do what I needed to do. The language itself was fine except that it easily allowed for this sort of architecture.
Also, the advantage of writing the frontend and backend in the same language should not be understated.
Hey — could you go into more detail about some of the times when inheritance and abstraction in Java were a problem, or when you had to deal with the gorilla issue?
I promise I'm not trying to catch you out or start a language war; as someone who lived in the Java world for a long time, I'm interested in which parts of it people who visited briefly have found annoying or painful to deal with.
This five year cycle turns up very often; that seems to be about the length of time that it takes for an idea to form, ride the hype wave, sink into the trough of disillusionment, and then be replaced by a new hot thing pushed by the next generation of fresh, idealistic programmers. Usually the new hot thing is the same basic idea as the hot thing one or two generations before the current hot thing, which always starts off bright-eyed and bushy-tailed before it runs into the wall of leaky abstractions and murderous corner-cases and the stark horror of being used in the real-world for real products. Then nearly all of the trade-offs and short-comings of the older iteration of the idea come back to haunt the new thing, and the gray-haired programmers over 35 who've seen this cycle come and go time after time have a chuckle.
All that said, I feel like documentation is getting better in newer technologies.
Automotive engineers in 2022 are still creating new designs of water pumps and alternators. The new design meets some new requirements for the particular application, be they thermal, spatial or cost-related.
The date picker designed in 2022 is to be used with 2022 applications, IDEs, toolchains, etc.
Software engineering is not a craft. It is a discipline. If society is paying people these exorbitant sums to be the equivalent of digital Etsy moms, we're getting a raw deal.
A few things that could justify rebuilding the wheel as it were:
1. The thing doesn't work for the environment it's being used in (corporate network, obscure mobile browser, screen reader, whatever)
2. The thing doesn't work for non-Gregorian calendars
3. The thing doesn't work consistency across various types of clients (this reason gives me some nostalgic flashbacks to the early days of jQuery)
4. The thing is fine, somebody just wants to see how the sausage is made to learn of it
There's probably other reasons one would want to re-invent a common UI element. It's really hard to say how objectively good or bad a particular attempt is without context and more clarity on what metric it's being judged by.
People in our industry need to know how to make the fundamentals from almost-scratch. In electronic engineering I learned how to bias transistors and build RC-filters. Heck, in chemistry class we even learned how diodes and resistors are made. When I studied music I learned scales, chords and rhythm... why should I, if I could just download some WAV loop off the internet, or sample some old song?
I don't see why software has to be special. Haven't we commoditised the profession enough?
But a new era has arisen, with a new God from this combined mind. Copilot rises and casts a great shadow on us all.
In past years, children were born with monads in their minds, functors at their fingers. Teens rose from sleep speaking in bytecode and LLVM IR. But now that is lost, mankind’s combined mind is being eaten by the power of Copilot and the children’s fingers only hit Tab.
We should never have known each other so intimately. The risen god of our inbred mind is a dark spiral. Thought itself fades.
Similarly in software there is not "one size that perfectly fits everybody". Therefore there is much need for customization.
When you write software you are in effect not only creating the program. You are creating a system that consists of the computer, its program, and its user. The way the task is divided between what the human does and what the computer does can be done in very many ways. In some systems that results in a system where users need to do more but thus also have more options for controlling the system. Therefore there are so many different ways any "applications" can be written.
No program is an island. It is something that must interact with its users. It is always designed to be a part of a bigger system consisting of itself and its users.
Tech organizations value their managers in terms of their budget, headcount and being in fashion. As such the tech business model has always been about getting "good enough" results through a brute force process, possibly with the latest tech. Expertise/deep knowledge is only added/valued when the development gets stuck.
Sure, GoLang could learn something like "don't make things nullable" but it would be easy to say, "yes I know that, but in our case, this creates a benefit because of X"
Just like people who believe in TDD, DDD etc. they might all work, but that doesn't mean they are objectively the best way to do things in all circumstances.
I think the better target is learning how to think and analyze more effectively so we know what we are trading off more clearly.
― Georg Hegel
