It's not clear to me why this is true. If LLMs are writing code, why are developers simply not orchestrating the completion of more features instead of moving up the stack to do product development work? Is there some implication that the existence of LLMs also enables developers to run user studies, evaluate business metrics and decide on strategy?
Additionally, if PMs can use LLMs to increase velocity in their work why not focus on all the things that used to be deprioritized? Why, with the freed up time, is generating code the best outcome?
These questions likely have different answers depending on organization size but I'm not sure I understand why orgs wouldn't just do more work in this scenario instead of blending responsibilities. It's not like there's infinite mental bandwidth just because an LLM is generating the code
In a (hypothetical; not quite there yet) world where SWEs are in surplus, there is no reason to have PMs.
The really eye-popping efficiency gains from LLM coding won't come from doing the coding faster but from consolidating the PM, SWE, and QA/SDET roles under the same person. Then you'll start seeing startup/indie level productivity-per-person inside large organizations. Imagine Google is like 50,000 Pieter Levels.
In my last job, PMs were responsible for identifying problems that were worth solving and align with the overall company vision and plans within the owned domain, and design+engineering decided how to solve those problems and what to build. Of course with collaboration w/ the PMs (and EMs).
The job before that, PMs wrote jira tickets and nagged engineering when tickets will be delivered. The "what problems to solve and what to build" questions came straight from CEO/CTO.
> Code review is being unbundled. Its four functions (mentorship, consistency, correctness, trust) each need a new home.
> If code changes faster than humans can comprehend it, do we need a new model for maintaining institutional knowledge?
The humans we have in these roles today are going to suffer. The problem starts at hiring, because we rewarded memorization of algorithms, and solving inane brain teasers rather than looking for people with skills at systems understanding, code reading (this is a distinct skill) and the ability to do serious review (rather than bike shed Ala tabs over spaces).
LLM's are just moving us from hammers and handsaws to battery powered tools. For decades the above hiring practices were "how fast can you pound in nails" not "are you good at framing a house, or building a chair".
And we're still talking about LLM's in the abstract. Are you having a bad time because you're cutting and pasting from a browser into your VIM instance? Are you having a bad time because you want sub-token performance from the tool? Is your domain a gray box that any new engineer needs to learn (and LLM's are trained, they dont learn).
Your model, your language, your domain, the size of task your asking to be delivered, the complexity of your current code base are as big a part of the conversation. Simply what you code in and how you integrate LLMs really matters to the outcome. -- And without this context were going to waste a lot of time talking past each other.
Lastly, we have 50 years of good will built up with end users that the systems are reliable and repeatable. LLM's are NOT that, even I have moments where it's a stumbling block and I know better. It's one of a number of problems that were going to face in the coming decade. This issue, along side security, is going to erode trust in our domain.
I'm looking forward to moving past the hype, hope and hate, and getting down to the brass tacks of engineering. Because there is a lot of good to be had, if we can just manage an adult conversation!
I don't think the industry is ready or has the manpower to deliver on all the promises they're making and a lot of businesses and people will suffer because of that.
> we have 50 years of good will built up with end users that the systems are reliable and repeatable
There's good yes, but also we've raced to build dystopian bullshit and normalized identity theft because most software is garbage. There might not be as much goodwill as you think. Software eats the world, and many simply feel helpless. The erosion of trust you're predicting has already happened, or never existed IMHO.
LLMs may not 1-shot reliable and repeatable systems, but they're a powerful tool that I hope will end up improving systems overall, for reasons you've mentioned, among others.
Here some story. Long time ago, i wrote a (software) accounting system. From 1st principles - nomenclatures, accounts, double-entry, transactions, balance (=current cached status), operations+reports on top of these. 5 tables (+1 for access control later). Very flexible and re-configurable into whatever one imagines. But anyway.
We deployed it at several places. The biggest one - retail with 50+ salepoints across whole region - was the most troublesome.. and after a month+ back-and-forth it dawned that.. they did have very well-working paper system of accounts/documents/data/values flow which was highly optimized for humans and the reality it was in (papers, remote places, delays, etc). Humans forget, make mistakes, displace things etc ; paper rots in time; distances make things out-of-sync - yesterdays invoices from village X will come tomorrow - maybe - .. etc. So their document flow - and even people-roles - were aligned with that system. Duplicating some things and completely avoiding others.
The new software had no such notions. There was no such thing as forgetting, displacing, out-of-balance. And while temporal stuff was fine, the document flow - even if consisting of same dot-matrix-perfect documents - was different to what they have used to. So.. it took them - and us - 3 months to retrain the personnel to unlearn their old system and to start actually using the new one properly, and enjoying the ride instead of fighting it.
Back to the topic.. i guess the old system of software engineering, built last 50+ years, has to be rearranged now. Not everything, but.. quite. Some things probably may wait for tomorrow, as the paper notes, but some - like roles and what they mean, and the cognitive/understanding chasm - is for yesterday..
Edit: after reading the whole paper, i think there are some things that can be "loaned" from hardware-design (chips etc) flows and processes. i see this analogy - the hardware's target environment (actual physical world, e.g. silicon etc) is also non-deterministic.. just mostly. Things like Requirements engineering, design-for-test ; all the enveloping (heat, power etc) and whatever else may come handy (i am not hardware dev, only seen these from aside, e.g. from a Verilog compiler)
Up the abstraction ladder; we conceive axioms and constraints; we define actors and objects; we direct rules, flows, sequences and say when and how each one of them lives and dies.
May you live interesting times (some say this is a curse)
I'll say it again and again and again: Rust is the best language for ML right now.
You get super strict, low-defect code. If it compiles, that's already in a way a strong guarantee.
Rust just needs to grow new annotations and guarantees. "nopanic", "nomalloc", etc., and it would be perfect. Well, that and a macro-stripped mode to compile faster. I'd happily swap AOT serde compilation (as amazing as Serde is) for permanent codegen that gets checked in and compiles fast.
This is a fairly easy to read doc discussing some of the challenges with using AI tooling in a forward thinking and disciplined way. Coming from Thoughtworks it also gives a bit of gravitas and legitimacy.
There's good stuff in here. It would be a shame for the larger HN community to miss out on this conversation.
Why? I thought the opposite. Consultancies, of which thoughtworks is one, publish thought leadership as marketing material.
> Engineering quality doesn't disappear when AI writes code. It migrates to specs, tests, constraints, and risk management.
These are generic "thoughts" you can get from any agency pushing AI SDLC. The pages I read through left me wondering if there was even a real retreat.
I agree that it's marketing material, but that doesn't instantly make it garbage. I've been reading their quarterly Thoughtworks Radar for a while now and it's clearly put together by people who understand the industry.
They at least put the effort into having the retreat and putting this together. Would other consultancies (who we know little about) have of done the same?
