The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and headcount you're willing to give, how much tech dept you're willing to take, etc., for the privilege of getting their business.
If you see it for what it really is, you'll see that it works pretty well actually. The business gets more out of you for less to them. It was never about having an accurate timeline or helping with planning or prioritizing, and always about negotiating a better contract with the dev team.
Now keep in mind that the "business" in this case is a person who need to report that through their amazing prowess of administration and management, they personally managed to get X feature out during their last review cycle at Y cost with impact Z. This person will not need to deal with developer satisfaction, retention and performance. They will not need to deal with the impact the lower margins they pushed for had on the next feature delivery, or the continued maintainance of the systems. And if the dev team had to lower the quality too much in order to meet the quote they put out, that will be 100% their fault, the "business" will know not to use them for their next contract, or they'll expect the dev team to take on fixing all the issues at their own expense once more.
These days, I only give estimates in terms of units but without numbers. Hours, days, weeks, months, quarters or years. Some relatively small number of those units. If you want a quote it will take an extra 1/4 of the estimate worth of time for an exact timeline. I really wish we would treat sales people the same way. "How much money is this contract for? What date will it be signed?"
This is an incredibly important point. I was "raised" in commercial software at a time when there was a lot of condescension internally towards ourselves as an industry because we couldn't do estimation and planning, and building was the favorite comparison. A $100 million building couldn't run late or over budget. That would be unimaginable, and it doesn't happen because those people are too serious and professional to let it happen, not like us unserious, immature software doofuses. Turns out that was a complete lie, but at one time it warped the profession, because it made people believe the answers already existed, right around the corner, something you would figure out pretty soon just like last year you didn't know regular expressions and now you did.
The problem with that was that when somebody walked in with a button-down shirt, oozing confidence and saying they knew exactly how to consistently deliver software on time and bug-free, those people weren't laughed out of the room. Elaborate heavyweight processes, and their "lightweight" (hah!) cousins, could sell themselves as proven, established solutions. We could believe that bullshit because we had been fed the lie that other industries had conquered these problems.
In well-run sales teams, that is exactly what happens. There is a constant move towards more accuracy and tightness in forecasting.
In poorly run sales teams, the reps basically set their owns quotas (padded, of course, to mitigate against massive uncertainty) and then are very imprecise at managing the WIP. Which sounds like a poorly run engineering team too. In sales, though, at least there is a very objective measure ($$$ closed) at the end of the period. Also, the period is fixed typically (months, quarters, etc).
Engineering is obviously different... what's the measure? LOCs? Obviously not...
Edit: just to be clear, I'm saying that well-run sales teams and well-run engineering teams probably have a lot in common, not that they are exactly the same.
Potential deal with client Foo will be for $X, to be signed on Y day, with Z% chance of closing.
On an individual level, it may be more precise than it is accurate, but it’s a fairly standard and important part of sales operations. And yes, the estimate dates are often just ways to express orders of magnitude (put next Friday as estimate dates not because it’ll definitely close next Friday, but bc it’ll probably take about 2 weeks). The % chance of closing are typically pre determined by what stage of the deal you’re in.
However, when applied across dozens, hundreds, thousands of potential deals, it allows the organization to understand their sales pipeline and sales velocity fairly accurately.
Of course the numbers are gamed a bit. And some sales people are much worse at estimating than others, or less consistent about keeping the numbers up to date. But the benefit of these practices are massive.
Then you must be either an independent contractor or have some magical source of job security. I couldn't get away with this kind of estimate.
A lot of the rest is Planning fallacy[1], particularly with individual contractors, but larger contracting businesses can absolutely come up with realistic estimates based upon past experience.
So having a rough or concrete time line / roadmap is helpful for these parts of the business.
I also believe that a feature request is incomplete without an estimate. It is like a friend asking you to go buy him a car. What does he need a 3000$ subaru from 2003 or a 2021 Ferrari? Of course the friend can give you an insanely accurate list of specifications, but giving you a budget + some guidelines will probably be more effective for your friend and more fun for you. Especially considering that you know much more about cars (read software) than your friend. Knowing the budget you will be able to make much better choices.
If you take the allegory of a quote (which I think is correct) then you need to consider that the haggling process always also includes scope of the product. „I‘ll give you 5000$ for the car, but only if you include a free car wash“. And you can always use an estimate as defense against feature creep. „We are running low on budget, I suggest descoping xyz into the next iteration. Otherwise we won‘t be able to ship on time“. If on time is not defined then you can never have this conversation.
I think where companies can get in trouble with software is when the initial rough estimate somehow gets blessed as the plan of record to cement in everyone else's critical path, and subsequent estimates are treated as engineering screwups rather than more informed predictions. I don't know what the solution is, because stuff like this can happen with everyone involved being super competent and having the best intentions. And sometimes there are engineering screwups, neither side is always blameless.
Power imbalances between organizations are where natural creative conflicts go bad. In a less healthy company, whichever org is on top politically tends to be blameless, whether it's fair or not, which ultimately hurts everyone's ability to do their best work because conflicts between orgs are less likely to be resolved based on what's best for the company or project as a whole. In ideal companies, orgs are balanced such that they can push back on each other for good reasons, but not for zero-sum, shit-rolling-downhill reasons. I'm pretty sure ideal companies only exist briefly or never though.
So then you goto build the car and everything's going great till you find out that the latest research shows the specific design of engine you're using doesn't work with the eco friendly fuel that is now being mandated by the government. And suddenly killing cows is uncool so you have to change all your chairs to fabric or source fake leather, but half of your company thinks that endorsing fake leather is hide-appropriation and you're now spending a third of your time in HR going through training.
And eventually you get about a week out from delivering the car and the customer calls your PM and says they want the car to have a top speed 30mph over the original specs while maintaining the fuel efficiency rating. And the doors should swing up, instead of out. Like billionaires'.
But because it seems so simple to estimate how long a piece of work will take for everyone else, it's really difficult to get other people to realise how hard this is.
I've sat in meetings saying "I don't know how long this is going to take" repeatedly until the rest of the organisation works out that they're not going to get a definitive date. Then suddenly the marketing team plans for "OK, well if it's done by this date then we'll have enough time to make this trade fair. If not then we can skip that and do the other one". They'd like to have a definite date because it makes their job easier. But if they can't have one, they can work around it.
> I also believe that a feature request is incomplete without an xestimatex delivery date attached
This is so true, as long as the devs have control over the scope. "We don't know and can't predict how long it's going to take to build the thing. If we can't build it all by the deadline, what's more important: delivering something on the deadline, or missing the deadline and delivering the entire feature set?"
To use your car haggling analogy: You've given me $5K to buy a 4-door car less than 5 years old. If I can't find one like that, what's more important: the budget, the doors or the age?
I think it's a lot more natural in business software to say something like "we need the best version of XYZ you can do in three months" rather than saying "we need XYZ" and hoping the estimate is three months. That way everyone's cards are on the table and can work together on getting what they need.
- does it have to be instant or can we run it as a batch
- maybe it’s enough to have this tool in English only
Etc.
The big cost tends to be software quality as it is less visible/harder to measure than dates and scope.
Sometimes they'll modify it, sometimes they won't but at least I won't have to argue about it or get in a situation later where suddenly they pretend like their estimate is my estimate.
But plenty of engineers do estimate based on what they think they can get away with or other negotiable ways.
Often I see this haggling down of estimates and then microagression if estimates are not met but no need to work for free to make up for it.
And I’ve also seen wise use of estimation but that is rarer!
I've got a question for you here, what is a wise use of estimation?
What problems are estimates the ideal solution for? In the context of software development?
I can think of only two: planning headcount and priorities (aka planning the order in which you do things).
When you focus on those problems, you realize there's no need for an accurate estimate. All you need is relative sizing between requests. This seems twice as much work as that but gives us half the benefit. Ok probably we shouldn't do this one first. And even then, often time impact is all you need for priorities. Like who cares what takes you longer, if you just focus on building the next most impactful thing you'll very often succeed.
The question of: "When are we going to be done this?" Is almost always irrelevant except for what I mentioned "asking for a quote". Like, who cares when it's done? What else you'd rather we do? I can tell you if your alternative will be any faster or not. And is this not the most impactful thing? If so, let's finish it and stop slowing us down with these irrelevant questions.
Now for headcount, it's the same thing. Look how much you did last year with what headcount. Ask yourself if what you're hoping to do this year is of the same relative size, twice as big, half as big, or smaller? And then plan headcount accordingly. If twice as big, double head count. Overtime you'll learn if twice as big requires doubling headcount or 1.5x or 1.2x or 2.3x etc.
I've almost always had business partners that were just that - partners - invested in the product and the team long term. I am the same way now that I am "the business."
Estimation is a useful forcing exercise for thinking through what something takes, identifying risks and hard parts and deciding upfront what to do about them.
It also becomes a good measure of people's integrity - what they do once it's apparent the estimate is off. 100% of the time, when engineering came back saying "we underestimated X, it's much harder" - it was totally fine.
It really made my day :)
TFA is talking about estimates inside projects, which are not time based anymore in any environment that cares about the problems you laid out.
Personally I offer my customers to either pay by the week, or pay by extremely fixed scope with a 3 month definition period up front and a 50% buffer above the estimate. I stress that paying by the sprint lets them change objectives at any time, gives them an always-current report of how much "effort budget" they have left to work with, and guarantees they only pay for what they use. That felt like a risky move the first time I did it, but I never lost any business over it.
For either, there is often a function from time spent to quality. 100% perfection is basically impossible and before that the function increases very slowly, seemlingly logarithmicly.
For SWE, expectations are often close perfect solutions. Too greedy effort estimations cause a lot of trouble. For DS/ML, however, perfect is usually off the table and this fact is widely (not universally though) accepted. When it is accepted to give estimates in this way, suddenly there no harm from being quoted on it and I really don't mind to give estimates anymore, where I just make a guess at a good 80/20 point. If I am wrong with that point, chances are nobody on the outside/higher up ever knows.
This may be different in domains where very clear targets have to be met (e.g., "self driving cars that pass lawmaker's requirements for use on the streets") and then I'd guess it is a true nightmare.
Like this, I never felt overly pressured by ML/DS deadlines over the last years. Some things were great successes, sometimes the quality wasn't great enough and projects were stopped or customers left. But there never really was a case where anyone thought that working extra long might have been an option to meet higher expectations.
I don't really have a solution for SWE, I don't really see how one would sell something like "I can do it in X time and it will only crash / stop working / make mistakes / become too slow / have vulnerabilities so often. More time will lead to fewer problems". This just isn't what's expected. But at least for complex systems and security vulnerabilities, I'd argue it is actually quite true. Guarantees for 100% perfection just aren't realistic. Avoiding the most obvious pitfalls is done rather quickly and the more time spend, the more is needed for further improvements.
Some parts seems to hint the devs are an outside consulting shop, and some parts hint that the devs are on an internal team.
Most internal teams can blow through dev deadlines over and over again for years without any real repercussions. And while outside dev shops can't blow through estimates like internal teams the individual devs usually can unless they're in a leadership role.
Now you have a recipe for a highly adversarial relationship between management and engineering throughout the planning and development process.
According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather.
If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.
Short of large and completely government funded development projects, I think it would be a struggle to get data. Few businesses would be willing to offer up development processes and surrounding data due to potential IP lost. Any organization that has good processes have it in their interest to striffle others from discovering how to improve theirs or learn from the success of others.
Part of the issue is that academia largely just doesn't pay for software development which they can leverage as an accessible cheap data source, it's done as a completely privatized exercise. Any research that requires highly protected commercial processes is pretty difficult to get any traction on unless you're inadvertently rediscovering the same processes (from my experience).
With that said, I feel like the amount of empirical data you'd need is going to be incredibly high, much if it not even currently being collected.
> The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).
> Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?
> Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.
Is this telling me that most theories about "sw estimation best practices" are cargo cults o-O ?
IMO the theories have sprouted more from management which unfortunately is probably the best equipped today to view the trends over time and compare them to actual results (given the lack of empirical data). If we admit that our ideas of estimation have come from management, we should also admit their conflicts of interest in the matter and the various management fads that come and go. I think as humans we all search for patterns and try to generalize rules to solve our problems, even when we have an incomplete view of things to begin with.
Finally, even with a generalized rule we should realize how different people are team-to-team. I haven't worked on two teams where the same exact processes worked for everyone. The best teams and managers I've had will observe a team over time and tweak the various processes according to what works at that point in time. The worst managers were the ones who idolized a specific style or person and copy-pasted their opinions into their workplace without listening to the team.
I'd wager that data is just as poor overall, but the sheer volume of data available might be able to help get some sort of consistent conclusions from them.
My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.)
But at that point nobody cares! People go on with further projects, and no data is ever collected.
Nobody learns.
When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away.
Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done.
It's insane.
The triangle is real[1]. Something's gotta give when a deadline can't move, and as we know from Brooks, software development speed doesn't increase linearly with more people. So it's primarily quality or scope that must suffer to hit a software deadline.
[1] - https://en.wikipedia.org/wiki/Project_management_triangle
And no doubt a great deal of crunch time, their fair share of delays, some major messes and the occasional outright cancellation. And if I'm not wrong their roadmap is not public information.
I personally find it hard to believe that a large company can exhibit all those adjectives you list. Everyone finds this hard.
Do they tell you at the beginning of that cycle what features to expect? Or do they constantly work on features, and then at presentation time do a freeze?
Requirements gathering, general communication, testing, production rollout, coordination, research, final approval - it all adds up. For a 2 week project you might only actually spend a day of ass-in-seat coding. And that ass-in-seat coding is what they tend to estimate.
That's what the research has previously found[0] decades ago -- engineers will generally correctly estimate the part they are best at, but will either neglect or under-estimate other work. The old-school ways of dealing with this are canned Work Breakdown Structures and checklists.
This is not unique to software.
[0] van Genuchten, "Why is Software Late? An Empirical Study of Reasons For Delay in Software Development" https://web.archive.org/web/20160914040151/https://lab.cs.ru...
For stuff that I'm doing on my own? I can estimate pretty well. I know more or less how long the writing will take based on length. I'll allocate more time for research if I need to do more than a base amount. And then I add some slack in case I'm struggling with something, priority interrupts happen, or I'm just not into it on some given day. (Underpromise. Overdeliver.) It only gets harder with bigger projects. It helps that I'm quite fast compared to most people so even with significantly underpromising people are usually very happy with the estimate.
But where it's really hard is work that involves getting buy-in and input from a lot of different people. Those always take forever. (And at this point I pretty much know they'll take forever, a lot of which is just latency rather than actual active work.)
Estimation is really hard, especially if you're dealing with new challenges, new people, and new expectations. Throwing moving targets (unplanned changes) into the mix, as you noticed, makes estimation even more difficult.
The thing is, it very rarely is the case that a wrong estimate will sink a project. Things get delivered late all the time. No one dies. Most stakeholders aren't like two year olds throwing a tantrum because an unforeseen problem delays a deliverable by a few days (admittedly some are close).
The actual definition of failure also needs to be considered. Is it considered a failure because a project manager ended a colored gantt bar at a particular date and the actual time was longer? Or is it a failure because the customer cancelled the contract because a deliverable was not available on the agreed-upon date? Or is it something far more nebulous, like a slow death march towards a misguided goal that burns everyone out and compels them to sip artisanal coffee and daydream about leaving instead of grinding onwards and hitting targets?
Ultimately, people "don't care" because accurate estimates are not the most critical problem on their mind. Most of the time an accurate answer to "When will it [the deliverable] be done?" has about as much gravitas as guessing the number of jellybeans in a jar at a kid's birthday party.
Doing a retrospective and analyzing what went wrong when a project delivery slipped expectations is certainly reasonable but it so easily slips into a blame storm and/or people making commitments that aren't realistic. Getting straight answers from folks for this kind of analysis requires a level of psychological safety that just isn't present in most workplaces.
There is a story that Microsoft Word was a staggering failure in terms of actual time and cost to implement versus its estimates.
It then went on to be among the most lucrative ($ billions) pieces of software ever created.
Was Microsoft Word a failure?
For everyone (n00b to senior) I would always recommend designing & breaking things down into no smaller 1/2 day blocks, where nothing went longer than 3 days (for inexperienced), but longer for more experienced. Senior devs should be able to be able to be trusted (to be accurate) for week-long blocks.
Use a Gannt chart (ms project or similar) and monitor the inexperienced, gauging their real-life times with their estimates, adjusting accordingly. Give semi regular feedback at the accuracy so that they (& you, the manager) are working with.
In one of my companies, the most inaccurate we were for any project was that we were late 2 days over 2 years. Build on trust, build on meetings and other time sinks. Include testing, sign offs, etc.
So, absolute n00bs need close mentoring so that the (senior) mentor can begin to gauge the growth rate... Which will be steep... But different for everyone.
You can get very high accuracy on estimates by spending lots of time creating smaller stories and encouraging people to over estimate. This may not be in the best interest of the project or the business.
For example say you have a CRUD application with a page for editing and searching complex objects. Rather than building the entire search page with all of its filters, and then building the edit page with all of its fields, you build a simplified version of the search page with only a few filters and a simplified version of the edit page with fields that correspond to those filters. You also make sure the functionality integrates with the backend to form a cohesive, vertical feature that can go through QA, and possibly the client. Once that is working you expand the list of fields and filters.
This approach tends to surface design issues much faster, and is more adaptable to change requests from the client.
another commenter sort of hinted at this, but this type of process has a downside of prioritizing short term design decisions that lead to working code. It takes time and sometimes a lot of thought to think about the system and the problems deeply and ask yourself if you've created the right primitives.
Add an estimate that you multiply by three in front of each, ie. if the dev thinks 1 hour then they should put 3. From my experiments: no multiplying factor turned to nightly and weekend work, multiplying by 2 turned to long (10 hours) work days, multiplying by 3 turned into comfortable work hours and quality work.
I've had success delivering quality code and respecting deadlines since 2012 or so with this system, but YMMV
Only problem: the customer might not like what he gets, even if it works exactly as planned.
Ah, but the customer does not know everything they want, due to the fractal nature of reality. The closer you get to the end product, the more detail is resolved, and more work is added.
A programmer can accurately estimate the schedule for only the repeated and the redundant. Yet,
A programmer's job is to automate the repeated and the redundant. Thus,
A programmer delivering to an estimated or predictable schedule is...
Not doing their job (or is redundant).
I don't think having the detailed requirements made estimates much more accurate. However it made creating those estimates take an order of magnitude longer - developers needed much much more time to absorb all the details.
At that point I decided I only ever want to work with senior developers who have good imagination and can give reasonable triplets (solution, # of devs to do it, approximate time) for requested pair (idea, quality level).
Then you need to train the IT professionals to build the dynamic nature into the conversation and process. You dont quote a number, you give a range. You communicate confidence levels and uncertainty clearly and continuously. You update assumptions as new information comes available. You _push back_ when the uncertainty is frowned on.
It can be done successfully on a micro scale sure, by working on it within an organization and/or team. But its not scalable and its not a 5-step plan. Its hard.
When people ask me how can they learn, I tell them to find old project schedules and other project documents. Looking at past projects makes the learning more objective, and it is not happening under time pressure.
Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word than estimate.
For many software projects, you simply cannot make meaningful granular estimations for parts of them. It doesn't matter how many story-point poker sessions you hold. Some software work cannot be reduced to a positive integer. In our business scopes creep, bugs plague us, and myriad issues make precise estimations pointless. It's engineering Numberwang.
I get that folks want to tame a chaotic world. But sometimes you can't. And the software engineering field in particular feels wildly neurotic about pretending otherwise. Sometimes I imagine applying our project management tools to other technical fields and laugh at how insane we must look.
Can I just say that is the greatest description of software estimation I have ever read? I am 100% stealing that.
No matter how little you know about something you want to estimate, there is an infinite difference between what you do know and knowing nothing. The right thing to do is to represent your knowledge without fabricating it.
Saying "I can't make a point estimate, therefore I can say nothing" is a false dichotomy.
Even if you say, x is between 0.001 and 10^10, you are narrowing things down.
There is a great paper mentioned periodically on HN, where they examined the famous Drake equation and show that when you get rid of point estimates and use reasonable ranges for each variable, the "paradox" that we don't see aliens vanishes. Despite many of the variables having uncertainty of many orders of magnitude.
I'd take that a step further: if software work can be reduced to a positive integer, you're doing it wrong.
It gets a little complicated partly because of the industry's penchant for reinventing its tools on a rolling basis. In the trades, technology remains largely unchanged over decades, and only truly useful new advances are integrated into workflows (for example, laser level vs plumb bob or spirit level).
In software, new technological approaches are adopted on a continuous basis, and so a lot of effort is spent on learning how to solve old problems with new tools.
Not really. The "building" part in software really is compiling and deploying your code. With CI/CD being set up, this takes a few minutes and requires almost no effort.
So what's left is not "building" the house but "designing" it. Unfortunately, the comparison you give fits more for buying a software and installing/configuring it. Then yes, this should be rather well estimatible, given that you have done it before.
But most software developers are not paid for that. They are paid to design a space station that has never existed before. Sure, they can use existing parts (libraries, frameworks) but it is still design in the sense that certain expectations are simply not possible. Or someone wants that the space station is connected to the other space station on another planet and we don't even know how the other space station looks like (it's built by another company). Who knows if it's even possible to connect them? And if not, the other space station might have to be adjusted to be connectable and this must be done by the other company. How long will that take? Who knows...
By the way, don't think about a space station like the ISS here. Please think about a space station like the star wars death star. Because that's the complexity of most software where estimates are desired and important.
And we are only talking about design, not building. The building part is easy - once we have designed it, we can actually copy it and have 100 of these exact space stations easily.
If you said there is very little fundamental research going on, I would agree with you; but even putting together building blocks in a new way is invention. If you look at patents (arguably a unit of invention), most of them don't have fundamentally new building blocks, just existing blocks put together in new ways.
1. The construction part, as you said, can be estimated.
2. I'd just call the other thing "allocated time" instead of "estimated time".
Any time someone asks me how long it will take me to fix a bug that I haven't really looked at yet, or to plan some new feature or something like that, and they badly need a number, I ask them how much time I should allocate to that. I can't promise to have something like that done by that time, but it gives us both an idea about how to treat that problem.
For example, we could allocate two hours to fix a bug, with the understanding that if that turns out to not be enough then we'll need to talk about workarounds. Or we can allocate two days to plan a new feature, and the best solution we can think of in that time shall be the one we use.
For me it often something like this:
I don't have a clue how the hell to do what you asking for. But maybe implementing .. might help. Can't guarantee but it might. I think I can confirm or disprove that spending .. on the prototype subject to following limitations.. If then we find out it actually works for you, we'll go from there but approximately gonna take .. extra to rework the prototype into production-quality stuff. If it won't, I'll think about something else to try.
Now I know "most papers are pointless" is a common complaint in science, specially in my area of focus- machine learning. But I can't shake the feeling that the situation is particularly worse in software engineering related academic research.
[1] I saw Mozilla attempt it, but not sure if it's currently in use.
There are just to many unknowns you cannot foresee. Software development is complex.
e.g. say you think it will take a day, so 2 days, 4 days, add another day, likely estimate is 5 days.
My pet theory is that when we estimate, we typically think of how long it will take to figure out a working solution to the problem, but forget about how long it takes to debug it, add tests, rework for changed requirements and unexpected nuances, and then roll it out and do any training, etc.
I make the most optimistic prediction and then * PI.
The part you didn't mention is that for most businesses it's better to be over than under in the estimation. I also explain this thought process to the various stake holders. We can certainly try to tighten up an estimate, but that runs a higher risk of being under which is usually a worse outcome (promised launch dates are missed, marketing is missed/happening, customers are told, etc...).
The problem in software though is that such a repeatable process would be immediately automated away by writing a function, library, framework or any such tools that programmers use on a daily basis without much thinking. Unlike in building construction, to which programming discipline is often wrongly likened to, where construction companies simply cannot "write a function" to deploy cookie cutter houses or bridges one after another.
Therefore software engineering is never a repeatable process, unless crappy tools are used, which don't allow for sufficient abstraction of repeatable parts.
Tasks in software disciplines therefore don't follow a normal distribution. They follow exponential distribution most often. Most issues go unnoticed. Majority are just so tiny and ofthen considered business as usual. Every time you get stuck and have to look up a solution in docs or stackoverflow technically is an issue, but never gets reported in an issue tracker for its triviality. There are however issues which are orders of magnitude larger than what management expects when they occassionally sampling issue trackers. Some issues lead to critical design flaws which could need a full blown rewrite for example, or ever lasting and expensive hackery in case the executive decision is to work around the critical design flaw. These issues can take years to accomplish or take massive amount of pain endurance.
Trying to estimate a process with such exponential distribution and making sense of averages or other statistics of such distribution is borderline insanity.
Why not just go to physics department and ask when the next Theory of Relativity will be invented and how much budget and story points those guys need.
This is the crucial point here.
Source code is a (almost) self-assembling blueprint.
The actual product that will be build is the software and software is a configuration of matter, in this case of a computer.
The source code/blueprint for a house is not self-assembling. Compiling such a blueprint requires you to configure the building materials in a way that they become a house.
With better robotics, we will probably get there at some point in the future.
And with software we will always be in a place where you either do new stuff the first time manually or with crappy tools the 100th time.
In every single case when you've done that much work, I seem to wind up with a reasonable estimate.
If we've done the job before and have data on it, also reasonable.
Double or triple anything else.
If it assumes remaining work is equivalent in time to done work (notion of velocity), it's (in my view) very optimistic. People learn the nature of the beast as time progresses. Collective learning happens slowly. A lot of important work is usually only discovered and planned in the second half of the actual (not original) timeline. A lot of important stakeholders are naturally only introduced into projects close to its finalization, which creates a flurry of new activity and discoveries. This makes projects late.
But where that real halftime is - most people I worked with rarely know. I was no better, I also rarely knew.
The only way to manage this I found was the basic rule: build, demo, decide what to do next. Don't get attached to original backlog and grand plans for the future. It won't work that way. Just build and make it work, launch ASAP, get decisions on a ~weekly basis done. Ignore long term planning. It's not helpful.
Sadly, few business folks want that of course :)
Secondly, telling business that we can't estimate accurately is more useful than saying we can estimate accurately and being wrong (which is guaranteed). It's a tough sell but it's necessary.
I’ve been trying to distill this into a framework that is accessible to both development and business folks [0]. I've used it a few times and it has resulted in a few projects getting cancelled due to a massive difference in initially expected cost and post-analysis expected cost.
0 - It's basically just Scrum oriented towards providing upfront estimates. https://jonpurdy.com/pm-framework/how-to-quickly-plan-and-es...
> Sadly, few business folks want that of course :)
Business folks want this usually because it's their money you're playing with.
> Ignore long term planning. It's not helpful.
So, no don't tell people to ignore long term planning otherwise people who give you money are going to stop giving you money to "build, demo and decide". Unless you're on your own dime (e.g. you pay yourself or are self funded), then I wouldn't take this advice.
The theory of story points (which originate outside of Scrum and are not part of Scrum proper) is that task-specific time estimates in creative intellectual work are extraordinarily unreliable and expending more effort on them doesn't improve them, but broad-banded complexity class evaluation mixed with empirical observation of the teams velocity produces time estimates that are (while still extremely fuzzy) both better and much lower effort, once you have the basic tracking in place, than task-specific direct estimates.
The “dogma” you report seems like something that might be a derivative of that that has lost track of rationale and purpose, reducing it to a cargo cult practice.
Planning poker and such are useful as they encourage the team to discuss differences and help identifying stories which are too large.
But that obviously won't work from the start, and it won't be accurate... Just better than nothing and (hopefully) better than what a programmer will estimate in their head.
And IME, it's a lot less stressful on the programmer to estimate points rather than time.
How to turn your organisation into a cargo cult 101.
So who are we kidding?
Another way to look at it: take a small one-person project and assign it to three different developers. You may get wildly different results. How could you have predicted those differences in advance? Let alone apply that type of prediction across a large team.
About a dozen years ago I gave a presentation to the Silicon Valley Software Process Improvement Network (does it still exist?) My presentation: "Unsolved Problems of Software Maintenance." You think predicting greenfield development is difficult? Try predicting maintenance work, where figuring out what to do can be more than half the work.
Can you share your presentation or at least some of your thinking behind it?
Based on what you know, how do you frame this problem? Imagine you had an impressionable audience of 10,000 software professionals (C-level people, managers, developers, UX people, customer support, and so on).
It's easy to detect this:
Gently begin a discussion of how much uncertainty is tolerable, do they want to know the number we are 50% likely to hit? 80%?
If you get emotional pushback to discussing uncertainty, they are looking for a promise.
If you are some combination of lucky, influential, and persuasive, you might have some ability to shape the contours of their expectations. :)
I found that on average, things take twice as long as expected.
So, I was like, now I know how to estimate any project. I figure out what seems reasonable based on known factors...and double it.
A way to look at this is the "unknown unknowns" of anything empirically average to a 100% overshoot.
But this doesn't fly with the project managers I work with, because they can only see that as illegitimately padding an estimate.
Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done.
Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to play a song on piano?
And, learning to play an existing song doesn't necessarily translate to being able to write one.
It's not hard at all if you are willing to be (and are allowed to be) honest about the uncertainty of the inputs and calculate the uncertainty of the final result based on that.
It's true that people may demand precision that you can't give them. But at the same time, you know something and it is simple to compute what you know.
It's like Fermi estimation, that everyone hates so much and claims is so useless to interview for.
Of course, this has been tried and doesn't work. The max time on many distributions is effectively infinity. Which, fails. For obvious reasons. And is why I assert you can really only talk of how long you are willing to work on something. Not how long it will take.
If you estimate, estimate a distribution.
I've had to fight to actually hold post mortems, and every time I've done this, the manager ends up asking, "hey, can I share this?"
So clearly, there's value, at least when we've done them.
I'm amazed at how few places even perform a complete feedback loop. It's just, "when can you get this done?" and, "is it done yet?".
Know how much a Honda Accord costs?
About 25 Grand.
Know how much a Mercedes S450 costs?
About three times as much.
They are both great cars, that will be rewarding to own.
The Mercedes doesn't have 3 times more parts, but it probably took four times longer to make, and they paid the folks that make it, a lot more than the Honda. It's actually, probably better "bang for the buck," although it won't seem like it, on the surface.
The reason is that all those little things that go into high quality take lots of time.
I can write a pretty complete app that does some cool stuff in a day or two. I do it often, when I write test harnesses for my libraries and whatnot.
However, if you want that app to be ship quality, and highly usable, you're looking at over a month.
The thing is, many folks would consider my test harness lash-ups to be their "shipping" product, and will use that as a basis for estimation.
Depending on why you are buying a car, the Accord is very likely much better bang for the buck than the S-class Mercedes. And depending on the situation, the prototype is often better value than the shippable product.
That is why Honda is a bigger company than Mercedes. They do decent quality (Hondas cost more than Kias), but at scale.
I worked for a "Mercedes-level" photographic equipment company for years. People were often quite surprised, when I told them the size of the company. It was like Roadhouse "I thought you'd be bigger."
Our prototypes were incredibly expensive. A $2,000 (retail) body prototype would be insured at half a million bucks.
I just find, in my experience, managers expect Mercedes-level quality, for Kia (not even Honda) prices, and (if you are lucky) mature Honda assembly line development speed.
That can lead to real Jurassic-scale disasters. If you want that level of quality, you need to plan for it.
I'm working on the app that I'm doing, because I was helping a friend of mine evaluate contractors for his dream.
The promises they made were absurd. After a couple of these, I just said "Screw it. I'll do it for you."
He'll be getting Mercedes quality, but he'll need to wait a bit longer for it. I am pretty good at doing damn good; damn fast.
But he also won't be paying a dime for it, so I think he's good with that.
You don't want to daily drive a 30 year old performance oriented Mercedes.
An Accord however, most generations can hit 300k miles easily.
The Mercedes is more expensive because they put more investement into refinement and luxury. Make the doors feel right. Ensure the torque curve is flat and starts at a low RPM, usually using turbo chargers that will fail by 200k miles. Lots of carefully placed sound deadening.
This isn't higher quality, it's extra features.
I doubt it. Luxury goods don't proportionately more cost much more to produce than commodity goods.
Instead they often derive their value from perceptions and exclusivity. That is people perceive the object to be more value for their own reasons, or they are made more expensive as a proof point of exclusivity / conspicuous consumption
[1] https://moznainaczej.com.pl/Download/Teal%20Doctrine/A.Blikl... (13.10 Quality and value)
From your link:
> The more the product promises the customer, the more luxurious it is; the greater the extent to which these promises are fulfilled, the better the product’s quality.
While this is an interesting way of thinking about it, I find personally it to be both non-standard and not useful.
I'm generally not inclined to force or coin a definition of a concept that is intrinsically messy. Instead, I think it is better to take almost an anthropological view: unpack what different people and cultures mean by a word.
With this in mind, I recommend the following article from MIT's business school, because it embraces the vagueness and complexity of quality:
https://sloanreview.mit.edu/article/what-does-product-qualit...
A car's primary function is to move its passengers from point A to point B, safely and timely. Both cars do this very well, but the Mercedes will probably be costlier over their respective lifetime (fuel, parts, service).
The only way for it to be a better bang for buck would be to turn its secondary functions (comfort, prestige, signalling) into a primary tool. For instance, if a broker or salesperson finds that owning the car gives them the added confidence to project the image they want during negotiations and thus contributes to their overall success, then it becomes a worthy investment.
Don't do an estimate, build a prototype.
Don't do an estimate... seriously... build a prototype.
And then throw it out.
And THEN do an estimate. It will probably be pretty accurate.
Not always easy to do this in a tech company but as an eng director I was able to get it done and it changed a seriously broken process based on multi-week scoping and estimation futility.
In my experience, when an estimate is spot on the world goes on as if nothing happened. When it's incorrect, all hell breaks loose and it's every man for himself. And at the end of the day, all of the blame ends up on the person who guessed wrong. I'm glad I don't have to deal with that anymore.
Bit long, but well-presented and worth a listen for any practicing software developer (or person who manages developers).
Yes, I get on a high level, one needs to be able to say "yes we can make this before [date]", when using big deadlines. How often do people really have to finish something before a critical deadline or decide to drop it right then and there? I'd wager most software devs do not, let alone biweekly deadlines. Isn't agile methodology supposed to help us fight against artificially tight deadlines (customer collaboration)? Isn't the SaaS model combined with "deploy any time" designed to be profitable in accordance to features implemented? Then, why push estimates so strongly in whatever flavor of the month Scrum version BigCorp wishes to use today? What do they even add at that point? If they are really so important, why do we always feel the need to introduce human error when we can extrapolate former experiences with computer models?
Maybe it is just me being cynical. The entire need to hold so fiercely onto an estimate reeks of micromanagement and desire to push responsibility entire onto the lower ranks.
Everyone wants estimates. For decent reasons. But until we've done a similar thing before, they're utter fabrication. We can take a semi-educated guess, but we know they're still guesses... so for things we don't have concrete answers for, we give small/medium/large/extreme markers for how uncertain we are.
It's fine to take on a big unknown or two. You might even get them both done in a half. But they better be worth it (or you have to decide when to cut your losses), because completing those two could consume all of your resources... and if that happens and you didn't commit everyone to it up-front, you won't get it done this half. Making that tradeoff more explicit managed to get us signed up for fewer low-impact-but-highly-uncertain projects that would inevitably balloon out of control but never be cut.
Known knowns: familiar tasks that can be reliably estimated from past experience. You can improve these estimates by better knowledge sharing (internal wiki, adding comments).
Unknown knowns: tasks that can be increasingly better estimated the more time you spend on estimating (for example creating Draft PRs with pseudo code).
Known unknowns: tasks that haven’t been done before, so estimations need a decent buffer (30-50%) to account for research and potential blockers. You can improve these estimates by benchmarking your previous efforts combined with the team's skill levels (aka sprint story point velocity).
Unknown unknowns: the unforeseen blockers that come out of nowhere, can't really be estimated, and can really disrupt a project's schedule. Improving these estimates is really hard. The key to improving these is by improving team/company communication and building agile feedback loops so you can identify these issues early and reprioritize as needed.
"a temporary endeavor undertaken to create a unique project service or result."
No "projects" in software fhat I know of are actually temporary. They only end when management fires the people behind them, it gets cut off or there is no adoption.
We in software think of projects really as things which we create and which need maintenance in order to live. There is never some "end" to it.
Because it doesn't even conform to its own definitions, we could therefore conclude that the whole PMP project management discipline, as applied to software, is a scam.
- the application is thoroughly specced. You might need WEEKS for this. - all variables are taken care of. Stack is known, and you’ve experience with all parts involved. If you don’t, get familiar with the parts first. Again, might take weeks. - there is no implicit functionality. It is either explicit or not included. - there are clear boundaries and rules to prevent feature creep. - you cannot estimate an estimate - all designs and UX are final
Now the problem is, this estimate is really expensive, because it’s actual work. It takes about 10-25% of the total project time to estimate the project.
More weeks than it will end up taking to build the finished product, in fact.
The second key mistake is forgetting that estimates (at least in software) are an upper bound. So high estimates are often “better”.
When in construction you prepare the building site, lay a foundation, frame the house, put siding and a roof on it, and plumb and wire it. This is equivalent to programming. In other words through the lens of the construction metaphor, the developer is someone who ultimately build someones house by working together with different disciplines (Architects, designers, contractors etc)
The problem with that metaphor IMO is that it's not actually what software development is.
The proper metaphor for software development is more "engineering and development of construction site equipment and material" i.e. a developer is not building buildings they are building the things necessary for building the building.
And so the developer will often find themselves in a situation where the material doesn't exist and they have to invent it it or the material exist but we dont know how it's going to work with some other material needed or the equipment used for that material isn't made yet or doesn't work with that material even though it normally does.
I.e. a developer is inventing, engineering and building all at the same time and that is what makes it impossible to estimate development regardless of process or metaphor.
Humans like to think via metaphor because it's a least-effort mode of thought but sometimes there just isn't one and it's just tough luck and start thinking from first principles instead.
Yet any company that does a lot of the same work could probably use that. You'd still need to do some extra work and tag your issues by things that you want to observe (and maybe predict), but then you could say things like "after we switched to that fancy new API client generator, our client development times went way down but debugging times went up a bit". And the system could display some time range while you're writing the issue. You could also look at the issues at the end of a sprint and then merge them with other items from the time tracker to see how meetings and other distractions played into the outcome.
To borrow the usual building a house analogy, if during software estimation we're actually given how many rooms, how many floors, where are the doors and windows etc, then I think estimates will be a hell of a lot more accurate. Instead, what we typically get is, build a house that can be used by the family of 4, and we'll discover exactly what the family wants as we build.
The broad strokes of the design work need to be done before the estimate (i.e. gather requirements, identify constraints, decide on direction and tools).
If work is unknown or in large chunks, it needs to be broken up into smaller well-understood chunks before estimation.
When new ideas arise, they should usually be kept for future work.
When work is added, other work needs to be removed or the estimate revised.
If the work runs late, something should be removed from the scope to keep it on time, then done after delivery.
If work involves integration with other systems this is much harder unfortunately, but for some classes of software (much business software), it is not so hard to estimate.
>...non-machine learning techniques, e.g., regression.
Is this where we are now? The connotation of "machine learning" doesn't include regression? Wow.
To be fair, this is a pretty common perspective in computer science academia and adjacent fields (and probably occurs in a lot of fields where people don't focus on statistical learning..ahem I meant ML).
This business model is normalized for other professions, and it should be for software engineering too. As a profession, we should move more towards partnering with organizations to realize business value through software rather than being simple "feature factories" (see also: Developer Hegemony[0]).
[0] https://daedtech.com/developer-hegemony-the-crazy-idea-that-...
You should never claim better precision than you know. It's often more accurate to say "more than an hour but less than a week" than to say "6 hours". We learn scientific precision in middle school... why is modern workflow management still so bad at uncertainty propagation?
The article ends by suggesting using GitHub. When I was a PhD student I recall open source projects being a standard data source for people who did software engineering research. But it’s not my field, so I’m not sure if effort estimation really has missed this.
The time and cost estimates do not measure the amount of technical debt in the outcome. it may good great on paper but that is because of a vaguely defined specification which says nothing about the quality, including amount of te4chnical debt (meaning mostly maintainability).
(Like "There's a 80% chance we'll finish this today." ) You immediately start to understand why it can take so much longer than expected (You've rolled a dice before; I want a six... How long will it take)
Time estimation on a macro or micro level is just educated feelings.
The thing I notice with people that are forced to estimate time and have nagging managers is that they start to report time according to how much the task was estimated to, to make the burndown chart nice.
I.e. if one task is finished early a late running one gets the hours giving the illusion of getting better at estimating. I think Joel's estimation method for Frogbugz had the same fate linked in somewhere above.
InternalEstimation = estimate the work as truly as possible (ie. in MD) ExternalEstimation = double the value of InternalEstimation, and increase units by one order
Example: 1 day (MD) = 2 weeks 2 weeks = 4 months
Update: working now.