Ask HN: Is there a way to get Agile right?

The first line of the Manifesto for Agile Software Development:

Individuals and interactions over processes and tools

Don't try to spend time deciding on spreadsheets versus Trello versus Slack versus etc for managing the project, instead focus on the problems your team has.

Simple example: let's say your project management spreadsheets are not working well because people keep duplicating work that someone already finished, so you decide to start using Trello instead. Turns out, your teammates had not been regularly updating their status in the spreadsheet, and you still have the same problem in Trello- people are not keeping their status up-to-date, so the problem is not solved, and you have a bunch of teammates angry because the problem of duplicated work has not been solved AND they had to learn a new tool.

Agile kind of has to mean "willing to hack your methodology". ("Let us be agile by following this rigid methodology" is self-contradictory nonsense.)

So, try something. If it makes things worse, stop and try something else. If it makes things better, keep that. Repeat. (Note, however, that things that work may not work forever.)

It's really hard to lose by following that approach. The only way you lose is when someone has a rigid approach that must be followed, in spite of obvious evidence that it isn't working.

(1) There has to be trust between management and people who do the work. You can get things done without trust but it is going to be expensive, late, broken, nerve-wracking, etc. You aren't going to paint process onto a low trust situation and see improvement unless the trust improves.

I'm going to go out on a limb and say that close to 100% of the time when people are in agony over agile there is a lack of trust.

(2) If you want to have good results you have to have a technical framework such that changes that management thinks are "a small change" are really a small change. Some people think agile means "not thinking ahead" -- you certainly shouldn't be borrowing trouble but you should be planning for sustained productivity which means that tomorrow's "small change" is not a 6 month project.

This is not to advocate for some particular, language, framework, database, whatever but to say that, from a system level, you are designing on the expectation that you're going to be doing "sprints" on this thing for a long time and that you shouldn't need major re-engineering.

(3) You've got to get realistic expectations about details, the "definition of done", etc. For instance does "finishing the sprint" mean that the developers throw a feature over the wall to testers? Or does that mean the testers say the feature is ready to go? If it's the latter, the testers will need to see the product long before the sprint is "over". Then what do the testers do before that? What do the coders do when the testers are testing?

I'm not advocating for a particular right answer but that there has to be agreement on the team for something real. I have been on some teams where we often finish 2/3 of what is supposed to be on the sprint but we do a good job of not letting things get stuck so we solidly deliver features. There are other people who would blow a fuse if there were 15 items on the sprint and only 14 got done.

(4) When it comes to very high velocity I think fixed length sprints are a problem instead of a solution. Sometimes there is a feature you could put in front of customers in two days but they have to wait two weeks because of the process. If there are multiple teams that have to line up with their own sprints a small delay can snowball into a big delay.

In teams without trust people really stick to the sprint structure rigidly because it is all they have. The most effective teams will figure out how to to bend the sprint structure to overcome its limitations.

Check early ID software (from 91-97) and you will get a clear picture of how to do high quality, very efficient development:

- Small team in which everyone knows everyone else and the chemistry is right

- Clear objective, all team members own something, could be share or part of project, whoever falls behind got maybe one or two chances and then gets cut ruthlessly

- Team is working on challenging projects

Most teams simply do not satisfy even one.

> tracking work efforts

Why are you doing this? How does it help improve delivery of value to those who are paying for the product/service?

Agile was intended to have people discover better ways to deliver something of value.

This is why the Agile Manifesto starts with People (vs. some unchangeable process handed down from above) and Working (Valuable) Software, instead of a book full of "requirements". And so on.

If you haven't read the Agile Principles, you're unlikely to be Agile: https://agilemanifesto.org/principles.html

The goal is: provide value, get external feedback on how it could be more valuable, as well as internal feedback on how people can deliver that value better.

That's it.

Everything else needs to either support that, or is just someone trying to micromanage you (and don't think that's not done on purpose).

Two rules to get agile right:

1. Make things that are easy to change

2. Change things

> So, my question: Is there a way to get Agile right? or are we doomed to fail in the same ways that has been described by countless articles discussing the downfalls of Agile?

Yes. Don't be dogmatic. Pick an initial set of processes/practices that match more or less what you already do, want to do (based on a reasonable belief of utility and practicality), or have done (with success) in the past. As you move forward, periodically reexamine what you do and change when something isn't working well (in Scrum this is the Sprint retrospective, which seems to be the first thing cut from every Scrum-derived process I've ever been part of or near to "save time"). Experiment not just with your code and systems, but with your processes and organizational structures themselves. If you only apply things like the ideas of Extreme Programming to your programs, you're only doing half (actually less) of what Agile aims for.

Sterman in his Business Dynamics book has a nice set of diagrams starting on page 15, talking about feedback loops/systems for learning.

Attempting to draw them as ASCII art, not that skilled at this:

Simplest feedback model:

         Real world
         ^      \
        /        \
       /          v
   Decisions<--Information feedback

This is a very common thing, people make a decision, try it, get info, and make new decisions. What they don't do is extend this feedback to their decision making model. He extends this model on page 16 with:

         Real world
         ^      \
        /        \
       /          v
   Decisions<--Information feedback
      ^
       \
        \
     Strategy, Structure,
       Decision Rules
           ^
            \
             \
          Mental Models
            of Real World

Again, note that the feedback does not go back to the mental models which feed into the decision making strategies and, ultimately, the decisions. This is the dogmatic approach to any process. You select it, and you run it regardless of its real applicability or utility but based on a belief that you don't update (because it's dogmatic). Doesn't matter if this is Agile (big-A) or big design up front (BDUF) or anything else, if you don't reconsider your models then you will not be agile, you will be stuck. This doesn't mean things won't work, you may have settled into a local optimum (on accident or on purpose). Maybe things are done this way because they've always been done this way, but 30 years ago someone did think about the business and selected something that worked (deliberately) and things haven't changed much so it happens to still work well. But once it fails, you have no deliberate way of adapting to the new reality (which is a key to being agile, responsiveness).

The final model he shows on page 19 brings that feedback to not just the decisions but also the models (this is the hard one to put into ASCII art):

         Real world
         ^        \
        /          \
       /            v
   Decisions<----Information feedback
      ^                       ^|
      |                       ||
      |                       |v
  Strategy, Structure,<----Mental Models
    Decision Rules         of Real World

This, to me, is the critical thing to agility and if you focus just on the manifesto and not the Agile Industrial Complex (and fuck SAFe and others like them), you will see that this is present (though less explicitly). Paying attention to feedback, trying new things, being adaptive. That is the essence of agility and Agility.

=======================

Regarding using a shared spread sheet or something else for managing your work, that's all about scale. A 4-person team can use a spreadsheet for status management, it may not be the best, but it will totally work. Scale up to 4 teams, each with 10+ people, possibly geographically separated, and you'll need something else. But take one 4-person team or 4-teams that are all located in the same space, you could manage your status with sticky notes on a whiteboard and probably be as successful as with the spreadsheet or the fanciest project management software (and maybe more successful in some cases because of the relative ease and flexibility sticky notes on a whiteboard afford compared to the fixed, or more fixed, workflows of most PM systems). What you record, how often you review and consider things, those are going to be more critical.

As someone who has worked in 5 different companies using some form of Agile, I have the following take on this (below). I know you said you're not strictly a development team, but that's where my experience is and the comments below are focussed. That said hopefully some of the points are transferable to your domain.

- Agile is about flexibility, but it can* come with a much bigger price tag if you don't do most of your analysis/design up front for big/technical projects. *Sometimes not always. More on this below.

- It's great having conversations with product owners and business representatives at various times as you progress to other features, but you can't have multiple developers going back and forth for this (as you loose continuity) and if you choose your lead developer or solution architect (if you have one), then they'll spend more time on admin than on actually designing and building things. Therefore it's good to have a business analyst or dev manager help do most of this, with the option for your tech guys to get some face time with product owners if they want/need it.

- I found that none of the business analysts charged with leading/facilitating the Agile requirement gathering really knew how to write good epics / user stories where I worked, and this can make life difficult / waste lots of time during refinement sessions and during development.

- The biggest challenge I found with Agile previously is how to incorporate design tasks (more on this below).

- Some of the business analysts, dev managers, project managers, portfolio managers that I worked with thought that once all the epics and stories were defined we'd just be able to build everything with zero design. Er, no. The lack of a definite 'design phase' like in waterfall doesn't mean no design is required, it just means that your design execution is more flexible and incremental if need be. You still should go through the design process for your separate features (as they get prioritised) because a) Like TDD for units of code, it forces you to think about the gotchas/edge cases/touch points/cross cutting concerns BEFORE you implement something. b) You have the design to refer to in follow up discussions with product owners / BAs / dev managers and your development team which could consist of several developers.

- By design above I mean a design document/wiki page that covers conceptual, logical, and physical aspects of the solution that will be built. The design might also include and overview of discounted designs initially considered but then not chosen. You want your conceptual/high level design section to be just that, with all the really tech stuff in your logical design section e.g. most your UML stuff and other tech points. But back on Agile, by design I ALSO mean Spikes and Proof of Concepts that you sometimes need to do to validate design assumptions.

- Even with an 'Agile' approach being dictated from above, you might still decide / realise that you need a waterfall approach of sorts, at least initially for the analysis and design (or enough of the design to de-risk things). For example, in massive or extremely technical projects if you don't consider the full requirements or full design up front, then you can have these incredible costs of re-work. E.g. If features 1 and 2 need to be re-written because their implementation is not compatible with features 5 and 6 which were always needed, but were just further down the backlog and not considered in the initial design work.

- This leads me to what I see as the the delayed cost and increased cost of Agile, for large projects with set business requirements. By that I mean again, if you have a large/very technical project with clear defined goals (example in next point), then you probably want to do your full design up front or at least 80%-90% of it. Definitely all of your conceptual design and most of your logical/detailed design and some physical design. Why? Well you don't have to, you can be fully Agile and do just in time / incremental design as you build new features, but the business and your tech leaders need to be told that this can come as a BIG cost down the line. As mentioned in my previous point, you could deliver certain features first but then discover you need to substantially re-write them at a later point because the design didn't factor in those features 5 and 6, that turn out to be just as important. If your business and tech owners aren't aware of this risk when they are sold on the Agile approach then I guarantee they'll be fairly pissed with you when you say you need to re-implement features 1 and 2 and they need to swallow the cost/delay of doing so.

- By large projects with set business requirements, think an entire ecommerce platform that is currently live but total spaghetti code that the business knows needs to be replaced due to slow / costly changes, or other technical risks. E.g. You know you have a search feature, product page feature, category page feature, basket feature, checkout feature, authentication/membership feature, fulfilment feature that ALL need completion.

- So how do you do Agile and Solution Design for large/complicated projects? I like this approach:

a) Your BA defines the high level features in the user story language ('as a', 'I want', 'so that') if the features at a high level lend themselves to this format, but if not, some general high level description of the features.

b) Your BA then starts building the backlog by defining the actual User Stories for certain features, in whichever priority they are likely to be built/released in. They interact mostly with the product owner, but ideally bring them along to at least some of the refinement sessions with the dev team.

c) You raise a technical story for Your Solution Architect / Tech Lead to do First Draft of the Conceptual/High Level design for feature A or features A-E if you want to de-risk things a bit more. You can have another story for First Draft of Logical/Detailed design too.

d) Your SA / Tech Lead then starts on the design under the relevant stories, but also raises Spikes, Proof of Concepts, and other composite design stories within your technical backlog, to cover off questions/unknowns that need to be answered for enough of the solution to be done. This is where you can get your dev team involved in the design process, by ticking off some of those technical stories. For example, you might as part of your design need to consider Capacity and Performance but decide to divvy up the work, and get the devs to flesh out that part of the design. Or you might decide a Spike is needed to determine if X is needed or not. Or you might want to prove a facet of your solution with a bit of POC work.

e) As both b-d progress above, your team will be able to start estimating some of the backlog being produced by the BA. You'll reach a point where you have both enough of the business requirements gathered, documented as User Stories or Technical Stories, and estimated.

f) Then you can start building the initial / highest priority features, while the BA and SA continue to progress with b) and c) respectively. You can do this because they've had a head start and your team has enough backlog & design to get started on development, while the BA and SA then try and keep ahead of the curve with the remaining backlog and design work.

- Part of the problem is that even with Agile development, certain businesses will want an up front estimate for a large project for costing purposes and budget sign off. The key to avoiding issues is honesty. Rather than giving some finger in the air estimate, you need to explain that until the precise requirements are known, solution devised, and backlog estimated, you can't give a precise estimate. Try to ask them for at least some of the concrete requirements and for a week or two for your SA / Tech Lead to give an initial estimate. Let them know that early on at best you can probably give only a +/-30% or +/-40% estimate.

- Most of the points above focus on Agile with big projects, but if you have a well established product / platform, and are making smaller feature changes gradually over time, then Agile preparation and development is much simpler too.

- So again, finally, honesty, honesty, honesty, is a big part of success. Your BA's, PM's, Dev Managers, SA's, Tech Leads, Dev's, all need to learn to be comfortable saying: 'I don't know the answer/estimate to that right now, but if you can give me x information and y days then I can get you an answer, or bring you much closer to having an answer.' and 'Because we're taking this approach of developing and delivering features A and B sooner with an Agile approach, it means that we might need to re-work some of the implementation at a later time when we consider and design features C, D and E.' and (as can happen with waterfall).. 'Unfortunately we underestimated X, and need Y amount of time to complete it. We've learnt Z, and this will help us avoid this situation in future.'. Good luck!