Abstract

In February 2015 Bertrand Meyer gave a presentation of his views about Agile methodology, based on his recent book of that name [1]. While he presents an insightful overview of Agile and accurately identifies its "brilliant" elements, I think he fails to penetrate to the true core of the Agile approach. This article seeks to reveal and discuss those core elements.

Small Steps and Tight Feedback

The difference between, say, an expert skier and a novice is that the expert has learned to detect tiny perturbations in the ideal trajectory and is able to make equally tiny (and rapid) corrections for them, which leads to smooth overall progress. Similarly one might suppose the dancer who graces the cover of Meyer's book has learned to detect tiny deviations from ideal form and how make equally tiny corrections.

Ideally, a software development project should proceed smoothly from conception to delivery. Agile suggests that the way to approach this ideal most closely is to proceed in the smallest practicable steps, and correct any deviations as soon as they are detected. This is done at several levels:

• While the complete set of functionality may be articulated in a general way at conception, the desired complete set is developed over the life of the project, by producing an initial minimal version and mutating it into the final version in a series of small iterative improvements.
• Implementation is divided into a sequence of short, time-boxed, phases, each of which is reviewed to determine how to proceed with the next.
• Code is generated in a continual series of small loops in which new code is written and tests are executed.

The Essence of Agile

The essence of the Agile approach is this:

Take a small step at a time, test each time and make appropriate corrections.

Why it Works

• By taking small steps you cannot go too far wrong, so the cost of any failure is small.
• The immediate detection of errors dramatically reduces the search space for diagnosis.
• Prior work can be relied on, providing a firm base for progress. Returning for a moment to the skiing analogy, at any given instant the expert skier is proceeding from a position of balance, whereas the novice is generally proceeding from a position of imbalance and is also likely to make things worse by clumsy attempts at correction that result in over-correction.
• For designing and coding rapid cycling is key [2].

From Theory to Good Practice

Let's consider good Agile practices in the light of the above principle.

• Short iterations – no further explanation is necessary.
• Frequent builds (including full regression tests) and continuous integration – errors are exposed as quickly as possible, minimizing the effort to correct them.
• Prohibiting increase in scope during an iteration – to do otherwise makes any estimation of effort meaningless and prevents any improvement in the accuracy of such estimation.
• Every feature must have a test – this is how deviations from the required functionality are detected.
• Refactoring – this is how deviations from the ideal design are corrected. We can see why it is important that refactoring should be done as soon as it is seen to be needed.
• Daily "stand-up" meeting – this is how deviations from the schedule are detected as the earliest possible moment, how possible impediments can be anticipated, and how actions to deal with known impediments may be initiated.

Agile is not a Panacea

Although they are generally valuable, Agile techniques do not cover the whole of the development process. Sometimes it is necessary to take big steps. Several of the bad parts of Agile as enumerated by Meyer result from an attempt to force all development into the straitjacket of small steps.

• Upfront requirements and design – Agile tells you not to do either, but this is just silly. It is certainly a fruitless endeavor to generate complete requirements and design before doing anything else, but right from the start you need to have a pretty good idea of what you are trying to do and how you are going to do it. Also, as Brooks pointed out long ago [3], conceptual integrity is the most important consideration in system design, and conceptual integrity will not result from piecemeal design.
• Test-driven development – This works reasonably well for a certain class of programs that mostly comprise a set of small and largely independent features (many web applications are like this), but it is a terrible way to develop algorithms. Mills discusses this point at some length, with several well chosen examples [4].

The Problem Agile Cannot Solve – Estimation

As Glass sagely observes [5], "... I don't believe in the software crisis. ... I do believe in a software estimation crisis ...". For software with any significant degree of novelty it is impossible to make accurate estimates. The rule of thumb is that an initial estimate will be wrong by a factor of 4 in either direction [6]. The initial estimate is unavoidably a big step. Agile cannot improve this initial estimate, but it can facilitate the detection of deviations from the initial estimate at the earliest possible moment and provide good guidance on the best way to resolve this situation.

Footnotes

1. Bertrand Meyer, Agile! The Good, the Hype and the Ugly, Springer, 2014.
2. Robert L. Glass, Software Design and the Monkey's Brain. Commun. ACM 51, 6 (June 2008), 21-22).
3. Frederick P Brooks, Jr., The Mythical Man-Month, Addison-Wesley, 1995, 42.
4. Richard C. Linger, Harlan D. Mills and Bernard I Witt, Structured Programming: Theory and Practice, Addison-Wesley, 1979, Chapter 7.
5. Robert L. Glass, Goodbye! IEEE Software 26, 6 (November/December 2009), 95.
6. See this article and references therein.

© Copyright 2015 Michael J. Harrison. All rights reserved.