A Drop In The Stream

Three months after our wholesale adoption of an agile development process there is enough time elapsed and experience gained for me to pen some thoughts about our successes and failures and the strengths and weaknesses of the agile approach to software development.  The punchline is that while not perfect in every respect, agile software development is far more effective and realistic a process than any other I’ve used.

First a bit of description of how we work.  We use one week iterations that include daily, brief status meetings (10 minutes or so).  Our focus is to define granular pieces of work that can be completed and demonstrated at the end of the iteration.  The guideline is that developers work in a Test-Driven way but the bottom line is that we now require unit tests for all code that is created or modified but don’t look ove anybody’s shoulder as to when they’re writing their tests.  We also use Fitnesse (PyFit) for our specifications and User Acceptance Testing.  

The most obvious strength of our process is that we are producing tested and testable code all the time.  In 3 months over 1000 unit tests have been written and we have over 10,000 assertions in the Fitnesse UAT.  The coverage tool I run shows that were at 39% coverage of our code base but I believe that the tool may be counting lines of code (LOC) in external libraries as well so perhaps we touch as much as 50% of our 70-80k LOC may be exercised.  Of course, there is a difference between exercising a line of code and validating its correctness but over time the completeness of the the testing will increase.

The transition to writing tests and running them regularly has been challenging for the development team.  While most software professional will tell you that tests are important, they don’t always walk the talk.  Writing tests first is an especially difficult notion for some staff members.  I think that until one feels the TDD mojo of the double entry accounting that the process embodies that some developers view the tests as extra work that “doesn’t do anything “for them”.  Nothing could be further from the truth but again, this is a tougher sell to some than others.  Likewise, writing fixtures for the PyFit (Fitnesse) UAT is extra labor that isn’t always welcome.

The UAT are supposed to be created collaboratively by the Product Team and the QA Team with input from the developer to ensure that the columns of the Fitnesse tables reasonably represent the API to the functionality that the UAT exercises.  In reality, our Product group has punted to a large degree on using the UAT.  They review them, however, the codification of the specifications by QA hasn’t been as collaborative a process as I had hoped.

Despite the bits of organizational process dysfunction that have surfaced since we started making our process more agile the tests have paid for the effort to create them on many occasions.  Failed tests have led to the discovery of bugs that would have escaped to production in the past.  A good number of developers have reported being able to more fearlessly refactor their work because the unit tests have given them confidence that they’re code still works when they’ve cleaned things up.  The extra effort to create the testable API to support unit and UAT is still groaned about occasionally but the intermittent discomfort isn’t anything like an insurrection to overturn the process.

The tougher part of the process isn’t the technical practices but rather the sociology of prioritization and planning.  While the developers can code tests for what gets built, our approach to being agile is to emphasize taking small bites of the apple to improve the predictability of our output.  Developers are experts in thinking through the functional chunks and alogorithms to build a solution but decomposing these things is often another story.  Our goal is to have automated Fitnesse tests that can be demonstrated at the end of the iteration. 

Taking the parts of a problem and deconstructing them into stories that can be completed in 3 days or less isn’t always an easy task.  At this point, we are circling around some reliable velocity ranges for the teams but the variability is still more of a pendulum swing than we’d like.  I’m optimistic that once we have some more infrastructure built for testing our more complex objects that the difficulty of breaking things down into testable chunks will be reduced because there will be more example API code available for reuse or copying and modifying as needed.

The other sociology part of the equation that isn’t going as smoothly as I’d like is that our Product Team doesn’t always come to the table prepared and clear about what they want.  Moreover, we’ve not accounted for defects work (both for in-process features and production bugs) well in our assessment of velocity.  The usual pressures from the business to release quick and dirty solutions is resisted to the extent we’re able, however, Product doesn’t have a master plan so the laundry list of priorities changes at a rate that sometimes negatively affects development’s throughput.

Overall, our adoption of agile practices has benefited our organization in myriad ways.  The changes haven’t always been easy to adjust to and there are still cultural issues that we’re working through but the net positive benefits of agile software development have been huge.

A commenter one of the Scrum posts inquired if I could clarify the use of story points so here’s my best shot at it.  Story points are an abstract unit of measure that aren’t formally/mathematically tied to any unit of time.  3 different teams in my shop all have different weekly story point velocities and the scales that two fo the teams use differs by a factor of 10.  There is no contradiction here because what story points offer is a way to estimate that works for a particular team.  The goal is that over a period of months, assuming the team composition and the type of work is more or less constant, the team will establish an average velocity measured in their abstract story point units.

Achieving some degree of predictability is the holy grail of software development but there are, of course, gotchas in getting to this point. The first trap is assuming that estimates really are some form of concrete prediction.  In one of Mike Cohn’s books (Agile Estimating and Planning, I think) he offers data that estimates of software stories vary from 60% to 160% correctness - this is a very wide variation and jibes with my experience.  Certainly, more experienced developers with a solid grasp of the problem domain are more likely to produce estimates that are close to the real labor expended but there is variability and things to be wary of with this claim too. Experienced developers can’t always break stories down into small, granular bits and the larger the chunk the less accurate an estimate is likely to be.

Even when stories are appropriately sized an experienced developer is sometimes more likely to over-engineer the solution and produce more fluff and features that aren’t actually part of the story.  So, the estimate might be reasonable but the predictability goes out the window due to technical exuberance. The important things to remember about using story points for estimating are that they are an estimating tool and don’t represent a guarantee of any kind.  Second, the usefulness of the points grows incrementally with the length of the historical record you’ve assembled.  These data are the source of the velocity report contents.