$Home|Elec|Tech|Misc|Pics

Being Agile for the Scrum

22 December 2013
It has been a few months since I started at my current company, which uses the Agile-orientated Scrum development framework. At first it seemed like a heavyweight management fad that overdid just about everything, especially meetings, but having operated under it for a while I have started to appreciate that sanity of the underlying motives. I am yet to be convinced that Scrum is a universally good idea, but it has given me more insight into what is good and bad practice in writing software.

Although I get the distinct impression that a lot of the gains of Scrum could be obtained by taking on-board things like the Joel Test, I am familiar with the uphill struggle that is getting set-in-their-ways colleagues to adopt even one of the things on this list, and don't even mention the difficulty of encouraging those further up the food chain. On the whole I am skeptical of significant parts of Scrum and Agile, but I get the feel that religious adherence to an off-the-shelf package of processes is the only way to instigate good practice.

Agile vs. Scrum

While Scrum is but one implementation of Agile principles, for practical reasons I am mostly disinterested in whether given cause & effect are due to underlying Agile ideas, or whether they are specific to Scrum itself. This somewhat assumes that any fault of Agile is automatically inherited by Scrum, which is perhaps unfair as Scrum actually predates the Agile Manifesto, and was only later labelled an Agile framework. While it is improper to use Scrum and Agile interchangeably, so far I have seen little incentive to make the distinction.

Its use-cases, stupid

As I have written previously about, the one thing that any software project absolutely cannot do without is use-case analysis, or in other words descriptions of what and how the system will be used. Without this, programmers do not have any real goal to aim for, and they end up having to speculate when it comes to deciding how to architect a piece of software. While this is not all bad, the burdens it places on developers are a recipe for burn-out.

A lot of Scrum is the handling of what it terms user stories, and these go through deliberately segregated prioritisation, effort estimation, and task breakdown processes. From my perspective the main thing Scrum gets right is that it forces proper consideration of what a system is supposed to do, and above all, writes it down. The latter point is of particular importance, as otherwise there is no guarantee that intended functionality is anything more than something plucked out of the air. The first company (a start-up) I worked for was particularly bad for ever-changing goals, the worst single instance being a director doing a major-league refactor of the mobile front-end over a weekend, which pretty much buried any real prospect of proper job planning. Specifically documenting requirements is one of the few practical ways to ensure consistency in goals, because without that doing a decent development job is a non-starter.

Roles

In Scrum, there are only three roles, and although notionally there is scope for overlap, to me keeping them all separate seems to be the ideal. The overall structure convinces me that it is largely designed as a coping strategy got projects with large numbers of people working on it, but with this in mind it has its logic.
Product Owners
Calls the shots regarding what gets done, with the aim of maintaining consistency, and there must never be more than one. From the developers' perspective, they are the interface with the customer, and as a result are the ones who make sense of feature requests. If a programmer realises that what they are about to do requires a fundamental decision, then it is the duty of the product owner to make it.
Scrum Masters
Scrum specifically states this role is not leadership, but it is also not secretarial. Seems to be about keeping the team running smoothly, in particular keeping non-programming tasks off developers' plates. There seems to be an implicit assumption that developers dislike organisation and administration, and one part of this role is keeping these distractions at bay.
Developers
Scrum is very specific in not having any hierarchy between programmers, particularly when it comes to job titles, and the developers in a team are deliberately left to their own devices in how they organise themselves. There is some encouragement for things like pair programming, but on the whole the techies are left to figure out for themselves how they best wok together.
A notable factor is the absence of a 'lead developer', on the basis that being both a project coordinator and someone who is to be coordinated is a bad combination. There are scenarios that this works, but keeping arrogance and resentment out of it is a fine balance at best. With Scrum the underlying idea is that developers write code and are isolated from distractions such as sales, politics, and organisation.

Work processes

Scrum is based around self-contained three or four week sprints, with the aim of having something that can be shipped to the customer once a month or so. Obviously more often than not there will be missing functionality, but the aim here is to keep them in the loop with something that at least partially works, so they can give feedback if the direction of the project is not to their tastes. This iterative approach is done because of bitter experience with waterfall development models. Scrum is quite heavy-going on meetings, but many of these are clearly designed to avoid SNAFU, and the daily stand-up meeting is something I rapidly did a U-turn on and decided is one of the best parts of the entire framework. At first I felt the way it constantly puts people on the spot is somewhat disruptive, but I rapidly found that it is a good forum to highlight immediate problems, and in cases collar a colleague about some obscure piece of code they wrote.

For the programmers, the key planning part of the whole process is grooming sessions (yes, a very bad choice of process name), where developers either make estimates on effort involved for presented user-stories, or they break down a given story into tasks with time estimates. Planning poker seems to be the most popular way of estimating effort, which is deliberately a separate metric to the task time estimates. At the end of each sprint is a retrospective session where developers give their feedback, although bitching session is perhaps a more accurate term, as it tends to concentrate on what the developers think is badly wrong.

Faults & pit-falls

If there is a fundamental issue with Scrum, and to a lesser extent Agile in general, it is assumptions it makes about both projects and the ecosystem with which it operates within. In particular, there are inherent issues that are independent of the influence of Scrum specifics, and there are others that Scrum makes no real effort to fix. Below is an overview of some of the grubbier aspects:
Coping strategy
Some of the principles behind the Agile Manifesto are statements of reality rather than good practice, in particular changing requirements and circumstances. However things are cut, having changes late in development is asking for trouble, and no experienced developer worth their salt would ever actually welcome such things. As a result, Agile is largely about coping with problems rather than properly addressing them.
Overheads
Scrum is quite heavy-going on meetings, easily taking up half a day of developer time per week, and then you have the overheads of a non-developer Scrum Master. For a 5-developer team, this results in 30% of the team's personnel budget being blown on non-development activities. At companies with less than 10-15 developers, there are clear amortisation and cost/gain issues. Ironically I have seen Scrum criticised as having poor scalability.
Customer involvement
While any sane software project has to keep customers in the loop, the way Scrum requires regular feedback on frequent betas is a major can of worms. For starters it assumes the client has the technical resources and infrastructure to be able to do proper evaluations, and things get particularly messy when the software is not intended for unrestricted use by the customer. Trust is a big assumption.
Task sizes
There is an implicit assumption that the project can actually be broken down into a large number of relatively small tasks, and this assumption is big. It only really works well in projects where relatively little of the effort is architectural. A large-scale website using a framework such as Django or ASP.NET is a good fit because the work is basically on mostly independent sub-sections, whereas an RTSP/RTP media streaming server would be a very bad fit as much of the work is on a tightly-integrated core.
Anti-specialisation
While I agree with the motives in having no hierarchy between developers, the concept of special generalists is an utter cop-out that is loaded with long-term job satisfaction issues, particularly for the ambitious. For someone like me who has a Ph.D, discouraging specialisation in itself is nuts, so I would not be surprised if Scrum becomes a red flag when looking at future companies.
Little strategic foresight
More often than not, there is little incentive to treat stories as something other than isolated tasks, so there is a tendency to kick than can down the road rather than think about architectural consequences of implementation. Result is disjointed development with wasted effort. User stories aside, there is very little in terms of planning documentation to help with consistency, which explains why on my project there are five different error-handling conventions.
Discourages technical documentation
While the idea that working software is more important than comprehensive documentation is fundamentally sound, its practical consequences are not. As a result little gets produced in terms of technical references, or even documentation of assumptions. This makes things particularly unforgiving for new developers hired mid-project.
Vested interests
Use of post-it notes in planning & retrospective meetings is clearly a business school import, and metrics like team velocity are the latest in a long line of attempts at quantifying something that is qualitative. Agile and Scrum fundamentally are marketed to management rather than actual developers, so it contains some stuff that is of marginal relevance to the actual code writing process. There is also a clear consultancy gravy train built around Agile and Scrum, with plenty of people out there happy to charge top dollar for training courses and certification.
As far as developers are concerned, Scrum does not add much to the actual process of writing software, and to an extent actually discourages some of the thought processes that go into good software engineering. While developers might like the work, they will never have any real love of it, as the latter requires responsibilities that will never be delegated to them.

Overall

The basic idea is that software developers should write software, and ought not concern themselves with things not related to this role. While it is very hand-off in terms of how the development team gets things done, the processes are based on quite rigid assumptions, and they also fail to address some fundamental underlying problems.