The Sociology of Software Development Organisations

Last week I traded some emails with a person (I will call him Dangerous) who still works at a company I wrote a recent post about. In that post I discussed an event where the company had benefited by acting on experimental evidence rather than following standard processes. I wondered if as a result of this event the company had learned to learn from experimental evidence, or if it had simply incorporotated the solution into its processes.

Dangerous told me he remembered things differently. He remembered me introducing some software development practices and a style of doing software development. There was the continuous integration and testing with high coverage tests and static analysis thatI mentioned in my post and pervasive unit testing, short development cycles, short and focused daily meetings and automatic reporting of predictors of code maturity  that I didn't mention. My style was to focus on measurably working software and address customer needs. He liked this approach because he thought it was mostly what effective development teams did anyway and it was easier for him to work with an approach that was mostly right rather than spend time trying to fix what was wrong with the approach.

Dangerous told me that after I left the company most of the practices I had introduced had been downplayed until a new GM arrived. The new GM was an Agile enthusiast and he introduced Agile by re-using many of the tools my group had developed. Dangerous said he liked this approach too because he thought it was mostly what effective developers did anyway and it was easier for him to work with an approach that was mostly right rather than spend time trying to make the approach perfect. He also told me what he didn't like: the buzzwords and the Agile consultants who knew the buzzwords but didn't understand large-scale software development. A small price to pay for sane management in his opinion.

What the new GM did made sense to me but there is not much to learn from people you mostly agree with. This post is about why the organisation moved in a completely different direction between my (and other people's) somewhere-in-middle-management) approach and the new GM's top-down approach.

The biggest resistance to the empirical customer-focused approach I favoured came from other technical middle managers. These middle managers had the key skills of management. They were well-liked  got things done, could deliver to a schedule, were well-organised and sensitive to risk, and were technically smart and good communicators.

So how could people who seemed smarter than me take a direction that seemed to be less productive than the one I had left them with?

My guess is that it only looked that way to me and the experienced developers, not to the other middle managers. Software development processes have at most a third-order effect on companies with mature software product lines. All the critical features have already have been added so new software development has only a second order effect on a company's success, and processes have only a second order effect in software development ,way below people. Once the development middle managers deal with the the first and second order issues they may not have time for the third-order issues like effective software development. And the higher they go up the organisational ladder in many development organisations, the less time they have. Thus increasing infuence correlates to decreasing interest. This is a rational reaction to an organisational structure.

The second issue the middle-managers had with the empirical approach was that it was disruptive to the organisational culture. This system I set up was automated and cut through a lot of the human communications that dominated the development work at the time. The middle-managers had spent a lot time making the communications-intensive style work so learning a new style was going to be a lot of effort for them.

Stated in that way, my approach sounds like an irrational reaction to my circumstances in the company. The best I can do to explain it is to trace it back to the time of previous GMs who were focussed on the first and second order issues of customer needs and retaining and motivating good staff. These GMs delegated all the third order stuff and that is how it became my ecological niche when I came across the technical problems of large scale software development.

Further Reading