• Is co-located
• Is always available
• Knows everything
• Can make decisions
• Can prioritise
• Knows how to write user stories
• Can see the ‘wood for the trees’ (is comfortable with abstraction)
• Can imagine a future way of working

A person with these qualities does not exist on the project I’m working on at the moment. I doubt anybody’s working on a project with somebody like that; if you are – work hard to get a contract extension because that’s as good as it gets.

Product owners are not requirements engineers, nor do they want to be. If we, as the delivery team, sit back and wait for a prioritised and properly sized set of stories to arrive in time for our planning game, we are going to be frustrated. This is a ‘push’ model; the requirements people push us the stories. What do we do if the stories aren’t coming through? Well, then we have to use a ‘pull’ model to get the stories into the planning game.

The business knows what they want; up to a point. As far as they are concerned, they have told us what they want. It’s not enough for delivery to get on and build it, but it’s certainly a good start. We say ‘where are the requirements?’ They say ‘you know what we want.’ So what’s the problem? The problem is the devil is in the detail.

If we don’t have enough detail, we need to model the scenarios, look at the data, consider the alternatives, come up with our best recommendation of how the system should behave and then present the business with the findings. So, a pull approach to getting stories into the backlog, when the cupboard is looking bare, is you do the modelling and scenario work upfront, then you have a conversation and then you get agreement to feed the sprint.

The sprint is a hungry story monster that needs to be constantly fed with good quality stories. The business can get bogged down in stories, value, priorities, acceptance criteria and release plans. It’s hard. The analysts in the delivery team need to offer the business analysts/product owners a hand – whatever it takes to get the job done.

Software engineering tries to excuse itself by saying that it’s a young discipline so it can’t be expected to get it right yet. But software engineering has been going for 50 years or more, and in the same time, if bridge engineering hadn’t managed to get its act together people would still be using ferries to get across rivers; they would have concluded it couldn’t be done.

There have obviously been software successes. It’s said the computing power in the Nasa rocket to the moon was much less than the machine I’m using to write this. Some teams are better at delivering software than others. How would it be if some bridge builders built bridges that stayed up while others built bridges that fell down. If you needed a bridge built you’d know where to go – to the successful team.

It’s easy to know if a bridge is successful (i.e. it doesn’t fall down), it’s easy to see what progress is being made over time and it’s easy to know if the bridge is going to come in on budget . If you were on the team whose bridges kept falling down, you would probably go and see what the successful teams were doing that was so different from what you were doing. Instead of working from a drawing scribbled on the back of a cigarette pack, you might find they had a good cohesive set of engineering diagrams and conclude that these were prerequisites of project success.

In the days of old, it wasn’t the bridge architect that drew the diagrams, it was the draughtsmen. Draughtsmen were the power behind the creative vision. Draughtsmen studied long and hard to learn their craft. Draftsmen would sanity check a design. They would ensure the maths were right and the bridge wouldn’t collapse. They figured out how much steel and how many rivets to buy, they drew diagrams that explained the order in which things needed to be built. They drew the bridge from different angles, they drew it from the top and from the side to provide perspectives.

Software engineering needs good drawings. Up until now good diagrams have defeated the software engineering community. The state of the art can describe approximately what they should look like, but the state of the practice is to forget them all together. In the old days we used to use Data Flow Diagrams (DFDs) and other arcane representations. Programmers found they were of no use. There was no way to go from the drawings to the code – so nobody could see the point. Instead, sections of the IT community decided it couldn’t be done, it was a waste of effort so they dreamed up the Agile way of working and somehow this got mis-interpreted as meaning the drawings weren’t important. They never really said that, but it was easy for programmers who always hated doing drawings anyway to interpret it that way. So they didn’t do any drawings, they didn’t believe in capturing the requirements upfront, they just believed in programmers getting their heads down and working as a band of heroes to somehow build software systems in a warm fuzzy collaborative environment where the requirements would simply present themselves as a consequence of the programmers building something and showing it to the users. Think about it – what I’m describing is a process where the making of the diagrams is too hard, so nobody bothered. And if that had worked we’d see the crisis in IT project delivery solved, but that hasn’t happened.

By the model described in this paper, Agile can be broken down into Agile Delivery (writing code and sprints etc.) and Agile Requirements. Agile Requirements is an approach that requires the breadth of the requirements to be articulated. It does not require the depth of the requirements to be articulated in advance; depth (or detail) can be deferred on a ‘need to know basis’. Another way of thinking about this is as a ‘just in time’ approach to requirements representation.

It is possible to draw good diagrams, but it can’t be done without training aimed at doing exactly that. It’s not something you learn on the job. Good drawings are mostly done in UML and are drawn according to internationally recognised standards. The drawings show the system from different perspectives including both data and behaviour. All the drawings are related, they are all cross-referenced and they all tell a different part of the story, but it’s all about the same story and it’s clear how they are related, how one starts where another ends, or how one provides an input to another.

Engineering drawings are divided into static and dynamic models. The important views are:

• Data view
• Use case model view (the ‘epic’ view)
• Activity view (or the process models)
• The wireframe view

These views are used for a host of purposes. The epic view is akin to the ‘table of contents and is especially useful in:

• defining the user stories or iteration stories programmers use to write code against
• informing release planning
• informing project progress reporting against the project plan
• informing user acceptance testing
• defining ‘done’ i.e. when the system is finished

It takes education to produce software engineering diagrams, and it takes training and motivation to read them. To the uninformed a UML engineering diagram might be right or it might be wrong. The uninformed can’t tell the difference between good and bad. Diagrams that are not validated as correct and which do not go on to be used effectively are useless. I’ve coined the term ‘Analyst Draughtsmen’ to describe the people who produce the diagrams. Ideally, two analyst draughtsmen should work together to verify the diagrams one of them produces. The diagrams will go on to be used by the product owner, the project owner, the project manager, the software designer, the architect  the programmers and the coders.

A sufficiently accurate representation of the requirements can be reliably constructed through the application of requirements patterns. All database driven applications are similar at an abstracted level and an understanding of the patterns allows the professional analyst draftsman (or woman) to create a requirements framework very early in a project, typically taking no more than six weeks. The approach taken is both top down and bottom up (meet in the middle) where a requirements framework is created using requirements patterns that are verified with the business through interviews/workshops. All that is required to apply the requisite requirements patterns is a natural language description of the system, ideally a scope document or project mandate. This is sufficient in most cases.

A typical IT team is a group of people educated as programmers and computer scientists brought together and led by business people. The business people receive no training in how they are supposed to manage the IT team. It’s a formula that shouldn’t work – and normally it doesn’t. Too often the project is reduced to mutual recrimination where the business blames IT for failing to deliver and the IT team blame the business for failing to engage with the project. But nobody’s trained the business to manage the IT project – that kind of training is not currently available. This is a big obstacle and a real opportunity for forward thinking educationalists.

In future I envisage a Masters level degree course with two components, the first exists to train business staff to manage IT projects. The second component is for business or technical analysts who want to become analyst draughtsmen. It doesn’t sound particularly high-powered, the role of draughtsman has never attracted great status, but it is where a real difference can be made in representing requirements properly and allowing managers to make informed decisions based on a common language of understanding. The analyst draughtsman is first a technically minded analyst who understands business process and is second an artiste with a CASE tool.

My job, as a technical analyst, is to quality control the stories arriving from the requirements team. The ‘stories’ themselves are held as issues in Jira. This illustrates the level at which the stories are broken down, as they are meant to be groomed to fit a single iteration, where an iteration is only one week. So, the stories are low level, and don’t represent business value in themselves. The stories are decomposed from higher user stories, but the link to the complete user stories has not been maintained. My job is to try and make sense of the existing backlog and try to map iteration stories onto the parent story. The other thing I’m tasked with, the over-riding priority, is to get enough stories into the sprint planning meeting to ensure the programmers are fully occupied; the aim being to have visibility over a four week period of upcoming work.

This is a challenging task, and why I’ve been too pre-occupied to write. I know that is no excuse, if somebody writes on a blog, they need to keep it up, or people will lose interest. This is a different style of blog then, a kind of ‘considered’ blog where I need more time to write anything I really want to say. So it’s a ‘clog. That’s it – I’ve become a clogger! After all people in clogs can’t run very fast (I don’t think…)

• Define features of the product (or desired outcome)
• Decide release date and content
• Ensure profitability
• Prioritise features/outcomes
• Adjust priorities
• Accept or reject work

Does this person really exist? Is it possible that somebody can be found to do this job; that has the subject matter expertise and the authority to make big decisions all wrapped up in one person? That’s a busy person. And isn’t it likely that somebody like that is too busy to participate in the Scrum team on a daily (insert time period you prefer here depending on how ‘Scrummy’ you are) basis?

So we come on to the subject of the proxy product owner. Where is this defined in Scrum? Roman Pichler at the Aglie Alliance writes:

Using a proxy product owner is an attempt to superficially treat a systemic issue. Rather than employing a quick fix, organizations should address the underlying issues. This might require freeing up the product owner from other obligations; colocating the product owner, ScrumMaster, and team; or even finding a new product owner.

http://www.scrumalliance.org/articles/168-common-product-owner-traps

So is this Product Owner a logical role or a physical role? I think it’s supposed to be a physical role i.e. one person does it. I was talking to the owner of an Agile consultancy recently, and he said he’d never met one. I’ve never met one either. Sometimes I get the feeling that Scrum dictates what the world should be like and the world struggles to fit the definition and that’s frustrating. Which is flawed, the world or the Scrum definition?

What was wrong with Project sponsor, Responsible owner, Subject matter expert and Story Teller? After all, these are the competencies the Product Owner is being asked to exhibit. No wonder it’s hard to find this guy/gal. Maybe s/he’s a unicorn.

‘We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.’

Or, a slightly longer version:

• Process is important, but not as important as people working in effective teams.
• Documentation needs to be appropriate and it needs to make the delivery of working software more efficient, because working software is what is important.
• If you have to rely on a legal contract, the relationship has probably already failed, so put a lot of effort into working with the customer as a partner.
• Things change, people change their minds: deal with it, rather than getting hung up on a plan that is imperfect

This is just a way of turning the original points around to give them added emphasis. It helps me to gain greater clarity anyway

To some degree the antipathy with which the PMO function views Scrum is mutual. This is captured in the 2001 Agile Manifesto, which can be viewed as a software engineer’s charter that bemoans the obstacles put in the face of programming teams by ‘Dilberesque’ corporations.

In order to succeed in the new economy, to move aggressively into the era of e-business, e-commerce, and the web, companies have to rid themselves of their Dilbert manifestations of make-work and arcane policies. This freedom from the inanities of corporate life attracts proponents of Agile Methodologies, and scares the begeebers (you can’t use the word ‘shit’ in a professional paper) out of traditionalists. Quite frankly, the Agile approaches scare corporate bureaucrats, at least those that are happy pushing process for process sake versus trying to do the best for the “customer” and deliver something timely and tangible and “as promised” because they run out of places to hide.

Jim Highsmith (2001) from [http://agilemanifesto.org/history.html]

The opinion expressed by Jim Highsmith is undoubtedly, in some cases, true. Where it is true, Scrum will struggle to make in-roads. Where the organisation is driven by the market and less by the needs of the organisation to service itself, Scrum has more of a chance in scaling to the enterprise. Ultimately for Scrum to work, things have got to change, and everybody knows that change is difficult. It is unlikely to happen driven from the bottom up without support and direction from the top. Perhaps one definition of an organisation that is Dilbertesque is the characteristic whereby it is dominated by layers of middle management and senior management becomes remote. So it is impossible to define scalable enterprise Scrum without first defining the characteristics of the organisation where Scrum can flourish. A seed cannot grow in soil that will not support it.

As an aside, the interested reader would do worse than to read:

(http://monster-island.org/tinashumor/humor/corpmemo.html)

I was watching a Jeff Sutherland video at (http://www.youtube.com/watch?v=9y10Jvruc_Q) where he talks about Scrum and Google building Adwords, and it seemed to me that the way they did that made sense for them. Not everybody is Google. We might classify Google as an ‘information company’, and Microsoft as a sealed package company, and PatientKeeper as a ‘managed service’ company. But that still leaves lots of organisations where software is not what they fundamentally offer, what they ‘do’, what they sell. So it stands to reason the way those organistions organise themselves to build software is going to be different. Not everybody is going to be an XP shop (clearly), not everybody thinks Scrum will do everything.

So with a meta-kernal if Bank of America want to combine [RUP, use cases, Java, MVC, cross-functional teams, Scrum, PMBOK] then why not? There are a lot more organisations out there like banks, insurance companies, stock brokers etc. manufacturers, wholesalers etc. who are never going to be cutting edge Agile houses, but maybe they can use bits and pieces from different practices that work for them and maybe that would be OK. Why not? Now that would make a meta-kernal really useful and whether or not it provides the foundation of a universal SE engineering doesn’t matter for this application because it provides a universal notion of how different practices in different fundamental areas of software engineering concern can be combined.

If you find this interesting read ‘Agile and the Universals of Software Engineering’ (an earlier posting…)

A company like Microsoft is likely to have a very different way of building software from a government department, or from a 3^rd party supplier of software who wins their business in open competition. One way of describing organisations based on how they use software is presented below:

• Sealed Packaged solutions e.g. Microsoft, Apple
  Build software products and sell them.
• Unsealed Package solutions e.g. Siebel, SAP
  Build software products, sell them and configure them
• Managed services e.g. PatientKeeper,
  Build a single software service and sell access to the service.
• Information provider: e.g. Reuters, The Guardian, Google
  Build and platform that gives access to valuable information.
• Professional services, (incl. Government, Manufacturer/Supplier): e.g. Aviva, Barclays, Lloyds of London
  Provide banking, insurance, order processing, manufacturing support, government services etc. and use IT to support the business.
• 3^rd party supplier: e.g. mPhasis, Tata, Logica
  Build software and services on behalf of other professional services or information providers.

How core software is to what the organisation does is going to affect the way an organisation configures itself with respect to software engineering.

The ‘Universals’

If you want to build software, what are the kinds of things you need to worry about? Have you got all the bases covered? That’s the question the group ‘Software Engineering Methods and Theory’ (SEMAT) have been asking themselves. SEMAT aim to come up with a design for a meta-method-kernel. This is perhaps not a fundamental theory of software engineering [1], it is however a useful thing to have [2] because everybody building software should have thought about the tasks they are going to be faced with and can at least make a rational decision on what practices they should adopt. The decision they make will be affected by the type of organisation they are and the centrality of software to their core mission.

SEMAT haven’t finished their work yet, (and perhaps they never will!) but we don’t have to wait to propose one view of what that kernel of software concerns might look like.

Figure 1: A version of the Universal kernel of software engineering concerns

The kernel in figure 1 is a collection of placeholders and it is up to the organisation to put some practices into the holes. It can be argued that there will always be something in the holes, even if the something is actually an ad hoc practice. Having watched Jeff Sutherland talk about the work at Google to build Adwords, one is left with the impression that Scrum pretty much does it all [3].

Figure 2: a populated kernel as Jeff Sutherland might have visualised it working on Adwords.

That could be absolutely fine for Google, because they are pretty good at building software and they have access to their product owners on a daily basis. The culture of Google is to have few managers and it is led by engineers. Not all companies are like Google though and therefore there is no need to copy them, because the way they do things may just not be right for the way you do things.

Consider a different kind of organisation, such as an insurance company. This company spends a lot of money on I.T., they used to do it in-house and now they’ve outsourced it to Eastern Europe. They have a Programme Management Office (PMO), they have people who want to manage risk, and are interested in ‘quality’. It’s not my job as a person interested in methodologies to tell them they should change before I can help them build better software. I can work with them to demonstrate that maybe there is a better way and if that is successful they can choose to change themselves.

In the meantime it’s perfectly possible for them to take advantage of the kernel of concerns and define a set of good practices that work for them to improve their software delivery.

Figure 3: a populated version of the universals kernel that would be appropriate for an organisation where software was not its primary product.

In Figure 3, our imaginary insurance company has chosen a set of practices that suits it, its culture and the degree to which software is integral to its business. One of the benefits of this way of visualising a software engineering practice is that it makes it easier to ‘hot swap’ one practice out for another without disturbing the others. This is the familiar concept of ‘encapsulation’.

My particular areas of interest in this topic include:

• Opportunity: Demand management, Ideas management, and Benefits Analysis
• Requirements: Agile Requirements Practice
• Way of working: Unified Process lifecycle
• Governance: PrinceLite www.princelite.co.uk

References:

[1] ‘Why We Need a Theory for Software Engineering’, Dr Dobbs, Jabobson I, Spence I, [2009] http://www.drdobbs.com/architecture-and-design/220300840;jsessionid=RZLXNZ2D4154PQE1GHOSKHWATMY32JVN

[2] ‘A Detailed Critique of the SEMAT Initiative’, Cockburn, http://alistair.cockburn.us/A+Detailed+Critique+of+the+SEMAT+Initiative [2009]

[3] Scrum Tuning: Lessons learned from Scrum implementation, Jeff Sutherland, [2006] http://www.youtube.com/watch?v=9y10Jvruc_Q

Set against a wider context of the end-to-end business process, actually delivering some working code comes far downstream. What needs to happen first? First ideas need to emerge in the organisation, gain momentum, become scoped, costed and valued and then, if all goes well, the initiative results in working system.

Given the project delivers a working service, we would be interested in a retrospective aimed at measuring success. For instance, certain assumptions were made regarding how much value the project would bring and how much it would cost to bring to fruition. We want to know whether it was ultimately worth taking this project forward once it has had a chance to ‘bed down’ because that information is valuable. Ideally a cycle can be established where the assertions and outcomes of initiatives are tested so that they may inform future estimation activity resulting in the ‘go/no go’ decision that governs all intiatives becoming more accurate. In this scenario, the estimates of future projects (cost and value) are compared with the resulting outcomes ultimately leading to the enterprise fully directing the IT function and reaping maximum competitive advantage.

Tags: idea management, governance, PMO, Programme management, Project management, effort estimation, value analysis, agile evolution