Multi-Vendor Software Development
- Solutions to Some Common Problems -
by Paul Hazelden

Contents

0.   Introduction

1.   Development Issues

2.   Project Management

3.   Project Scope

Introduction

Whenever a piece of software is being developed or maintained by more than a single person or a single tightly integrated group of people, a number of standard issues arise.

The issues raised by software development in a multi-vendor environment are essentially the same as those in any complex development, with the additional problems raised by the fact that the different parties involved are not all working for the same employer.

This document addresses some common problems experienced in the development of large or complicated software products.   Almost by definition, any product which includes software from more than one vendor is going to be complicated.

1.   Development Issues

It should be noted that the following factors are commonly found in complex developments which run into difficulties.

• inadequate documentation
• inadequate testing
• over ambitious scope
• attempting to deliver too much at a time

This document attempts to address these factors, and other common related issues, which cause software development projects to run late and over budget.

1.1.   General Development Problems

The following problem areas are common to most complex developments.   The answers to these problems are straightforward and well know, so I do not intend to address them further in this document.

1. Simultaneous Updates

Inadequate version control results in several people changing the same module, and some of these updates being lost.

2. Configuration Clashes

Different parts of the development establish different configuration requirements for shared aspects of the delivery environment.

3. Unavailable Resources

Different strands of the development all need access to key areas of the system, but there are not enough people with the knowledge and experience of these areas to support all the simultaneous development.

4. Extending Scope

Sometimes the developers realise that it would be 'easy' to do this... and that... as well as the things originally requested.   More often, the Users think of more 'essential' features the longer the development goes on.

5. New Application Context

An application (/subroutine, etc.) which works perfectly well in one context is used (because 'we know it works') in a different context.   Some of the original assumptions no longer hold true, and the application fails - often in subtle ways which are difficult to spot.

6. Uncovering Old Problems

An application which appears to work in one context is used perfectly correctly with different data, loadingpatterns, etc,. and errors which were always pressent suddenly come to light.

1.2.   Multi-Vendor Development Problems

In addition to the general problem areas identified above, multi-vendor developments often suffer as a result of the following areas.

1. Competition

The different vendors are in competition with each other.   As a consequence, they are often unwilling to help each other, or to share information and resources.   Issues of confidentiality, and the desire to sell help to each other rather than give it away.

2. Priorities

The reason why each vendor is involved in the project will often be different, so the different vendors will each have different agendas and priorities

3. Contracts

The different vendors will need to have contractual arrangements with each other which costs money to define and maintain, and interferes with the normal process of co-operation which can be expected between people working for the same employer.   An 'internal market' is not the most efficient mechanism for keeping costs down: the most efficient way is to motivate the people involved.

2.   Project Management

2.1.   Simultaneous Development

The key is to restrict simultaneous development to projects which can be developed in parallel - this is defined more precisely below.   The pressure to allow more projects to be developed simultaneously 'to save time' must be rejected.   The cost of making two groups of developers each reliant on the other group to deliver software which functions as required before they can complete their work will always outweigh any potential saving of time through working in parallel.

1. Analysis

Any number of projects can work on the analysis phase of related areas as long as they each keep the other projects informed of their work.

There is a cost involved as each modifies their ideas to take account of the work being done by the other groups, but this is offset to a degree by the synergy of access to independent but relevant ideas coming from the other groups.

2. Development

Projects can be developed as long as the version control system allows them to have control of all the modules they require to develop and test their work.

• The easy part of this is development: you do not - under normal circumstances - ask several people to update the same module at the same time.

• The more difficult part is testing.   There is no point in people developing software which they cannot effectively test.

'Control' can come in two forms: active and passive.   Active control involves total ownership: the module can be changed in any way which satisfies the documented requirements.

Passive control involves the ability to prevent change.   Interfaced modules which are outside the scope of the project need to be locked.   The project does not need to change them, but it needs to ensure that they are not changed by any other project.

At any one time, each module can have one of three control states:

• Free - no project has control of it.
• Active - one project has active control of it.
• Passive - one or more projects have passive control of it.

Each module must therefore be regarded as a resource to be allocated to the project for a period of time.   For modules where active control is required, they are allocated to the project in exactly the same way that people are (or should be!) allocated to the project.   For modules where passive control is required, they must be flagged as unavailable for update until the project is complete.

2.2.   Phased Delivery

How do you eat an elephant? One bite at a time.   How do you deliver a large or complex product? One bit at a time.   Correct division of the project into phased deliveries is an essential tool in project planning, but a tool which people often fail to understand and use.

Various tools and methodologies exist to support phased delivery.   Some popular approaches, such as RAD - 'Rapid Application Development' - emphasise the need to achieve strict deadlines, even at the expense of some functionality.   Others emphasise the need to develop the right code first time, or some other aspect.   The following is intended to be applicable to any of the ways in which phased delivery can be implemented.

1. Typical Single Phase Development

The typical project plan goes like this:

• Design
• Build
• Test
• Deliver

The reality is not like this.   In any development, you always hit unexpected problems.

You often get through the 'Design' part without any significant nasty surprises.   You generally get through the 'Build' part without any surprises: people have been writing code for a long time now.   The initial part of the testing goes well, because the programmer has already done a fair amount of unit testing before releasing the program.

At this point, the project is 95% of the way through the work that needs to be done before the first delivery to an End User.   A few minor errors have been identified and fixed - things like spelling mistakes and screen layouts - but everything is more or less on schedule, and hopes are high.

Then another error is identified and fixed - and half the system fails to work.   The fix to one module has had a knock-on effect on several other modules, and they all need to be changed accordingly.   Some of the changes uncover or create new problems, but these are soon fixed.   The testing starts again.   The project once again reaches the '95% complete' stage.

Another error is identified, and this time it is a design problem: one part of the system assumes records will be created in a certain sequence, but the sequence is not guaranteed by the part which creates them.   Most of the time the sequence is as expected, because of the way people tend to use the system, but not always.   There is no easy way to constrain the sequence.   A design decision is required.   Fundamental parts of the system architecture are shifted around, the programmers go off and make the changes.   There is no time to update the specifications, the project is several weeks late, and everything has to be done as quickly as possible.

Mistakes are made in the changes because they have not been thought through or specified sufficiently.   Where they are made correctly, new knock-on errors are created, and, when two programs refuse to work together, it is now impossible to tell which one is at fault as neither are now doing what they were specified to do.

Eventually, the system becomes stable enough to ship to the first End User.   And then you discover that what was designed was not what the User wanted or expected, so parts of the system need a major re-write...

2. Learning From the Traditional Experience

So what has gone wrong, and how can a phased delivery approach help to avoid these problems? Here are three preliminary observations.

First, we have to recognise the problem is not that errors were found in the programs, or even that the design was not perfect first time round.   People - even the best people - make mistakes, and any project plan which fails to take this into account deserves to fail.

Second, any approach to project planning which tells you the project is 95% complete when perhaps less than half the total work has been performed is fundamentally flawed.

If we choose to use the traditional approach described above, we have to recognise that no matter what the project plan may say, there is absolutely no way of telling how much work remains outstanding.   On the project plan, a job is done, it is signed off, and it is therefore complete.   In reality, modules are written, changed, and changed again.   Unless you can tell how many times a piece of code is going to be changed, you have no way of telling how complete the project is.   Human optimism tells you that each change is going to be the last, so projects are planned on the basis of wishful thinking and not hard facts.

Third, we must recognise that cost-cutting never works.   Any time which is saved by not updating the documentation, or by releasing a program before it is fully testing, will be wasted many times over further down the line - and will be lost when you are closer to the delivery date and less able to negotiate another date or provide a plausible explanation.   Once you have established a set of procedures which work, they describe the best way to do the job.   Therefore any attempt to 'save time' by ignoring the procedures will inevitably cost both time and money.

3. The Phased Alternative

If cutting corners and 'going faster' do not lead to a faster delivery, what can? The logical, if counter-intuitive, answer is that you can deliver the product faster if you do more work and if you do the work slower.

The two key features of the traditional scenario are: firstly, while minor errors are detected quickly, major errors tend to be identified only late in the project lifecycle; and secondly, the major errors have knock-on effects on other areas of the system, which cause changes to parts of the system which are thought to be complete.   Phased delivery addresses both these issues by identifying major errors earlier in the project lifecycle, and to minimising the knock-on effect of errors when they are found.

Firstly, in breaking a large or complex project down into a number of deliverables, it becomes essential to specify both the deliverables and the interfaces between them.   This additional work, which has to be done up front, removes most of the design errors, and identifies the remaining ones much earlier in the project lifecycle.

Secondly, the early delivery of some parts of the system means that at least some mistakes in understanding the User's requirements can be picked up much earlier.

Thirdly, isolating parts of the system means that modules which have been built and shown to work will generally not need to be modified even if the requirements for other parts of the system change.   The definition of the interface between different parts of the system and the provision of a way to test that interface ensures that knock-on effects of changes are minimal.

Finally, the tools built to test the phased deliveries will also enable all future developments to be undertaken with confidence and effectively tested.

3.   Project Scope

Many projects fail because they are defined with an inadequate or incorrect scope at the outset.   Unless there are good reasons for excluding them, the following aspects must be covered:

3.1.   Requirements

All the requirements for the project to be a success must be identified at the start of the project.   Once the project gets the go-ahead from management, the only remaining question is whether the product as delivered fulfils the stated requirements.

1. Platform

The platform on which the product is to run must be clearly defined.   This is an aspect of the system pre-requisites.

2. Functionality

What the product is to do must be defined.   Also, what the product is not to do must also be clear: this is one of the main weapons in the fight against scope creep.

3. Performance

The minimum acceptable performance on a specified reference platform (hardware and software) must be defined.

4. Partitioning

The distinct deliverables must each be identified, and - crucially - the interfaces between them.

3.2.   Documentation

The project must deliver all the documentation which is required to enable the Users to correctly and effectively use the product, and to enable enhancements to the product to be specified and implemented.

1. End User

In general, this will need to be provided in two formats, both on paper and as on-line help.

2. Technical

Written by the programmer and analyst explaining what they would need to know if they were asked to maintain the system.   Crucially, this needs to describe the design and architectural issues which were considered in the development, the decisions made, and the reasons for those decisions.

The usual practice of using the technical documentation to descibe what the system does at a technical level is almost entirely pointless: all this information can be gained (much more reliably!) by looking at the source code.   What the people who follow desperately need to know is not what the system does, but why it does it that way.

3.3.   Implementation

The project must deliver everything which is required for the product to be used.

1. Distribution

The product must be distributed: supplied to the people who will use it.

2. Live

The product must be capable of running in the live environment.

3. Training

The Users must be capable of using the product.

3.4.   Testing

The project must deliver everything which is required to effectively test the product to ensure that it conforms to the documented requirements.

1. Automated

Automated testing must be defined as part of the product definition so that there is no uncertainty about whether the product performs to specification, and so that errors cannot slip in unnoticed because there is no point in paying for a full product test after every minor enhancement.

2. Test Stubs

There are basically two situations where test stubs are essential.

Firstly, when building a system to get the full range of functionality before the full depth is implemented.   This is essential when a phased delivery is being adopted.

Secondly, when building a client-server system, to test the parts which are used in several places.   The test stub defines whether the common part is functioning correctly.