In the end, most software project failures come down to politics. We grind through endless methodologies to prove we are doing everything 'correctly', but no methodology yet devised supplies common sense where it is lacking in the people, and the conclusion of the report is never one the boss will disagree with.
In theory, everything we do is rigorously cost-justified. In theory, where I first worked, every program was written using Jackson Structured Programming. Everyone was taught the basics, and the Job Description said it would always be used, on pain of death or worse. In practice, nobody used it, and nobody was expected to except at the Annual Review. In practice, the project is justified by a conversation with the Chairman on the golf course, or to keep a senior manager quiet.
The few times we do try to cost justify some work, we can't. For lots of reasons. You can know something will save the company money, but proving it to the accountants who really run the joint is another matter. A realistic estimate of the cost is unacceptable, so an acceptable estimate is given, pared down, and accepted. There is never enough money to do it right, but always enough money to fix it once it is done.
That is why 95% of software projects come in after time and over budget. They wouldn't get started any other way. We think of that as a 5% success rate, but were those 5% of projects really successes? Do millions of peoples' lives really benefit as a result of them? Or were they the projects nobody cared about enough even to change the spec? Some of the 95% are awful disasters, but some of them are the real success stories of our industry: projects that through ingenuity and imagination have improved the world people live in. Projects that took about as long and cost about as much as the people actually involved always knew they would. What does it matter what the plans said if, at the end of the day, it actually works?