Novopay Technical Review for Dummies

By | March 20, 2013

I’ve worked on “enterprise grade” software for a large chunk of my career. Do not get me wrong: I could not build a payroll system capable of running the NZ School payroll without error, but I do know a grade-A clusterfuck when I see one.

This post attempts to cut through some of the veiled (albeit admirably blunt as far as political documents go) language in the Novopay Technical Review (heretofore known as the NTR)

Bear in mind: the NTR is focussed specifically on the current situation and how to get out of it, not how we got here. However based on the findings, it’s pretty easy to see some of the how.

So, let’s run through the “key findings” and break those out:

1. In some areas system functionality does not adequately support the business processes.

The system, as built, does not do what it was meant to do. This could be because the requirements were poorly documented from the outset (probably), but also because the system is an “off the shelf” system that has been “customised” to meet the requirements.

At some point, a decision would have been made that the “off the shelf” software was an 80% (or 70% or 90%) fit for requirements, and the rest could be built as customisations. Hold that thought and read on.

2. Usability issues and lack of data input validations contribute to processing errors.

To me, this is the most egregious finding. As I understand it, the entire purpose (or at least a key selling point) of Novopay was to reduce costs by moving from Datacom’s human-intensive workflow to a system whereby schools do all the data entry, and the “system” just needs to calculate pay.

For a system like this to work, a massive amount of time and focus has to go into usability of the front-end system. Later on in the review there are comments about tab keys not working properly, and links between pages of the same input form not working. This is fundamental stuff, and points to Talent2 and ALESCO not giving a flying shit about end-users.

This, while fairly typical of an “enterprise” system build on Oracle Forms, is completely unacceptable if it’s so utterly important that data is entered correctly and quickly. It is an absolute core failure of a system intended to be user-driven.

As an aside: this is what “enterprise” software users work with every day. Usability and User Experience are pretty much dirty words, with SAP and Oracle making their money from systems that are actively hostile to end users.

3. School management visibility and control is limited by reports that are sometimes poorly presented or inconsistent.

See above.

4. Data quality has been affected by system issues, raising the risk of future errors.

Shit’s gone bad, which makes future shit worse. There’s a comment later in the report that data integrity is often managed by code, rather than by database constraints. Not inherently evil when done for the right reasons, but this sort of code approach more often than not points to coders that don’t understand the underlying system.

Think about a situation where your pay has been calculated incorrectly, then fixed up by a manual adjustment, what happens when the pay calculation is fixed? Does your tax liability for the year include the adjustment?

10 points if you answered “I have no fucking idea”. Perhaps we just cross our fingers? With no test suite there’s no way to know.

5. Quality controls on data entry have not adequately prevented errors.

Too much reliance on the computers, not enough overview by experts.

6. A high degree of customisation in high-impact areas has made on-going development more difficult.

This is another one that really sets my alarm bells ringing. Talent 2 sold Novopay to the government as an ALESCO system with some customisation (see point 1). This is the dirty not-so-secret of any “out of box” enterprise software installation: the base system is next to useless without massive amount of customisation.

More often than not, when you add up the cost of an “out of box” solution PLUS the required “customisation”, a greenfield development fine-tuned to your own requirements becomes a much more comparable solution.

In the case of Novopay, the core system itself has been “customised”, which makes a complete mockery of the “out of box” solution. Expect upgrades to come with significant pain. Also expect ALESCO to wash their hands of any errors because the system is no longer “as-sold”.

 

7. Aspects of the application architecture make customisation difficult.

See above.

8. Service support processes have struggled to manage the volume of issues.

Shit’s gone so bad that Talent2 don’t have the people or expertise to get it under control. Expect the costs to continue to escalate while the government throws money at a solution.

Conclusion

Like I said on Twitter: Novopay appears to be the wrong solution sold by people with a poor understanding of the problem, implemented with the typically lax approach of enterprise software development.

In other words, pretty much business as usual for Enterprise IT.

If you think this thing has run its course, you’re kidding yourself. I’ve seen Oracle systems quoted as costing $6m run up $30m of cost before being completely scrapped – as in yes, uninstalled and reverted to the system they were meant to be replacing, completely wiped away. Or there’s the  $1bn spend for “no significant capability” on a US Airforce SAP system.

How do we fix this?

Come back next week and see me get on my high-horse with a real solution =)

 

 

17 thoughts on “Novopay Technical Review for Dummies

  1. David Cole

    I want to know how much the Government is getting back in penalty clauses from Talent2 continually getting it wrong.

    Any half decent service provider contract includes SLA reporting and penalties for errors and failing to meet SLA conditions.

    Reply
    1. Stephen Bell

      They’ve not started on more than an exploratory stage of “who’s to blame & how much does who owe whom”. “We (govt & Talent2) have our positions,” says Joyce “and not surprisingly, they’re different.”

      And whether penalty clauses can be invoked, he says, depends on “who signed off on what”.

      At present, the focus is on putting it right, not sheeting home blame -yet.

      Reply
  2. Henry Maddocks

    @david cole: Probably nothing. If they were this bad at spec’ing the system what makes you think the contract would be any different.

    Reply
  3. Dylan

    The mention of the core system having been subject to customisation is, in my mind, the scariest of all (actually that’s a big call, there’s a lot of scary shit)…

    As soon as you modify the core system you can say goodbye to being able to incorporate security patches and other vital updates without some major re-evaluation and implementation process.

    Everything I ever learned from various small web applications development projects makes it clear to me what a clusterfuck this was destined to be. It’s insane that this stuff can (and frequently does) happen at a high level.

    Reply
  4. Zef

    It seems that there’s still a perception in some quarters that Usability and User Experience is just about organising user interface elements – but it’s so much more than that.

    It’s about observing and understanding the users of the system, including their habits, equipment, job pressures and where it fits into their day. We identify patterns of use and spend a lot of time refining the user interface to remove obstacles, frustrations and the possibility of errors.

    It’s crucial that the interface gives the user clarity around what action they’re about to take, feeds back to the user about what they’ve just done and, if possible, allows a quick way to correct an error should one slip through.

    That’s why they call the field “HCI” (Human-Computer Interaction).

    Interaction is a kind of action that occurs as two or more objects (e.g. human and computer) that have an effect upon one another. The idea of a two-way effect is essential in the concept of interaction, as opposed to a one-way causal effect. So it’s not just about a user whacking data into a system – but sadly – many enterprise IT systems are just that. A one-way interaction – enter data, enter data, enter more data.

    For example, does any decent airline website let you book a flight between destinations that don’t connect or book a return flight on a date earlier than the departing flight? A good system won’t allow the user to select these options in the first place. An airline website will also check (and sometimes double-check) the booking with you and then send you confirmation. It’s a complex process but the user understands what’s going on.

    I don’t know the complexities of Novopay and what the user interface designers had to deal with (probably a lot of constraints and push-back), but I was flabbergasted when the company started pointing the finger at the users. A well-designed system should not allow users to fail at their job using their tool.

    Reply
  5. Guy Sherman

    Customizing the system is certainly a bad idea, and certainly usability would have helped reduce the error rate with the data-entry, but I think the problem starts sooner, and it is a classic: “fixing” a broken process by replacing the system that supposedly supports it.

    Putting in SharePoint won’t solve your document management issues if you haven’t worked out how to manage documents.

    A financial system won’t help your finance woes if you haven’t got sound financial management practices.

    A new timesheet system won’t solve your payroll problems if your payroll process is unmanageably complex.

    I have extended family who work in education, and as far as I can make out, conceptually, paying teachers is not that complicated.

    The government tried to bandage a broken process with a new system, and then selected someone to do it without stopping to ask if the massive bargain was too good to be true.

    Reply
  6. LB

    I think a large part of the problem is that we have decision makers in government who are there not as a result of merit but length of tenure, signing off on expenditure they do not understand, to build systems sold by snake oil salesmen.

    These bottom feeders of our industry know exactly which words will conjure visions of efficiency and cost savings, things to make the public servant look good when first announced, before any actual proof is furnished. Meanwhile, fast forward a few years, and the result is overruns and non-functional products.

    Honestly, it feels like the majority in our industry are too busy coining it to consider applying their skills into increasing the capability of government, and to be fair, government itself does not do much to attract top tier talent.

    Because government does not have hard-arsed, technically savvy leaders who can push back when they’re being bullshitted for under delivery, this saga is going to repeat itself.

    This is not a problem unique to New Zealand. In every Western democracy, there exist legions of consulting scumbags who exist to suck the public purse dry, building solutions on costly edifices that require constant handholding to limp forwards.

    As an aside, my father who is a headmaster in our school system, for the past three months has been paid as if he was his first year in a teaching job, having to take a mortgage holiday to make ends meet. Bloody ridiculous.

    Reply
  7. Stephen Bell

    “Clear instances where the system does not meet specification, as well as example where specification may be met but the result is not fit for business purposes.” (NTR P6).

    The spec’s wrong in places, but in places they didn’t follow it. Can we call Talent2 to account for the latter? I suspect that depends, as Joyce said at the Beehive media conference, on “who signed off on what”.

    The part of my input RadioNZ chose to highlight (possibly because it was the bit that overlapped least with what Ben had to say) was the apparent lack of adequate regression testing once they started fixing things.

    As the NTR puts it: “In some cases, remedial system changes have caused unexpected problems in other areas”. Surprised this doesn’t figure in your top 7, Ben.

    RadioNZ clearly wanted a piece on the causes of the failure and while there were significant hints of this, detailed diagnosis is not really what the NTR was about, and I tried to make that point.

    That aspect in full will have to wait for the Ministerial Inquiry.

    Reply
  8. Ani

    The whole thing is a massive fuck-up and an embarrassment.

    As a project manager, I can’t possibly imagine letting a system go live with so many critical and high defects, nor recommending to a client or project stakeholder that customising base code was a good idea! Sound likes there was a whole project team on both sides that were working on a project well above their expertise….

    Reply
  9. Clifford Heath

    “a greenfield development ” is a career risk, but anything based on an accepted product (note: I didn’t say a *good* product) is not. I wrote about this in

    Reply
    1. Trix

      That is actually depressingly true.

      It’s precisely the gap – assessing cost/benefit of a properly-geared greenfield dev job vs customising a off-the-shelf product – that good business analysis should fill.

      And why doesn’t that happen, so often? Poor understanding of a BA’s role? Refusal to take on their recommendations? (Sadly true, much of the time.) Bad BAs? (There are definitely some cowboys out there.)

      I’m not a BA myself, but I believe that a good one engaged at the right point is just as influential on project results as a good PM.

      Reply
  10. Ashley

    Couldn’t they just say to each school:
    “Here’s $X, go buy whatever usual payroll software package a company with 10-200 employees uses, (or whatever one you want to use)”

    Then once a month the school invoices the ministry for the sum of the payroll payments.

    There’d be some 10-30 staff member schools where the ‘payroll system’ could probably just be Xero or some similar service, the govt. could have negotiated a pretty good volume deal.

    Seems like it’d cost less to de-centralize the whole mess.

    Reply
  11. Trix

    Great article, and I couldn’t agree more on your disdain for “enterprisey” systems like Oracle Financials and SAP having interfaces that are actively hostile to their users, while quite often (mostly?) failing to deliver promised functionality.

    And to add insult to injury, implementing these systems costs a significant amount, not least because of the inflated wages of their “consultants”. I’ve yet to see one of these implementations come in at anywhere near on budget – I can’t help wondering if their business model actually depends on a lack of understanding of the concept of “sunk costs”.

    Also, all this focus on the actual software tends to indicate to me some serious blame-shifting by the Ministry from their obviously-inadequate requirements gathering process. Having to rewrite core components indicates massive FAIL right there.

    Reply
  12. Garion

    Simply put, this is a clusterfuck. It makes no sense to me that the government would put such a critical job in the hands of a clearly inexperienced, incapable company. It’s a massive project, and certainly not one I could undertake single-handedly, but it’s also not that difficult to get right, and passing the reins to a decent New Zealand development firm would more than likely have resulted in a much better outcome.

    Politicians that are incapable of making intelligent decisions in their ‘area of expertise’ shouldn’t be employed in that role. At the end of the day it only serves to hurt the citizens of our country.

    As a side-note, I’m a rookie developer, and even I can’t understand editing the core of a framework or software package. The only foreseeable outcome is disaster and there are better ways to do what they wanted – namely, start from scratch, or EXTEND the existing code, rather than editing it.

    Reply
  13. AndrewNZ

    What really gets me about this whole thing, is that they did the exact same thing to us at NZ Post 12 months beforehand. Our payroll system can’t be anywhere near as complex as the School payroll.

    It was a complete disaster with all of the same end issues I’m reading about in the news. Even now, 18 months down the track, there are still significant issues.

    There is no way Tallent2 should be paid, or allowed to continue supplying payroll systems to public sector organisations.

    I sincerely hope they do the parliamentary payroll next.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *