Last month I described research being undertaken by Judge Business School MBA students on behalf of ClearBooks as ‘bizarre.’ A few weeks later, the results are in and now I describe it as even more bizarre. At the time I quoted from the blurbs saying:
Their goal is to explore the online accounting market, its size, how quickly it is growing and to compare it with the traditional accounting software market.
It turns out that was at best a misrepresentation and at worst a lie. The report states its purpose is:
This reports looks at the UK market for accounting software targeting small and medium enterprises (SMEs). There are two major types of deployment of the software: traditional on-premise solutions
and software as a service (SaaS), available from anywhere with via the internet.We investigate the competitive landscape, players in both segments of the market and provide an estimate of the current sizes for both on-premise and online accounting software. The size of the online accounting software market was estimated to be around 200,000 paying users currently generating £40 million of revenue per annum. The size of the on-premise accounting software market was estimated to be over 1.14 million small businesses generating over £213 million. In addition, we examine on a case by case basis how some companies have financed their operation and expansion to date.
We also compare the usability of our client, Clear Books, in comparison with its competitors with a variety of user surveys.
At the time I was approached I surmised the report was being sponsored either directly or indirectly. It now turns out that ClearBooks is a JBS client. There is a massive perceptual difference. The people who called up didn’t state anything about their secondary purpose which, when viewed in light of the overall report was in fact its primary purpose. In other words, respondents were duped. I called up David Terrar to find out what he was told. He said categorically: “They never told me this was being done for ClearBooks.” I’m sure they didn’t. While we live in an age of transparency, competitors are always wary of providing market data to those researching on behalf of competitors. And as I said at the time, simply checking in with and relying upon vendor statements is precarious.
In one sense that doesn’t matter. It can be helpful in research to keep respondents in the dark. Usability is one example. However, as I looked through the report, it was clear there are glaring errors.
I know for example that AccMan was used as a prime source because the weekend after they called, I could see page views coming in from .ca.uk addresses. Hundreds of them in fact. (Thanks for the spike in traffic people. Hope you learned something.) I am a primary source in some situations but a secondary in many others. That’s because on the one hand I do my own research while on the other I am relying on information I receive. Therefore any researcher using AccMan needs to discern the difference and come to their own conclusions.
The second major problem comes in the way the research was defined. Since this vendor led, you can be sure that ClearBooks agenda figures into the equation. I’ll return to that later. Next we have the problem of how the stats were compiled and their interpretation. While JBS used multiple sources including corporate reports, they made some fundamental errors in understanding the data they were looking at.
For example, on page 20, they estimate IRIS annual revenues at £119 million of which they ascribe £77 million to accounting across 14,000 UK users. IRIS annual report for 2009 states that revenue from continuing operations was £114.8 million. The £119 million figure is global turnover of which the notes to the accounts clearly state that UK turnover was £115.4 million. Of that amount, IRIS says that 70% of its revenue comes from maintenance. This creates a problem in its own right because whenever making comparisons with SaaS solutions, the question of maintenance figures heavily in discussions around innovation and delivery. This could be a topic of discussion on its own. In the same report, IRIS breaks out the following: £11 mill in payroll services, APS (accounting professional services) is stated at £33 million, Legal is put at £22 million and not for profit/major business is said to be £19 million. That totals £85 million leaving somewhere between £30-34 million for the remainder depending on which number you want to choose. Given that the stated aim was to look at SME’s I’d suggest JBS has failed to understand what it is looking at and made obvious mathematical errors. There are other examples but that should suffice for readers to do their own homework.
Furthermore, while the report singles out Xero, Kashflow, FreeAgent, Winweb and FreshBooks it says nothing about ClearBooks financial performance. That might be because there is no entry under that name at Companies House. A commenter found the entry. I must have made a mistake when entering the search term – my bad.
I could go on but we’ll leave it there for the moment.
But it is in its analysis of Google and Alexa stats that I find the most egregious problems.
The above image comes from Alexa and was snapped today. Notice the spike in Clearbooks traffic? Now compare with what JBS discovered at around the time of their research:
Anyone notice the obvious problem? ClearBooks knew the research was underway, knew the terms under which it was being conducted and probably provided considerable assistance in framing the research. It doesn’t take a genius with knowledge of web stats more than 2 seconds to figure out that setting up a bot to drive traffic would be the obvious next step. There is no other reasonable explanation for the traffic spike. Now let’s look at what Google is saying:
The graphic is a bit difficult to see (sorry – page size constraints) but right at the top is Xero with a sponsored link. ClearBooks (which I’ve underlined) is ‘above the fold,’ which is important but unless a seeker of information is doing serious comparisons, they are unlikely to go much beyond the first 3-4 solutions they find. Even then, it is far from clear whether Google ranking alone let alone page views is any indicator of popularity or – as the report states – ‘fairing well.’ Why? Each of the players the report singles out has a login button on its home page. Therefore, it is reasonable to assume that sites where the existing numbers of user are relatively high will have a correspondingly high number of page views/visits.
Now we come to usability. JBS approach is not unreasonable. However, usability testing is a notoriously unreliable indicator of what keeps users selecting or running the software they do. I think back to the time Microsoft launched Vista and the next iteration of its Office suite. Microsoft spent many millions on usability testing in the hope they could improve the customer experience. Vista is now widely regarded as one of the worst OS releases Microsoft ever made available. The ribbon style navigation was thought to be a great productivity booster but those transitioning to the solution found it confusing.
Yesterday I was talking with a practitioner who made this telling statement: “The solution we’re using has a weird interface that takes some getting used to but functionally it beats the more whizzy competitors hands down.” Sage for all its ills and for all the gripes manages to keep people stuck to its solutions and that despite its interface is far from great. On the other hand on Xero, FreeAgent and Kashflow sites, I see linked references to customers who rave about the services. Customers speak far louder than any academic testing lab assessment. Even then though we run into difficulty.
The research talks about things that are important to customers as follows:
Preferred attributes are on the left, desired accounting software features are on the right. But that’s not what they tested. Instead, they worked on a create customer, sales invoice, credit note, mark as paid and support inquiry scenario. That’s not a bad way to look at usability but what results did they achieve?
In all the years I have been conducting and poring over research, I have never seen a radar diagram of the kind that is shown at bottom left. It is as close to ‘perfection’ as anything I have seen. If you can find one that looks similar then please let me know. The other three seem far more realistic. How could this have been achieved? Anything I say at this point is pure speculation but what I can say is that I have watched countless dummy setups that look realistic, I have seen fake tests, I have watched skewed shoot out presentations. Any crash test dummy dolly will tell you that it is surprisingly easy to blow smoke and manipulate mirrors to achieve a given result. I have seen executives pulling out cheque books based on cleverly assembled Camtasia demos. I have done it myself when entering a development competition. We concocted a fantastic presentation backed up with not a single line of code. We were short listed. Need I say more?
What can we conclude?
All research is flawed. The countless arguments I’ve had with vendors who when looking at independent research seem baffled and confused that what we find doesn’t align with their own research is enough to tell me that arguments will always arise. Even taking that into account, the fundamental flaw with vendor led research of this kind is that it is always self serving. The outcomes are so predictable the research almost becomes irrelevant. On this occasion I think we have something else going on.
- Interviewees including myself were not adequately informed.
- The researchers either had insufficient experience or guidance in understanding what they were looking at
- I suspect the researchers themselves have been led up the odd garden path.
At the time, I made mention of the fact that the researchers were looking for information on matters I consider to be either private or competitive and at a depth I would never disclose to anyone. Given they were MBA students, I am not convinced they had the experience to ask certain questions without being primed. That was enough to put my BS detectors at full alert. On the basis of what I have seen and on the cursory examination I have performed, I conclude that the findings are riddled with major problems. The bigger problem is that ClearBooks will use the kudos of an academic institute to add gravitas to something that I see as of questionable value and worth.
UPDATE: As always and as predicted, ClearBooks shoots back telling me I’m an idiot. They fail to address any of the factual errors but make it worse for themselves where they say:
Now to set accmanpro straight on a couple of points.
Go for it tiger…
“Their goal is to explore the online accounting market, its size, how quickly it is growing and to compare it with the traditional accounting software market…It turns out that was at best a misrepresentation and at worst a lie.”
All vendors were emailed by me and told, “All those who particpate will receive a copy of the finalised comparison report. And to be completely open, the students are doing a separate section specific to Clear Books which you will not be receiving, but this is not relevant to the comparison or the data you provide anyway.” In the end we decided to just published the entire thing.
I hate to sound overly cynical but this doesn’t hold water. Commenters tell me they were duped. Your own words hang you out to dry because respondents say the researchers were silent on the usability study. If the usability material was not relevant then why publish? It doesn’t make sense. What’s worse you say WE decided. In other words there was influence as to the outcomes. In light of your own words, that cannot be denied.
“It now turns out that ClearBooks is a JBS client.”
We did not pay the students – they are students afterall! They were our clients in as much as we set their objectives and that much was obvious from my original post.
I didn’t say you did pay the students and in any event I doubt you could have done so under the rules that apply in these cases. You might have paid the academic institution either by way of donation or fee but none of that is clear. In the alternative, these are stupid students – which I doubt. Either way, they SAY ClearBooks is a client so they think there is a relationship of a kind that others will discern for themselves. It seems you saw the draft report and chose not to correct this perceptual error?
“you can be sure that ClearBooks agenda figures into the equation”
Untrue. We set the objectives and let the students get on with it. We gave them access to data and customers but had no impact otherwise. It’s not in our interest and this is where a fundamental point is missed about the nature of our business. We are interested in feedback to improve the system not in any hidden agenda.
My emphasis added to their comment. In setting the objectives you are setting the ground for the methodology which we know to be flawed for the reasons I outlined. I can point to thousands of similar examples. They appear in my inbox almost daily. And…
“Furthermore, while the report singles out Xero, Kashflow, FreeAgent, Winweb and FreshBooks it says nothing about ClearBooks financial performance.”
We have been open about our performance since we launched so there is nothing afoot here. As a reminder our user numbers are here. Fair enough that Clear Books wasn’t included in the report, but I guess the students expected us to know our own numbers! As I have already mentioned, Clear Books had no input aside setting the objectives and marking a presentation at the end of the project.
My emphasis added to their comment. I was not referring to your user numbers but your financial performance. So the students didn’t have the gumption to put your figures into the mix? Why would they not do that and then compile a comparative usability report that was included? It doesn’t make sense. If as you say they were under the impression the report was for you then why say it will be made public? Another mystery. And…
“ClearBooks knew the research was underway, knew the terms under which it was being conducted and probably provided considerable assistance in framing the research. It doesn’t take a genius with knowledge of web stats more than 2 seconds to figure out that setting up a bot to drive traffic would be the obvious next step.”
I had to laugh out loud at this one! The truth of the matter is that we had a very large survey conducted which was responsible for the spike. As outlined here 37,459 people took the survey but in the end more than 50,000 did. Again, we had no input on the students’ research.
So the students were not aware of the survey impact? Was that fact deliberately withheld or omitted? Since you obviously have some understanding of SEO then one assumes that would be a pretty important part of the discussion. You now say you knew the reason for the spike but had nothing to say about the draft report findings? Surely you must have known that someone would check this out? The way you are trying to spin this, anyone would think you threw a few facts and objectives over the wall and that was about it. If so then it is the most kack handed way I’ve ever seen a vendor instruct a research group. Are you surprised that in light of what you are now saying that I am even more incredulous or confused?
