On reviews

by admin on January 19, 2009

in General,Innovation

Who sucks most
Image by dahowlett via Flickr

Duane Jackson eviscerates the ICAEW report of online accounting systems:

The ICAEW (Institute of Chartered Accountants in England and Wales) have recently released a guide to online accounting for accountants.

Emily Coltman expressed surprise on her blog that KashFlow wasn’t listed.

“I realise there’s a limited amount of space in these reports, but I was surprised not to see KashFlow, a recent winner at the Software Satisfaction Awards.”

I wasn’t surprised. I had a rant about it back in September on UKBF. Essentially, we didn’t want to pay the £3,5k to have a “review” in there.

To say that Duane’s a salesperson he clearly didn’t think to negotiate. -:)

Having said that, I sympathize with those who are critical of the pay to play model of reviewing. It can never be unbiased or at least as unbiased as is reasonably possible. Duane refers to the reviews undertaken by Nigel Harris over at AccountingWeb. Yep, Nigel does a reasonable job but the thing that is always missing from these types of thing is an assessment from the end user perspective coupled to a broad appreciation of the market.

For example, Duane thinks Twinfield is ‘enterprise’ class. I don’t. That’s because my definition of enterprise is coloured by the fact I get exposure to some darned big systems. CODA is enterprise, so is Oracle, SAP, CedAR, a bunch of the Infor stuff and even Microsoft Dynamics NAV now falls into that class.

I can visit any number of web sites and get glowing user testimonials but it is only when you’ve got your hands grubby from the non-accountant (as well as the accountant’s) standpoint that you get a genuine sense of what a product or service can do for you.

If you’re going to do any objective assessment then you need a set of standard test data that you can grind through the system. It needs to be ‘real world’ so that you can effectively simulate what someone does on a day to day basis. Otherwise you’re comparing apples and bananas. That’s what we used to do wherever possible on Information Week when I was running apps testing. That meant our reports ran 10,000 words on a basket of products and not 800 or less as is usually the case in blog posts on a single product or service.

Finally, peer review is essential. We all bring biases to the table and as is often the case, one man’s meat is another man’s poison. Similarly, when you’ve got a few heads on the case then things are less likely to get forgotten. Like a certain company’s $1 billion cash balance. -:)

With all that in mind, and given that evaluation is much more broad in its requirements these days, I’m planning to undertake a series of reviews that will be pay to view. The vendors won’t get a say in what is written (hence they don’t pay to be in) though they may be asked to fact check. I have a partner in mind and we’ve discussed broad parameters and how it might work.

The idea is to take the pain out of the review process for decision takers by making evaluation as accessible and complete as possible. We’re working on a volume model which means individual low cost to the end user but can be a subscription to include updates.

I fully expect a significant amount of vendor bitching and moaning but that goes with the turf of offering independent opinions. We’ll see how it goes.

Reblog this post [with Zemanta]
Comments have been disabled for this post.
Sort: Newest | Oldest

For me, in writing reviews, one must look into both the advantages and disadvantages of a certain product or material. Like what I read about warehouse management software in a review website. They both pointed out and discussed the advantages and disadvantages of using it in your business.

Martyn,
I haven't got a problem with OSS either. It definitely has a place, but in general requires the company to have more technical expertise in house or with a consulting partner alongside to help. I just want to chip in a few comments using your numbering:

1. Commercial SaaS products will usually offer try before you buy too.

2. Any half decent commercial SaaS product will have a community of users you can tap in to as well.

3. On changing the source, that's where some risk comes in. Hopefully the product (OSS or commercial) will have some configurability and options to allow flexibility and change without getting in to the code.

Dennis

Sorry to cross post, I know its a pain.

“I guess mags like Linux format et al do this very well in the consumer space so why not in the business space.” - there’s a few reasons:

1. they don’t know what they’re looking at
2. business is boring - except for those who are in business
3. it’s not a huge volume draw for paeg view based advertising"

Yeah, exactly, which is why if somebody (like you??) provided the same sort of service but for business users it would be useful. Would've been for me.

M

Dennis

Agreed. I guess what I'm trying to say is that the reviews would "add to" a more rounded view that you could get by using the unique properties of the OS model - i.e. the ability to poke around under the hood.

Which is why I put question marks on my opening statement.

Thinking back to my own selection experience I spent a lot of time doing my own "reviews" of OS software before going it alone. So some third party input to get a start point would have been very useful.

I guess mags like Linux format et al do this very well in the consumer space so why not in the business space.

Martyn

"I guess mags like Linux format et al do this very well in the consumer space so why not in the business space." - there's a few reasons:

1. they don't know what they're looking at
2. business is boring - except for those who are in business
3. it's not a huge volume draw for paeg view based advertising

OK so where does Open Source fit here? Does it break the reviewers' business model as well as well as the vendors?

1. You can try before you "buy", so you don't have to rely on reviews. It costs nothing but time. You could even give it to users and let them play on a n extended demo.

2. Any half decent open source project will have a vibrant community (developers and users) so you can get a feel for the issues before you take the plunge.

3. Because you get the source means you can change it if your business model changes - if that is what you want to be able to do.

OK, so this may not work for the mega-enterprise but in the 10-50 user range, it may be an option (I'm now expecting some heat!!!)

Martyn

Not at all. OSS doesn't install or configure itself so there is a business model into which others can tap. Or not. But if we don't take a peek then we're hardly doing users a service.

Dennis,
You're right to quote the blurb, but as it says "typically, small and medium enterprises (SME) or small and medium businesses (SMB)"... it's just that Twinfield was designed from the outset to scale up to larger companies too. We've purposely included the COA flexibility of great products like CODA or Cedar.

David,
Agree completely that reviews should be looking at a product's long term use over time, as well as the flexibility to change with the business. From that point of view, we come from exactly the same philosophy (if you know what I mean :) ).

I love a plan! Looking forward it...

Reviews should be one element for potential clients to look at; so should doing their own testing. However, it is ESSENTIAL that they talk to long term users and ask the right questions - how easy is it to make the system fit you? How easy is it to get the system to change when your business undergoes rapid (and often unexpected) change? How much does change cost in the long term?

I know these are mantras from Agresso and CODA, but it appalls me to see so many companies getting stuck in the quicksand of big ERP systems that they have implemented only to find their business model has changed - and the system can't. At least, not without you calling in 'the experts' and shelling out vast sums of money. In these economic times in particular, companies need to make the right systems choices, and simple product reviews won't tell you the whole story.

That's part of the plan...

Heh - it all depends on your POV. But from Twinfield's own blurbs: "Twinfield users are, typically, small and medium enterprises (SME) or small and medium businesses (SMB), often using the product in conjunction with their accountant." -:)

You won't get any bitching and moaning from this vendor. We need more objective reviews from many sources just like you say.

On Duane classifying Twinfield as being Enterprise, it depends on the definition. Duane picked up on Gartner's recent comments highlighting them as the sole European SaaS vendor they expect to emerge as a worldwide player in core financial management. Similarly, IDC picked them out as the major SaaS vendor in their market report ‘Western Europe ERM Applications Competitive Analysis’. Twinfield's customer's include companies like EB, a Finnish company who have implemented Twinfield as a (multi-lingual) financial and consolidation solution in over 20 countries around the World - that sounds pretty Enterprisey to me.

I look forward to your sequence of reviews.

Previous post:

Next post: