Surveys show 9 out of 10 surveys can't be trusted

From the very start of Virtually Free we have been determined to try to get feedback and also research whether our apps are effective. We have spoken in previous blogs about how we would want to do this and also what constitutes good quality research. 

We have started as we mean to go on and have had an abstract accepted for the Royal College of Psychiatrists International Congress. The abstract was a survey of how acceptable people found the Stress Free app. We felt this was a good place to start because it doesn't matter how effective the app is if no-one ever uses it.  

The results showed that 83% of people found the app useful and 70% intended to continue using it. We also got some good specific feedback on the user interface and how the sessions are presented.

The pitfalls of surveys

We are often presented with this kind of information in adverts and newspapers and you should be very cautious about just accepting it; without any information about how the survey was done it is meaningless.

When looking at a survey there are two major sources of error:

1) Asking the wrong people (sample errors)

2) Asking the wrong questions (invalidity)

The idea behind a survey is that it is impractical to ask everyone what they think of a president, a shampoo or an app so you ask a much smaller number who are representative of the larger target group i.e. the same age ranges, gender mix etc. This is easier said than done and there have been some major gaffs. Often quoted is the 1948 US Presidential Election in which all the major polling organisations predicted the wrong candidate to win. One of the main causes of this was they failed to get representative samples. 

President Truman winning and the the Editor of the Chicago Tribune getting a new job!

Photo by Byron Rollins, Associated Press

Sometimes in surveys people can be dishonest rather than mistaken. For instance you ask people to use your product for six weeks and then survey how satisfied those who use it for the stipulated time are. Well, surprise, surprise the people who keep using a product mostly like it. The fact that many stopped using it and aren't included is conveniently left out.

Asking the wrong question

Developing questions can have the same problems; either your make a mistake and people misunderstand the question or you ask the question in such a way as to get the answer you want.

For example:

Q1 The government should force you to pay higher taxes

Q2 The government should raise taxes to pay for better services

Q3 We should all contribute more in taxes to provide care for those in need in our society

Although all the questions basically ask if people would pay more taxes it is easy to see more will answer Yes to Q3.

Finally there can be another problem when people are unwilling to give an honest answer. As a doctor the classic is asking people about their smoking and drinking habits as they almost always underestimate.

Now you've had survey methodology 101 in my next post I'll take you through the positives and the limitations of ours.

In the mean time we'll post the abstract on the site so you can check it out for yourself.