Interpreting results of your RCT and other pitfalls
By: Andres Fonseca
On: 16th September 2014
Organisation name: Virtually Free
I wrote a previous post where I go through the basic steps when designing a randomised controlled trial. There is one more very important step which is the analysis of the data. RCTs usually compare means of a variable of interest. In our own trial we compare the mean score on the Panic and Agoraphobia Scale mount of energy consumed or they look at the chances of an event happening in the control group versus the chance of it happening in the intervention group. A lot of harm can be done if the wrong statistical method is picked and this may lead to drawing completely the wrong conclusion. There are a few concepts when reporting outcomes that are important to understand.
So there is a difference between your intervention and doing nothing but, is it big enough to matter? Effect size is a measure of this.
Numbers needed to treat
Another measure of whether the difference you have made matters. Simply stated this is how many people do you need to go through your intervention for one extra person to get a favourable outcome. So if you are reducing youth unemployment this would be how many young people need to go through your programme to get one of them employed that would not have otherwise found a job.
Depending on what it is that you are trying to run a trial on you will probably need to obtain informed consent. This is actually quite a tricky process that would require its own blog post. Suffice it to say that simply asking is not enough in most cases.
Privacy and data management
There is a piece of law, the Data Protection Act 1998, that you need to comply with if you are gathering any data on anyone for an experiment like this. You will also very likely need to register with an organisation called the Information Commissioner’s Office.
If you are doing anything that is remotely health-related you will need to gain ethical approval from either the NHS or the university which sponsors your trial. This is a tricky process that takes months. I have done this many, many times and it still took me 6 months to get my RCT through ethics.
For an excellent guide on how to design an RCT here’s a 2003 paper written by J Kendall in the emergency medicine journal. It is medical as the RCT is usually used in medicine. Just ignore the medical bits, the methodology applies to almost anything where you think an intervention might bring about change.