Representative Sample

If we are studying problems in the brain which are not due simply to the makeup of individual neurons, but rather due to the way those neurons are interacting (which one would expect to be the case with anything remotely psychological), then it is the configuration of neurons that is pertinent, and thus a representative sample needs to be representative of the possible configurations. And that is a problem: a representative sample of a mind-bogglingly large number of possibilities naturally also needs to be of a mind-bogglingly large size, way more than there are people on the planet, let alone people with a particular psychiatric diagnosis.

So your sample isn't representative of the population, and isn't representative of all people with the same diagnosis, so what is it representative of? One can only answer this by reversing it: it is representative of the subset of possibilities that it is representative of, and this subset is a very small subset of the total space of possibilities. If you try something out on 100 people exhibiting symptoms of mania, then the 101st person you meet exhibiting symptoms of mania may be with the set that the 100 people are representative of, or they may not, and you have no easy way to tell.

Thought Experiments

Efficacy of Musical Instruments

Suppose we wished to measure the efficacy of playing a musical instrument for improving depression. So we gather together a group of people suffering with depression. We randomise them into three groups: one is given an electronic keyboard, one is given a guitar, and the other is given nothing. We then give the experiment subjects a multiple choice survey to assess their depression once every 4 weeks. The trial lasts 6 months.

Note: We do not control for anything like what musical background a person has, whether or not they can play the instrument they were given, what sort of music they listen to, if any, and so on.

What sort of results will we get? And what value are those results? (When it comes to trying to apply RCT's to things like Yoga, Tai Chi, Mindfulness or Meditation, I see this sort of problem as inevitable.)

What If Sequence Matters?

Suppose we have some diagnostic label, such as 'mania with psychotic symptoms'. We collect together a group of people with this diagnostic label. (We have no information beyond this diagnostic label.)

Suppose we have 5 treatments we could use, A,B,C,D,E. Suppose that only A,B,C are the subjects of the trial. Suppose that order matters, in the sense that for some test subjects, the only effective treatment is A followed by B followed by C, and that the treatments in any other order will be no more effective than no treatment. (We can not safely assume that 'placebo' = 'no treatment', as a 'placebo effect' may play a role in outcomes.)

Suppose that the test subjects are grouped into four groups, one for each treatment, A, B, and C, and a placebo group Z. Let us consider only those subjects for which the only effective treatment is A followed by B followed by C. It is easy to see that no matter which group they are assigned to, the outcome will be the same as the placebo, as each patient will receive only one of the three treatments, rather than the sequence of A followed by B followed by C. So the objectively correct treatment (so far as this thought experiment is concerned, that being A followed by B followed by C, which I'll abbreviate to A→B→C for now) will appear no more effective than a placebo.

As such, if effective treatments involve anything complicated like the order in which treatments are applied, then a simple randomised controlled trial giving each group a single treatment will not pick such things up.

Patient-Specific Outcomes

Let us now consider all test subjects, and assume that some will show benefit from a single treatment out of A, B, or C, but not others. So a patient requiring treatment B, if given treatment A, will show no more benefit than a placebo.

In this case, a patient requiring A will have a 1/4 chance of receiving treatment A. What happens if such a patient is randomised to a group other than that receiving A? What if such a patient is randomised to a group receiving A?

While this is unlikely if randomisation is truly random, consider that if no patient for which A is the only effective treatment is randomised to group A, then A will appear to be ineffective.

The trouble here is the assumption that, e.g. mania=mania, in the sense that two patients diagnosed with mania have the same underlying problem (and not just outwardly similar symptoms). It is assumed that if, say, B is more effective, then those patients exhibiting e.g. mania, will do better if randomised to group B. Quite possibly statistics will detect a significant correlation with better outcomes from group B. But in reality, any patient for whom A works but B does not, will get no benefit from treatment B, despite the 'science' showing that B is more effective.

Fire Analogy

To see the inspiration for this line of thinking, in terms of a more everyday metaphor, consider fire. (I've said similar to this elsewhere, but I'll repeat it here.) Different fires need to be treated in different ways. That's why there are different kinds of fire extinguisher. Putting water on a chip-pan fire won't put it out, for example.

Fixing a Car Analogy

Now consider a mechanic fixing a car. Various steps may need to be done in a specific order (in engineering scenarios this is the usual case). If we take the individual steps in isolation, and try to test them in isolation, we won't get sensible results.