I am so sorry to do this.
A lot of people never really understood Webs of Substance. For a start, I picked a bad title. It was based upon a quote from Francis Bacon but, stripped of this context, it sounds pretentious.
And everyone assumed that I was out to convince; that I was making a case in order to persuade. It wasn’t that at all. It gave me the freedom to talk into the void. That’s why it was anonymous; freedom. As it grew, I saw it as a resource for those people who already agreed with me: Do you think knowledge is important too? Great. Here are some useful links.
I know that some of you did find it useful because you said so. Thank you. It lives on as an archive in a beautiful walled garden. If you have a genuine use for it then ask me for the key. But I might say no. Don’t take that personally.
If you are an anonymous blogger then have an exit plan; a better one than mine. I have heard horror stories about bloggers being exposed. It’s weird. People thought my anonymity was about them. They didn’t stop to wonder why I would make that choice. Instead, it was an insult or a slur or the sign of something shady.
Anyway, Goodbye. It’s been a pleasure. Genuinely. But all good things must come to an end. This post, too, will disappear soon. Such is life.
Harry Webb is dead.
Effect sizes from uncontrolled studies have all sorts of confounding factors. For instance, keen teachers often sign up to deliver an intervention whereas the less enthusiastic ones are left in the control. Teachers in the intervention group also know that they are part of an intervention and often so do their students. This has a potential placebo effect where positive expectations about an intervention become a self-fulfilling prophecy. It is for this reason that Hattie chooses a cut-off of d=0.4 for effect sizes. However, this number is quite arbitrary and he applies it equally to both well-controlled trials – such as Sweller’s trials of worked examples – and the standard kind that I have described. He even applies it equally to time based effects (comparing before an intervention with afterwards) and group based effects (comparing an intervention group with a control).
There are other problems, such as the fact that when the test subjects are quite homogeneous you are likely to generate larger effect sizes. So, if you are testing in a selective school or a group of engineering undergraduates, your standard deviation is likely to be small relative to the improvement in mean scores. Given that the effect size is the latter divided by the former then you are going to get a big one.
Unlike some, I don’t think that this renders effect sizes completely useless – they are a sincere attempt to enable effects to be compared across study designs – but we do need to bear in mind their limitations.