Is John Hattie Incorrect?

I don't remember where I stumbled across this but it was about John Hattie and his work. As you known, he's written several books which contained conclusions based on synthesized data on how to improve learning. My district uses his information out of his "Visible Learning books to talk to us about what we should be doing. In fact, I set my personal goal based off of something I found in his "Visible Learning in Mathematics.

There are quite a few articles out there questioning the methodology used. One claim is that Hattie conducted a meta-meta-analysis of over 1200 meta-analysis studies that synthesized over 50,000 studies which covered all sorts of topics from interventions to pay performance teachers. These studies had sample sizes from one to several hundred, or carried out with everything from great parameters to bad. It was also claimed that any biases contained in these meta-analysis studies into his final work.

In addition, several people argued that since many of the original studies had sample sizes of one or lab studies using paid volunteers and other issues, the results may not be valid. Supposedly, Hattie synthesized all the results and applied a metric of confidence so they could utilize the date.

At least one statistician claimed the method of calculating averages and statistical deviation used by Hattie and his people shows a lack of sophistication. Many questioned whether this method of calculating averages and statistical deviation can provide enough data for accurate conclusions. It has also been said that he does not use proper baseline comparisons and has also used comparisons of other factors in correction. Furthermore, they allege that Hattie didn't really understand most of the normal methodology used which lead to incorrect conclusions.

Another person raised the issue about using a multitude of studies that do not share the same methodologies, implementations, or having students who are in the same socio-economic group. These differences can make comparing results even more problematic. Others have asked if conducting a meta-meta-analysis of meta-analysis studies is the best way to obtain information since one is synthesizing information that has already been synthesized.

I don't know if his conclusions are correct or incorrect. For me, my district follows the results published in "Visible Learning", and I have to accept that they want me to use the information. Let me know what you think about this topic. I'd love to hear. Have a great day.