I have been reviewing some work this week and found the authors promoting a concept known as the Hawthorne Effect (of which more later). I read an article in the Economist in 2009 which pulled the rug from the whole concept of the Hawthorne Effect and underlined the danger of extrapolating a generic concept from a single instance.
Doing this is known as induction, and was (until the discovery of Australia) the method proposed for scientific discovery in the civilised world (i.e. western Europe at the time). The idea is simple, you take an instance and suggest a theory that explains your instance; then every occurrence of another instance reinforces your theory. The favourite example was ‘all swans are white’. I suspect you can see the link to Australia coming!
Every white swan confirmed the theory and all was well with the world. Then explorers returned from Australia with their iconic black swan. Just one contrary occurrence and the white swan theory is dead! And in this case the use of induction as a scientific method.
Yet, the psychological world built a generic concept on a single instance, and didn’t even try to repeat the experiment to test the results. The Hawthorne effect is based on experiments in the 1920’s to determine the effect of lighting on factory workers productivity. The experiment was carried out at a factory in Hawthorne near Chicago. It seems that no matter what the experimenters did to the lighting (including reducing the lighting) they found that productivity rose. They came to the conclusion that it was the attention paid to the workers during the experiments and that having something interesting and different happening raised the productivity (given factory conditions at this time you can understand how this might happen). This ‘effect’ has been used to argue that the results from change pilots, or tests, are unreliable indicators of long term performance improvements. So why bother with pilots, just get on with rolling out the change the first way you thought of.
Well, the Economist article reported that two researchers had found the lost data and records from the original experiment and set about doing some modern econometric analysis. And they found that there was no actual performance change attributable to the lighting. What they did find was a very simple experimental design error which was not allowed for. The experimenters always took baseline performance measures before the change on Saturday morning (yes people worked Saturdays then); changed the lighting on Sunday when the factory was closed and measured again on Monday and Tuesday. So what they really measured was the state of employees at the end of the working week against the start of a new working week after a day off. What do you expect to happen? The researchers found the data clearly showed a consistent difference in performance at the factory between Saturday and Monday/Tuesday. And this was all that the experiment confirmed. So the Hawthorne Effect is very suspect if not not a myth.
Which means that well designed (note the reference back to the Hawthorne experiment design) pilot is still a very useful tool for helping to test and understand how to carry out change in an organisation.