Sunday, September 28, 2008

The Drunkard's Walk...How Randomness Rules Our Lives..



My daughter, Raena (who recently graduated with a degree in Mathematical Science from Edinburgh), on noting that I have read and blogged on the book "Fooled by Randomness" a few months ago, bought me this book in Scotland when I went there for her university graduation in July.





The author , Leonard Mlodinow Ph. D , is a faculty member at the California Institute of Technology (Caltech).





I read this book during the 12-hour flight home from Heathrow to Kuala Lumpur and read it again recently and found it to still be equally interesting.





.

The greatest challenge in understanding the role of randomness in life is that although the basic principles of 'randomness' arise from everyday logic - but many of the consequences that follow these principles ( of randomness) prove counter-intuitive.

As an example -in behaviour modification study - the conventional wisdom is that a positive reward will reinforce a positive behaviour and punishment of a mistake does not improve the behaviour.In psychology labs, experiments with animals prove that rewards work better than punishment.

But in observation of trainee pilots of the Israeli Air Force, the instructors always reported that everytime a pilot was praised for a good landing, they did worse the next time. On the other hand the pilot who did poorly and was being scolded always improved the next time. So the instructors observation and conclusion is a challenge to the basic theory of behaviour modification theory.

Daniel Kahneman is a behavioural psychologist and a junior professor at the Hebrew University. He agreed that the instructors observations were correct and went about to understand and explain this apparent paradox. He then realised that it was true that the scolding did preceed the improvement, but it did not cause it!

The answer lies in the phenomenon called 'regression towards the mean'. That is, in a series of random events, an extraordinary event (like a good near perfect landing) is most likely to be followed, due purely to chance, by a more ordinary one.

Here is how it works. Each pilot has a certain personal ability to fly the fighter plane. Their skill will slowly improve through training but the change would not be noticeable from one maneuver to another. Any exceptionally good or bad performance was purely a matter of luck. If a pilot did an exceptionally good landing today, one far above his normal level of performance, then the chances are high that he will perform closer to his norm ( which is worse) the next day. And if his instructor had praised him for the good landing, it would appear that the praise had done no good to him. Whereas if the pilot had done an exceptionally bad landing, the chances are great that he will perform closer to his norm ( which in this case will be better) the next day. So if the instructor were to scold him for the bad landing, it would appear as if the scolding had done him good.

The instructors had made the conclusion that the scolding (punishment) was a powerful educational tool when in reality it made no difference at all.This error in intuition spurred Kahneman's thinking. Are such misconceptions universal? Kahneman's work and research in this area of 'randomness' won him the Nobel Prize in Economics in 2002.

This book went about introducing theories and concepts in Mathematics and Statistics through a historical narration of the lives and works of important contributors of these fields of science. The life or Cardano,the doctor from Milan who became a mathematician who wrote the first book on the "Game of Chance" and the lives and works of Galileo, Pascal, Bernoulli, Bayes and many others made very interesting reading.




This book would be of interest to doctors (like me) to understand how our interpretation of false positive or false negative measurement of tests or treatment outcomes can be flawed due to our poor understanding of statistical concepts.





Similarly, lawyers would find it interesting in the author's discussion of the "prosecutor's fallacy" and how the Harvard Professor of Law had used this statistical (flaw) to argue and convince the jury to favour O. J. Simpson in the infamous murder case!





How I wish my statistics teacher at university during my undergraduate and postgraduate years had taught me this subject in such an interesting way...




1 comment:

wannabequant said...

Daniel Kahneman and Amos Tversky seminal paper "Prospect Theory" on how a person's risk appetite changes in the face of risk or loss is a must read for any student of behaviourial finance.

Their findings rocked the classical assumption of the reasonable agent who seeks to maximise profits and/or minimise risk.

"One very important result of Kahneman and Tversky work is demonstrating that people's attitudes toward risks concerning gains may be quite different from their attitudes toward risks concerning losses. For example, when given a choice between getting $1000 with certainty or having a 50% chance of getting $2500 they may well choose the certain $1000 in preference to the uncertain chance of getting $2500 even though the mathematical expectation of the uncertain option is $1250. This is a perfectly reasonable attitude that is described as risk-aversion. But Kahneman and Tversky found that the same people when confronted with a certain loss of $1000 versus a 50% chance of no loss or a $2500 loss do often choose the risky alternative. This is called risk-seeking behavior. This is not necessarily irrational but it is important for analysts to recognize the asymmetry of human choices." http://www.sjsu.edu/faculty/watkins/prospect.htm accessed 14th Oct 2008

For a most enjoyable read of risk and an insight into "prospect Theory", I would highly recommend Peter Bernstein's "Against the Gods"...a most enlightening and good read, indeed.

Source:

Peter Bernstein, Against the Gods: The Remarkable Story of Risk, John Wiley & Sons, New York, 1996.