Tuesday 23 August 2011

Popular inaccuracies: A review of Malcom Gladwell’s “The Tipping Point”

Malcom Gladwell’s book The Tipping Point has itself become something of a fad. It is hugely popular and this might lead one to believe that the author really has discovered the social laws that explains why some things catch on, while other books – or diseases, or rumours, or habits – seem to stumble before they get off the ground. Is this true?

Despite the popularity of the book I remain sceptical of much of its content. First of all, the arguments are far too anecdotal to be convincing. Anecdotes makes for easy and entertaining reading. However, precisely because it is so easy to remember anecdotes they can convince us when we should be sceptical. Second, one of the major messages is at best an oversimplification and at worst very misleading. It is simply not true that it is necessary to have some highly connected individuals to get an epidemic going. Third and finally, at some points – such as the importance of zero-tolerance in fighting crime - detailed investigations reveals more doubts than the enthusiastic picture that is painted in the book.


A brief summary

How do epidemics – social and real – get going? The answer, Gladwell claims, is three principles. First, he assigns a crucial role to some individuals. The most important individual would be someone who are well-connected (have many friends, sexual partners and so on), are knowledgeable, and those who are able to sell the message. Second, the thing that is to be spread has to have “stickiness.” If it is information, it has to be easily remembered. Finally, the context can sometimes be such that an epidemic spreads easily, for instance by encouraging certain kinds of behaviors.

Obviously there is some truth to these arguments. Gladwell’s main point, however, is simply not just that these are important principles- He also argues that the theory unite a lot of rather diverse phenomena and that it shows how small things can have large consequences. The spread of rumours, HIV, crime, syphilis, smoking and lots of others phenomena are used to illustrate the importance of the three principles and in all cases he argues that small changes can have large effects. For instance, small things making a message glue itself easier to the brain can make a a large difference to the sale of a product.

How are we convinced?

To convince the reader of his message, Gladwell gives us illustrative examples. He tells us about the homosexual flight-attendant who supposedly had 2500 partners, about a successful businessman who had an extremely large circle of friends, and how the spread syphilis in a city could be traced back to a small group of individuals. To be fair, Gladwell also cites scientific research. The reader is told about a wide range of experiments and results. However, the research is used selectively. He uses articles that support his views without pausing to mention potential weak spots or rival views. In sum, the combination of anecdotes and scientific references can be quite convincing at first, but on closer inspection they tend to reveal several weaknesses. A good example includes the role he assigns to “special” individuals.


The role of special individuals

The flight attendant with many sexual partners is used to convince us that some extremely well-connected individuals are crucial to the spread of diseases. It is clearly true that such individuals could spread a disease rapidly. Still, we must ask whether the existence of these individuals really are necessary and important for the quick spread of a disease. It turns out that they are not. To understand this, think first about a community in which everybody knows their neighbours and nobody else. In this imagined community everybody has the same number of friends and there are no special individuals in the sense of a person who is extremely well connected. In such a community it would take some time for a disease to spread from one random person to another because it would have to go through the whole chain of neighbours.

Now imagine that we create some friendships across the community so that our inhabitants meet some people in addition to the neighbours. There might be some differences in the number of friends each individual has, but this is not important. The important point is simply that we allow some more or less random connections between people other than to their neighbours. It turns out that in such a community a disease or a rumour can spread very quickly (the small-world property). Note here that the key is not that there are some people who are extremely well connected. It is the fact that there are some cross-cutting connections that is important for the speed by which something spreads. Reading the Gladwell’s book one gets the opposite feeling: That some individuals are extremely important. This is misleading in an important way. It makes us give more prominence to individuals than we should and less to the structure of the system.

The devil is in the details

The spread of crime can be viewed as an epidemic and the question then becomes what it is that is driving the epidemic. In this book Gladwell seems to favour the so called zero-tolerance or “broken-glass” argument and he cites researchers who have argued for this. However, while it is intuitive that our surroundings affect us and out behaviour, closer investigation reveals that there are weaknesses in the empirical evidence connecting the drop in crime with the introduction of zero-tolerance policies. In several places the drop appears to have started before the policy was investigated. Also, while Gladwell may be excused for not knowing about Lewitt’s work on the potential importance of abortion in explaining this drop, he must have known that there are many alternative possibilities that we should explore before simply accepting the zero-tolerance argument just because it fits the argument so well (that small things can have big effects).

In sum

One might complain that it is unfair to use such strict criteria for a popular science book. I sympathize with this. One clearly cannot write a book with lots of statistics and expect to sell a lot of books in airports. One could also argue that at least the book stimulates the mind and may serve as an appetizer to more rigorous arguments. Still, there are lots of popular science books that manage to be critical and popular without compromising too much on substance. Simon Singh’s book Fermat’s Last Theorem is an excellent example of this. Although not a bad book, I think The Tipping Point ends up compromising too much on substance.

Thursday 4 August 2011

Convert text to numbers in excel

Excel is nice (much better than word), but sometimes it drives me mad. Like today. I found a table with numbers on health expenses from 1946 until 2006 on a web page and wanted to use the numbers in some calculations. Copied the whole thing into excel. Everything works fine, except excel believes the numbers represent a text string. This is a common problem because the numbers had an initial space and a space as a thousand-delimiter. Now, there are some solutions (trim, clean and so on), but they did not work in this case. And just to make matters worse it is not possible in excel to search for a blank spaces and replace them with nothing. Annoying! Solution? Well copy the table to word, search and replace all spaces with nothing, copy the table back to excel. Voila!

Thursday 16 December 2010

Addiction, free will and wanting not to want something?

In a paper on addiction and freedom of the will, Olav Gjelsvik makes some useful distinctions and clarifications. The basic question is whether addictive behaviour should be viewed as the outcome of a free choice. Olav first distinguishes between freedom of action and freedom of the will and argues the addicts act intentionally and have freedom of action, but that they typically lack freedom of the will because they do not want to have the desire to take drugs.

After reading the paper, I have two comments. First, Jon Elster has argued that the very notion of "wanting not to want x" is incoherent. If this is true, Olav's argument breaks down. However, I am not sure that Elser is correct. It seems common at least to experience feelings like "I wish I did not have such a strong desire to do x." Now, it may be conceptually incoherent to think this and still do x at the same point in time, but across time it may be more coherent.

Second, when defining free will we need a criterion for what a good definition is. The criterion, in turn, depends on the purpose of the term. What is the purpose of putting the label "freely willed and chosen" on some actions (or patterns of action) and not on others? I think there are at lest two major reasons. First, when we use the label we also imply that the agent is responsible and that the action has welfare-relevant information. The purpose of the term cannot be divorced from the intuition behind these concepts. The criterion for whether the definition is good, is whether it puts actions into categories which conforms to our reflective intuitions about whether the person should be held responsible.

Friday 12 June 2009

Heroin only, no tobacco smoking

In Norway addicts can inject heroin in special government sponsored rooms. I am not objecting to this, but I do find it slightly ironic that they are not allowed to smoke tobacco there.

Thursday 11 June 2009

Absurd tax to increase demand?

Here is a rant about something that seems bizarre: In Norway we sometimes watch commercials about the benefits of eating meat. It turns out that these commercials are financed by a tax on farm products. This is regulated by a law from 1936 in which the government promised to help the industry by increasing demand. The result was a tax on meat which finances advertisements for meat consumption. The tax amounts to about USD 2 per capita every year. This sounds a bit absurd. There is no lack of desire to buy meat. Could such a tax be justified?

Wednesday 3 June 2009

Rational self-torturer?

W. Quinn has argued that there is something wrong with rational choice theory since it implies that a person given the choice between a negelectible increase in pain (too small to be perceptible) and a reward, will continue to select the reward until she ends up in excruciating pain. Chrisoula Andreou has written several papers on this and here are my comments on her paper.

Tuesday 3 February 2009

A defense of classical statistics against the bayesians

by Hans Olav Melberg

Introduction
There are lots of attacks on classical statistics, and only a few brave souls seem wiling to defend it philosophically. So, although my sympathies are Bayesian, I would like try to present a couple of arguments against Bayesian statistics and in favour of the classical approach.

The main argument is practical. When I read an article I am simply not very interested in the priors of the author or any other experts. All I need to update my beliefs and my priors, is the result of the experiment. In addition to this practical defence of the clasical approach, there are also some conceptual problems with the Bayesian method such as the problem of how to update on beliefs with probability one or zero.

Brief definitions
A Bayesian interprets probabilities as subjective beliefs and updates her beliefs using Bayes theorem when she receives new evidence. A classical statistician interprets probabilities as a measure of relative frequency and has little use for Bayes' theorem since they believe the priors are subjective and therefore they should not be used as inputs in order to arrive at a objective and scientific conclusion.

In practice
To examine the practical differences between the two approaches, consider how you would arrive at a probability about the effect of a new drug. The Bayesians will start by forming a prior belief about the effect of the drug, then conduct an experiment, then use the results from the experiment in order to update the prior beliefs and arrive at the final estimate of the effect of the new drug (the posterior distribution). The classical approach is to go straight to the experiment and present the results from this experiments without forming or updating on prior probabilities.

Now, philosophically I have little doubt that the Bayesians are right that probabilities should be interpreted as subjective beliefs. However, this does not mean that I am very interested in the prior beliefs of the author about the consequences of the drug and their probabilities. I have my own priors and all I need from the article is the result from the experiment in order to update my own beliefs. At a practical level therefore, all I want is the information contained in the classical approach.

One might argue that this is a bit too harsh and that it would be interesting to know the priors of a group of experts. I agree! However, although presenting expert opinions is an advantage, I still do not think this should be aggregated to a prior belief and used as an input in an updating process before presenting the final verdict. Why not? Because there is no such thing as a "group subjective belief." Only individuals have beliefs and there are lots of aggregation problems and psychological biases in a group process leading to the "expert consensus." So while I would like to be informed of these beliefs in an article, I do not want the author to use an expert prior, or his interpretation of what the expert consensus is, and update on this. Based on the information available and my reading of the opinions I have to form my own opinion of the initial priors and all I really need and want from a paper is the classical results: the relative frequencies.

What about beliefs with probability one or zero?
Let's say that you make a mistake and that you falsely believe that something is impossible. An experiment then shows that it is possible. Clearly you should change your mind! The problem is that it is impossible to do so if you are a true Bayesian. You simply cannot update beliefs which are assigned probabilities that are zero or one. They stay likie that forever regardless of the evidence.

This is clearly absurd and I guess that most Bayesians would like to shout at me that this simply means that you should not assign something a probability of zero or one. So instead of being a problem, they turn it into a partial virtue: As Bayesians we are careful not to bee too sure because we are aware of the foolishness that would result from being too certain. And the argument that Bayes therem does not work for zeros and one is simply a demonstration if this foolishness.

Although I am attracted to this argument, there are still some problems. First of all, to put it in general terms: What if I happen to be a fool at some point about some issue and assign it a probability of zero or one, I would to have a system that allowed me to recover and not be stuck! To point out that it was stupid of me to do something like that in the first place, does not eliminate the problem that it is impossible to get out. True, I should not have done so in the first place, but what if! That is the problem, and it is a problem the Bayesians cannot escape. (At least this is what I believe with a certainty of about 0.87)

Let me add a second argument that makes thiungs even worse. Stating that you should never assign something a probability of zero or one (unless it is an analytic truth, like a mathematical theorem), is easy to do if the list of possible outtcomes is well defined. However, what if the state space is infinite or there are some states you simply have not thought of? In theory this is equivalent to assigning a alternative a zero probability and in practice this is a much easier mistake than explicitly assigning something a probability of zero or one. We may be fools for being absolutely certain about an outcome we have heard about, but it seems much more common and less foolish simply to fail to list all the possible and relevant outcomes. Logically the two are euivalent for Bayesians. They must assume a complete description of the relevant state-space. This, I think, makes it more difficult to dismiss the zero/one argument as foolish. Instead it becomes a conceptual problem for the Bayesians: How to do statistics in a world in which we often fail to list all the possible outcomes?

Conclusion
I am a Bayesian in the sense that I think beliefs are subjective. However, I am most interested in the result of experiments when they are untainted by other people's subjective beliefs. In this way it becmes easier for me to update my own beliefs using my own priors. So in this sense I like the Bayesian philosophy, but I still like articles to have a classical approach. However, even the Bayesian philosophy givens me some worries sine it seems to lead to the absurd conclusion that we shouold maintain beliefs that are obviously false given that we at some point made a foolish mistake or, slightly less foolish, made a mistake when imagening the possible outomes.

Literature
There is a ton of literature on this topic, mostly by Bayesians pounding on the Classical approach. A philosophical discussion from both perspectives can be found in "Bayes or Bust." An interesting and more formal attack on Bayesianism is "The Limitation of Bayesianism" by Pei Wang.

Popular inaccuracies: A review of Malcom Gladwell’s “The Tipping Point”

Malcom Gladwell’s book The Tipping Point has itself become something of a fad. It is hugely popular and this might lead one to believe that the author really has discovered the social laws that explains why some things catch on, while other books – or diseases, or rumours, or habits – seem to stumble before they get off the ground. Is this true?


Despite the popularity of the book I remain sceptical of much of its content. First of all, the arguments are far too anecdotal to be convincing. Anecdotes makes for easy and entertaining reading. However, precisely because it is so easy to remember anecdotes they can convince us when we should be sceptical. Second, one of the major messages is at best an oversimplification and at worst very misleading. It is simply not true that it is necessary to have some highly connected individuals to get an epidemic going. Third and finally, at some points – such as the importance of zero-tolerance in fighting crime - detailed investigations reveals more doubts than the enthusiastic picture that is painted in the book.

A brief summary

How do epidemics – social and real – get going? The answer, Gladwell claims, is three principles. First, he assigns a crucial role to some individuals. The most important individual would be someone who are well-connected (have many friends, sexual partners and so on), are knowledgeable, and those who are able to sell the message. Second, the thing that is to be spread has to have “stickiness.” If it is information, it has to be easily remembered. Finally, the context can sometimes be such that an epidemic spreads easily, for instance by encouraging certain kinds of behaviors.

Obviously there is some truth to these arguments. Gladwell’s main point, however, is simply not just that these are important principles- He also argues that the unite a lot of rather diverse phenomena and that they show how small things can have large consequences. The spread of rumours, HIV, crime, syphilis, smoking and lots of others phenomena are used to illustrate the importance of the three principles and in all cases he argues that small changes can have large effects. For instance, small things making a message glue itself easier to the brain can make a a large difference to the sale of a product.

How are we convinced?

To convince the reader of his message, Gladwell gives us illustrative examples. He tells us about the homosexual flight-attendant who supposedly had 2500 partners, about a successful businessman who had an extremely large circle of friends, and how the spread syphilis in a city could be traced back to a small group of individuals. To be fair, Gladwell also cites scientific research. The reader is told about a wide range of experiments and results. However, the research is used selectively. He uses articles that support his views without pausing to mention potential weak spots or rival views. In sum, the combination of anecdotes and scientific references can be quite convincing at first, but on closer inspection they tend to reveal several weaknesses. A good example includes the role he assigns to “special” individuals.

The role of special individuals

The flight attendant with many sexual partners is used to convince us that some extremely well-connected individuals are crucial to the spread of diseases. It is clearly true that such individuals could spread a disease rapidly. Still, we must ask whether the existence of these individuals really are necessary and important for the quick spread of a disease. It turns out that they are not. To understand this, think first about a community in which everybody knows their neighbours and nobody else. In this imagined community everybody has the same number of friends and there are no special individuals in the sense of a person who is extremely well connected. In such a community it would take some time for a disease to spread from one random person to another because it would have to go through the whole chain of neighbours.

Now imagine that we create some friendships across the community so that our inhabitants meet some people in addition to the neighbours. There might be some differences in the number of friends each individual has, but this is not important. The important point is simply that we allow some more or less random connections between people other than to their neighbours. It turns out that in such a community a disease or a rumour can spread very quickly (the small-world property). Note here that the key is not that there are some people who are extremely well connected. It is the fact that there are some cross-cutting connections that is important for the speed by which something spreads. Reading the Gladwell’s book one gets the opposite feeling: That some individuals are extremely important. This is misleading in an important way. It makes us give more prominence to individuals than we should and less to the structure of the system.

The devil is in the details

The spread of crime can be viewed as an epidemic and the question then becomes what it is that is driving the epidemic. In this book Gladwell seems to favour the so called zero-tolerance or “broken-glass” argument and he cites researchers who have argued for this. However, while it is intuitive that our surroundings affect us and out behaviour, closer investigation reveals that there are weaknesses in the empirical evidence connecting the drop in crime with the introduction of zero-tolerance policies. In several places the drop appears to have started before the policy was investigated. Also, while Gladwell may be excused for not knowing about Lewitt’s work on the potential importance of abortion in explaining this drop, he must have known that there are many alternative possibilities that we should explore before simply accepting the zero-tolerance argument just because it fits the argument so well (that small things can have big effects).

In sum

One might complain that it is unfair to use such strict criteria for a popular science book. I sympathize with this. One clearly cannot write a book with lots of statistics and expect to sell a lot of books in airports. One could also argue that at least the book stimulates the thought and may serve as an appetizer to more rigorous arguments. Still, there are lots of popular science books that manage to be critical and popular without compromising too much on substance. Simon Singh’s book Fermat’s Last Theorem is an excellent example of this. Although not a bad book, I think The Tipping Point ends up compromising too much on substance.

Thursday 11 December 2008

Tipping in Tanzania

I just came back from Tanzania. The tour made me think about lots of issues, one of which was the practice of tipping:

1. Should I tip based on what I think it is worth or should I follow the local norm?

Here is an example: I once quarreled with two people who carried my luggage (3 pieces, 20 kg each) and helped me get on board a ferry. It took something like 15 minutes, their help was important because it was a lot of hassle - lots of people pushing and a confusing loading system. The daily minimum salary is around USD 3, but there are substantial variations. My safari guide earned about $100 a month in basic salary + at least another $100 in tips. My driver had told me not give more than $2. After finishing, they demanded $10. I protested, arguing that the driver had told me not to pay more than $2, but that I was willing to pay $4. They were not happy. We finally agreed on $5, but they were still not happy. However at this point I said: "Look. I am not stupid. $5 is quite a lot of money here and it is a lot more than you usually get for this job." They were stil not happy, but eventuely they left.Their unhappiness might, of course, simply have been an act to make me give more, but that is not the point here. The interesting issue is why I refused?

Thinking about this now, I am a bit surprised at myself. Giving a few dollars more would not have been a problem for me. And had it been in the US I would have done so since the standard there is different. The reason I refused to give more was simply that I did not want to give too much more than the local norm. Why not? I am not sure. Maybe it was because I do not want to appear to be stupid, that I do not want to feel like I was being exploited (!), fooled or taken advantage of. But then again, its a little money for me and a lot for them, why not just give it? I still do not know what the ethically right course of action would be. Pay what I think it is worth based on my rich world standard or follow the local norm?


2. Is is bad to tip? The economic consequences of tipping

By tipping I clearly make some people a little less poor and presumably a little more happy. This sounds good. On the other hand, in a poor country it sometimes has some strange effects. First of all it may imply that lots of people hang around harbours and airports and other places with tourists, doing nothing for a long time because in these places they occasionally get quite a substantial sum. On average this pays more than steady work, so we'll get a lot of people allocated to jobs that could be done by a few. Individually rational, but a collective waste of resources.

Moreover, some talented people who speak well English may well end up doing this kind of work since it pays better than other more worthwhile jobs - like teaching (Lets not make this too complicated: It is clearly a value judgement, I simply think that is is a bit sad when a system encourages talented teachers to hang around airports to carry luggage for rich white peole once or twice a day). Or good cooks become waiters because they earn much more as waiters. Now, there is little to do about this in a free economy (an economy which I like!), but it seems to create a paradox. Tipping encourages an economic structure and allocation of resources that may not be good. Maybe I should tip less? At least I should discard the guilt giving, redistributive motivational aspect of tipping and and just give more to the local school instead?

In any case, much tipping, I believe, is not really driven by a desire to reward good service, but by a norm. people may sometime give slightly more or less, but these variarions are often small comparet to the sum you usually give regardless of the quality. I also tend to dislike tipping because it solidifies and symbolizes a power inequality: Me as the important rich person who gives an inferior poor person a few crumbles of my wealth. Yes, I know this is a left wing argument, but I do not care. I think there is something to it regardless of the label. I would much rather simply pay a slightly higher bill at the hotel or the restaurant.


3. Tipping as a signal, but signals should be send before, not after the event!

Why did my guide insist that I only tip after I have received the service? Yes, I know that this keeps people on their toes until the end, eager to provide good service all the way until you leave in the hope of getting a good tip? For one time events like going to a restaurant, this may make some sense. But what about when you stay in a hotel? Would it not be more optimal to first signal that you are a person who will give a large tip if you get good service by tipping generously initially if you receive good service? You could tip like this every day, but i guess the rational strategy would be to tip a lot initially, or punsh a lot for bad service, and then announce that you will give all the rest of the tip the last day and thenthe shamelessly rational economists just leaves without giving more in tip.

Or even betterl: People can free-rided on the practice of tipping. It seems to me that this is rational and that tipping itself is similar to voting. There is one significant difference because tipping it is a face-to-face interaction, but in a hotel in Tanzania it is easy to get around this. Just announce that you will use the "tip box" for tipping while leaving without doing this. Then there is no social sanction and the no-voting rational man has also become the no-tipping man. In practice, of course, we tip and vote. it is a norm, as well as a way of reaffirming ideas we have about ourselves and the persons we want to be. Expressive rationality!

In the end
Although tipping quite generously, mostly out of a selfish desire to appear to be a nice person as well as a guilt driven rich white man, I did refuse once. When leaving, at the airport. Going through the security check with detectors and everything, one official requested a tip. He may have been poor, but there is something slightly worrying about corrupt people being in charge of airport security. So I left, thinking that tipping is something that deserves and article. This was, of course, not very original since lots of people have done it before (Eg Coyote blog, MR, History of tipping). So I'll just settle for a brief blog post right now and moe on to other things. Like the science of particle physics or something.