Definition

Also called Biased Statistics, Loaded Sample, Prejudiced Statistics, Prejudiced Sample, Loaded Statistics, Biased Induction, Biased Generalization.

This fallacy is committed when a person draws a conclusion about a population based on a sample that is biased or prejudiced in some manner. It has the following form:

• Sample s, which is biased, is taken from population p.
• Conclusion c is drawn about population p based on s.

The person committing the fallacy is misusing the following type of reasoning, which is known variously as Inductive Generalization, Generalization, and Statistical Generalization:

• x% of all observed a's are b's.
• Therefore x% of all a's are b's.

Explanation

The fallacy is committed when the sample of a's is likely to be biased in some manner. A sample is biased or loaded when the method used to take the sample is likely to result in a sample that does not adequately represent the population from which it is drawn.

Biased samples are generally not very reliable. As a blatant case, imagine that a person is taking a sample from a truckload of small colored balls, some of which are metal and some of which are plastic. If he used a magnet to select his sample, then his sample would include a disproportionate number of metal balls (after all, the sample will probably be made up entirely of the metal balls). In this case, any conclusions he might draw about the whole population of balls would be unreliable since he would have few or no plastic balls in the sample.

The general idea is that biased samples are less likely to contain numbers proportional to the whole population. For example, if a person wants to find out what most Americans thought about gun control, a poll taken at an NRA meeting would be a biased sample.

Since the Biased Sample fallacy is committed when the sample (the observed instances) is biased or loaded, it is important to have samples that are not biased making a generalization. The best way to do this is to take samples in ways that avoid bias. There are, in general, three types of samples that are aimed at avoiding bias. The general idea is that these methods (when used properly) will result in a sample that matches the whole population fairly closely. The three types of samples are as follows:

• Random Sample: This is a sample that is taken in such a way that nothing but chance determines which members of the population are selected for the sample. Ideally, any individual member of the population has the same chance as being selected as any other. This type of sample avoids being biased because a biased sample is one that is taken in such a way that some members of the population have a significantly greater chance of being selected for the sample than other members. Unfortunately, creating an ideal random sample is often very difficult.
• Stratified Sample: This is a sample that is taken by using the following steps:
  1. The relevant strata (population subgroups) are identified,
  2. The number of members in each stratum is determined and
  3. A random sample is taken from each stratum in exact proportion to its size. This method is obviously most useful when dealing with stratified populations. For example, a person's income often influences how she votes, so when conducting a presidential poll it would be a good idea to take a stratified sample using economic classes as the basis for determining the strata. This method avoids loaded samples by (ideally) ensuring that each stratum of the population is adequately represented.
• Time Lapse Sample: This type of sample is taken by taking a stratified or random sample and then taking at least one more sample with a significant lapse of time between them. After the two samples are taken, they can be compared for changes. This method of sample taking is very important when making predictions. A prediction based on only one sample is likely to be a Hasty Generalization (because the sample is likely to be too small to cover past, present and future populations) or a Biased Sample (because the sample will only include instances from one time period).

People often commit Biased Sample because of bias or prejudice. For example, a person might intentionally or unintentionally seek out people or events that support his bias. As an example, a person who is pushing a particular scientific theory might tend to gather samples that are biased in favor of that theory.

People also commonly commit this fallacy because of laziness or sloppiness. It is very easy to simply take a sample from what happens to be easily available rather than taking the time and effort to generate an adequate sample and draw a justified conclusion.

It is important to keep in mind that bias is relative to the purpose of the sample. For example, if Bill wanted to know what NRA members thought about a gun control law, then taking a sample at a NRA meeting would not be biased. However, if Bill wanted to determine what Americans in general thought about the law, then a sample taken at an NRA meeting would be biased.

Examples

Bill is assigned by his editor to determine what most Americans think about a new law that will place a federal tax on all modems and computers purchased. The revenues from the tax will be used to enforce new online decency laws. Bill, being technically inclined, decides to use an email poll. In his poll, 95% of those surveyed opposed the tax. Bill was quite surprised when 65% of all Americans voted for the taxes.

The United Pacifists of America decide to run a poll to determine what Americans think about guns and gun control. Jane is assigned the task of setting up the study. To save mailing costs, she includes the survey form in the group's newsletter mailing. She is very pleased to find out that 95% of those surveyed favor gun control laws and she tells her friends that the vast majority of Americans favor gun control laws.

Large scale polls were taken in Florida, California, and Maine and it was found that an average of 55% of those polled spent at least fourteen days a year near the ocean. So, it can be safely concluded that 55% of all Americans spend at least fourteen days near the ocean each year.

To see how Canadians will vote in the next election we polled a hundred people in Calgary. This shows conclusively that the Reform Party will sweep the polls.
People in Calgary tend to be more conservative, and hence more likely to vote Reform, than people in the rest of the country.

The apples on the top of the box look good. The entire box of apples must be good.
Of course, the rotten apples are hidden beneath the surface.

Counter-examples

None.

Advices

Show how the sample is relevantly different from the population as a whole, then show that because the sample is different, the conclusion is probably different.