Cheating on Sample Size

Object of this page:
Be aware when people cheat on sample size.

Using the square root law, or calculating confidence intervals, or almost any inference method: Watch that independence requirement. The N items composing the data should be independent. Otherwise, its cheating on sample size, and it`s as dishonest as it is common. Here are examples:
——————————
Suppose 10 students measure the same thing in a science lab. The average of the 10 measurements in the 10 lab reports should be more precise than a single measurement - sqrt10 times as precise. But are undergraduate labs done in the same period are really independent? How many reports are there really? Is "N" really 10, or is it, perchance, 1 or 2 or 3?   (One class I asked told me N really equals one.)

Artificial instance: A sample has five items but each item is listed twice. The ten numbers in the list are not independent. Five items are five items: Using N=10 in formulae is cheating. Penmanship is not a substitute for research.

Suppose the right and left leg lengths of 5 people are measured. This produces 10 numbers, but, again, they're not independent: Almost, not quite, they are duplicate pairs. This is the above again, thinly disguised.

For a really wild example have 4 observations, but write down each number 100 times. Then plug 400 into the denominator of the square root law as N.

A psychology professor once said of a thesis - "Sure there are 400 observations, but only 4 dogs."

What about a forestry research project which samples 400 trees - all of them in one of 4 woodlots? (Hypothetical, case, but this sort of thing really happens.)
  • Suppose the population of woodlots comprises hardwood lots and softwood lots in about equal numbers. About one time in 8 all 4 woodlots of the sample will be of one kind.
  • Or suppose about half the population of woodlots is upland and half is lowland.

The term cluster sampling is used when random groups (clusters), rather than random individuals are sampled. For example, to sample the students in large school, one might pick at random 4, say, classes. If each class has, say, 30 students, each of who receives a questionairre, is the sample size 120, or is it 4?   Items in the same cluster are not independent.

Cluster sampling is not always recognized as such:
In Development of Survey Methods to Assess Survey Practices by Barbara Bailar and Micheal Lanphier, © American Statistical Association, 36 surveys were investigated. Of these, 24 had "at least partially clustered" samples. From page 45 of the report: "In the clustered samples included in the pilot study, . . . . With only four exceptions, estimates of variance, when made at all, were computed as though the data had come from a simple random sample."

Tale heard during a party at a meeting of the Statistal Society of Canada:
Each ocean bottom sample cost some thousands of dollars to dredge. Then, the sample required hours processing in the lab to reduce it to 1 c.c. of purified material to be inserted into an instrument from which a value is read: All that money and effort for one number.
Despite the cost there were pages and pages of data
Statistician: "How could you ever afford all that data"?
Oceanographer: "What's the problem?   After the specimen is collected and processed, I put it in the instrument, and Push the button and get a number, and    Push the button again and get another number, and    Push the button and get another number, and    Push the button and get another number,    and . . . "

Pseudoreplication:
In ecology cheating on sample size has become widely recognized under the name pesudoreplication because of this paper:
Pseudoreplication and the Design of Ecological Field Experiments
by Stuart H. Hurlbert   Ecological Monographs: Vol. 54, No. 2 June 1984
and a follow up paper in 1996
Pseudoreplication Revisited, by Robert A. Heffner; Mark J. Butler; Colleen Keelan Reilly Ecology 1996 p 2558-2562
 
Exercises
William Knight
University of New Brunswick