|
Statistics Can Be Deceptive |
| |
Selection bias
From
Wikipedia, the free encyclopedia (which carries
numerous hotlinks and cross-references)
Selection bias is the error of distorting a statistical analysis by pre- or
post-selecting the samples. Typically this causes measures of statistical
significance to appear much stronger than they are, but it is also possible
to cause completely illusory artifacts. Selection bias can be the result of
scientific fraud which manipulate data directly, but more often is either
unconscious or due to biases in the instruments used for observation. For
example, astronomical observations will typically find more blue galaxies
than red ones simply because most instruments are more sensitive to blue
light than red light.
There are many types of possible selection bias, including:
Spatial:
- Selecting end-points of a series. For example,
to maximise a claimed trend, you could start the time series at an unusually
low year, and end on a high one.
- Early termination of a trial at a time when its
results support a desired conclusion.
- A trial may be terminated early at an extreme
value (often for ethical reasons), but the extreme value is likely to be
reached by the variable with the largest variance, even if all variables
have a similar mean. As a result of that early termination, therefore, the
means of variables with larger variances are overestimated.
-Partitioning data with knowledge of the contents
of the partitions, and then analyzing them with tests designed for blindly
chosen partitions (see stratified sampling, cluster sampling, Texas
sharpshooter fallacy).
-Analyzing the lengths of intervals by selecting
intervals that occupy randomly chosen points in time or space, a process
that favors longer intervals.
Data:
- Rejection of "bad" data on arbitrary grounds,
instead of according to previously stated or generally agreed criteria
Participants:
- Pre-screening of trial participants, or
advertising for volunteers within particular groups. For example to "prove"
that smoking doesn't affect fitness, advertise for both at the local fitness
centre, but advertise for smokers during the advanced aerobics class, and
for non-smokers during the weight loss sessions.
- Discounting trial subjects/tests that did not
run to completion. For example, in a test of a dieting program, the
researcher may simply reject everyone who drops out of the trial. But most
of those who drop out are those for whom it wasn't working.
Studies:
- Selection of which studies to include in a
meta-analysis
- Performing repeated experiments and reporting
only the most favourable results. (Perhaps relabelling lab records of other
experiments as "calibration tests", "instrumentation errors" or "preliminary
surveys".)
- Presenting the most significant result of a data
dredge as if it were a single experiment. (Which is logically the same as
the previous item, but curiously is seen as much less dishonest.)
Selection bias is closely related to:
- sample bias, a selection bias produced by an
accidental bias in the sampling technique, as against deliberate or
unconscious manipulation.
- publication bias or reporting bias, the
distortion produced in community perception or meta-analyses by not
publishing uninteresting (usually negative) results, or results which go
against the experimenter's prejudices, a sponsor's interests, or community
expectations.
- confirmation bias, the distortion produced by
experiments that are designed to seek confirmatory evidence instead of
trying to disprove the hypothesis.
See also
bias (statistics)
Berkson's paradox
|
|
From Skeptic Magazine's e-mail newsletter
comes this intro:
In response to my Scientific American column on Michael Drosnin's "Bible
Code" (June, 03) in which he claims to have predicted 9/11, and his
subsequent letter to the editor, published in the most recent issue (go to
www.sciam.com to read any of the past columns and issues), we received a
wonderful letter to the editor about predicting the future. This really says
it all. It is from John Byrne, ([e-mail address deleted]), the comic book
writer/illustrator of Spider Man and other super heroes (who is also a
reader of Skeptic).
And the letter itself:
Reading Michael Drosnin's response to Michael Shermer's column on the Bible
"code" and its ability to accurately predict the future, I could not help
but laugh. I have been a writer and illustrator of comic books for the past
30 years, and in that time I have "predicted" the future so many times in my
work my collegues have actually taken to referring to it as "the Byrne
Curse".
It began in the late 1970s. While working on a Spider-Man series titled
"Marvel Team-Up" I did a story about a blackout in New York. There was a
blackout the month the issue went on sale (six months after I drew it.)
While working on "Uncanny X-Men" I hit Japan with a mjor earthquake, and
again the real thing happened the month the issue hit the stands.
Now, those things are fairly easy to "predict", but consider these: When
working on the relaunch of Superman, for DC Comics, I had the Man of Steel
fly to the rescue when disaster beset the NASA space shuttle. The Challenger
tragedy happened almost immediately thereafter, with time, fortunately, for
the issue in question to be redrawn, substituting a "space plane" for the
shuttle.
Most recent, and chilling, came when I was writing and drawing "Wonder
Woman", and did a story in which the title character was killed, as a
prelude to her becoming a goddess. The cover for that issue was done as a
newspaper front page, with the headline "Princess Diana Dies". (Diana is
Wonder Woman's real name). That issue went on sale on a Thursday. The
following Saturday. . . I don't have to tell you, do I?
My ability as a prognisticator, like Drosnin's, would seem assured --
provided, of course, we reference only the above, and skip over the hundreds
of other comic books I have produced which featured all manner of
catastrophes, large and small, which did not come to pass.
That about sums it up, no? |
| |
|