Summary of a study done by Peter Austin PhD, University of Toronto, on the Evaluation of medical ailments associated with each Astrological sign.
Here is what was in the study.
1. Prof. Austins' group randomly divided the adult population of Ontario in the year 2000 into two samples. The first sample was called the derivation sample. The second sample was called the validation sample. Each sample consisted of about 5.3 million adult residents of Ontario.
2. They classified each resident of Ontario according to their astrological sign of birth.
3. Using patients in the first sample (the derivation sample), they examined diagnoses for hospitalizations, until they found two diagnoses for which residents born under each astrological sign were statistically significantly more likely to be hospitalized in the year following their birthday in 2000 compared to residents born under the remaining 11 astrological signs combined. In order to do this, they had to search through hospital admissions for 223 different diagnoses. Note, that they did not begin with any pre-specified hypotheses, but searched through diagnoses until they found two diagnoses for each astrological sign for which residents born under that sign were more likely to be hospitalized.
4. They then examined whether these 24 associations (2 diagnoses per astrological sign x 12 astrological signs) were confirmed in the second sample (the validation sample). Note that the second sample is independent of the first. The 24 associations between astrological sign and health outcomes were "discovered", "uncovered" generated or derived in the first sample (the derivation sample).
5. Of the 24 potential associations 'discovered' above, only 2 remained statistically significant in the second sample (the validation sample).
An important statistical point is that if you test a hypothesis, then 5% of the time, you will detect a statistically significant association even when none truly exists. Let us assume that they test two associations simultaneously, and furthermore, assume that both associations are in reality false. Then the chance of falsely detecting a significant association for each is 0.05 (5%). The chance of correctly concluding that the association is false is then 0.95 (95%) (1 ≠ 0.05 = 0.95). Therefore, the chance of concluding that both associations are false is 0.95 x 0.95 = 0.9025 (90.25%). Therefore, the chance of mistakenly identifying at least one as truly existing is 100% - 90.25% = 9.75%. Now, in their study, they examined 24 associations in the second sample (validation sample). If a particular association were truly false (in other words, if there were no association between that astrological sign and the given health outcome), then they would have a 0.95 (95%) chance of correctly concluding that. The probability of correctly concluding this for all 24 associations is (0.95)^24 = 0.292 (29.2%) (here (0.95)^24 means 0.95 multiplied together 24 times - or 0.95 to the exponent 24). Therefore, there would be a 100% - 29.2% = 70.8% chance of mistakenly concluding that at least one of the astrological associations was truly present. In the second sample (the validation sample), they found that two associations were 'statistically significant'. However, if one takes into account statistically the fact that they examined 24 associations, then one can demonstrate that none of the associations were found to be replicated in the second sample (the validation sample). Therefore, a complete statistical analysis would show that none of the associations generated in the first sample (derivation sample) were confirmed or found to hold in the second sample (validation sample).
One of the primary lessons they wanted to illustrate from their study was that the more associations that one examines, the greater the risk one takes of finding false associations. Stated another way, if one examines enough patterns that are truly random, eventually some of them will appear to have a have a discernible pattern. If one uses standard statistical tests, then 5% of the time one will identify as present an association that is truly absent. As one increases the number of associations that one examines, the chances of making an incorrect conclusion increases dramatically.
Peter Austin, PhD
Senior Scientist, Institute for Clinical Evaluative Sciences. Associate Professor, Departments of Public Health Sciences and Health Policy, Management and Evaluation, University of Toronto.
Institute for Clinical Evaluative Sciences
G106 - 2075 Bayview Avenue Toronto, Ontario, M4N 3M5
ICES Web Site: www.ices.on.ca