Year: 2000
Type: editorial
Abstract: Mining medical dataHuge bodies of medical and pharmaceutical data have accumulated and are continuing to accumulate.Clinical trials carried out by pharmaceutical companies and research institutions typically record many measurements, taken at multiple times, on many patients.Patient records contain information on illness and disease going back over the lifetimes of almost everyone in the country.More recently, automated data capture is leading to the compilation of vast bodies of data: the human genome project being an obvious and topical example.The existence of such large bodies of data naturally prompts one to suspect that there may be interesting, and unsuspected, nuggets of valuable information concealed within them.After all, rare diseases, of the kind which affect only 1 in 10 000 or more of the population, are only discovered -the common characteristics of the sufferers are only noticed -when large quantities of medical data are collected and examined.Indeed, as science progresses, so the more overt and obvious structures have been discovered.Those that remain are less obvious, and will need to be teased out by more subtle examination of the data.There is a parallel here to the way in which science advances by a leapfrog process of refined measurement technology permitting the detection of subtle phenomena, which in turn lead to the genesis of new scientific theories, in turn leading to the development of improved measuring tools, and so on.This applies just as much in the medical sciences, where, often, the refined measurement procedure goes hand in hand with clarifying the definition of the concept being investigated, as in the physical sciences.The advent of the computer, and the storage of these large bodies of data in electronic form which permits rapid search and processing, permits one to search for these subtle patterns in a way which was inconceivable before.It has also led to the accumulation of astronomically vast bodies of data.Gigabyte and even terabyte data sets are no longer unusual.The science of searching large bodies of data seeking interesting and unsuspected patterns and structures is termed data mining.Clearly data mining owes much to other scientific disciplines.In particular, the most important base discipline is probably statistics: exploratory data analysis to find novel relationships, and statistical inference to see if the relationships are chance events, lie at the heart of much data mining.However, data mining is not merely statistics.It also uses ideas from other areas and has gone on to develop its own unique identity.Some examples of nonstatistical aspects are the necessary concern with data manipulation (if there are billions of data points, this cannot be ignored), emphasis on algorithms rather than models, the importance of adaptive and sequential estimation procedures, a concern with detecting small 'local' relationships in the data as well as larger scale statistical models, and the fact that the data may not be numerical but are often given in terms of the presence or absence of a series of properties.These points mean that technologies such as machine Ó
Author: David J Hand
Institution: Imperial College London
Language: English
Subfield: Health Information Management
Field: Health Professions
Domain: Health Sciences
Open Access status: bronze
OpenAlex ID: W2035632382
