Woonie: Hi Doc, I have a fever and a terrible sore throat.
GP: Ah, you must have been sleeping late, playing with your iPad2 till the wee hours.
Woonie: How did you...
GP: And you have been to the recent PC Show, squeezing your way through the crowds.
Woonie: Hey, how did...
GP: And you hope to get an MC so that you can catch up on your sleep on Monday right?
Woonie: How...
GP: You're the 88th male patient I've seen this week with similar symptoms having an iPhone4 and dark circles under your eyes. Nurse Tan, use prescription F1, next patient please!
Theoretically data mining could have very cool applications in clinical medicine. However, I am deeply skeptical we will ever in my lifetime come up with a truly practical and widely useful data mining tool. For starters the data is intrinsically very noisy and therefore not so easily to categorize. Next the complexity of biomedical systems which routinely interact with one another, often in ways we have yet to determine, creates a much more complex matrix than any existing data mining challenge. Governments obsession with the privacy of patients data makes accessing it increasingly more challenging. Finally interpreting and acting on any answer is a challenge in itself. If you were to tell a patient that they are at 60% risk of getting alzheimers disease, a disease with no treatment, what is a patient suppose to do? Do you as a patient even want to know? You can pick a few simple scenarios where data mining might help diagnose a child's simple cold, and in fact such things already occur in many public health systems, but a tremendous number of advancements (& government regulation) will be required before data mining might be truly helpful in medicine.
ReplyDeleteThanks for your invaluable comments and insights in the healthcare domain where I have minimal knowledge!!
ReplyDeleteI'll try my best to address all your concerns which can be summarized into 4 main issues:
1. Noisy data: This is a classic problem in data mining and in fact, a most critical problem because it's a fine line between an outlier (noise) and a real anomaly. Many noise filtering techniques exist and both data mining experts and domain experts must work together to find the most appropriate one to apply.
2. Data complexity: This is precisely where data mining comes in and statistics walks out! Non-linear correlations, unknown inter-dependencies, real-time interactions - these are the unique challenges that data mining algorithms are designed to solve! In fact, the stream version of traditional data mining algorithms is the key here to handle massive data with both gradual and sudden concept drifts.
3. Privacy issues: I absolutely agree that this is one stumbling block with no practical solution at the moment... To me, there are only two solutions, both of which may take years/decades: either governments become less paranoid or Secure Multi-party Computation becomes tractable.
4. Actionable interpretation: This is one of the key reasons why many industries don't dive into data mining - they don't see immediate value creation. The crux of the problem: data mining and domain experts don't interact enough because both are too busy with their own stuff! My proposed solution: lock them away on a deserted island for a week! In the case of your alzheimer's example, indeed, there is no point to tell patients that they have a 60% risk - it's more important to advise them how to reduce the risk given their current conditions and based on historical data of other patients with similar conditions.