Researchers have raised concerns about questionable data sets being utilized to train AI models aimed at predicting stroke and diabetes risk, according to a preprint report on medRxiv. There are indications that some of these models might have seen use in clinical practice, although it’s uncertain if this has resulted in inaccurate diagnoses. At least two academic journals are currently examining studies that incorporated these controversial data sets.
Adrian Barnett, a statistician from the Queensland University of Technology in Brisbane, and his team discovered 124 peer-reviewed papers that reported using one of two open-access health data sets to develop machine-learning models. However, there is little transparency regarding the origins of the data.
Upon analysis, the team noticed several anomalies that, in their view, raised suspicions about the authenticity of the data. “It was quite surprising to come across something like this,” Barnett remarked.
Evidence suggests that at least two of these models have been implemented in hospitals across Indonesia and Spain. One has even been referenced in a medical-device patent application from 2024, alongside two publicly accessible web tools where individuals can assess their risk by entering personal information.
“Models based on data with unknown origins should not be part of clinical decision-making. They are fundamentally unreliable,” commented Soumyadeep Bhaumik, a public-health researcher at the George Institute for Global Health in Sydney. If these tools don’t rely on real-world data, they could lead to mistaken predictions and poorly informed clinical choices, potentially resulting in inappropriate treatment decisions.
Bhaumik advocates for stricter standards requiring institutions and funders to demand researchers disclose data sources for AI health models. He believes journals should reject research that fails to meet this transparency requirement. In line with this, Barnett emphasizes that these flagged data sets should be removed to hinder further misuse.
Data sharing
The two data sets under scrutiny, which have yet to be peer-reviewed, were uploaded to Kaggle, a platform for developers to access data for machine-learning purposes.
The first one, called the Stroke Prediction Dataset, includes details that claim to feature “11 clinical characteristics for predicting stroke events.” It contains health data from 5,110 individuals, addressing risk factors such as heart disease history, marital status, average blood glucose levels, and body mass index (BMI). However, when the team examined the average blood glucose levels against participant identifiers, numerous inconsistencies emerged.
For instance, Barnett noted that rather few data points were missing, which starkly contrasts real-world data that usually has gaps due to participant drop-off or death. “No real-world data set is ever entirely complete,” he pointed out.
They found that 104 research articles utilized this data set for stroke-prediction models, including one in a hospital in Indonesia and another tested on a small group. A separate study from the U.S. indicates it might be in use at a “local heart clinic.”
This stroke data set was uploaded by Federico Soriano Palacios, a data scientist in Madrid, who mentions on Kaggle that the data should come solely from a confidential source and be used for educational purposes only. Palacios has not responded to inquiries regarding the data’s origins.
More unreliable data
The second data set, known as the Diabetes Prediction Dataset, is described as “A Comprehensive Dataset for Predicting Diabetes with Medical & Demographic Data.” It purports to hold information on 100,000 individuals, including BMI, smoking history, and blood glucose levels. However, Barnett’s team found only 18 unique blood glucose values across all those claimed participants, which Barnett argued is implausible. They also noticed thousands of seemingly duplicated entries.
The team identified 21 studies utilizing this data set for diabetes predictions, though none of these models have yet been applied clinically. One study employed both data sets.
This diabetes data set was uploaded by Mohammed Mustafa, a data engineer in Chennai, India, who states on Kaggle that it is based on aggregated electronic health records. In response to a user inquiry about the source, Mustafa claimed he couldn’t disclose specific details due to confidentiality constraints. He has not replied to questions regarding the concern for the data’s use in research.
Kaggle’s media team declined to provide comments on whether they would investigate the data sets or take any further action.
