Scientists have tested the ability of AI to evaluate the work of the heart

Artificial intelligence (AI) is able to analyze an MRI scan of the heart in a few seconds, but the speed of image processing does not mean that the results obtained can be fully trusted. The researchers compared the automatic measurements with the doctors' estimates and found that the algorithm was only marginally better at predicting patients' risk of death. At the same time, in some cases he made mistakes, and the results of his work differed from medical measurements. See the Izvestia article about what the study showed and why AI still needs to be monitored by specialists.

How did scientists compare the work of AI and doctors

The researchers examined data from 27,382 patients from the UK and the USA who underwent magnetic resonance imaging (MRI) of the heart. They compared the results of the automatic image analysis with the measurements of specialists. The focus was on the left ventricular ejection fraction, an indicator that reflects how much blood the heart expels with each contraction. It is used to assess cardiac function and determine health risks.

The work was published in The Lancet Digital Health (LHD) journal. During the follow-up period, which averaged 5.6 years, 13% of the participants died.

The algorithm needed about 3-4 seconds to analyze the images, whereas manual processing by a doctor took an average of 13 minutes. However, the study was retrospective: scientists analyzed already collected medical data, rather than checking how the use of the program affects the treatment of patients in real practice.

How accurate were the results?

According to the results of the analysis, the indicators obtained using the algorithm predicted mortality from all causes slightly better than the measurements of doctors. Before age and gender were taken into account, the forecast consistency index was 0.669 for AI and 0.637 for specialists. After taking these factors into account, the difference decreased significantly: 0.781 versus 0.777. Thus, the advantage of the algorithm turned out to be small, especially after taking into account the main characteristics of the patients.

At the same time, automatic measurements did not always coincide with medical ones. The median value of the ejection fraction calculated by the algorithm was 57.8%, while the experts obtained 62.1%. In an additional check of 600 randomly selected images, segmentation errors were found in six cases. More often, difficulties arose at extreme values of the ejection fraction, especially very low ones. This is important because it is precisely such indicators that can be of great importance in assessing the condition of the heart.

Scientists have questioned the willingness of AI to help in healthcare

Why didn't the algorithm handle all cases?

The results also varied depending on the disease. In patients with obstructive coronary artery disease, AI measurements predicted mortality better than medical assessments. In non-ischemic cardiomyopathy, the advantage was less pronounced, and in hypertrophic cardiomyopathy, the specialists showed slightly better results.

The authors suggested that one of the reasons may be due to the peculiarities of the structure of the heart. For example, the algorithm does not always correctly take into account such anatomical details as enlarged papillary muscles and pronounced trabecularity of the myocardium. This shows that the quality of automatic analysis may depend on the specific condition of the patient, and average accuracy indicators do not guarantee equally reliable operation in all clinical situations.

Why it's too early to trust AI without verification

The authors of the study do not claim that the algorithm is capable of replacing doctors. The work showed that automatic measurements can be useful for risk assessment, but did not prove that the use of AI improves diagnosis, helps to choose treatment or reduces mortality. This requires prospective studies in which the technology will be tested directly in clinical practice. In addition, the scientists did not evaluate the calibration of the forecast — how accurately the calculated risk corresponds to the actual probability of an event.

There are other limitations: most of the study participants were examined at the same medical center in the UK, and the accuracy of the algorithm was not checked separately for different ethnic groups. In addition, several authors had financial ties to the company involved in the development of the technology used.

Therefore, the results obtained require independent verification. So far, the most reasonable approach is to consider automatic measurements as an additional tool, the results of which should be evaluated by a specialist.