Annotations from Radiology Report Impressions Reliable

Critical findings that are annotated from the impressions of radiology reports are reliable, according to a study presented at the annual meeting of the Society for Imaging Informatics in Medicine, in Portland, OR.

Researchers from the University of Utah in Salt Lake City sought to develop an annotation schema and train and test reader agreement between trained readers in the detection of critical findings from the impression section of radiology reports.

One-hundred eight randomly sampled radiology impression sections were used to develop a schema that represented 41 unique critical findings. Annotation tasks for each impression included recording critical findings reported in the impression, context values including the existence of findings, historicity of findings, and urgency with which of the findings should be communicated.

Following development of the schema, two senior medical students who were trained readers performed the annotation of 60 randomly sampled radiology reports. The inter-annotator agreement was based on whether both readers identified the same findings, and if both readers identified the same findings and the same context.

The findings showed high inter-annotator agreement (greater than 80%), and it remained high (70%) when evaluating agreement of matching critical findings with each context for all metrics.

The researchers concluded that critical findings can be reliably annotated from the impressions of radiology reports. The group will continue with their research in order to develop a knowledge base of lexical terms and phrases, as well as to annotate a larger corpus of 1,000 reports for further development and evaluation of the algorithm.