Independent evaluation finds problems with top three PACS

A comparative evaluation of premier PACS has uncovered "significant deficiencies." The problems are fundamental to the very mission of picture archiving and communications involving DICOM compatibility, HIS/RIS integration capability, maintenance, and security. The evaluation, conducted by ECRI, a nonprofit international health services research agency and a collaborating center of the World Health Organization, also found considerable variance in the level of professional services offered and supported by the vendors.

The evaluation focused on three PACS, one each from Agfa (Impax), GE (PathSpeed), and Siemens (Sienet). ECRI examined system design and operation, assessing system integration with imaging modalities, information systems, and other applications, such as speech recognition systems. Also examined were the systems' reliability and ability to recover from internal problems.

"This is important because an independent comparative evaluation of PACS has not been conducted in the industry until this study," said Kenneth Olbrish, a senior project engineer in the ECRI Health Devices Group. "PACS may be a several million dollar investment, but few evaluations, if any, look at PACS prospectively."

Most evaluations in the radiology literature assess the impact that an installed system has had on a facility or the planning and implementation programs that enabled the PACS. The ECRI evaluation focused on the ability of each system to improve workflow, paying particular attention to external integration capabilities-with less than laudatory results.

Representatives for Agfa and Siemens questioned the relevance of the study's conclusions and their usefulness.

"The assessment doesn't help the average user, because the top three (PACS) came out as adequate with similar shortcomings," said Dean Kaufman, director of Strategic Marketing for Impax at Agfa. "It basically confirms that Agfa, GE, and Siemens all have good systems, and the stuff they tested doesn't matter when it comes to making a purchasing decision."

Kaufman did not dispute that the issues addressed by ECRI are important. Rather, he said that the study did not reflect real-world requirements.

"For them to give us some negative marks on our support of DICOM or interoperability is indicative of ECRI's lack of savvy when it comes to PACS," Kaufman said. "We're aware that ECRI has a lot of credibility in other markets dealing with stand-alone modalities or patient monitoring systems. Unfortunately, I think this is going to fall off many people's radar screens."

Siemens also expressed doubts about the evaluation and its conclusions. Yet in the same breath, Siemens' director of new business development, Rik Primo, intimated that ECRI's comments did not go far enough.

"You may want to ask your manufacturer not just what it can support today, but what it will support tomorrow," Primo said. "'Are you going to support that DICOM service class that just came out? Are you going to support JPEG 2000?' Knowing what your manufacturer can do in DICOM is important, but it's even more important to understand what will be available in two, three, or four months."

ECRI could have put the question of monitoring security in perspective by adding something regarding HIPAA (the Health Insurance Portability and Accountability Act), Primo said. Vendors are waiting on more specific recommendations on how HIPAA will be implemented.

2/28/01, Issue # 1504, page 3.