Zero-downtime PACS upgrade takes careful planning

Striking fear into the heart of a radiologist is easy. Just say there will be a system upgrade over the weekend and downtime will be around eight hours. The PACS industry is rife with horror stories of the infamous eight-hour upgrade that lasted days, wreaking havoc on a clinical practice. Physicians who have become reliant upon information systems conservatively call those downtimes a waking nightmare.

One of the dirty secrets in the PACS industry is the so-called scheduled downtime. Many PACS vendors will guarantee an uptime in the 99.9% range, but if you read the contract carefully, they do not count scheduled downtime in that calculation. But fear not. Scheduled downtime doesn't have to be as certain and dreaded as death and taxes.

Dr. Carrera, chief of radiology at the Medical College of Wisconsin, doesn't care what you call your downtime.

"Downtime, whether planned or not, can be a terrible thing for a radiology department," he said. "Besides the obvious problems with workflow within the department, disruptions and misperceptions from consulting services can cause all sorts of trouble. Our institution houses a Level I trauma center and large ICU, so our services have to be consistently available and flow without interruption. The standard of comparison the referring services use is film, which is fast, available, and reliable. PACS has to be able to work at all times at least as well as film, or there is sure to be trouble."

Upgrades are a necessity to make improvements and change with the times. Most PACS are based around a central architecture where the images and the database come from a central location. Taking those services offline renders all workstations useless and prevents any access to images. During an upgrade, the database and image services typically need to be taken offline to replace the software.

At Froedtert Memorial Lutheran Hospital in Milwaukee, a lengthy downtime during an upgrade was not a palatable solution. Jeff Rehm, equipment supervisor at FMLH, worked out an alternate solution with PACS vendor McKesson.

RECIPE FOR THE OLD SWITCHEROO

? Step 1: Get a spare server. We had a six-year-old Pentium III server that was discarded from the old ultrasound system. We were using it as a test server. During the weekend, the system load is 1/10 to 1/20 the load during peak time. It does not require the same high-end servers needed at rush hour. So we used the old Pentium as a temporary server during the upgrade of the production servers. To do this, we loaded the new software on the spare system first.
? Step 2: Load the new upgraded software on the spare system.
? Step 3: Two weeks prior to the upgrade, route all incoming studies from the modalities to the test server as well as the production system. An additional benefit about sending data to the test server running the new software is that this provides a grace period to evaluate the new system and do some acceptance testing to see if it will work.
? Step 4: On the evening of the upgrade, point all the workstations to the test system. It will only have the previous past two weeks, but that is usually sufficient for after-hours emergency and inpatient needs. The system is operational to new arrivals, and some priors are available. We called this a limited operation mode.
? Step 5: Take the production system down and conduct the upgrade. The McKesson upgrade team had the freedom to take some extra time to make sure they did everything correctly. In a traditional upgrade, an eight-hour window can be like a gun held to your head. If there are unforeseen complications, this new technique provides a site more time to focus on the problem instead of having to deal with panicky users.
? Step 6: After the upgrade is complete, point the workstations back to the production server and route all the studies created in the interim to the production server.

"It was the best upgrade we ever had, without any user inconvenience," said Bryant Mascarenhas, PACS coordinator at FMLH. "The only hurdle was that we needed to import the user account database into the test system, so all the same users had the same access during the time the system was running on the test server."

This method takes foresight and planning on the front end. But if you are a Level I trauma center and take your uptime seriously, this could be a great way to maintain your level of service to the hospital. This technique should be applicable to any central architecture PACS out on the market today. Planning for change is a much saner approach than living in fear.

Dr. Nagy is on the faculty of the Medical College of Wisconsin, which is affiliated with Froedtert Memorial Lutheran Hospital.