When an issue arises in the world of healthcare IT that affects the exchange of clinical data, it is crucial to act fast. If not, it could result in a domino effect throughout a healthcare organization that negatively impacts patient care.
So how does an organization ensure they use they leverage their support team’s valuable time and resources in the most strategic ways possible?
For the University of California Irvine Medical Center, it meant validating alarms and eliminating their manual process of ticketing, which they accomplished using Corepoint Integration Engine to write CATS: the Corepoint Automatic Ticketing System.
Prior to the creation of UCI’s CATS, the integration engineers were receiving more than 50 alerts per day. With each alert, a team member had to manually create a ticket to send to various analysts and application engineers.
They were also finding that the majority of these alerts were not interfacing problems that needed attention, but instead would resolve before someone could even dial in to address it.
The result? A group of application professionals tired of being disrupted throughout the day and woken up in the middle of the night to “fix” problems that didn’t actually require attention.
CATS allows for a buffer time between when an alarm is triggered and when a ticket is created so that the interface has an opportunity to self-correct and deactivate the alert. This can be anywhere from 10-15 minutes, or up to an hour depending on the interface.
If the alert does not deactivate during that delay, a ticket is automatically created and notifies the appropriate individual, who knows there is a valid issue and can address it immediately.
“What we did was we set up different alarms to call an action list, and what that action list did was write to a SQL database. Then we had another process that read the SQL database and compared the time that the alarm was written to the amount of time passed, and it would then write to our ticketing system and create a ticket. If the alarm deactivated, then it would delete that record from the database so that the second process did not see that record. We went from 50 alarms in a day down to one or two a week because the alarms were correcting themselves.”— Mike McNair, Programmer IV at UCI
In the CATS system, an outbound connection can trigger one of four alert types: queue depth 1 (low activity interfaces), queue depth 10 (moderately active interfaces), queue depth 300 (highly active interfaces), and queue depth 300 with no connection restart (charging interfaces).
The alert sends a message to an Action List that reads a SQL database of tables containing records for every outbound connection. It gathers defined parameters (i.e. the length of delay) and writes to a tracking database where it then sits.
Another Action List calls this tracking database every five minutes and reads each record sitting in it. When it finds a record that has reached its time limit (meaning it has not self-corrected and been deleted from the database), it formats a message with information pertaining to the interface and emails it to ServiceNow, UCI’s ticketing system.
From there, ServiceNow automatically generates a ticket and notifies the appropriate on-call analyst who can then begin addressing the problem.
CATS has other features, too, such as reminder tickets. Each interface has a specified amount of time before a reminder ticket is generated, with the default time being two hours. They also have the ability to configure different blackout times (i.e. scheduled downtime, holidays, weekends, or evenings) to prevent tickets from generating.
“Everyone is happy,” McNair said. The team doesn’t have to manually create tickets anymore, and when someone receives a ticket, they can jump on it immediately knowing it’s valid and needs to be addressed.
“In fact, it's working right now at home,” McNair explained, “but I can tell just by the number of tickets that are been generated that I don't even need to look at the monitor.”
Sharing and Learning at Corepoint Connect
Because UCI had been live on Corepoint Integration Engine for only a year, McNair was a first-time attendee at Corepoint Connect 2018, where he presented the UCI CATS solution.
In addition to presenting, he was eager to learn from the other presentations, classes, and Corepoint users in attendance. Because like many, he prefers to not reinvent the wheel when possible.
“I learned a lot,” he said. “There were a lot of things that we had seen in Corepoint and not really used. Now that I know exactly how they work, so I can now use those things to help fine-tune our interfaces.”
So what’s next for the UCI CATS system? Turns out, there may already be new enhancements in the works.
For starters, they want to build an API to enable two-way communication with ServiceNow. Doing so would allow them to receive a ticket number back and update it with additional comments, rather than creating a second ticket.
The outlook for achieving this looks optimistic, because as McNair explained, “We haven’t found a single problem that we have not been able to solve with the Corepoint Engine.”