Update about the temporary slowdown to unavailability of the system during the last days

The verification of products in the national system has been significantly reduced in the last few days or completely interrupted. The first incident of this kind appeared on April 10, 2019. Then, the problem occurred several times, with the situation rising from Wednesday, May 22, 2019, when most end users were affected by the unavailability of the system.

With the exception of a three-hour outage on Wednesday 22 May 2019, the system's operation was not interrupted, or its components were available within the monitoring. However, for end users, the response time was extended to tens of seconds and was unavailable for timeout. This inability to verify the medicinal product occured about one in two to three packs. Problems were recorded by different end users with different IT systems.

From the escalation of verification problems, there were intensive investigations into the possible causes of system problems. At first, it was not clear where the problem came from, so a number of hypotheses were examined: from application error, to infrastructure problems to targeted attack.

We still don't know the exact cause, but on Monday, May 27, we managed to locate the area in the main database that generates problems. The error is expected to be in the load-distributing module on individual nodes, or in combination with the node code.

After this finding, adjustments were made to the configuration of the solution, so CZMVS is now available and the packs can be validated from the 27th of the early afternoon.

Examining the cause did not confirm that it was an attack or misuse of the system.

Also, after re-deployment, performance tests have been conducted to show that the national system is able to handle a much larger burden than currently generated by the Czech Republic.

In addition, code revisions and simulations are run in a test environment to reveal the exact cause in order to correct the error. make changes to eliminate it.

 

This incident was certainly unpleasant, but its solution also tested the reactivity and cooperation of the participants, which led to effective identification of the defect and it will continue.

NOOL would like to thank all those who work together to solve the problem and provide information. At the same time, we apologize for the complications that system failure has caused to pharmacists.