Classified Information and Human Error Caused Trade Register’s Collapse Mind the horns!

After talking with experts, Bivol managed to establish with a significant degree of reliability the events that had led to the collapse of the Trade Register. “The burning of the four discs with 25 terabytes of information” as Deputy Prime Minister Tomislav Donchev put it, is neither a natural phenomenon nor a cyberattack, but most probably a result of administrative chaos and human error.

Fatal error with LUN

The main cause for the collapse is that someone, either an administrator at the Registry Agency, or someone from “Lirex BG”, the company supporting the Trade Register, has placed the Oracle-based main software system on the same LUN (logical unit number) that is keeping the backup copies in the backup hub in the Sofia “Business Park”.

This has caused a risk of losing both the main system and the backups at the same time in case the system malfunctions.

The hardware is not new –it is from 2012, with long-expired warranty. It was configured on RAID5, probably in order to increase the efficiency and size of disk space. RAID5 has a low level of protection, it is only resistant to burning one disk in a database. This may be enough for a place storing backups that are not used in real time but is at risk for a basic real-time storage that functions best with RAID10, complemented by a large number of spare discs.

Once Oracle’s main software “bites” this LUN, we can assume that the backup disk database had operated under a heavy load. The latter and the likelihood of the presence of unsuitable for RAID or defective disks had caused their rapid wear and tear and a simultaneous defect in an amount that the RAID5 configuration cannot compensate for. Both the backup of the database and part of the real time one had been lost. What exactly and how much information is lost, and whether it can be recovered from tape archives, is yet to be established.

Whether there is a malicious intent in this action is a matter that must be clarified by the authorities in charge. We bet on the hypothesis of error due to negligence or taking unnecessary risks due to the systematic lack of sufficient resources and free disk space, rather than the one of deliberate sabotage.

The database operates on old hardware due to secrecy

As it became clear, the supply of new, state-of-the-art equipment – servers and storage – for the Registry Agency and the Trade Register began in 2018. The equipment provider is Telelink, associated with controversial businessman Spas Roussev.

Who is behind the companies and how the public procurement had been handled is another matter. We were told that Telelink had not been allowed to migrate the Oracle base of the Trade Register to the new servers because the information is classified, probably because of the large amount of personal ID numbers (social security numbers) and other personal data of Bulgarian citizens.

It is yet to be clarified why there was such an administrative obstacle to this necessary and important technical operation. Did the Telelink experts have the necessary level of access to classified information? If they did not, why was it not provided by the management of the Registry Agency in due course?

The result is that until the collapse, the information from the Trade Register had remained in storage that, at best, dates from 2012 (possibly even from 2008), which is simply out-of-date and is the weakest place in the system and this had been known. It has been quite possible to prevent the risk of a mishap caused by human error by upgrading the hardware in a timely manner.

If only…

Probably, all of this would not have happened if the Registry Agency followed its own instructions to secure and support the Trade Register; if it maintained sufficient IT staff; if the hosting and support company also provided enough specialists and staff; if the Agency updated its hardware in a timely manner, with a sufficient vision for the necessary resources in the future with a surplus and with secure configurations. According to Bivol’s sources, only two experts from the Agency and two others from Lirex BG have been involved in the Register’s support, which is far from sufficient for such a vital system.

Finding and retaining competent staff is a well-known problem, as wage levels in the State administration are at times lower than in the private sector. There is money, however, it is distributed in the form of additional material stimulus (a bonus system) where the lion’s share of these bonuses goes to government officials, and not to IT specialists.

“You see there are countries where their elections had been hacked through cyberattacks. Here, in Bulgaria, there is a huge outrage over everything. Yes, it is not pleasant, but everyone is working actively to safeguard the system,” Prime Minister Boyko Borisov said on Sunday in commenting on the Trade Register’s collapse.

The difference with these countries, however, is that Bulgaria had been hacked from inside and not by a foreign enemy. It is obvious that a key system for the country’s business has collapsed not because of malicious hackers but because of the incompetence and the negligence of the government and its employees. This should be laughable for hostile cyber armies that are looking to hack our country, but are not even given that chance.

Because before they attack, we are hacking ourselves.

***

If you find this article useful, support our work with a small donation.

Pay a Bivol Tax!

We will highly appreciate if you decide to support us with monthly donations keeping the option Monthly