I recently landed at Newark Airport and when I turned my mobile phone on, it caught on fire. Not the kind of fire with flames and sparks shooting out of it. Rather, one of my clients got infected by ransomware whilst I was in the air – and there was panic in the dozens of messaged I received.
I had a lot of work planned during my stay in the States – but not so much that I was going to be jammed end-to-end. This was not on the list.
One of my technologists was already on-site doing triage. His initial plan was to wipe-out the systems, reinstall the operating system, and restore from backup.
But unfortunately, the ransomware had also infected the backup system and the most recent valid backup we could find was from November of 2019 (a whole 16 months prior). We also feared that the virtual server within the server was also infected. It was going from bad to worse.
Note: My firm did not setup the technology infrastructure nor was it ever hired to perform a security review – even though we suggested both many times over the course of a decade or longer.
If there was a shinning light, a glimmer of hope, my client had closed his books the previous day and had all of his major reports; financial statements, accounts receivable and accounts payable, inventory stock status report, and a materials requirements planning (MRP) report that showed outstanding demand and replenishments (planned and issued). The bad news was that we would have to recreate this information in a re-implemented system. This was “Plan-A”.
Recreating the financial statements and the accounts receivable and payable was going to be rather simple because there was not that much to recreate (measured in hundreds of records). But the inventory stock status report was thousands of records, and the MRP was going to be massively difficult because the data was strewn about. For instance, one purchase order might have dozens of items on it, each item appearing in different places on the MRP report. So recreating the open customer orders, open purchase orders, and open production orders was going to be difficult, if not impossible.
We decided that, for customer orders and purchase orders, we would just contact the customers and vendors and perform an audit of all open orders – trying to keep a lid on why, least people start getting nervous (and who needed that).
As for the inventory, our technologists wrote an application that would convert the PDF of the inventory stock status report into raw data, strip the headers from the report, and cherry-pick the data we would need to repopulate the inventory database.
Even so, after this was all recreated, there will be a 16-month gap in the data, so analysis of history would have to be from manual records (if ever done at all).
No bueno. But still a path forward.
But we also wanted to discover more of the nature of this particular ransomware. And during our investigation, we came across a company that worked with people infected with ransomware who advocated on behalf of the victim towards a resolution. It was a full-service company that would negotiate with the criminals for a reduction in the ransom demand (providing full transcripts of all conversations and correspondence), purchase and send the cryptocurrency, and receive the decryption utility (in the hopes it would work).
This became the new “Plan-A”, with the old Plan-A becoming Plan-B.
We proceeded to execute Plan-B (recreating the system manually as appropriate and automatically, as able) while simultaneously executing Plan-A. After all, we did not want to lose the time if the negotiations or the decryption utility provided by the criminals were not successful.
It was a nail-biting week while the negotiations between the advocacy firm and the criminals took place, But at the end of the day, the negotiations with the criminals were successful. The advocacy firm was able to negotiate a ransom that was substantially less than originally demanded and the utility provided by the criminals as a result decrypted both the server and the virtual server.
What happened? How did the company’s systems become infected?
Our client truly was concerned about cyber threats. So much so that they took the extraordinary step of having two workstations for each user; one workstation was connected to their internal business network and had no access to the internet, and the other workstation had access to the internet but not the internal business network. The users would switch from one workstation to the other by use of an A/B switch for the monitor and keyboard.
Even so, the vulnerability to their business system was to be found in a terminal server running the Microsoft XP operating system (which Microsoft ceased supporting in 2014). Although there were some unscheduled security updates afterwards, the company never installed them (and it is unknown whether this would have prevented the breach if they had installed the security updates).
In addition, the terminal server was exposed bare to the internet and not behind a firewall with remote users accessing the system using a Virtual Private Network (VPN).
The combination of these to critical implementation flaws made the systems vulnerable and the breach possible.
Contributing factor; Lack of a system backup.
The client had a back-up system, but never checked that it was actually backing up. It wasn’t. Remember to “trust but verify” and “if your backup system and backups are in the same place, you don’t have a backup”.
Contributing factor; Overconfidence.
The client is an engineer who, unfortunately, held an exaggerated level of confidence in their knowledge of information systems and believed the hype that these systems are “plug and play” as presented by the computer industry. It’s easy to fall into this trap because the industry as a whole downplays how truly challenging it is to have a system working in an optimally deployed manner – not to mention keeping up with the latest technology and knowhow. If someone says, “it’s easy”, or “no problem” they are lying.
What can you do to protect yourself?
First and foremost, please take cyber security seriously – a “belt and suspenders” approach where you don’t take anything for granted.
Keep your systems up to date. The operating system patches that are released from the various software providers oftentimes include security updates to plug leaks. The very nature of leaks means that someone, somewhere, had an event that exposed the leak, and the software providers updated their software in response. Don’t be left behind. If your operating system is obsolete and no longer supported, upgrade to the most recent version if the hardware can support it. If your hardware can’t support it, replace the hardware.
Be sure to have anti-virus software installed and running. Most operating systems (and hosted providers of solutions or infrastructure) have anti-virus software included in their operating systems or platforms. But it is often difficult to determine if it is running as intended or includes countermeasures to the most recent threats.
I run Microsoft Windows 10 and it is set to check for updates automatically. Even so, I don’t trust that Microsoft keeps it’s defenses up to date and I can’t tell if it is running. So in addition to Microsoft’s defenses, I use an additional anti-virus application; one that I can tell is running and is updated. Most importantly, I can see that it scans every device that I connect to the computer for threats – especially any USB device. I can’t see this automatically happening with Microsoft (and I don’t know that it does).
Have a backup system. As noted in the case above, don’t assume your backup is working. Check the backup logs daily and take immediate action if the system is showing an error. Test that the backup is working by restoring files. I copy a folder of pictures to another folder and restore the pictures; if they restore properly, I know the system is working. Then I delete the copied folder.
But remember the lesson from above; if your systems and your backups are in the same place, you have no backup. Make sure to take the backups off-site and have only that day’s backup is onsite.
Create and distribute computer policies and procedures. This will at least make people aware that protocols exist and should be followed. But the problem with policies and procedures is that they are only good if the people follow them. I don’t trust this to be the case, so I make sure to have anti-virus software and backups.
Create and use proper passwords. I am not a fan of storing passwords on my computer. I had a laptop stolen once in Athens, Greece. If I had kept my passwords on my computer, some hacker might have cracked the code and I would have had worse day than I did.
I am also not a fan of using those giganormous auto-generated passwords. They are impossible to remember and you usually have to store them someplace or on some device. Doom on you if you lose the device and need the password; you will be forced to reset them one at a time. The internet is littered with stories of people who lost their access to their Bitcoin wallet and have millions of dollars locked-up; forever beyond their reach.
Create passwords you can remember, but that are still complex. A mix of numbers, letters (using both small and capital letters), and symbols should do you well – and at least 10 characters long.
Make sure your internet connection is secure. Public WiFi is just that, public. Don’t go connecting to just any WiFi, and especially don’t send any information you intend to be secure or private (such as login details). Better to create an encrypted WiFi hotspot with your phone for such needs.
The bottom line
I shared with you some of the basics that everyone, whether an individual or a company, can and should do to protect themselves.
So don’t believe the hype coming from the information industry. Nothing is safe and secure. Nothing is plug and play. It is up to each of us, individually, to keep safety and security in mind when we are using our information system; because I guarantee you that your information system partners will wash their hands of anything bad that might happen should doom befall you.
And it’s not hard, it’s not expensive, and it doesn’t take a great deal of imagination. Just a heightened awareness that there are a lot of bad hombres out there that want to gain from your loss and that your systems are never as stable and secure as you imagine them to be.
GovInfo; Information from the United States Government
Small Business Information Security: The Fundamentals
United States Federal Trade Commission
Start with Security; A Guide for Business
About the author
Paris is an international expert in the field of Operational Excellence, organizational design, strategy design and deployment, and helping companies become high-performance organizations. His vehicles for change include being the Founder of; the XONITEK Group of Companies; the Operational Excellence Society; and the Readiness Institute.