Hark! Good gentlefolk of industry, lend me thine ears, for I shall tell thee a tale most dire—the tale of patches neglected, of systems laid bare to villainous cyber rogues, and of noble operators who, by wisdom and prudence, didst fortify their fortresses against the dark tide of digital treachery. Verily, in this age most modern, where machines do govern our mills, our lights, and the very breath of commerce, a silent war doth rage. ‘Tis not fought with sword nor steel, but with cunning code and whispering specters that creep through unguarded gates. The stoutest of defenses matter not if a single crack remaineth unpatched, for a fortress undone by neglect is but a castle of sand before the tide.
I'm pretty sure the above was somewhere in William Shakespeare's canon. We'll include his wisdom as we move through the topic of Operational Technology and Software patching. Nor-Cal Controls understands the critical importance of robust cybersecurity in power generation and industrial settings. We see firsthand the challenges our customers face in maintaining secure and reliable operations. That's why we're committed to sharing insights and best practices to help you navigate the complexities of OT security.
The Bard has touched on a topic that couldn't be more contemporary, especially in the space of power generation.
We do not have the luxury of relegating the mitigation of cyberattacks to the IT team; it's the responsibility of an entire organization and must account for security, uptime, and risk management. One critical element (though not the ONLY critical element) to this defense is a non-negotiable regular patching schedule. Let's explore why it's non-negotiable versus just "important" - like everything else you do.
- The Ever-Watchful Foe
Lo, let none think their peace secure!
There have been innumerable volumes written about how dangerous the Internet is. If you're in charge of a networked computer system and you're sleeping well at night, you might not be paying attention.
We deal with ransomware thugs, nation-state actors, and even idealogues with computer skills targeting industrial control systems. Attacks like Stuxnet, not to mention Colonial Pipeline, highlight the need for patching - and the dire result of ignoring or avoiding it. We do not own the present, we are stewards of the future, and the only way to ensure a smooth transition to the future is to keep our systems patched against the things that are ‘known’ to be better prepared for the future ‘unknowns’.
- The Law’s Stern Hand
Mark well, ye keepers of industry, for those who govern and decree have set forth commandments most clear. The laws of NERC CIP, the strictures of IEC 62443, and other writs most weighty do demand that all manner of defenses be maintained, lest thou fall afoul of compliance and find thyself in shackles of regulation.
Regulation feels uncomfortable, to be sure, and it is well beyond the scope of this post to argue its benefits or lack thereof. For this post, we will simply acknowledge its presence with the further point that the portions of regulation that require you to perform basic system and software maintenance surely can't be too odious. Regulations aside, IT and OT best practices should be guiding principles:
- We don’t use stock passwords and we explore multi-factor authentication
- We're careful about who has access to the controls systems
- We’re aware of the dangers of exposing systems to the Internet
- We patch thoroughly, and often (at least quarterly)
- We do not assign functionality to a device beyond its core capabilities
- We back up critical systems and ‘test restore’ those backups regularly
- The Balance Betwixt Time and Fate
Aye, ‘tis true, the wheels of industry turn ceaselessly, and to pause their motion for mere maintenance seemeth folly to some. Beware, ye who say, “All is well, and naught need be done!” For the foe striketh not where thou watchest, but where thou dost not look. A system left untouched is as a castle whose moat hath run dry—the walls may stand tall, but the enemy shall find his way within.
In the system administration/network engineering world I came from, uptime is very near a badge of honor. For instance, a statement like "I've run my XYZ server for 12 years with no reboots" sounds super impressive - but it also indicates to me that you might not have patched it. Fortunately, the braggadocio around uptime has calmed recently, but some still subscribe to it.
If I'm an attacker, I'm looking for those kinds of claims. Or, more accurately, for systems that reflect that claim. If there's one thing I've seen in our industry, it's that many industrial devices were not designed with security in mind - and they're prime targets. We must be prepared to think of ALL possible attack vectors, because the adversary is definitely doing exactly that.
- Use unique passwords at the site level, at least; preferably at the device level
- Shut off unused switch ports to prevent access to your network
- Restrict traffic between the IT and OT networks to just what is absolutely needed
- Don't assume you're too small (or too large) to be attacked
In today's world, products are deployed with a perfect security record, then attacked, and then found to be insecure, much more like Will's castles and moats. Regular patching ensures that it "ain't broke" for at least the next patching cycle, then we all begin again.
Microsoft's 'Patch Tuesday' is an excellent reminder that we need to be patching systems on a routine basis; it does not need to be on Microsoft's monthly cycle, but it needs to be at a frequency that works for your organization. I would recommend no less than quarterly for Windows-based systems, and twice yearly for non-Microsoft systems.
- The Wages of Neglect
Consider well the price of thy choices. To patch is but a trifling burden, a small tax upon time. Yet to fall to cyberattack? That is ruin unmeasured! A ransom demanded, a kingdom laid low, reputation cast to the winds, and fortune lost in the abyss.
The operational cost of patching everything in the fleet can be daunting. Many of our customers have dozens of sites with literally hundreds, if not thousands, of devices to manage. We make our best effort to determine the cost of patching an entire fleet, but there are so many factors that can modify that cost; a company's existing cyber-security stance, the level of effort to patch a single system, and corporate comfort with a site not producing for the patching duration are just a few of the myriad ways that this initiative can be a greater cost than anticipated. All of that, however, pales in comparison to the cost of a cleanup after a cyber-security incident.
Colonial Pipeline's incident started at $4.43 Million; this is the ransom that they paid to ransomware 'providers'. This is just the initial cost to unlock the data. Absent any regulatory fines, it is simply additive to the cost of the work that must now be done - to evaluate security status, patch devices, train personnel, and set up regular testing.
Save future massive costs by doing these things BEFORE the incident, because you will be doing them after -
- Evaluate security status
- Patch devices
- Train personnel
- Set up regular testing
Thus Sayeth Wisdom: Patch Ye Well and True!
Patch, dear stewards of industry, and patch with diligence, for the hour of reckoning cometh unbidden. Those who attend to their defenses shall sleep sound, whilst those who tarry shall find themselves cast into a pit of woe.
Couldn't have said it better if I tried, Will.
Contact us today to discover Nor-Cal Controls' project capabilities.
