By Kevin Struck, UW-Madison Extension Sheboygan County
It was national news when a computer interface at a water treatment plant in Oldsmar, Florida was briefly taken over by a hacker earlier this year. The cyberattacker substantially raised the setting for sodium hydroxide, commonly known as lye, which is added to the water supply to minimize corrosion. Contact with sodium hydroxide at excessive levels can kill skin and cause hair loss, according to the National Center for Biotechnology Information. Ingestion can be fatal.
The FBI and the Pinellas County Sheriff’s Office, which are jointly investigating, uncovered multiple shortcomings in the plant’s network security, which was set up to allow authorized users to access it remotely for troubleshooting. Lapses included the lack of an internet firewall, the use of shared passwords and outdated software, and the absence of two-factor verification. Experts fear such lapses may be typical among our nation’s 151,000 public water systems.
Additionally, security professionals have long advised keeping information technology (IT) and operational technology (OT) networks separate for maximum security, while limiting, if not eliminating, all connections from OT technology systems to the internet. In other words, the same tech system that handles email and web browsing should not also be able to adjust a setting on plant filtration equipment. Oldsmar’s systems were not separated.
Another standard security feature that is sometimes neglected is the practice of disabling user passwords and accounts for employees that have left the public works department.
Following the cyberattack, Oldsmar officials maintained the public was never in danger. Even if the onsite operator hadn’t initially detected the attack, it would have taken 24 to 36 hours for the contaminated water to start flowing out of public faucets. Further, the plant had automatic backup systems monitoring the water’s chemistry that would have sounded alarms long before then.
The executive director of the federal Cyberspace Solarium Commission, which Congress established in 2018 to upgrade the nation’s defenses against major cyberattacks, isn’t so sure. Retired Admiral Mark Montgomery warned that a more sophisticated hacker could have wreaked havoc: “If the attackers could break into the lye controls, don’t you think they could break into the alarm system and alter the checkpoints? It’s a mistake to think a hacker could not introduce contaminated water into our water systems.”
In testimony before Congress on March 10 of this year, Eric Goldstein, cybersecurity chief for the federal Cybersecurity and Infrastructure Security Agency, said the Oldsmar breach should be “a clarion call for this country for the risk that we face from cyberintrusions into these critical systems.”
Goldstein wasn’t the first to sound a warning. A 2020 Journal of Environmental Engineering review found “an increase in the frequency, diversity, and complexity of cyberthreats to the water sector”; and the Cyberspace Solarium Commission’s March 2020 report warned that water systems in the U.S. “remain largely ill-prepared to defend their networks from cyber-enabled disruption.” This is partly due to a lack of in-house IT staff, which is not a surprise in an industry traditionally focused on physical and mechanical risks like natural hazards, burst pipes, and onsite intruders.
The defense of water systems is mostly the responsibility of local utilities. Although Congress included a provision addressing cybersecurity in a 2018 water bill, the requirements were simply that every water system serving more than 3,300 customers had to conduct a self-assessment of its physical and electronic systems and prepare an emergency-response plan. Water systems with fewer than 3,300 customers, of which there are tens of thousands, were exempted from this requirement.
The 2018 legislation also called for $30 million to help water districts deal with problems such as potential cyberattacks, but Congress never appropriated the money.
The reality is, most public water systems must function with small budgets, few employees, and aging infrastructure. To address this situation, many have turned to software systems and digital monitors to increase efficiency and cut costs. There are instances, however, when this strategy is implemented without sufficient safeguards and employee training.
The exact number of attacks on water utilities is unknown. Some attacks go undetected, while others are unreported. No federal law requires disclosure to regulators or law enforcement.
With all of this in mind, UW-Madison Division of Extension Sheboygan County sent a survey to the 11 water utilities in the area to ask how they are addressing this important issue. Recipients included the Plymouth Water Utility, Adell Municipal Water Utility, Waldo Waterworks, Oostburg Municipal Water Department, Cedar Grove Municipal Water Utility, Cascade Waterworks, Random Lake Municipal Water Utility, Elkhart Lake Water Department, Glenbeulah Public Utilities, Town of Sheboygan Water Utility, and City of Sheboygan Water Utility (which also serves Sheboygan Falls and Kohler).
Seven of the 11 surveys have been returned. The survey was set up to ensure respondents would be anonymous, so there is no way to know which of the utilities returned or did not return the survey. Although it’s possible to conclude that the four unreturned surveys are related to poor security measures the four utilities were reluctant to confirm, there is no evidence to support such a conclusion. It is also possible that the four unreturned surveys were misplaced, forgotten about, or never received.
Since some water utility plants in the U.S. are still controlled with analog/manual systems, the first question asked whether the utility uses a computerized control system. (If a utility plant is not using a computerized system, there is little or no opportunity for a cyberattack, since there is no digital operating network to disrupt.) The survey revealed that three of the seven utilities do not use a computerized control system; consequently, nine of the 11 questions were not applicable to the three responding utilities without computerized control systems.
Questions 2 through 9 focused on specific security practices that have been recommended for safeguarding computerized networks from cyberattacks.
Out of a total of 12 responses from four utilities to three password related questions, 75 percent of the responses indicated the recommended best practices were “Always” followed and 25 percent indicated they were “Usually” followed. None of the respondents indicated “Occasionally” or “Never.”
There were a total of 7 responses to a pair of software related questions, with 71 percent of the responses indicating the recommended best practices were “Always” followed and 29 percent indicating “Usually.”
All four of the utilities using computerized control systems responded that they “Always” have an internet firewall activated.
Two-factor verification, which is a somewhat newer best practice, was used sporadically by the four utilities for granting access to critical systems.
All seven of the utilities indicated they have adequate backup systems and alarms.
Finally, two of the utilities have required some or all of their employees to participate in training related to preventing potential cyberattacks. Although the other five utilities have not had any staff participate in such training, it should be pointed out that three of the utilities have little need for such training, since they do not use computerized control systems.
Overall, the survey would seem to indicate that our local water utilities are not as vulnerable to serious cyberattacks as plants in other parts of the nation may be. This is good news for local water utility customers. Nevertheless, the tactics of cyberattackers continue to evolve, always seeking new ways to infiltrate networks and systems. Our utilities must continue to prepare for the challenges that are likely to arise in the future.
Sheboygan Water Utility in the process of replacing critical infrastructure to avoid catastrophic failure
The City of Sheboygan Water Utility treats water drawn from Lake Michigan, then sends it back out to city residents, as well as Kohler and Sheboygan Falls, which are wholesale customers of the city’s water utility. Sheboygan is currently in the process of replacing some of its aging critical raw water infrastructure, which will help avert catastrophic failure.
All the water that is sent out to the three communities passes through two intake pipelines – one installed in 1959, and another installed around 1909. If either of the pipes failed, the other would struggle, or even fail to keep up with pumping the water, leaving many residents and businesses without water. There are no neighboring water utilities big enough to provide backup if Sheboygan’s system does fail.
Plans have been in the works for many years to overhaul the water intake system. The project includes a new intake pipeline system, shore well, and low lift pumping station, which pumps raw, untreated, water to the plant. One of the existing pipelines will possibly serve as backup.
Preliminary engineering design work began in 2020 and construction work is expected to begin in 2022, with completion expected by 2024. The project is estimated to cost anywhere from $30 to $40 million. Various financing options were explored to help fund the project, but a small rate increase in the future is expected as well.
For more information, visit sheboyganwater.org and click on “Projects.”