Item - 2021.AU10.5

The Audit Committee recommends that:

1.  City Council direct that the confidential presentation (November 2, 2021) from the Auditor General be released publicly at the discretion of the Auditor General, after discussions with the appropriate City Officials.

The Audit Committee recessed its public session to meet in closed session to consider this item, as it pertains to the security of the property of the City of Toronto or one of its agencies and corporations.

In 2019, the Auditor General became aware of attacks on critical water systems in the U.S. and other jurisdictions. In addition, there were a number of alerts issued by the U.S. Department of Homeland Security (DHS), the U.S. Federal Bureau of Investigation (FBI), the Canadian Centre for Cyber Security, and other agencies.

These alerts included attacks and ransomware campaigns by foreign states, including an alert in March 2018 from the DHS and the FBI about a multi-stage intrusion campaign by Russian government cyber actors who targeted small commercial facilities' networks. The attackers staged malware, conducted spear phishing, and gained remote access into energy sector networks. After obtaining access, the cyber actors conducted network reconnaissance, moved laterally, and collected information pertaining to Industrial Control Systems (ICS).

The Auditor General became particularly concerned when she learned of a threat published by the U.S. Attorney General Cyber Digital Task Force describing that:

"Iranian hackers… gained access to the Supervisory Control and Data Acquisition ("SCADA") system of a dam in New York, allowing him to obtain information regarding the dam's status and operation. Had the system not been under maintenance at the time, the hacker would have been able to control the dam's sluice gate."

The Auditor General's concern was that if hackers could gain access and remotely move the doors on a dam, they could possibly also do other damage like manipulating chemicals in a water system.

After considering the increased risks, the increased number of alerts and the importance of cybersecurity at our own critical systems, the Auditor General fast-tracked an audit of the Toronto Water SCADA network in November 2019. The Auditor General had just completed a cybersecurity assessment of the City's overall IT infrastructure.

The audit of Toronto Water's SCADA system was the Office's first audit of the City's critical infrastructure Operational Technology (OT) systems. The objectives of the audit were to assess the adequacy of controls in place to address potential threats to the SCADA network, systems and applications. The results were tabled at the February 10, 2020 Audit Committee through a confidential report.

Following the initial audit, there were increased attacks on water facilities and other critical infrastructure systems. Those attacks are becoming more sophisticated and focused.

Recent Cybersecurity Incidents on Water Facilities/SCADA Systems:

1.  Compromise of U.S. Water Treatment Facility
 

An alert from the U.S. Cybersecurity and Infrastructure Security Agency warned water system operators that there was a remote attack where the attacker tried to change the chemicals in the water supply. According to the Agency:

"On February 5, 2021, unidentified cyber actors obtained unauthorized access to the supervisory control and data acquisition (SCADA) system at a U.S. drinking water treatment facility.

The unidentified actors used the SCADA system's software to increase the amount of sodium hydroxide, also known as lye, a caustic chemical, as part of the water treatment process.

Water treatment plant personnel immediately noticed the change in dosing amounts and corrected the issue before the SCADA system's software detected the manipulation and alarmed due to the unauthorized change… The cyber actors likely accessed the system by exploiting cybersecurity weaknesses, including poor password security, and an outdated operating system. Early information indicates it is possible that a desktop sharing software, such as TeamViewer, may have been used to gain unauthorized access to the system, although this cannot be confirmed…"

2.  Ransomware Attack on SCADA Systems at Three Water Facilities in U.S.
 

The October 14, 2021, alert from the above-referenced U.S. government agencies describes recent ransomware attacks that impacted industrial control systems (ICS) at water facilities:

- In the first incident, cybercriminals used unknown ransomware to target a water facility in Nevada in March 2021. The malware affected SCADA and backup systems.

- In the second incident, hackers deployed the ZuCaNo ransomware, which made its way onto a wastewater SCADA computer in Maine in July 2021. The treatment system was run manually until the SCADA computer was restored using local control and more frequent operator rounds.

- In the third incident, threat actors deployed a piece of ransomware named Ghost on the systems of a water plant in California in August 2021. The ransomware was discovered roughly a month after the initial breach, after the organization noticed three SCADA servers displaying a ransomware message.

Following up on Toronto Water's progress

The Auditor General regularly reviews the implementation status of recommendations and reports the results to City Council through the Audit Committee. This follow-up review assessed Toronto Water's progress towards addressing issues and recommendations raised in the February 2020 report so that the SCADA network, systems and applications remain protected.

To verify the implementation of audit recommendations, we undertook significant work to re-test the physical security at selected water facilities, network security and user access management of the SCADA network, systems and applications to identify any remaining gaps.

Testing Results – Progress made by Toronto Water

The initial audit was timely, and based on our testing, we found that Toronto Water has implemented many recommendations and made substantial progress in many areas. The following are some key areas where the Auditor General found significant progress:

- Physical security at water facilities and IT equipment;

- Implementation of technical fixes related to cybersecurity;

- Discontinuation of outdated systems and devices; and

- Staff training and awareness.

 
The results of the testing will be provided to City Council through the Audit Committee in a separate confidential report.

Of note, we noticed a culture shift at Toronto Water in the level of awareness and importance of staying vigilant for cybersecurity risks. Going forward, however, cybersecurity risks will continue to evolve and change. Toronto Water needs to finish implementing the recommendations and directly monitor for and address any new security risks.