Colorado Springs Utilities is the nation’s largest municipally owned, four-service, non-profit utility. The utility provides electricity, natural gas, water and wastewater treatment from one source.
Employing nearly 2,000 people and providing utilities to more than 600,000 combined metered customers in the Pikes Peak region, Colorado Springs Utilities’ service area spans 500 square miles and includes residential, industrial, commercial and military sectors. As a J.D. Power and Associates award-winning public utility in multiple categories, Colorado Springs Utilities meets some of the industry’s highest standards for dependability in service delivery.
Challenge
In the post-9/11 world, operating a public utility has become more complex. Utilities today must comply with strict security and operational procedures to reduce or eliminate potential for catastrophic mishaps related to intentional (or unintentional) human acts. Under the Bioterrorism Act of 2002, congressional authorization requested that all public utilities conduct a vulnerability assessment on their facilities, develop an emergency response plan, and submit the certified plan to the Environmental Protection Agency (EPA) within six months of completing the assessment.
For Colorado Springs Utilities, complying with these regulations and conducting a critical analysis of its automation control practices and policies was no easy task. According to Jay Hardison, customer operations superintendent, the completed assessment revealed that the company’s electronic, computer and automation systems lacked the necessary control and network security. It also revealed that the utility needed a better emergency recovery plan to outline the steps and procedures the utility should take if it should need to complete a large-scale startup following a disaster.
“We needed to prepare an emergency recovery plan and implement a system that would notify us of security breaches before any of our operational or delivery processes could be harmed by outside threats,” Hardison said. “If we didn’t do so, our control systems would be vulnerable to unauthorized changes or sabotage, leaving our supply resources and delivery processes — and ultimately public health — at substantial risk.”
Implementing a solution to address the shortcomings uncovered in the assessment without jeopardizing the service that its thousands of customers depended on was going to be a challenge. In order to meet the certified plans submitted to the EPA, the utility needed a robust solution that would meet not only its current needs, but would be scalable enough to meet future growth demands.
Solution
Colorado Springs Utilities turned to Rockwell Automation and its change management software, Rockwell Software Maintenance Automation Control Center (RSMACC). It gives Colorado Springs the added security required, while offering supplemental features of authentication, auditing, archiving and verification. These features allow the utility to better manage its processes, and use the tools and analysis capabilities to manage its maintenance activities.
Before the RSMACC solution could be integrated, there was plenty of work to complete. To prepare for the installation, the utility formed a dedicated cross-functional team to gather a list of all the utility’s automation controllers and develop a security matrix for control system authorization.
“In order for the solution to work, it was important that we had complete participation and cooperation between our automation and IT teams,” Hardison said. “Rockwell Automation did an excellent job educating both teams on what we needed to do to streamline our equipment operations to prepare for the integration of RSMACC.”
The utility developed site preparation documents to systematically collect data on the equipment for the RSMACC installation and database setups. In its data gathering process, the team was responsible for getting information on: Allen-Bradley ControlLogix programmable automation controllers; Allen-Bradley PLC-5 controllers; Allen-Bradley SLC 500 controllers; Allen-Bradley MicroLogix controllers; and 10 workstations across six different facilities. To further simplify operations, common Rockwell Software RSLinx drivers were coordinated with all facilities to enable collaboration in real time.
According to Hardison, taking inventory was a necessary, time-consuming exercise that was absolutely critical to ensure that services to Colorado Springs Utilities’ customers were not interrupted the day the solution was integrated.
In late 2004, the RSMACC solution was integrated seamlessly with the existing infrastructure without any interruptions in service delivery.
Results
By implementing RSMACC to enforce security across controllers, Colorado Springs Utilities has reduced the potential for mishaps related to incorrect/outdated programs being uploaded to critical applications. In addition, the utility now has the ability to quickly recover in the event of an emergency interruption in services.
With the RSMACC system, engineers are reassured the files they are using are up-to-date because there is one central repository for all files and documentation. The utility also now has a method of enforcing accountability for file possession, location and a change management trail. Even if there is a user error, the change is tracked and the original program can be quickly recovered.
There is no room for downtime or security mishaps in an industry as critical as public utilities service, and the RSMACC solution has exceeded what the utility set out to accomplish in terms of EPA regulations. Colorado Springs Utilities now has a public utility operation with multiple levels of critical controls access and an approved and filed emergency response plan, while still continuing to deliver the award-winning level of service for which it is known.
“Our labor budget is lower than many municipalities because we incorporate automation products that optimize our procedures and drive down costs, which means lower rates for our customers,” Hardison said. “And in our industry, that’s the bottom line.”
The results mentioned above are specific to Colorado Springs Utilities’ use of Rockwell Automation products in conjunction with other products. Specific results may vary for other customers.