Wouldn’t be nice if every new piece of equipment or system worked flawlessly from the moment it was installed and initiated until the first breakdown occurred many years later. That would be an exception with most industrial equipment.
When it comes to setting up equipment, we may find some components were damaged during shipment, the field engineer has little expertise, or we’re doing the beta testing on a new design (whether we agreed to this or not). The same can apply to rehabilitated equipment being installed in the original application and position.
Why is it that cars come off an assembly line and rarely run into serious warranty issues? The reason is that car manufacturers exhaustively test, improve and inspect both the design and the production processes on thousands of cars. Industrial equipment tends to run in small manufacturing lots; hence not all the bugs get worked out.
This is why commissioning industrial equipment is an art, and a necessary one at that. I call it an art because, by definition, setting up new equipment requires creativity, craft skills and deductive reasoning. These very same attributes apply to an art in every sense of the word.
My facility, Upper Occoquan Service Authority (UOSA), has matured its maintenance and asset management approach to the point where focusing on commissioning is the next logical step. This thinking arrived as part of our overall pursuit of best practices in reliability and asset management.
Failing to follow a commissioning process is tantamount to installing your next failure. Proper commissioning of equipment can extend useful life, which is an important goal in asset management. Therefore, commissioning is an essential component of a good asset management program.
After several years of designing and implementing maintenance support processes, we will verify improved reliability by tracking mean time between failure (MTBF), mean time to repair (MTTR) and equipment availability.
A Commissioning Philosophy
The basic challenges in setting up new or rehabilitated equipment are fairly universal. They boil down to anticipation of problems, perception of problems and correction of problems. Let’s take those words (anticipation, perception and correction) and examine how they apply to the commissioning process.
It may be difficult to imagine how to anticipate problems, but a good starting point is searching out repair histories. Repair histories can tell you what broke down, when, why, how, where and who was involved. This assumes your enterprise has accurate repair histories in a CMMS. If repair histories are not of sufficient quality, improve them as needed. Interviews with your operators and others who own and operate the same equipment are priceless.
As part of a commissioning plan, this valuable knowledge needs to be recorded in a retrievable format. This is a core part of the plan – the framework for recording information. Think of it as the rules of a game. The framework for recording should be available to participating users. The ability to change entries in the records must be monitored and controlled to ensure consistency and accuracy.
If the equipment to be commissioned is made up of subassemblies of familiar components, then anticipated failures can be defined by individual components. For example, a centrifugal blower should have a performance curve graph, an allowable vibration limit and other specifications. When the blower operates outside of any of these values, an investigation is required and followed by appropriate corrective action.
Some problems creep up so slowly that they are not detected until damage has occurred. This is a failure to perceive. This is where determining the vital test points and establishing the acceptable values is necessary. Sometimes we don’t know what is acceptable; this is where conferring with the manufacturer or others with similar equipment provides guidance.
Once the source of a problem has been decisively established, the next step is correction. The solution should treat the core problem. For example, if a one-times-rotating speed imbalance is detected, balance the rotating body. If the structure of the machine has a harmonic resonance so that the slightest excitation causes destructive vibration, address the structure rigidity.
When the corrective measures are completed, verify the anticipated improvements are perceivable. Use measuring tools to verify the correction did what it was supposed to do. This is where PdM tools shine – vibration analysis, ultrasound detection, motor current signatures, oil analysis and thermography.
None of the aforementioned efforts will produce desired results unless action is taken. Do something and document what was done. This enhances the value of the commissioning project.
Preparing for Commissioning
It is essential to become fully familiar with the equipment that is being commissioned. This means reading the manuals in advance (if possible), reading the cut sheets for components, studying the operations manual, getting the manufacturer to field your questions and contacting others with the same equipment to learn how they were successful.
It is useful to visit facilities where similar equipment is operating both successfully and unsuccessfully. Talk to the operators and maintenance people, as they have an intimate connection with the equipment.
I was talking with an electrician friend about some of the more interesting and difficult jobs we’d done over the years. We were discussing how to fit wires into a congested junction box. My friend said, “You’ve got to have a plan.” This stuck with me because it shows that even the most mundane task, such as dressing wires in a tight space, requires a plan.
Commissioning also requires a plan. A plan is nothing more than a well-thought-out set of guidelines which may incorporate rules, drawings, checklists and procedures. A commissioning plan is something like a cooking recipe; you follow it but adjust as needed based on experience.
The Plan
-
Select a project leader – one who will have the time and temperament to command resources in an effective manner
-
Define the absolute requirements for acceptable operation – what work must the equipment do?
-
Identify and address anticipated failure modes for the overall system and all subsystems.
-
Verify operating conditions (as specified by the equipment seller) will be met – i.e., operating environmental conditions (temperature, humidity, pressures, voltage, etc.).
-
Define the testing steps necessary and the order of those steps in the commissioning process. These steps are derived from experience, knowledge and research.
-
Define the skills sets necessary – mechanical, electrical, controls and engineering
-
Schedule the resources and personnel to accomplish the defined steps
-
Record the observations and test point values measured (perceived) during each step
-
Review the results (convert data into information)
-
Take corrective actions as needed
-
Verify that the corrective actions produce the desired results
-
Train the operators to make certain they will know how to operate the equipment properly in all circumstances
-
Implement a preventive maintenance (PM) program that addresses best practices
-
Repeat these steps as necessary until all absolute operating requirements have been met
-
Record the commissioning process describing what brought success and what did not for future reference.
-
Create a body of knowledge for commissioning and update it when new information is revealed as part of the commissioning processes.
The Commissioning Leader
A commissioning project also requires a leader to manage the steps and execute the plan. This leader acts as the central hub of communication for the personnel assigned to the commissioning project. Similar to root cause investigations, it is wise to not choose a dominant personality as a leader.
People of dominant persuasions can corrupt the process with their predispositions. The optimal leader is one who follows the plan and verifies all steps are accomplished and documented. This leader must also have good organizational and communication skills in addition to being assertive.
Select a competent leader who will have sufficient time to dedicate to the project without getting pulled off to yet another project. This only makes sense and is how successful enterprises advance.
Failure to Get It Right (and Knowing Right from Wrong)
It is a fact of our times that we are expected to dramatically increase efficiency and accuracy with the feast of computer-based tools that are laid before us. Commissioning is not a process that can be rushed. It must be conducted in a thoughtful manner with adequate resources. “The faster we go, the behinder we get” is a time-worn expression that embodies both experience and folk wisdom.
There are occasions when commissioning teams dig too deeply looking for problems and overlook problems that are blatantly obvious. It is important to review all available information and take all reasonable steps when commissioning. Don’t be afraid to have an independent outside observer look at your processes. Expect some root cause discoveries during commissioning.
Many problems can be anticipated before new equipment is started for the first time. Getting critical tasks into the PM program is as much a part of commissioning as any of the other steps. Any required maintenance tasks that are burdensome, time-consuming or messy tend to get put off. Anticipate problems with filters plugging and fan blades loading up with dirt and becoming imbalanced.
Lessons Learned at UOSA
During the last few years, we have learned a few valuable lessons. We have uncovered issues with the lack of a proper commissioning process as well as the benefits of having proper procedures in place. Our lessons highlight the beneficial use of predictive maintenance technologies (PdM), such as ultrasound, vibration, oil analysis, thermal imaging and motor current signature analysis, but the human senses are also part of the evaluation process.
Electric Motors in Critical Applications
Whether a motor is new from the manufacturer or reconditioned, it is essential to check its condition with a motor current signature analysis instrument as soon as it arrives onsite. If this testing is not performed immediately, it tends get forgotten or done in a hurry just before the motor is installed. Who wants to find out a motor has a problem just before it’s installed, or worse yet as soon as it is powered up?
Our experience has been that installation is a bad time to discover equipment problems, because personnel and rigging equipment are mobilized and there may be operational constraints to deal with if the motor is not installed on schedule.
A full set of procedures must be established for quality control inspection of motors, new or rebuilt, when they are received. Vendors of motor test equipment are a good resource for developing these procedures. A set of guidelines is also needed for repair of a motor to verify what was done during repair. There are many online resources on how a motor should be repaired.
Compare recommendations from various motor shops to get a broad overview. Motors are the prime movers behind industry and must be given critical attention or process failure is certain.
Basic off-line tests should be performed before a motor is connected to its power circuit. A polarization index test and a surge test are essential. When the motor is powered up for the first time, measuring the current in all three phases is vital. If online testing is possible, it should be performed. If an ultrasound instrument is available, the bearing sound signatures should be recorded to obtain baseline decibel levels and time-domain graphs for future comparisons.
If the motor can be run prior to coupling it to the driven component, it is desirable to measure and record vibration characteristics. If any of these tests reveal problems, it will be necessary to halt installation and take corrective actions.
Vertical Turbine Pumps (VTP)
The water industries frequently employ VTP for processes. These pumps have their impeller mounted in a volute sitting up to 30 feet below grade. The impeller is driven by a long shaft, which is stabilized by a number of “spider” bearings along the inside of the pump column through which the water flows. The motor is mounted above ground on a pedestal in the vertical axis.
On the suggestion of a PdM technician from a neighboring water facility, we installed two hermetically sealed vibration sensors on the volute of a new VTP prior to lowering it into the wet well. The sensors were installed in the horizontal plane, positioned 90 degrees radially from each other. When the sensors detected high vibration during initial startup, the installing contractor was surprised. They’d never seen anyone test for vibration at the volute before. The motor was not vibrating significantly, so they assumed the installation was satisfactory.
Brand New Motor was Overgreased
The detection of this vibration led to the discovery that a group of mounting bolts in the sole plate was too long and bottomed out into blind threaded holes. The split-lock washers on these bolts were not fully compressed, and this allowed resonances, which manifested as the vibration detected at the volute. We are now installing these vibration sensors on all vertical turbine pumps when they are pulled for maintenance.
We have also learned that there are a number of special considerations involved in setting up these types of pumps. The sole plate must be level and the pump column perpendicular. The sole plate must also be firmly grouted to prevent vibrations. The pump shaft sections should be within a nominal 0.005 inches of the total indicated runout.
The Hydraulics Institute (HI) vibration standards 2000 version provided a value for allowable vibration measured at the top of the motor for this type of pump. Later HI standards did not include this value. The owner must determine the allowable value. Our experience is that the pump manufacturers rarely specify a value. Generally speaking, less than 0.2 inches per second is acceptable based on our experience.
Centrifugal Blowers
A centrifugal blower (for induced draft application) was supplied with a regenerative thermal oxidizer system. The blower wheel was mounted directly on a 7.5 HP motor shaft, and a pressure gauge calibrated for inches of the water column was installed on the blower output piping. The blower vibration increased significantly after three years of operation, the motor failed and a critical process was shut down.
After the motor was replaced, vibration continued to be high. Surge resonance was suspected of causing the blower wheel to delaminate, which resulted in dynamic imbalance and the high vibration. Before further action was taken to correct the issue, the wheel jammed in the blower housing as a result of delamination. The wheel failure caused the critical process to shut down again.
During commissioning the contractor did not specify a vibration limit for this blower. When inquiring with the blower manufacturer after the failure, we found that the vibration limit was less than 0.2 inches per second. The blower was found to be operating at 0.7 inches per second before the failure, i.e., way out of specification. No one had taken an initial baseline vibration reading, so there was no good data for comparison.
The gauge reading on the output piping was never recorded during commissioning. No one knew what the gauge reading should be during normal operation. The gauge needle had oscillated violently during surge conditions until it jammed at 19 inches of the water column (WC). A new gauge was installed, and the pressure at normal operation was recorded as 39 inches WC.
The commissioning lessons learned are to record vibration and output pressure values, comparing these to the manufacturer’s allowable limits and instructing the operators on what conditions could cause damage. Trending of vibration was also added as a preventive maintenance (PM) task.
Sludge Dryer Mixer Bearing Failure
When a new sludge drying system was installed, every step of commissioning was carefully performed by the contractor. However, the large ball bearings on the mixing drum shaft still failed in a spectacular manner. They were completely lacking grease from the factory and chewed themselves up into misshaped balls of scrap metal. A simple measurement with a structureborne ultrasound instrument would have detected the absence of lubricating grease while the bearings were still salvageable. Ultrasound inspection has now been added to the commissioning processes where applicable.
Submersible Pump/Motor Units
Submersible pumps have become very popular in wastewater applications. They can be lowered into process wells where they slide into a coupling flange with relative ease and then be connected to electrical power. They are as close to “plug and play” as industrial equipment gets.
We have found it to be prudent to test the power wiring with motor current signature analysis instruments prior to connecting the pump motor to the circuit. This allows the technicians to prove that the power circuit has no resistance to ground issues which may generate variable frequency drive faults. The pump motor should receive both off-line and on-line testing.
Some of these submersible pumps have an oil reservoir for the shaft seals. As part of the PM, this oil must be pumped out and checked for the amount of process water that enters through the volute seal. The PM requirement for this check is generally around a 365-day cycle. However, when a new pump is installed, we have found that it is smart to check the water content in the oil after the first 90 days. It has revealed excess water on occasions.
Wastewater Clarifiers - Gear Drives
When the contractor installed a new, large (more than 7 feet in diameter) gear drive, they filled it with the manufacturer’s specified oil. One of our tests prior to startup was to take an oil sample. When our tech sampled the oil, it was found to be critically low in viscosity compared to the specification.
After a bit of investigation, it was learned that the manufacturer ships these gear drives half-filled with diesel fuel. Unaware of this, the contractor added oil to the fill line, which resulted in low viscosity when the oil was tested. Rigorous oil testing was added to the commissioning checklist of all clarifier drives.
Rotary Screw Compressors
These types of compressors offer energy conservation benefits but come with important maintenance responsibilities. Particularly, the oil condition must be closely monitored. Satisfying the warranty requirements often relies on being able to prove preventive maintenance was performed.
Many rotary screw compressor manufacturers require owners to send oil samples to their designated oil test lab. Failure to do so may void the warranty. Commissioning of these compressors includes making certain a vigilant PM program is put in place with special attention to oil analysis.
After Commissioning
After commissioning, periodic checks must be done to verify that operations is running the equipment in a way that prevents failures. The best commissioning process will fail if the operators try to use the equipment in ways for which it was not intended.
Verify PM procedures are correctly performed or not performed at all. Failures resulting from these quarters may be unfairly attributed to faults in the commissioning process. You should review metrics, key performance indicators (KPIs) and PdM trends on a regular basis to verify the equipment is operating within specifications.
Final Thoughts
To some degree, equipment commissioning is educated guesswork when first attempted. The guessing becomes better with each experience and as more information is added to the body of knowledge. There will be a good deal of root cause discovery. In reality, what gets done is common sense. With each project, the body of knowledge grows to become a better reference for future commissioning projects. Equipment commissioning truly is a valuable art.
