Once upon a time in the land of Ooze (where sticky, gooey hydrocarbons flow through pipes and vessels), there were two processing plants that boiled oil to make fuels and various other valuable petrochemical products. On one side of the river, rests a site called Perfecto Process Plant (PPP), while just across the river lies another plant called InZayna Zylum (IZZ). Perfecto Process Plant is rated 9 out of a possible 10 points on the fixed equipment mechanical integrity (FEMI) scale of operating excellence, while InZayna Zylum is rated a lowly 1 on the same scale of 10. As you might expect, they have very different approaches with regard to inspecting and maintaining their fixed equipment assets.
After reading about the many differences between Perfecto Process Plant and InZayna Zylum, owner-operators will have an opportunity at the end of this article to rate their own site performance in terms of where each of you believe it stands on the FEMI scale of excellence. Rest assured, your responses will be kept confidential and all eligible participants will receive a summary of the responses. So be thinking about your own site(s) as you read this article.
The goal of this article is to help readers identify where their facility stands on this FEMI spectrum. And while I fully expect the majority to fall somewhere in between these two sites, all facilities should be looking for ways to improve, to be better than they were before, and to get one step closer to FEMI excellence.
The Orchestra and the Hockey Game
First, I have a couple of analogies I like that seem to fit these two different companies. PPP is like a premier, world-class orchestra with the fixed equipment inspection manager waving his wand in front of very capable, competent musicians who are making great music from sheet music that has been well planned and rehearsed. While at IZZ, they have a bunch of rag-tag street musicians banging on pots and pans making noises.
Another analogy that seems to fit is that of an ice hockey game. Over at PPP, the hockey team has a strategic team plan with many different well-designed plays for getting the puck into the back of the net and remaining on top of their division (strategy and tactics). While over at IZZ, there is not much of a strategic plan – all of the players are simply slapping the puck around the ice all day long hoping they can score occasionally. At 5 o’clock when the bell rings, the puck is left somewhere out there on the ice until the next morning when the team returns and starts banging it around again (lots of tactics without any real strategy).
For those of you who read my previous article entitled, The Three Tiers of Most Fixed Equipment Mechanical Integrity (FEMI) Programs (1), PPP would be located up near the top of Tier 3 illustrated below in Figure 1, while IZZ would be struggling to get out of the lower Tier 1.
In each of the important FEMI topic areas below, I will briefly describe various performance characteristics of these two very different plants (PPP & IZZ), which are on opposite ends of the FEMI performance spectrum. I have witnessed these performance characteristics at various operating sites while participating in over 75 FEMI performance assessments worldwide; some characteristics of which you may recognize at your own operating site.
Management Leadership for FEMI
At PPP, the site leadership team understands and supports the entire depth and breadth of the necessary FEMI program – all 101 Essential Elements(2) and the role that every department plays. Plant management is committed to and holds the Operations and Maintenance Managers accountable for the quality of the FEMI programs at the site and doesn’t solely put the inspection group under the gun if and when a leak occurs. There is truly a team effort and a lot of cross-functional cooperation. The FEMI group supports those primarily responsible for the integrity and reliability of fixed equipment. Management understands that gaps (holes in the Swiss cheese) in the “Swiss cheese” model(3) can allow FEMI incidents to occur, and therefore emphasizes the need for maintaining all layers of protection (hardware-based and human systems-based) to avoid FEMI related incidents.
Meanwhile over at IZZ, the Maintenance Managers/Supervisors have some of those old-time, maintenance cowboy personalities that want to run roughshod over inspection and FEMI personnel. They demand things like ending a TAR on time no matter what work is left to do, send contract inspectors offsite before all the necessary records are adequately documented, and make arbitrary decisions to decline some inspection repair recommendations. At IZZ the advice and counsel of the FEMI group is not highly regarded. In fact, they often regard the FEMI group as a burden instead of an asset.
At PPP, because of their high reliability, the FEMI group has plenty of time available for pro-active approaches to continually improving their asset integrity management (AIM) practices because they are not always reacting to problems as they arise. While at IZZ, the FEMI group struggles to keep up, having to respond to one FEMI problem after another reactively (i.e. fighting fires (both literally and figuratively) because they cannot get the necessary support and budget to do be more proactive).
At PPP, they have an enlightened senior management team that recognizes the risks associated with their fixed equipment and as such, they are more than willing to spend their resources wisely to improve the results of their AIM program. While at IZZ, the management team is constantly looking for ways to control/reduce their FEMI budget since “FEMI investments don’t seem to make any money for them” One of the reasons for that is because at IZZ, the FEMI group is made up of too many employees just collecting a paycheck, doing what is expected to keep their management satisfied by chasing things like buried firewater piping leaks, heat exchanger tube leaks, being “hey boys” for maintenance personnel and reacting to other “hot potato” issues of the day as determined by operations and maintenance personnel. Over at PPP on the other hand, a highly skilled, high performing team of FEMI engineers and inspectors with a real passion for FEMI, keeps their management informed of their highest FEMI risks(4), their progress toward mitigating those risks, and alerts them of emerging risks.
Excellence vs. Staying in Compliance
At PPP, where they have tied FEMI excellence to the bottom line, the Inspection Manager has a five year plan for continually improving PPP’s FEMI performance. This plan is updated each year to provide justification for budget funding for the next phase of the proactive FEMI risk reduction plan. Meanwhile at IZZ, the Inspection Manager just accepts what budget funding he is given and tries his best to “get the job done” with his budget.
At PPP, the management team is oriented toward achieving excellence in FEMI (top circle in Figure 1) in order to have their fixed equipment always available to meet the business plan. While at IZZ, the management team is simply oriented toward “staying in compliance” with rules and regulations; whatever that means. I always liken the “staying in compliance” mentality toward climbing only to the third rung on a ten rung ladder, while achieving excellence in FEMI is at the top of that ladder.
Shared Stewardship of Assets
At PPP there is a clear philosophy, proactively led by the plant manager, that equipment integrity is everybody’s job; not just the responsibility of the FEMI group manager and his team. Operating and maintenance personnel have equipment integrity responsibilities written into their job descriptions. For example, maintenance foremen are responsible for the quality of repair work (e.g. less than 2% weld reject rates), operators are held responsible for staying within a comprehensive set of IOWs, and operations are responsible for quickly communicating any/all operating upsets, deviations, and abnormalities to the FEMI group for potential follow-up. Additionally, at PPP process engineers work closely with the FEMI group to assess any changes that they envision for operating conditions to increase yield or operability that might impact equipment integrity. Each process area manager at PPP conducts a quarterly meeting with representatives from inspection, operations, process engineering, and reliability to discuss the top ten FEMI risks in each process unit and plan the appropriate mitigations to reduce those risks.
At IZZ, pretty much every group works in their own silo, doing the best they can with their assigned work, but not communicating very well with the FEMI group. As a result, too many leaks and areas of accelerated corrosion find them, forcing unexpected repairs, replacements, and clamping. Then, by-and-large, the senior management team holds the FEMI group responsible for those leaks and unscheduled repairs. At IZZ, they don’t track their top risks in each area and aren’t even sure what they are.
Documented FEMI Management Systems
At PPP, they have an effective repertoire of documented FEMI management systems (MS) and work practices (WP), which expand to cover all the vital AIM activities needed to achieve excellence in FEMI(5). This first class set of documented procedures and practices serves them well to provide consistency across the entire plant and sustainability over the long term despite the usual employee turnover. At PPP the FEMI team reviews and refreshes these MSs and WPs every three years to make sure they are kept up-to-date and are the best they can be. Periodically, PPP audits themselves on key areas of performance to provide assurance that each documented MS is being properly implemented.
Meanwhile, over at IZZ, they don’t have time to document their MS. They just get out there each day and “get after it,” doing what operations and maintenance wants them to do. “After all,” management thinks, “because we have certified API inspectors – they know what they have to do – they don’t need procedures.” Hence, at IZZ they have only a few very generic procedures that may be out-of-date and not very useful in terms of specifics and guidance. Some of the inspectors don’t even know where to find them. But those inspectors stay busy, busy, busy. Site management does not place a high value on having a robust, comprehensive set of documented FEMI management systems at IZZ. Their attitude seems to be “we hire smart people and certified inspectors – they know what to do.”
At PPP, the site is moving rapidly into full implementation of risk-based inspection (RBI) for pressure vessels, piping, and aboveground storage tanks. Their RBI process is complimented by the existence of comprehensive corrosion control documents (CCD) for each process unit, containing all the information shown in section 10.2.2 of API RP 584(6, 7). CCDs are especially helpful in designating where each different damage mechanism might occur, what is driving it, and how it can effectively be inspected and controlled. Hence, PPP is enjoying the benefits of being intensely focused on the higher risk equipment with detailed inspection plans for each piece of equipment, while reaping the cost benefits of longer inspection intervals on lower risk equipment.
Meanwhile, over at IZZ, the site is still using time-based and rule-based intervals, built mostly on historical records rather than a thorough understanding of each damage mechanism that might afflict their fixed equipment. Much of their equipment is being opened for inspection at each TAR, with instruction plans that aren’t much more than: “Open/clean/inspect.” IZZ is also significantly over-using spot-UT for thickness monitoring without understanding that their chance of finding most localized corrosion damage with that technique is slim-to-none. They simply don’t know what they don’t know.
Integrity Operating Windows (IOW)
At PPP, each process unit has a comprehensive, high quality set of IOWs implemented for each process unit in accordance with the new API RP 584(6). Those IOWs were identified with the assistance of subject matter experts (SMEs) in the process unit’s technology, operation, corrosion, and inspection history. Each IOW is classified as Critical, Standard, or Informational, depending upon the risk of failure associated with each. And each has a specified operator action associated with it that varies from immediate operator corrective action to control the process variable or shut down, to the other end of the spectrum - simply notifying a designated SME that an exceedance has occurred. All IOW exceedances are reviewed by an appropriate and designated SME on a daily basis.
Over at IZZ however, they have heard about IOWs and know they need to do them someday, but don’t have the budget or specific plans with milestone targets in place yet for accomplishing the task. As a result, operators are not yet aware of all the process variables that need to be controlled in order to avoid accelerating existing damage mechanisms or introducing new ones. Hence, IZZ has more than their share of leaks.
Management of Change (MOC)
Both PPP and IZZ are able to recognize physical changes to hardware and process them through MOC(8). However, PPP does their MOC with highly experienced SMEs who understand the issues and can recognize the risks associated with each. While at IZZ, MOC is a somewhat undervalued work process run by less experienced personnel who are just going through the motions.
The bigger difference between the two sites though, is in conducting MOC for process changes. With a very functional IOW program in place, PPP knows clearly what a process change is in order to compare it to a documented IOW, and puts anticipated changes in process variables through MOC utilizing experienced process engineers and operators. Meanwhile at IZZ, they are not really sure what changes in operating variables need to be processed though MOC because they don’t yet have a full set of comprehensive IOWs, with the exception of some legacy design operating limits that have been in place for many years. Hence, some changes and deviations are made in process variables that eventually lead to accelerated corrosion or other damage mechanisms.
Inspection Data Management Systems (IDMS)
At PPP, they are using a high end IDMS with extensive search and reporting features to help them understand and track all aspects of their data/information collection(9). Data quality is excellent because of a focus by each experienced inspector who frequently analyzes data for any anomalies that might creep into the database. There is one IDMS “super-user” on site to help with unusual issues, but all inspectors are capable of understanding and manipulating the data, using the reporting and search features, and leveraging all IDMS modules. Today, there are very few hard copy records needed, as everything that was not in the IDMS for the past 10 years has been scanned into an inspection shared drive linked to the IDMS. NDE technicians taking thickness data are not only certified, but well-trained in all of the variables that can affect thickness data quality. All baseline and construction data/information is entered and the inspectors have great faith in its accuracy. Inspectors enter and analyze their own data instead of it being entered by a data clerk who doesn’t really understand when he/she is entering questionable data.
Meanwhile, over at IZZ, the site struggles with its record-keeping systems because of data quality issues, missing and unorganized files, hard copy data that has not yet been entered, and a lack of inspector skills in data analysis and manipulation. Hence, record-keeping issues are impacting effectiveness and efficiency of the whole inspection work process. Record keeping tends to vary somewhat from inspector to inspector as there are no documented requirements for where all inspection records are to be kept, as well as how each inspector must record and analyze the inspection data and information. There are deficiencies with the security and organization of hard copy files and numerous files are missing.
Special Emphasis Inspection Programs (SEIP)
At PPP, nearly all of their catch-up SEIPs have been completed and successfully merged with run-and-maintain inspection programs, including those SEIPs for CUI/CUF, Soil-to-Air Interfaces (SAIs), HTHA, Low-Si Sulfidation, Retro PMI, injection points (IPs), mix points (MPs), deadlegs (DLs), and critical check valves (CCVs). For example, all IPs, MPs, DLs, SAIs and CCVs have been identified and have individual inspection plans, individual tag numbers, and a few rounds of inspection histories recorded. CCVs are identified with the help of a process hazard analysis (PHA) and the PHA team knows which valves must operate successfully in order to avoid a process safety incident.
Over at IZZ however, some of these programs are still in the early stages of identification and/or completion, and at current funding levels may not be completed for a few more years. Not all IPs, MPs, DLs, and CCVs have even been identified, let alone inspected. CCVs are not connected with the PHA program. Retro PMI has not been completed for the higher risk applications, like the potential for inadvertent substitution of carbon steel into a low alloy Cr/Mo piping system and some of the records of previous retro-PMI are not adequate or cannot be located. Buried piping is not drawn up on layout drawings and the inspection program for SAIs is lagging. Inspection for corrosion under fireproofing (CUF) is purely reactive when rebar or underlying structural corrosion causes refractory spalling.
Bolted Joint Programs
At PPP, they have well-defined bolted joint assembly programs with documented procedures and work process based on ASME PCC-1 to make sure that all critical joints are properly assembled, tightened, checked, and tested prior to reintroduction of process fluids. All critical joints are identified and provided with the proper torqueing or tensioning values and procedures. Gasket selection is carefully managed to ensure that only the right gasket type and size are actually installed and then verified. Owner-user and contract fitters use the same carefully applied procedures. And finally, there is a documented QA/QC and leak testing work process to inspect and sign off on all bolted joints that have been reassembled by multiple stakeholders to ensure that they are properly assembled and leak tight. As a result, PPP encounters very few leaky joints after each startup.
However, over at IZZ, they don’t have a rigorous torqueing practice. The joints that get torqued or tensioned are often the result of undocumented tribal knowledge by a few senior people, or left up to the bolting contractor to handle. Most joints are still “rattled tight” or “fitter tight” and the site has little or no knowledge of ASME PCC-1. Operators go out and “ping” bolts prior to startup and look for the proper gasket color code, but still find loose bolts at this stage. As a result, the potential for joint leakage on startup is higher than it could and should be.
At PPP, the QA/QC programs for welding, NDE, refractory materials, coating applications, etc. are all highly structured, documented, consistent, and producing excellent results. The FEMI group handles the QA portion of the work processes (i.e., setting requirements and doing audits), while those actually doing the work are handling most of the QC aspects. For example, maintenance employees are fully capable of witnessing and documenting hydrotest results without any hand-holding from inspectors. Maintenance employees can summon radiographers to do the specified RT shots for welding QC and make appropriate repairs, while inspectors read and approve the final radiographs. A NACE level 2 or 3 coatings inspector is able to handle any critical coating applications.
Across the river at IZZ, inspector time is largely consumed with a lot of lower-value QC activities, instead of focusing on the many in-service inspection activities covered in the API Inspection Codes/Standards and the 101 Essential Elements of FEMI(2), such as inspection data approval and analysis, inspection planning, record-keeping improvements, risk analysis, NDE contractor overview/auditing, etc.
Continuous Improvement Programs
At PPP, because of sufficient staffing and high reliability, there is plenty of time to focus on continuous improvement of FEMI activities (10). As such, PPP has established high value FEMI KPIs that they follow on a monthly or quarterly basis to measure and track progress and improvements in their FEMI program. Members of the PPP FEMI group endeavor to network with their colleagues at other company operating sites and nearby process plants to exchange experiences and learn from each other. All FEMI leaks or near-misses are subject to one of the three levels of investigation outlined in API RP 585(11). There is a process in place in which the entire FEMI group participates to audit the effectiveness of several of their documented FEMI management systems.
At IZZ however, they are so busy with the daily grind and reacting to problems that they have little time to focus on continuous improvement. They try to track overdue equipment inspections, but their IDMS data quality is not as good as it needs to be in order to ensure that no inspections are actually overdue. They don’t track overdue inspection work requests because they are entered directly into the site CMMS and inspection does not have control over that work process. They don’t interact much with their colleagues in other process plants to discuss lessons learned. API RP 585 has not yet been implemented and there is little or no auditing of the few documented FEMI management systems in place.
At PPP, the FEMI group is keeping up with the many revisions and new standards coming out of the API Subcommittees on Inspection and Corrosion/Materials(12). Whenever a new API standard or new edition of an existing standard is published, the FEMI group at PPP holds a training session led by one of their own where they spend 1-2 hours reviewing the latest editions of the standards. As a result, all inspectors have a thorough understanding of the contents of API RP 583(13) on CUI, API RP 584(6) on IOWs, API RP 585(11) on Investigation of FEMI Leaks and Failures, as well as the 45 substantive changes in the 10th edition of API 510(14). Each inspector and engineer keeps a tabbed, highlighted copy close at hand of each updated standard that applies to their job.
Meanwhile at IZZ, they have heard about these new and revised standards, but have not yet had the time or inclination to study them in any detail and apply all of the good information in them. Hence, at IZZ they are missing out on significant advances in FEMI technology and methodology put together by some of the leading minds in the industry. At IZZ, the inspectors even struggle to find copies of the latest edition of each API In-Service Inspection Standard.
Advanced NDE Applications
At PPP, the site is utilizing advanced NDE techniques for a number of applications like:
- phased array UT for nozzle weld inspections,
- pulsed eddy current for locating CUI,
- guided wave UT for soil corrosion on piping buried under road crossings,
- real-time radiography for finding welds under insulation and CUI,
- smart pigging of furnace tubes,
- IRIS for heat exchanger tubes,
- laser scanning of their coke drums,
- and many more advanced techniques which can cost-effectively detect and/or size localized corrosion and cracking.
In order to properly apply these more advanced NDE techniques, PPP is consulting with independent NDE SMEs who advise them on the best technique(s) to use, including their advantages and limitations.
Over at IZZ, they are using liquid penetrant testing (LPT) and magnetic particle testing (MPT), but are not keeping up with most of the more advanced NDE methods, and are still over-using spot UT for nearly all inspections; including where localized corrosion may be occurring and spot UT has little or no chance of finding it. IZZ is not even using sufficient profile radiography on small bore piping and deadlegs.
Training, Qualifications, and Performance of Inspectors
At PPP, the FEMI folks really enjoy their jobs, are constantly making good progress, and are well-recognized for their contributions to the site’s overall performance. As a result, personnel study on their own time to learn more about fixed equipment mechanical integrity. Higher performing candidates are standing in line to get a job at PPP, while at IZZ, they find it difficult to recruit the best talent because of their reputation. At IZZ, FEMI personnel are always struggling to keep up with the daily grind and always feel under the gun whenever something leaks or fails.
At PPP, site management recognizes that FEMI jobs are very technical, and high performance in those jobs is necessary for maintaining a high level of process safety. As such, there are always sufficient funds in the budget to continue training inspectors, and for comprehensive on-the-job training by highly experienced and competent senior inspectors. Over at IZZ, training budgets are often the first to get cut, so inspectors and other FEMI personnel are not able to keep up with new developments and technology advances.
Inspection Work Requests (IWR)
PPP has a fail-safe work process for all IWRs that are entered into the CMMS in order to track each one to completion. The master database for IWRs is kept in the FEMI group so that none can be changed, deleted, or marked complete until the inspector that wrote the IWR is satisfied that it has been properly completed with all appropriate QA/QC. Any deferred or rejected recommendations are signed off by the IWR originator or a manager of appropriate responsibility level.
At IZZ, the inspectors enter their work orders directly into the CMMS and then are dependent upon those in maintenance to manage them to completion. But they also know that some get changed, some get deleted, and some get put on a “forever” backlog list without their approval and knowledge of how they are being handled. As such, inspection does not have good control over which IWRs are getting properly handled and which ones are not.
Engineering Support for Inspectors
At PPP, the FEMI group includes a mechanical engineering and a corrosion/materials engineer working “hand and glove” with inspectors to recognize and resolve equipment integrity issues before they result in reliability impacts. The engineers sit in the same area as the inspectors, which facilitates a close working relationship. When they need extra SME input on unusual or complicated integrity issues, they are able to readily get that help either from a corporate group or engineering service contractors that have knowledgeable, experienced FEMI support personnel.
At IZZ however, the inspectors and engineers work in separate departments and don’t work very closely together. There is no qualified corrosion engineer at the site. Sometimes inspectors have to submit a work request for engineering assistance when they believe they need it and sometimes that assistance is slow in coming. As a result, inspectors often do not have expedited access to an SME when they really need it.
By and large, Perfecto Process Plant (PPP) has very few leaks and hazardous material releases because of the degree of excellence that they are maintaining in their FEMI program. Whereas, over at InZayna Zylum (IZZ), they still have too many leaks, occasional fires, near-misses, and surprise findings, but have been lucky so far that they have not had a really significant FEMI incident. Over at PPP, they strongly believe that it is “better to be good than lucky”.
- Reynolds, J., 2014, “The Three Tiers of Most Fixed Equipment Mechanical Integrity (FEMI) Programs,” Inspectioneering Journal, 20(3), pp. 11-14.
- Reynolds, J., 2015, 101 Essential Elements in a Pressure Equipment Integrity Management Program, Inspectioneering, LLC, The Woodlands, Texas.
- Alvarado, G., Nichols, M., 2014, “Key Performance Indicators – Understanding is Key,” Inspectioneering Journal, 20(6), pp. 19-22.
- Reynolds, J., 2007, “Why Some Sites ‘Just Don’t Get It’,” Inspectioneering Journal, 13(3), pp. 17-20.
- Reynolds, J., 2012, “Site Procedures and Work Processes for Achieving Excellence in Pressure Equipment Integrity and Reliability,” Inspectioneering Journal, 18(1), pp. 8-19.
- American Petroleum Institute, 2014, “API RP 584 - Integrity Operating Windows,” Washington D.C., First Edition.
- Reynolds, J., 2012, “Corrosion Control Documents – One High Priority Approach to Minimizing Failures of Fixed Equipment,” Inspectioneering Journal, 18(5), pp. 6-9.
- Reynolds, J., 2010, “Integrity Operating Windows and Management of Change for Fixed Equipment Integrity,” Inspectioneering Journal, 16(2), pp. 4-9.
- Reynolds, J., 2012, “The Role of Record Keeping and Data Management in Achieving Excellence in Pressure Equipment Integrity and Reliability,” Inspectioneering Journal, 18(1), pp. 8-19.
- Reynolds, J., 2012, “The Role of Continuous Improvement in Achieving Excellence in Pressure Equipment Integrity and Reliability,” Inspectioneering Journal, 18(4), pp. 14-23.
- American Petroleum Institute, 2014, “API RP 585 - Pressure Equipment Integrity Incident Investigation,” Washington D.C., First Edition.
- Reynolds, J., 2011, “The Role of Industry Codes and Standards in Achieving Excellence in Pressure Equipment Integrity and Reliability,” Inspectioneering Journal, 17(4), pp. 7-13.
- American Petroleum Institute, 2014, “API RP 583 - Corrosion Under Insulation and Fireproofing,” Washington D.C., First Edition.
- American Petroleum Institute, 2014, “API 510, Pressure Vessel Inspection Code: In-Service Inspection, Rating, Repair and Alterations,” Washington D.C., Tenth Edition.