Can You Diagnose Problems in an Industrial System with No Observability?
I spent part of a summer on top of a little place in northern Idaho called Moose Mountain. It came with several families of moose who spend part of their year with us, grazing on our lawn. Counting the road up the ridge from the prairie floor, there are eight families, each with some lawn or other cultivation that enjoys regular irrigation, all supplied by a well at prairie-level.
Can you imagine how disappointed everyone, including the moose, was when our always-adequate water well, feeding our mile-long pipeline supplying our top-of-mountain cistern, just couldn’t get the job done? A similar situation occurred in prior years, resulting from a leaking coupling in the PVC pipeline buried four feet below the surface along a rambling path up the mountain. Speculation centered on the similarity of this present situation and the previous one. Was it another instance of the same sort of thing?
How hard could this be? A submersible pump, controlled by level in the cistern, cycles on and off to produce a normal flow of about 10 gallons per minute up a 2.5-inch PVC pipe. Last time, the leak reduced flow to about 2.5 gallons per minute, carefully measured by means of a bucket and watch at the cistern.
Pressure was described by the homeowner who usually took care of the well as “normal,” which meant the pointer on the only pressure gauge located at prairie level pointed at 150 psi. Fixing another pipeline leak would involve figuring out where to dig, digging down four feet to the pipe, and digging along the pipe until the leak was found. Last time, an acoustic device helped get to the leak quickly once we were close, and “close” identified itself by an amazing growth of grass and plants in a place normally devoid of that stuff. There were no such indicators evident this time. Three-quarters of our normal water flow weren’t making it to the cistern at the top of the mountain. Thus began a drama of observability.
Both people who worked on the last leak and understood the system were gone; one moved and the other finished his work here. Without any memories, notes, or records about “normal” pressure, it became apparent no one knew how much it should take to get the water to the top of the mountain, a rise of about 400 feet above the prairie. Static pressure should be about 173 psi and more would be required for normal flow, yet flow (albeit not much) was reaching the cistern with the gauge indicating exactly 150 psi, which was understood to be “what it always showed.” Hmm.
All costs are shared by the eight owners, and this kind of work gets expensive very quickly. This motivated a search for the most likely and least expensive solution. This could be a water table problem, a pump problem, a well problem, or a pipeline problem.
The contractor who installed the pump a few years earlier was called. Their crew lifted the pump and its piping up and out of the well casing to make sure all was “as installed.” Well water level was checked and determined to be higher than when the pump was installed, thus making the pump’s job easier. If the problem wasn’t at the pump, it must be in the pipeline, so attention focused there.
Think about observability. We had a single pressure instrument of unknown accuracy or calibration and no way to test it. We “bucket measured” flow at the cistern. Anecdotally, all conditions were more or less the same as they were for the last pipeline leak. The contractor that sold and installed the pump assured us there was nothing wrong with it. He suggested that perhaps we needed a bigger pump, even though the one he’d sold us had been adequate for three years of unchanging use. Hmmm.
The first lesson in all this is to ask the burning question: “Is anything that we think we know actually true?” The second question, of course, is the economical way to proceed on the basis of what we know and the relative cost of investigating other things.
The homeowner who had taken over resolution of this, a wonderful neighbor with a beautifully cultivated yard that was dying from dehydration, ran the pump to fill the pipeline to the cistern, then shut it off and watched the pressure decay. He observed the pressure decay rate seemed to decrease to a head about 100 feet above the prairie. Unfortunately, the need for domestic water cut short his test program before it could be determined if there were other similar occurrences along the way to stabilization.
The guy installed an iPhone elevation app, calibrated it against the pressure gauge, and determined the elevation of the probable leak location. A thorough inspection found no evidence of plant growth or erosion. Another contractor was brought in to do a gas injection test for a leak, but his results were inconclusive. He wanted us to install some valves to segment the line into smaller intervals. During all this, expenses for each of the eight homeowners mounted and patience began to wear thin. Can you imagine if this were an off-shore leak, and the irritation were not dying grass, but oil-soaked birds and dead fish?
Another homeowner brought in a relative who had a business some distance away that dealt with systems like ours, but mainly larger and more complex ones. They sent a good and experienced crew who did this work daily under diverse conditions. They thought about the pipeline leaking, but wondered why a static column of water, isolated from the well by check-valves, should drain all the way to empty in less than a day. Quick investigation determined the bottom-of-pipeline valve was leaking back into the well. Interestingly, the previous inspection missed that.
The crew leader also estimated (aided by extensive experience) that about 190 psi of pressure would be required at the pressure meter location to produce our historically normal ten gallons per minute at the cistern. He had his crew pull the pump-and-pipe up again, section by section. He ran the pump with each section looking for some leak at a joint on along the riser pipe. A section above the pump shot a stream of water out through a hole in the pipe. The contractor replaced the defective pipe section, tried again, and all worked as it always had. Our normal flow rate was restored, no new pump was required, and evidence of a pipeline leak evaporated. The hole in the pipe involved no evidence of corrosion and was obviously external impact, which somehow occurred on a 2.5-inch steel riser pipe inside a 4-inch casing that showed no damage. The pressure gauge was recalibrated and displayed 196 psi when the line was running at normal flow with the current water table.
There are huge lessons here.
- First, if you are going to run a fluid transfer system, know what you are doing.
- Second, be sure your equipment is in good condition.
- Third, be sure the responsible person (or people) know what they are doing and why.
- Fourth, competently observe the work of contractors.
- Fifth, it is usually easier and cheaper to resolve new problems while they are small, rather than in an emergency when they become unignorable.
It all seems simple, but transgressions in these areas, especially meaningful observability and consequent analysis, can make a small problem difficult and expensive. Also keep in mind that the cheapest help you can get might be the most expensive path to solving the problems.
I’m used to situations in which I can see real-time data sampled as often as 0.1 second intervals on a computer screen that presents trends and diagnostics. It’s a long trip back to having only one instrument that may or may not be broken; challenges become evident. The real challenge, though, is maintaining an easy-to-understand service environment. Concentrate on that.
In any case, it was an interesting diversion in a delightful summer-the “homeowner” version of an industrial pipeline failure. Fortunately, it’s a good group of people, and they maintained sensitive and productive efforts to bring everything to conclusion. It was a good lesson in the importance of keeping track of things you’ve learned or relied on in the past.
Originally published at https://blog.isa.org.