Data centres: Why They Need Water Leak Detection Systems

Steve Dilloway writes for Kent’s Invicta Chamber of Commerce on 5^th April 2019, and he cites research that shows “the consequences of total data loss means that 43% of companies never open up again”…and he writes that 29% of the companies that do recover end up closing 2 years afterwards. Therefore, water leaks pose a serious threat to business survival and business continuity. He also finds that water damage is the cause of insurance claims in hotels, retail premises and commercial buildings, and that it is costing the UK insurance industry upwards of £800m. Yet, in fact, the Association of British Insurers (ABI) puts the cost of water leak claims as being much higher than this figure.

Be proactive

Michael Wakley, CEO of water leak detection solutions provider Leaksafe, says the high cost of Escape of Water claims in the UK, over £930 million in 2018 according to the ABI, is leading insurers to “actively encourage their policyholders to proactively manage risk in their properties.” Nowhere is this more crucial, given that data is the new gold and what everybody’s lives revolve around today, than in datacentres.

“In our arena, the quicker a leak is identified, the less damage and disruption is caused, at least cost to owners and insurers.  Our LPWAN and Narrow Band Internet of Things (NB-IoT) technology-based leak detection and Remote Risk Monitoring (RRM) systems will identify even a drip leak, and immediately notify owners or property managers of its exact location so that can act swiftly to prevent further damage.”   

He elaborates: “Without the use of these technologies, leaks could persist for some time until they finally become noticed either visually or by odour, by which time significant damage may have been caused to the fabric of the building.  The location-specific data Leaksafe’s RRM system provides, cuts down on the amount of time needed to investigate where a leak has occurred; and in most cases, when caught early, the cost of repairing the leak and reinstatement will be far less.” 

The data emerging from NB-IoT devices over low-powered wider area networks (LPWANs) can be incorporated in property owners, property and facilities managers own management and maintenance platforms. Wakley says this permits them to have a consolidated and co-ordinated approach to maintenance issues. This potentially includes datacentres.

Avoid loss

Chris Mackenzie, Senior Risk Analyst and Paul Reddington, Major Loss Property Claims Manager at Zurich Insurance elaborate: “The loss risks associated with data centres relate to the physical loss of equipment, but also to the loss of business following that loss and any potential financial penalties that may be applicable. Also, the impact of a system or equipment failure involves the loss of public image and general concerns about information privacy and loss.”

“Data centres are becoming increasingly high tech, highly controlled environments, reflecting the complexity and sensitivity of the equipment they contain. Data centres require precise management and have to be protected from fire, smoke, flood, moisture, temperature change, dirt and pollution.”

They rightly point out that, today, most datacentres operate out of purpose-built facilities, and the vast majority of them will have leak detection systems fitted as standard on “chilled water piping, air conditioning units, cooling towers etc. which are centrally managed and linked to the business mainframe system for process control and monitoring.”

There are nevertheless many datacentres or server facilities that have been located in buildings or in rooms that have been retrofitted to accommodate IT activities and equipment. These rooms are often, they explain, “located below or adjacent to plant rooms where water sources, or tanks may be present, or where hot or cold water feeds may flow.” Plumbing and air-conditioning are, however, not the only sources of potential water damage. They can also suffer from water damage from leaking roofs and blocked guttering.

Plan for the unexpected

Yet Zurich have found that water damage has not been fully considered in many incidences. This leads to the risk of escape of water and leaks generally being unmitigated. They therefore advise: “It is critical that whenever data centres / sever rooms are designed or retrofitted, or wherever they exist, that a water damage risk assessment is conducted to consider the level of risk present and which mitigation measures are to be applied; a risk assessment should be conducted and reviewed periodically and when change happens.”

Exploit the benefits

Zurich Insurance says there are several benefits associated with leak detection systems (and other sensing equipment) including;

·         Data analytics: Many modern leak detections systems include smart applications that offer occupant, owner or facilities manager the opportunity to capture and analyse water flow data, in order to assess water consumption.

·         Early intervention: Leak detection systems indicate a problem in the early stages, so it can be stopped and or investigated before it become serious and significant.

·         Reduced number of incidents: Leak Detection systems highlight problems in the plumbing infrastructure, which are then subsequently repaired, resulting in healthier infrastructure and the reduction in number of incidents experienced.

·         Maintenance program: Leak detection activations can highlight an unhealthy plumbing system, helping to steer maintenance programs and prioritise budgets.

·         Favourable insurance terms: Comprehensive and effective leak detection systems, proven to reduce losses, will be viewed favourably by insurance companies.

·         Client assurance: companies with data centres storing client data and having comprehensive leak detection systems, offer an increased level of assurance to their clients on the safety of their data.

·         Business continuity.  Datacentres and sever rooms protected by leak detection systems are more resilient to water damage and business interruption as a result.

Consider external threats

David Trossell, CEO and CTO of Bridgeworks says during his experience in the IT industry, he’s seen “a fair number of incidents with water in in computer rooms mainly from air conditioning units with blocked condensation pumps.” Thankfully, in most cases he explains that floor cavity water sensor saved the day, preventing any calamity. Yet there is one incident that sticks in his mind. He describes it as being more dramatic than other incidents he’d seen before, and it was caused by an external source. It’s therefore, he argues, a good lesson in change management.

“I was called out in the middle of the night with a request from an international courier based in Heathrow, which asked: “How much humidity can the systems withstand before it becomes a problem? After a lot of questioning, they finally admitted that the computer room was flooded and there was an immediate need to restore the systems as nothing could be shipped and they had planes waiting.”

“When I arrived, it was total and utter chaos! Floor tiles were up and the whole of their IT team were using wet ‘n’ dry vacuum cleaners (they had bought every vacuum in a twenty-mile radius) removing the water from the floor cavities. The first problem to tackle was to replace some of the floor tiles to stabilise the floor which was ready to collapse.”

The incident was caused by poor planning and poor change control. He adds: “Three floors above the computer room, the offices where being changed around to a new layout and one of the sprinklers had to be moved. For those unfamiliar with this procedure, they freeze the pipe around the sprinkler to prevent water flow. In this case they froze an air pocket.”

“What happens is the plumbers freeze the pipe (and the water inside of it) to act as an “ice plug” to stop the water coming out of the sprinkler system. Many insurance and fire brigades do not like you turning off the stopcock to the sprinkler system just in case there is a fire. Hence, they freeze the pipe by the sprinkler outlet so they can change or move it. If there is an air pocket instead of water when they freeze the pipe to form the ‘ice plug’, then when they remove the sprinkler outlet there is nothing to stop thousands of litres of water spewing out at high pressure.” The issue was compounded because nobody knew where to find the stopcock for the sprinkler system.

He says it took everyone involved over 30 minutes before they found the location of the stopcock, which was far too late. By the time they found it, he claims that “water had flowed down three floors and into the computer room.”

So, from his perspective, the moral of the story is that while organisations often focus on the threat within datacentres, there needs to be more planning to consider the unexpected external threats because they are the ones that can often be costly from a financial, reputational and business continuity angle. These incidents often cause the most chaos too, and they are difficult to plan for.

Find solutions

Bryan Powell, Senior Loss Prevention Consultant - XL Risk Consulting, at AXA XL offer some possible resolutions: “With detailed and up-to-date pre-emergency and business continuity plans, the business interruption could be drastically reduced to the point where any one occurrence will have minimal effect.”

“The detectors should be positioned directly below the computer-room air-handlers (CRAHs) and in those areas where the water supply and return lines are routed. In addition to monitoring, floor drains and curbing should be present to control and route any escaping liquid. Other precautionary designs should include no process or domestic water lines above or near the IT area. Building and roof drains should also be outside the IT area.”

He adds that moisture detection “within an IT area should provide early warning of a situation that has the potential to cause extensive damage. Upon activation of an addressable moisture detector, trained personnel could reduce the possibility of a cooling system line or fitting total failure.”

Focus on prevention

Prevention is often better than a cure, so he stresses the importance of having up-to-date pre-emergency and business continuity plans. Ideally, data should be backed up to at least 3 disaster recovery sites to ensure uninterrupted business and service continuity. In other words, if a datacenter is affected by an escape of water incident to the extent that it floods and damages essential IT equipment, it should be possible to failover to one or other disaster recovery site to keep the business operational. That can only be achieved though if data is regularly backed up in real-time.

Powell adds: “There should also be a contractual agreement with systems and server equipment suppliers, which includes the use of readily available equipment resulting in minimal down time.” He also advises datacenters, or organisations that have server rooms, to avoid locating water supply and drain lines above or in close proximity to the “IT area”. There should also be floor drains and curbing below the CRAH units and any associated piping. The CRAH units should be properly installed too, and the media, supply and return piping, valves and fittings need to be correctly positioned.

He adds: “Moisture detectors should be addressable and monitored by a system that is continuously attended by a trained individual, [and it’s important to] develop a well implemented and documented preventative maintenance programme.” His last tip is to have trained employees or to work with specialist contractors that have “the ability and authority to secure, troubleshoot, repair and return any failing system back to normal operating conditions in an expeditious manner.”   

Powell suggests that the IT industry is evolving at a rapid pace to the extent that many datacenters are removing air handlers from the IT area to re-position them in separate areas. He says datacentres are also shying away from raising floors, which he claims can reduce the number of unforeseen issues. In addition to this, power and data cable are being routed overhead – perhaps away from any potential water sources.

Use a sensor mix

Mackenzie and Reddington conclude by pointing out that leak sensors are often being used with other sensors that detect heat and humidity: “This is important in respect of data centres and server rooms [because] direct water damage to servers is the obvious hazard, but so is a resulting environment created by the moisture.” They find that humidity is often the biggest threat because moisture affects sensitive IT components, while also corroding them. Corrosion is often an invisible threat that may not be found for some time.  It would therefore pay dividends to invest in all types of sensors now to protect datacentres and server rooms from all kinds of water damage – from water leaks to corrosion.