WEFTEC 2010 New Orleans, what a great show!

The show started off in its usual fashion – introductions, leads, demos and apparently for New Orleans little sleep and a lot of people talking about the fun they had the night before.  This year’s WEFTEC was easily surpassed previous years, with over 17,500 people attending and 984 exhibitors.  The exhibit hall itself was nearly a mile long end to end.

There was a lot of chatter this year about energy efficiency, sustainability and the wastewater infrastructure. A big ticket issue was the EPA’s new Clean Water and Drinking Water Infrastructure Sustainability Policy. This of course ties right into our philosophy – save operating and energy cost, work towards a sustainable water infrastructure and become greener through conscientiously managing your pump stations more efficiently.

Overall the show was amazing, we had great booth traffic, awesome help from Brendan’s reps – Coastal and Odessa Pumps, Hydra Service, Automatic Engineering Inc., Pete Duty and Associates, Instrument and Supply, and Nedrow and Associates. And of course our very own MultiTrode crew, Sandy in particular who did a great job organizing and coordinating WEFTEC 2010.  We are looking forward to taking on The City of Angels at WEFTEC 2011!

General News DuoProbe, efficiency, environmental, MultiSmart, probe, pump station, Pump Station Management, Pump Station Optimization, SCADA, WEFTEC

What is a Reservoir?
A reservoir is a body of water collected and stored for community use. A reservoir can be a natural lake or an artificial lake.

Why Do We Need Reservoirs?
The vast majority of Americans rely on centralized water systems for potable water. Centralized water systems typically include:

• Watershed – A Water Source
• Collection Point – Such as a lake, river, aquifer
• Treatment – Purification and Water Treatment Facilities
• Storage – Reservoirs, Storage Tanks, Water Towers, Standpipes
• Distribution – Network of Pipes and Pump Stations

 What is a Reservoir Monitor?
Centralized water systems consist of hydrologic and hydraulic elements including a Reservoir Monitor. A Reservoir Monitor is a device used to transmit water level information from a Reservoir to a Pump Station.

Why is Reservoir Monitoring Important?
Monitoring the level of water in a reservoir is very important to the general population because it influences decisions ranging from water appropriation to agriculture to hydropower generation. In addition, during times of heavy rainfall, water level information is crucial to planning and flood avoidance.

This growing need for optimizing the use and management water can be met by MultiTrode’s Remote Reservoir Monitor (MTRRM), which gathers water level and I/O data from a Reservoir to be used in a Pump Station at a different location.

MultiTrode’s Remote Reservoir Monitor (MTRRM)
MultiTrode’s Remote Reservoir Monitor (MTRRM) has two analog inputs, one analogue output, six digital inputs, three digital outputs and an input for monitoring the DC supply voltage. The MTRRM can link to a Pump Station via a SCADA system, or in a stand-alone configuration.

In a SCADA network, the CMF (Central Monitoring Facility) typically makes the decisions on activation of pumps. However, when the CMF is not operational for any reason, the MTRRM and remote Pump Station sense this and transmit in a peer-to-peer configuration, increasing water supply reliability. (When the MTRRM detects that the SCADA system is running again, it will revert back to normal operation via the CMF).

The MTRRM can also be used as a forwarding site for messages from SCADA to other units. This is useful for getting information to sites that cannot communicate directly with the SCADA system. Read more…

General News MTRRM, pump station, Remote Reservoir Monitor, Reservoir Monitor, Reservoir Monitoring, SCADA, SCADA System

The MultiTrode DuoProbe is built on the concept of combining MultiTrode’s ultra reliable 10 Sensor Probe with the high resolution of a pressure transducer.

The DuoProbe has 10 metallic sensors, just like MultiTrode’s standard Probe, with the addition of a built-in pressure transducer integrated to the end of Probe. After initial configuration, the DuoProbe will appear to operate exactly like MultiTrode’s standard Probe. After just a few level cycles, the unit will “Auto-Calibrate” and go into High Resolution mode using its built-in pressure transducer.

MultiTrode’s innovative Pump Station Manager – the MultiSmart – will periodically recalibrate the DuoProbe thus ensuring accuracy. In the unlikely event the pressure transducer fails, the MultiSmart will automatically switch over to operate off the 10 Sensor Probe. The DuoProbe will only function with the MultiSmart Pump Station Manager.

Innovative features include:

• High Resolution Level reading can be very useful in VFD systems or any system where monitoring small changes in level is desirable.
• Ability to monitor liquid after it has exceeded the top of the Probe.
• Level can be monitored all the way to top of well even if the DuoProbe is at bottom of well enabling Operators to know the exact well level (especially significant in emergency conditions).
• Particular useful to sites with a SCADA connection because Operators can determine which well(s) are at their highest levels and dispatch emergency generator equipment accordingly.

 Tim Agrillo
May 12, 2010 Read more…

General News DuoProbe, pressure transducer, probe, SCADA, VFD

If you are relatively new to DNP3 you might under-estimate the value of certification.

Compared with a protocol like Modbus, DNP3 has way more features. That’s another way of saying that it is a lot more complex.

Modbus compliance is very easy to test. Suppose you have an RTU with Modbus Slave – a unit which will be polled by a master PLC or RTU. To test it, you can download any one of a number of free or demo Modbus tools and check that you can read a set of registers and write to a set of registers. That’s not the complete functionality of Modbus but it does confirm that the basic functionality works – and because the protocol is so simple, the product vendor is unlikely to have got it wrong. One tool readily available is the Kepware OPC server and you can download a demo version from their website.

As the end-user or system integrator implementing DNP3, if you have to put together a master from one supplier and a slave from another, when it doesn’t work who do you call? Or if you are taking advantage of the fact that DNP3 is an open protocol supported by many vendors you might be choosing a number of different field products – each one suited best for the application at hand.

DNP3 has the flexibility you need for reliable, secure telemetry but there’s a lot of features to test.

The smartest approach is to get the vendor to do it for you before you start using the product or software. If you ask for an independent certification you know that the protocol has been tested by someone who’s reputation is on the line if they miss something.

To get an idea of what gets tested in DNP3 certification you can see a copy of a full test report on the MultiSmart RTU on our main site, under the product manuals section of MultiSmart – www.multitrode.com/pump-station-manager/product-manuals.html Read more…

SCADA & Telemetry DNP3, RTU, SCADA, telemetry

This continues from the earlier DNP3 posts -

Part One: Date/Time Stamping – or Less Guessing
Part Two: Communications Options – Polling and Unsolicited Reporting
Part Three: Security

The DNP3 protocol also supports guaranteed delivery. What does this mean?

Suppose you want to send a command to start a pump. How do you know the RTU at site received the command? With some older and simpler protocols the only way to check is to read the status of the pump at a slightly later time – and hope you catch it in the act.

Or suppose you want to ensure that the message High level alarm or All pumps unavailable sent from the RTU was not missed by the master station or SCADA? With some protocols, like Modbus, there isn’t any mechanism for ensuring this.

DNP3 provides message acknowledgements. With unsolicited reporting, the RTU might send all changed data every half hour, or if the event buffer was full. The “message” that the DNP3 protocol sends includes all the tags that have changed with the date/time of each, and also includes a sequence number. The master station would send an acknowledgement to the RTU – or “outstation” – that that sequence number had been received.

In the event that the RTU / outstation didn’t get that confirmation, it would retry. And after a certain time period the site would go into a Comms Fail mode with probably a longer retry delay. I say “probably” because that depends on how the user sets it up, but that would be the sensible approach.

As you can see if you’ve been following this series on DNP3, the creators of DNP3 designed it for the challenging world of telemetry where communications is always suspect and often problematic.

There’s more to configure in the protocol of course, but each element is there to ensure data integrity:

• you know what happened
• exactly when it happened
• you can guarantee that the SCADA system knows about it
• and you can ensure that data is genuine and not from a hacker

Read more…

General News DNP3, reliability, RTU, SCADA, telemetry

The magazine Control Engineering ran their monthly Information Control email today which included a tutorial from MultiTrode and Parasyn:

www.controleng.com/article/CA6640186.html 

They describe the tutorial:

SCADA systems aren’t scalable out of the box; you have to plan with the final footprint in mind,” says Tony Poole, managing director of Parasyn, a system integrator specializing in water/waste water applications. Steve Carson is with MultiTrode, a manufacturer of MultiSmart pump station manager units, which are replacement devices for pump controllers or PLCs/RTUs for lift stations. The devices add more monitoring and control capability to SCADA systems, and can also add 400 to 500 tags (data points) per site. In this tutorial, Carson and Poole provide best-practice advice for designing large tag count SCADA systems so they are manageable.”

Control Engineering  have a number of email newsletters that you can subscribe to. There’s plenty of good quality articles that make it worth the free subscription. Just visit http://www.controleng.com and you will see the Newsletters menu item on their site.

There’s a story behind how we came to write that article together, a subject for another day.

We find the subject of large tag count systems very interesting – we’ve run into it a number of times and in different elements of the SCADA solution. Parasyn’s approach made a lot of sense when they explained it to us and we certainly learnt from their experience.

“SCADA systems aren’t scalable out of the box; you have to plan with the final footprint in mind,” says Tony Poole, managing director of Parasyn.

Have a read of the article. It will be in the print edition of Control Engineering  in April. Read more…

General News large tag count, reporting, SCADA, telemetry

Our White Paper on SCADA reporting also appeared in Pumps & Systems – for those who don’t know it, an excellent US industry magazine. You can find the online version of the magazine at www.pump-zone.com.

What’s the White Paper about? Essentially it looks at where SCADA and telemetry reporting currently stands for a typical lift station network – compared with where it could be – and where the utility asset managers and operations managers would like it to be.

Operational Cost Breakdown

Anyone who has implemented a SCADA system, or even played with a SCADA package can tell you how easy it is to create an animation on the HMI that depends on a data point. For example, displaying a well emptying and filling as the level “tag” falls and rises in value. Or a motor changing from blue to red as the temperature tag increases in value.

It’s probably the first thing that you get taught on the SCADA course, or get shown by the salesperson when he/she demonstrates ease of use of that SCADA platform.

Reporting is a whole different question. There is the challenge of finding where the real data exists in the system, understanding data integrity and getting to grips with the reporting interface. But the bigger challenge for many end users and SI’s is that creating a good report means getting to grips with relational database concepts, and that’s a lot harder than linking a process value to a graphical animation.

As Paul Francis, MultiTrode CTO, said:

Reporting has always been a challenging aspect of any SCADA platform. Visualization and trending tools were some of the earliest adopted elements of SCADA systems and hence matured long before reporting interfaces did. Developing useful reports can also be more of a technical challenge than making a nice graphical interface for your plant or collection system; in part due to the nature and structure of the underlying data.”

Away from the mechanics of reporting, there are other challenges. For example, data acquisition in the field. Typical PLC and RTU systems aren’t bringing 100’s of tags back to the SCADA system. That requires a level of design that is rarely catered for. With minimal data, it’s hard to generate much in the way of reports – and this may well have contributed to the low expectations of asset managers.

Reports - Pump Efficiency Changes

And in the US, the de facto standard for water and wastewater telemetry protocols is Modbus. This works against any attempt to get asset rich data. (See, for example, DNP3: Part One or DNP3: Part Two).

None of this means that it’s too difficult to get reports that will cut operational costs or provide asset reports to allow better capital allocation. It’s just important to understand the different roadblocks that lie in the way.

If you want to read the SCADA Reporting White Paper you will have to fill in a short registration form (unless you already have a MultiTrode website login). Read more…

SCADA & Telemetry reporting, RTU, SCADA, telemetry

This post continues the themes from Part One and Part Two.

The subject today is security, and also why proprietary protocols aren’t the answer.

Security in communications is a hot topic, but in practice in the water and wastewater industry, not many people are actively implementing it.

It’s important to differentiate between “hacking the comms” and “hacking the server”. If there is a greater problem for the organization, it’s surely someone hacking your server through a firewall –  or from within your building – because now they can take control as well as present the operations staff with a completely false worldview.

However, if the SCADA server is highly secure and someone was very motivated to take control of your system, then they could potentially do a lot of damage by hijacking the communications. Imagine if they turned off all of the sewer pump stations in a city? You can send your staff out to put every station into manual over-ride – but only once you knew about it, and it would take some time to get to every single station. You would have lots of overflows, and you would have your whole team racing around from station to station. If it happened in a time of high inflows – e.g. a storm – then the problems would be much worse. In a water supply system it might be possible to burst pipes.

There are a lot of articles about communications security that start: “Since 9/11″ – probably because it gets higher exposure. But how much of a risk is it?  And is the risk greater from other sources than terrorism – like disgruntled ex-employees? It’s certainly getting attention from governments, but not much practical attention from the utilities themselves.

This article doesn’t try and address the risk factor. Instead, we’ll just explain a little about how communications to remote sites can be secured.

Security in Communications -  Are proprietary protocols the answer?

One subject that the promoters of proprietary protocols majored on in recent years is security. This is because they didn’t have a lot else to hang their hat on.

What am I talking about? Open protocols have been the perceived way forward for a long time, but especially in this century/millenium. For the last 5-10 years in the water & wastewater industry, almost anyone writing an engineering spec, or an operations manager or utility director who had done a small amount of research, knew that you needed to specify an “open protocol” for a new or upgraded system.

This presented a challenge for a number of companies who had their own protocols in their RTU and used these protocols to lock in customers. What to say to show they were progressive?

“At least no one knows how to hack our protocol, that’s an advantage..”

By the way, I’m not including in this list, companies with their own protocol who made them public. There are many companies, including ourselves, who in the 1990’s had their own telemetry protocol in their RTU because it seemed – rightly or wrongly - to have some advantages at that time. The important point is, once the move towards open protocols became desirable or a requirement, and high quality telemetry protocols became available, what did those suppliers do? The responsible ones published their protocol and made it easy for other parties, including competitors, to copy them.

In fact, most proprietary protocols aren’t that hard to reverse engineer.

In the world of encryption and authentication, the experts will tell you that openness is what allows the audit. Don’t tell the world that your protocol is “secure” because it is proprietary, unless you have invited a few hackers to break it. It probably won’t take them very long.

A good recent example is where one of our partner companies, Trihedral, reverse engineered a proprietary protocol from another supplier to allow them to break into their market – to replace the SCADA server software while still interfacing to the RTU’s in the field.

Only last year (2008) I read an article in a water industry magazine by a supplier saying how their RTU protocol was more secure than DNP3 because it wasn’t published..

Time to move on..

DNP3 Security – How Does it Work?

Security is one of those tricky subjects that most people actually don’t want to understand. As a user you just want to know it works. So I’ll stay away from the more technical aspects.

DNP3 is a published protocol with a very strong and a very technical user group. You can be sure that the people in the user group who published the security specification knew what they were doing.

Very simply, DNP3 security doesn’t encrypt the message, it authenticates the message.

If someone intercepted a command to an RTU: ”turn on Pump 1″ which might look like “digital tag 15 ON”, they could read it!

But if the bad guys then wrote a command to send to the RTU - ”turn off Pump 1″ – “digital tag 15 OFF” – the DNP3 authentication mechanism would reject it. The security mechanisms in DNP3 can determine when the command is a valid one by a trusted party.

This gives an insight into why the oil and gas suppliers might want RTU encryption, not authentication. In a highly competitive commercial environment you don’t want others to know how much volume you pumped  -for example.

In water and wastewater, even in the privatized but regulated market of the UK, it’s hard to see how anyone reading your pump station commands could cause you a problem.

The key point of DNP3 security is that while others can see what you are doing, they can’t pretend to be you and tell your system to do the wrong thing. That’s what authentication means.

If you want to know more, please take a look at our White Papers section. You need to fill in a short registration form to download any of the papers. The title of that paper is “Keeping SCADA systems open and secure from cyber-attack”.

There’s a lot more technical data available – check out the DNP3 Users Group.

And if you’re one of those people who like to understand more about “how everything works” – for the confusing world of encryption and authentication, a personal recommendation is Cryptography Decrypted by H.X. Mel & Doris Baker. It was published in 2001 so it might be a little dated, but I found it made subjects like public key encryption finally understandable. Read more…

SCADA & Telemetry DNP3, RTU, SCADA, security

We posted a news item on our main site about Trihedral and MultiTrode.

Background

This brief blog post gives a bit of background. Canadian-based Trihedral have been making inroads into the water and wastewater industry – our main market – for some time. From our perspective, we have seen them win a lot of customers in the SE of the USA. They might be winning customers in lots of other areas too.. we’ve just noticed the SE region.

The Florida market has had a major supplier who locked up the customer base with a proprietary protocol. What Trihedral did was reverse engineer the protocol. As a result, the Trihedral VTS platform can be implemented by these customers who can continue to use their existing field hardware but are now free of their restraints! They change their SCADA platform, keep their old hardware (so don’t have to change everything overnight), but they can start using new products.

These customers who have switched to VTS can now introduce any product they want in the field – or the plant – so long as it supports open protocols like Modbus or DNP3. 

The Partnership

That’s the background, but the news item is about what Trihedral have done, in partnership with MultiTrode. VTS will shortly have an “add site” function for MultiSmart.

This function essentially removes a lot of the legwork, or integration, out of bringing a MultiSmart site with 100’s of tags (data points) into the VTS system.

The new version of VTS isn’t quite out yet – expected in February. But I saw a demo of it a few months ago and was very impressed. The MultiSmart pump station manager was on an ethernet link (could have been a radio link) to the VTS SCADA server and the whole process was automated.

The remote MultiSmart site created a compressed version of the XML configuration (we use XML as standard for all configuration files), transmitted it using DNP file transfer (industry standard), VTS loaded it, unzipped it, and presented the user with a few checkbox options for configuration – including graphics, alarms and controls.

I know some of the Florida customers are very keen to get their hands on it. And some of our engineers are very interested in what Trihedral have done.

2+2 is more than 4 – if you choose the right combination

That’s the beauty of working with other companies who are great at what they do and have a passion for making things better! New ideas that spur our engineers on to come up with even more innovation.. and make life easier for the customer. Read more…

SCADA & Telemetry Add new tag, DNP3, MultiSmart, SCADA, telemetry

This follows on from the first blog post on DNP3 where we covered Date/Time Stamping – or Less Guessing - and a little bit of a comparison with Modbus.

This post is about some of the mechanisms for communicating between the SCADA, or master station, and the remote site. By the way, in DNP3 speak, you often hear the term “outstation” for the remote site. We’ll tend to stay with RTU (depending on who you talk to it either means “Remote Telemetry Unit” or “Remote Terminal Unit”), which is essentially the device that communicates.

Polling 

Let’s start with a Modbus comparison again, just to give a little context. Modbus is a totally polled environment – that is, the master station (typically a PLC) requests data from a site. The remote site can’t choose to send some data to the master station, it has to wait until the master requests it. So a very typical arrangement for a collection system is where the master PLC polls each lift station in turn and when it is has finished, starts again.

This has a lot of disadvantages as anyone with a growing network of lift stations has found. The time taken to get around the network goes up and up. You might get to a point where it’s 15 minutes between polls and decide to prioritize certain sites to get polled more frequently, or add a radio frequency (and therefore a new base station and repeater radio) to split the network into more manageable pieces.

It’s not an ideal situation, because Modbus isn’t an ideal telemetry protocol. (It’s a great protocol for other applications).

Unsolicited Reporting

If waiting until the master station asks you how you are doing isn’t the best way, what about unsolicited reporting. What’s unsolicited reporting? The remote site, also known as the RTU, sends data to the master station without being asked.

If you are used to Modbus polling systems, this might sound like anarchy. And if you have every site sending a message every time an event takes place, it could well be anarchy. Will the radio network stand up to lots of sites all trying to communicate at once?

The only way you might be able to prevent chaos, is by having a very limited amount of data getting sent, or a high radio bandwidth. One of the guys in MultiTrode told me about a system he was involved in with a previous company where they used unsolicited reporting for every event, and during a major storm the entire network shut down due to “collisions” in the radio network. The solution was for someone to go and visit every site and reset each RTU. Then the network started communicating again. But they lost all the data for their critical event.

That’s not great. What’s the solution?

Communication Choices and Different Classes

The DNP3 protocol has some great features to avoid the problems above.

Firstly, you can group events into different classes. Then secondly, you can choose how those different classes communicate from the RTU to the master station.

An example is the best way to explain it. Suppose you want to know about all high level alarms within 1 minute, and otherwise you are happy for all stations to be communicating at least once every 30 minutes.

You would put the alarm “High level” into class 1, and everything else into class 2. Then you would set class 1 to unsolicited reporting immediately. And class 2 you might set one of two ways – either to report every 30 minutes or sooner if the event buffer was full; or you might have the master station polling every 30 minutes.

Now you can find out immediately about high level alarms without risking the network turning into treacle and you still find out about all the changes in your network.

It’s the flexibility in DNP3 that is one of its great features. As Paul Gibson, one of the key people behind the development of the MultiSmart pump station manager, said:

“If we didn’t have DNP3, we’d be trying to design a protocol just like it to put into the product.”

The Communications Network still needs Design

Just because there’s a flexible protocol doesn’t mean you don’t have to do anything. Every communications network needs design. How much data? What is the bandwidth? What are the delays?

Read more…

SCADA & Telemetry DNP3, Modbus, MultiSmart, RTU, SCADA, telemetry