IP-based Protocols and Building Automation Security

When installers build automation systems getting it working is their main focus. They see building automation security as an obstacle to overcome rather than the goal. If authentication is getting in the way, they disable it. When firewalls are blocking communication, they disable that too. Even more problematic is the fact that several popular automation protocols do not even support authentication or auditing of any kind.

What Is Security?

Security is more than just access control or encryption. In Understanding Cryptography (a great book for anyone interested in the subject) these services of security are defined, which I am paraphrasing:

  • Secrecy – nobody can listen in
  • Integrity – ensures that attackers have not tampered with messages
  • Authentication – ensures the sender is who they say they are
  • Undeniable – the author of a message cannot deny that they sent it

All of these are valuable for an automation system, to varying degrees.

Building Automation Security Services

Secrecy is not always important, depending on what you are doing. You probably won’t mind if a motivated attacker knew your room temperatures, but in an industrial setting you might care about keeping your process secret. You may consider electrical metering information private as well as that leaks information about your business.

Integrity is important for an automation protocol when commands have critical effects that could harm your business if mishandled. If an attacker is able to modify your commands that could give them a subtle way of harming your business. Some encryption methods do not protect against this; a malicious person could modify even an encrypted message with unknown content so long as they know something about that content.

Authentication is an important security requirement for an automation protocol that I think everyone can understand. You should need some kind of username and password or other means of authentication to affect the field. Sadly, few protocols support this and those that do have very meager support.Your front end may have a login, but the protocol that ultimately commands the system may not.

In the case of a common “operator” password valid across an entire network, commands are deniable. To prevent this, we can use digital signatures where the user must enter their personal password to sign a message, validating that it was in fact them and not someone who knew some common code. This is valuable in auditing; if a problem occurs as a result of a command, we want to know who did it.

Protocol Security Support (in practice)

Advances are slow to come as security generally not a priority except in applications where the automation is considered a critical system. Manufacturers often do not support more advanced security when its available.

OPC (legacy, non-UA)

OPC is a DCOM-based protocol and as such inherits support for authentication. In fact this support is on by default and is a frequent problem when integrating these systems. Typically the client authenticates with Windows credentials that must be valid on the server. Access rights are broken up as: local access, remote access, launch and activation. As far as I know, you can’t selectively provide access to only some tags; if the user is authenticated by the system they can do anything to it.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – Yes.
  • Undeniable – Depends?

OPC UA has much improved building automation security, though it is still quite new. It’s looking like it will be the most security-conscious protocol on the market.

Modbus

Modbus is an extremely basic protocol and does not support authentication of any kind. If you are on the network, you can read or write anything you like. Some memory addresses are read only, but they are read only for everyone.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – No.
  • Undeniable – No.

BACnet

BACnet is a relatively advanced protocol.makes its lack of even authentication puzzling. All modern front-ends have segmented user access permission levels, and yet this modern protocol allows the network to be a free-for-all. If you’re on the network you can override points at priority levels above the front end. Read or write access is handled by the end device and is not on a per-user basis.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – No.
  • Undeniable – No.

LONworks

LON does support authentication for the sender of messages though I have yet to see it in practice. The sender of LON requests has a 48 bit authentication key. When a device sees an authenticated request from the sender it sends back a 64 bit random number as a challenge to the sender. The sender uses their key to modify this random number and sends it back, providing some manner of authentication.

This at least protects against arbitrary clients commanding field devices as would be the case with Modbus or BACnet.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – Yes.
  • Undeniable – No, the key is shared.

Conclusion

Building automation security is still in its infancy. Protocols are not built with security in mind, though the new OPC UA protocol looks promising. When you need to secure an automation system, look to defense in depth rather than relying on protocol security. Protect access to your network, lock up your controllers, and segregate your network.

Visual RS-485 Grounding and Termination Differences

RS-485 bus grounding and termination seems like a magical rite at times. Some people get lazy with the details because buses often work even when done incorrectly. I had some untwisted, shielded AWG22 lying around, so I used it to show a quick comparison between bus wiring in a controlled environment. I used a single MS/TP BACnet controller and 120 Ω termination resistors. The cable is of inconsequential length and I don’t have any weird equipment near my desk, so the electromagnetic noise should be minimal.

Shielded, Grounded Reference, No End-of-line Termination

This is with a single controller and a tiny cable length. The only effects we see with a lack of end-of-line termination is higher amplitude signals (~7V peak to peak) and capacitance in the cable resisting voltage changes as you can see at any of the “corners” that get rounded off.

Shielded, No Reference, Terminated Shielded

Grounded Reference, Terminated

The terminated version of the same waveform is shown above in two versions: with RS-485 grounding and no reference. Peak-to-peak amplitude is about 4.5 V. Strangely, it looks like there might be more noise in the example with the grounded reference which is not all that implausible considering there are no big noise sources around besides the ancient electrical system in this building. It’s not a big enough difference to take seriously, though.

Note that the waveform is inverted – that’s just because I had the probes the other way. Sorry about that!

Please note that this is a simple look on a desk in a residential area. All of the important things to do with RS-485 buses really show up in larger systems with real electromagnetic noise. The lack of noise in these captures is not similar to real-world applications. I wanted to have this as a baseline for reference. If you want to see what happens when you don’t ground your bus when you need to, check out this post!

What is a BBMD, and Why Do We Need It?

As BACnet networks get bigger and network infrastructure gets more complicated, BACnet/IP Broadcast Management Devices (BBMD) become necessary. Installers often configure these devices incorrectly despite their importance which creates the conditions for a broadcast storm.

What is a BBMD?

To understand what these are and why we need them, we first need to understand what a broadcast is. Network protocols use broadcasts to efficiently communicate public information. A broadcast is sent once and received by all devices on the same network. As your network gets bigger, broadcasts start to become a problem due to their increased frequency as they have to be processed by all devices. To address this problem, IT infrastructure network routers that connect different networks together block broadcasts from crossing networks. This presents a problem for us with BACnet because it requires broadcasts for discovery of devices and objects.

Each broadcast management device has a list of all others on other networks in its broadcast distribution table. They act as co-conspirators, getting around the broadcast blockade by smuggling the broadcasts into directly sent packages that routers allow to pass through them.

Here’s a point-form description of how they function:

  1. A BBMD packages BACnet broadcast messages that it hears on its own network into an envelope.
  2. It sends these packages directly to the other networks using its distribution table.
  3. When a BBMD gets a broadcast package from another BBMD, they open up that envelope and rebroadcast the messages on their own networks.

This allows your BACnet network to get around any routers in between that would block the broadcast traffic.

Visual Example of Broadcast Communication

Below is an example network architecture with three separate IPv4 networks and three routers. I chose the IP addresses in the example to be similar to what you’d be familiar with at home.

Example Network Architecture

If one of the devices on the 192.168.1.0 network sends a broadcast message (like a Who-Is address discovery), the other device on that network will see it but it will be blocked at the router and reach no other devices. This is standard router practice: they keep broadcasts local.

BACnet Broadcast Message Sent

When a BBMD receives the broadcast message it encapsulates it and sends it a direct message to every other BBMD in its broadcast distribution table. These other BBMDs open up the package and rebroadcast the contents locally, permitting BACnet broadcast traffic to permeate the entire network.

BACnet Broadcast Message Encapsulated and Delivered by BBMDs

Common mistakes with configuring BBMDs

  • Multiple BBMDs configured on the same BACnet network is a huge, common problem. You must only have one per network unless you have a controller that supports redundant BBMD configurations which is rare. This is a cause of broadcast storms.
  • Using different broadcast distribution tables on different BBMDs.
  • Using the same network number for networks separated by routers.

Further Reading

Modbus Is Hard, Then Tedious

Modbus is one of the simpliest protocols you’ll find in the field. It’s the closest protocol “to the metal” of the chip on a controller since it often just directly exposes the controller’s memory. There’s no point discovery here; there are no point names or descriptions to help you integrate. All you have are datasheets and fat-finger data entry. There are hardly even data types.

The labour involved with Modbus integration can be high since point configuration is so tedious. When working with a Modbus controller it is very important to have accurate documentation of its:

  • addressing convention,
  • register layout,
  • and data formatting descriptions.

Some controllers start their addresses at 1, others start at 0. Your front end will have its own convention that you’ll have to match up accordingly (off by one errors are pretty common). The register layout will show you the addresses of the data you want. The data formatting descriptions will tell you about the data’s “endianness” and data types; the modbus protocol is just a binary data storage and retrieval protocol, it doesn’t know what a integer or a floating-point number is and it’s up to you to interpret the results.

Modbus can be seen using RS232, RS422, RS485, or IP for connectivity. People frequently make the mistake of assuming one of the above when they hear a device is Modbus. If it’s RS422 and you didn’t plan for that, you’re going to have a bad time.

When getting documentation for the above from a vendor, make sure you get them to help you interpret it! Most communication cards that come with equipment are generic and apply to multiple applications, which means that much of the data available on those data sheets is meanless for your particular application. They should be able to go over the list and highlight points that actually will mean something in your case.

Once you’ve set up a modbus integration it’s robust and fast. Adding new points shouldn’t be too hard either. It’s the initial stages of poring over data sheets and the trial and error of data types that’s the hard part. Once you’ve integrated two systems over modbus once, you should be able to replicate that integration anywhere with relative ease depending on the tools you’re using.

I’ll update this post with links to more detail, especially around data types which are the trickiest part. I’m working on some straightforward real-world examples that should make it a bit clearer.

A Case Study on the Importance of RS-485 Grounding

RS-485 grounding is frequently missing from commercial automation installations. It’s especially egregious when you are using your shield as your signal reference, because that reference is now noisy as heck. The shield’s job to collect interference. It’s up to you to make sure that interference goes somewhere.

The Case Study

I was looking at the RS-485 (MS/TP BACnet) buses for variable frequency drives (VFDs) terminated on a BACnet/IP gateway today. Devices on this bus kept going offline and online periodically, so a bus issue was the first thing to look for. Below is what one conductor of the RS-485 bus looked like by itself on my oscilliscope:

VFDs are noisy. It doesn’t help that the bus cable goes through these before they terminate in the low-voltage side:

It’s going to pick up a lot of that interference from these high voltage things.

The cable is twisted pair and shielded, so interference on one conductor mostly cancels out the interference on the other one. Here’s what they look like together, and what a controller would see:

Much better, but that’s still very bad. This is because this cable is shielded, and the shield was terminated at the BACnet MS/TP to IP gateway in the VREF terminal. This is the reference “ground” that the A and B conductors are measured with respect to. The shield is going to pick up interference along the bus run, because that’s what it’s for. The interference is going to drain to this reference and throw off the voltage of the bus signal.

Here’s what the shield looks like by itself:

As you can see, I am measuring spikes greater than 1.7 V on the shield. This will throw off our measurement of A and B by the same amount, causing frame errors.

The Solution

Since this is a shield, I wired it like you’re supposed to. Let the interference drain to earth ground:

Now the combined signal looks like this:

Much better. Still, now I’m not using anything for a reference. Ideally the bus cable used would have a free conductor, but it doesn’t. I could earth-ground it too and do the same at all the VFDs in the field, but I don’t want a thunderstorm to blow up controllers. It works much better now, so I think I’ll just leave it as is. Not too happy about it, but not the worst outcome.

The Bus Problem Superweapon: The Oscilliscope

I had the opportunity to troubleshoot an MS/TP BACnet bus using my Rigol DS1052E oscilloscope. Oscilloscopes are much more useful for bus diagnostics than a multimeter. Although they don’t tell you where the problem physically is, they can immediately confirm that you are in fact looking at a bus fault instead of a software problem and also identify when it is corrected.

The Problem

The bus in question appears to have a short. Communication does still work, but it takes a long time to poll points or override them. I presume there’s a lot of error-correction and trying in the background making up for the bad bus on the platform I’m using here (Tridium AX).

You can’t really tell there’s a problem from the software side besides the slowness and some strange errors about frame errors printed in a log. With the oscilloscope I can see immediately that there is a problem and I know what conductor the problem is on. That doesn’t exactly solve it for me, but it’s a clear indication at least which has a clear, tedious solution.

Investigation

The picture above is a good-quality frame. There’s a bit of rounding from capacitance, but it shouldn’t interfere with communication as it’s quite minor. The square wave signal is clear and the peaks are consistent. I took this sample in the middle of the problem bus with the ground clips for both my probes on the reference conductor. I told my oscilloscope to invert one of the channels, and I am displaying above the addition of both. That’s essentially what an RS-485 transceiver will do in your controller.

The above is not a good frame. It’s not that clear that it’s bad. You can see a bit of capacitance, you can see that the peaks are not that consistent, and maybe a bit of noise. The total amplitude is lower than with the good one. I was surprised to see this result on the bad side of the bus, so I looked at the channels separately. Then, I saw what is clearly a problem – here’s one of the conductors (A or B) of the bad side of the bus:

Yeah, that’s not a square wave signal. That looks like a signal that wants very much to stay near to ground. I.E., this wire is shorted to ground somewhere. Given that this is an underfloor ventilation system and we have a lot of wires bell-tied to struts we get a lot of damaged cables when people are over-zealous with pulling.

I’ll update as I investigate!

The Solution

Yup, that conductor was shorted to the shield which is grounded. This is a floating floor system, and the bus is cable-tied to struts under the floor. It looks like someone pulled a little too zealously. Very easy to repair.