Wireshark Now Includes an MS/TP Capture Tool!

Wireshark is a fantastic tool for network diagnostics. It’s been the best tool for BACnet/IP network troubleshooting for years. It’s even been able to dissect MS/TP traffic dumps if you knew how and had the right tools. Well, those tools come standard now! Check out the new Wireshark BACnet MS/TP capture tool, included in 2.4.4 (and possibly earlier):

A screengrab of Wireshark's MS/TP capture tool

You’ll have to be connected to the BACnet MS/TP network directly. BACnet/IP routers will not work. I’ve used StarTech‘s successfully and was very happy with it. An MS/TP to IP gateway won’t do the job, since it hides the MS/TP side of things from you. You can use MS/TP capture with Wireshark to diagnose communication problems. It will also reveal if your token passing is being handled efficiently. Wireshark won’t do any automatic diagnostics for you unfortunately, you’ll have to use your understanding of the protocol to troubleshoot or identify problems. Wireshark does have some handy statistics generating features that can be helpful at a glance, though. It’ll break down the number of packets by the type of message they are sending: Who-Is, I-Am, etc., and what devices are communicating the most.

What Is BACnet?

The Basics – What is BACnet?

BACnet is a popular building automation protocol used worldwide to enable communication between devices in commercial automation systems. Industrial automation has historically used other protocols such as OPC and Modbus. BACnet standardizes communication across vendors and simplifies the integration process.

The name BACnet stands for building automation and control network. It was created by the American Society of Heating, Refrigerating and Air-Conditioning Engineers by a committee of industry professionals to modernize communication for automation systems using an open protocol.

Open – To Intepretation

It’s a open standard, anyone can get a full copy of it if they are willing to pay for it. Closed standards are proprietary communication systems created by vendors for their own devices. Open systems allow multiple vendors to utilize a common communication backbone and automation front end. It’s also a very easy standard to comply with, as it requires very little of you. This can be a challenge for system integrators.

A device does not need to support much of the BACnet standard to be a compliant BACnet device. So long as it knows how to say hello to other devices, it’s a BACnet device. This flexibility presents a challenge for integrators, as vendors can implement as much or at little of the standard as they like. Sometimes a vendor won’t bother to put any useful metadata in the standard fields for points. You’ll see analogObject_1 through 500 with no way of determining what anything represents. Lazy BACnet might as well be Modbus.

When implemented well by the vendor, BACnet is damn near pleasurable to work with. It makes some hard things easy since it is designed for our industry. The old school controls guys are not fans of BACnet because larger systems get into the realm of computer networking.

 

Overview of Commercial Automation Protocols

In automation there are several communication protocols. Some are entirely proprietary and others are open. Despite the friendliness of the word, “open” isn’t always good news. Some of the older proprietary protocols are easy to integrate with off the shelf integration products. Two versions of the same protocol often look different. It’s a zoo out there, so here’s an overview of commercial automation protocols to help you sort it out.

Modbus

Modbus is about as bare bones as a protocol can be. It’s simplicity is a strength and a weakness. Since it requires so little it’s easy for vendors to support Modbus, and as a result it’s quite popular. Modbus does little work for you, which makes it difficult to integrate. To integrate a Modbus device you’ll need good documentation and a good understanding of how computers store numbers. If your tools allow you to copy integrations you’ll only have to do it once, as the addresses don’t vary between identical devices. Modbus is more of a simple data transfer protocol without much in common with other automation protocols.

We wrote an article that goes into more depth on Modbus.

OPC

Like Modbus, OPC is common in industrial environments. It can be straightforward to integrate as it doesn’t require you to know low-level details. Discovery is possible, and advanced features such as alarms and historical data access are possible for devices that support them. The tricky part with OPC is connecting devices. The usual form uses a proprietary Microsoft technology that relies on DCOM. DCOM settings can be difficult enough that Matrikon developed an OPC Tunneler that allows you to skip all of that headache for any OPC client-server relationship.

LONWorks

Lonworks is a protocol developed by Echelon. The protocol understands an air handler and what kind of points they should have. Many vendors implement their Lonworks interfaces in a generic manner though and don’t take advantage of this standardization. Other times they will use the standardization, but some of the points will be duds or their implementation will be more confusing. A Lonworks installer needs to understand the protocol deeply since it’s an advanced protocol. If you work in a company with Lon products, you’ve probably had someone bring an entire building down because they synchronized the database the wrong way. It can be a very labour intensive protocol.

BACnet

BACnet is a popular protocol created by ASHRAE that is “open” but not very standard in practice. It’s replacing other commercial automation protocols. There’s a grab-bag of object types and parameters in the protocol for fire monitoring, lighting, scheduling, and so on. BACnet only requires that vendors are implement a few object types. Some will implement the bare minimum in strange ways. When done right, BACnet is a pleasure to work with. Objects are discoverable and contain lots of data that help the integrator understand what they are looking at. Points can be overridden on different priority levels, allowing for schedules to command a fan directly and an operator to override that without any extra programming logic.

Add a Remote Device Without a BBMD

You may encounter a situation where you must add a device on a different network. You may have no control of the BBMD configuration or you may have no BBMDs available due to poor device support. In these situations, you may have to add a remote device without a BBMD. It’s a handy trick, but not an ideal long-term situation.

How is that possible?

The BACnet Broadcast Management Device is required to route broadcasts between BACnet networks separated by routers. BACnet uses broadcast messages for device and point discovery, which are public requests by nature. However, if you know what devices are out there you don’t need discovery! You can add a remote device without a BBMD so long as you know the MAC addresses of the device in question.

What do we need?

You’ll need the device’s BACnet MAC address and BACnet instance number. The BACnet MAC address isn’t the same as the usual MAC address we’re familiar with in IT. A quick way to find it is to discover the device while you’re connected to its BACnet network directly and your BACnet client should tell you directly. Tridium’s N4 platform has a column dedicated to this field. For BACnet/IP devices, the MAC address is a combination of the IP address in hexadecimal and the BACnet/IP port in use.

For example, for an IP address of 192.168.1.6 and a default BACnet/IP port of 47808 (“BAC0”), the MAC address is C0A816BAC0 (192 -> 0xC0, 168 -> 0xA8, 1 -> 0x1, 6 -> 0x6).

Just add the device with the appropriate instance number and MAC address. With any luck, you should be fully communicating. However, the implementation specifics will vary wildly depending on what software you’re using.

Potential Problems

Though this technique can work well for diagnostic purposes, some devices will continue to assume good broadcast communication. If that communication isn’t possible, they’ll assume something is wrong. It’s common practice to use a Who-Is broadcast to check the network status of all devices on the network. If you manually add a remote device without a BBMD handling broadcasts, it will not see or respond to these requests and appear to be offline. You’ll need to get a BBMD in place to ensure your network runs smoothly in that scenario.

 

When and Why Do We Need an End of Line Termination?

End of Line Termination Concept

Most of the buses we use are specified with end of line termination in mind. In the field these terminations are not always present, especially in short runs. Do we need them?

Here’s an analogy I like that illustrates why we need end of line terminations: a bus behaves like a rope held between two people. When one person wants to talk they whip pulses down the rope towards their partner. The other person feels these pulses and figures out what the message means from them.

Those pulses don’t go away, though – they’ll reflect back at the person who whipped it down the line. It’ll keep reflecting back and forth, a little weaker each time as losses eat it up. These reflections are confusing to both people, who waste time trying to decode this false message over and over again.

A bus terminated on both ends is like a rope fixed on either side with springs. When the pulses reach the end, the person there can feel them and then the pulses are absorbed by the springs. If the springs are appropriately sized, the pulses will be “fully dampened” and there will be no reflection back.

RS-485/RS-422 End of Line Termination

Since we’re typically using RS-485, the end of line termination is selected based on something called the characteristic impedance of the cable you’re using. This is something the manufacturer stamps on the box the cable comes in, so it’s no secret. Typically we use something around 108-120 ohms. You don’t need to use exactly that value, but try to be close and don’t use resistors that are much less than it! Some other bus electrical standards will use more complicated devices as their end of line terminations.

Do you always need an end of line termination? Not really, no. It depends on two things: baud rate and bus length. The lower your baud rate, the less reflection will matter. The greater the distance, the more reflection will matter.

High speed buses much more sensitive because everything happens at a much lower time scale – a single bit transferred at 38400 baud lasts only 26 microseconds. For a 9600 baud bit, it’s 104 microseconds. A voltage spike in the wrong place that lasts the length of one bit length can cause communication errors.

Longer buses are more sensitive to reflection because any reflection will tend to dissipate after several trips back and forth down the bus; for a longer bus, that dissipation takes longer. If it’s long enough, controllers could send new messages on the bus while the older reflections are still bouncing around.

Getting Away With Lazy

You can reach a ballpark estimate of when end of line termination is needed can by calculating two things: the propagation delay and the bit pulse width.

Propagation delay is how long a bit takes to go down the cable. Since a voltage pulse travels at a fraction the speed of light (let’s say it’s 66% the speed of light) the propagation delay is 0.66 * c / (length of cable).

The bit pulse width is how long a single bit pulse lasts in time, which we addressed earlier. It’s as simple as 1 / (baud rate). Since baud rate is in Hertz, which is cycles per second, 1/(baud rate) gives us the seconds per cycles (bits).

We now have two time values we can compare. A general guideline is that you can avoid using end of line termination when your propagation delay is much less than the bit pulse time. I.E.: pulses travel so fast across the bus that a reflection will dissipate before it interferes with the interpretation of future bits.

It’s not pure science and could still depend on other factors. Generally speaking, there’s not much reason to cheap out on a few 108 ohm resistors. Bus problems are very labour intensive to troubleshoot.

If you want to be cheap, here’s a table of rough estimates I’ve made of maximum lengths for different baudrates before termination is required:

 

Baudrate Max Length for PD << BP (m) Max Length for PD << PD (ft)
9600 1440 3740
38400 400 1300
76800 200 650
115200 120 400

 

RS-422 – The Other Bus-meat

I encountered an RS-422 bus for the first time recently and it threw me through a bit of a loop! The device was a Modbus/RTU slave device and so I anticipated a RS-485 connection as per usual, but that was not the case. It’s the same cable, and the same essential signalling standard, but if you try to interface with RS-422 as if it’s RS-485 you’re going to have a bad time.

What’s RS-422?

It’s similar to RS-485, but it has some key differences:

  • Only one device gets to be the big talker, and everyone else has to listen. If you have multiple receivers, they can’t talk back. If you only have two devices, no problem. They can actually talk at the same time, which is faster than your usual half-duplex RS-485.
  • It can have one driver (talker) and 10 receivers, RS-485 can have 32 “unit loads” which can mean many more than 32 controllers (not many manufacturers would recommend it, though)
  • It’s a four-conductor bus. Just like RS-485 has a single balanced pair, RS-422 has two balanced pairs: one for transmission, and the other for reception.
  • You wire the driver (talker)’s transmit to the receive terminals of all of the slave devices

Some RS-485 adapters support RS-422, but don’t assume it.

Based on the standard, RS-485 can be “full-duplex” too. An extra pair of wires allows a bus to send and receive at the same time by using four wires, but it’s unnecessary and rare for RS-485.

RS-485 Bus Details

Low-voltage communication buses are critical to networked automation systems and when they fail they are a pain to troubleshoot. We use them because the cables are cheap and you don’t need expensive switches, but what you save in parts you may pay in labor later. Ethernet/IP based communication is on the rise, but is still expensive as an up-front cost. RS-485 is the typical low-voltage communication electrical standard used for the bus we see in commercial and industrial automation. You’ll see it used for BACnet MS/TP, Modbus, and various proprietary protocols like Honeywell’s C-Bus and JCI’s N2 bus.

What is RS-485?

RS-485 isn’t a protocol, it’s an electrical standard that describes how to transmit bits with volts. The protocol allows multiple devices to co-exist on a long, fast bus. It’s similar to the RS-422 protocol, except with RS-485 any device can talk, and any device can listen.

Is RS-485 a Two-wire Bus?

When you want to measure a voltage with your multimeter you need both probes because voltages only exist between things. The parts that “do” RS-485 aren’t looking between your A and B ports, they are looking at each wire relative to a common reference point. The voltage reference is often the power source’s ground, earth ground, a third independent reference wire. Some choices are much better than others.

RS-485 is polarity sensitive, having an “inverting” (negative, “A”) and “non-inverting” (positive, “B”) wire. Depending on the application the conductors will be shielded or twisted-pair.  Some installers will use the shield as a signal ground to save on cable costs. This can work well, but there are caveats. Not all grounds are equal, so be careful how you terminate your shield; one lightning storm and your bus could explode. If you forget to terminate your shield in one place and you’ll offset all your signals by the noise the shield picks up, as we found in this case study.

The two-wire applications that have no ground may actually be using the power source for the controllers as the voltage reference, but that can be problematic if you mix up the transformer polarity or if you are dealing with multiple power sources. It also looks freaky on an oscilloscope – you’ll see the bus riding on a 60 Hz carrier! It’s best to avoid this kind of setup if you can, but it’s cheaper.

It takes a lot more sustained thought and consistent effort to make two wires work. When you add a different controller or use a different source of power you may find yourself with an expensive problem. If you want a solid bus, it’s best to use a twisted pair of conductors for A+B, a free conductor for the reference, and a shield to improve noise immunity.

Interference – Why We Twist Our Pairs and Use Shielding

As you can see in the above case study twisted pair works extremely well; despite a noisy environment the bus signal can still punch through pretty clearly.

Slower buses (~9,600 baud) tend to be less sensitive to interference. Twisted pair alone works fine for slower buses especially in low-noise environments. BACnet MS/TP which typically runs at 38,400 baud is much more sensitive to interference and must be properly shielded, especially in a high-noise environment.

Interference can come from most electronic equipment but big offenders are VFDs and any kind of switches (MCC cutoffs, relays, etc.). Devices that are improperly shielded or have failing components emit interference. Shielding your cable correctly protects your bus from these difficult to locate and potentially impossible to fix issues.

If you look at your shield by itself on an oscilloscope you’ll see the stray voltage picked up by your bus, and it’s largely just sharp spikes that usually do not last very long. Higher frequency buses are more sensitive to noise. Take 38,400 baud for example. At 38,400 bits per second one bit is sent down the bus in a flash of 26 microseconds. Any interference that lasts that long will screw up the data stream; for 9,600 baud one bit lasts for 3 times as long as that. Many applications can sacrifice some bus speed without a noticeable change in performance, but wiring your bus right is a lot more effective.

Hidden Depths

You know how we talk about how RS-485 is a two-wire bus? It’s not true. Not even 2.5, or 3, or 3.5! RS-485 can communicate in full-duplex mode using four signalling conductors in several typologies. I’ve yet to see this or a controller in the field that supports it, but the transceivers exist.

IP-based Protocols and Building Automation Security

When installers build automation systems getting it working is their main focus. They see building automation security as an obstacle to overcome rather than the goal. If authentication is getting in the way, they disable it. When firewalls are blocking communication, they disable that too. Even more problematic is the fact that several popular automation protocols do not even support authentication or auditing of any kind.

What Is Security?

Security is more than just access control or encryption. In Understanding Cryptography (a great book for anyone interested in the subject) these services of security are defined, which I am paraphrasing:

  • Secrecy – nobody can listen in
  • Integrity – ensures that attackers have not tampered with messages
  • Authentication – ensures the sender is who they say they are
  • Undeniable – the author of a message cannot deny that they sent it

All of these are valuable for an automation system, to varying degrees.

Building Automation Security Services

Secrecy is not always important, depending on what you are doing. You probably won’t mind if a motivated attacker knew your room temperatures, but in an industrial setting you might care about keeping your process secret. You may consider electrical metering information private as well as that leaks information about your business.

Integrity is important for an automation protocol when commands have critical effects that could harm your business if mishandled. If an attacker is able to modify your commands that could give them a subtle way of harming your business. Some encryption methods do not protect against this; a malicious person could modify even an encrypted message with unknown content so long as they know something about that content.

Authentication is an important security requirement for an automation protocol that I think everyone can understand. You should need some kind of username and password or other means of authentication to affect the field. Sadly, few protocols support this and those that do have very meager support.Your front end may have a login, but the protocol that ultimately commands the system may not.

In the case of a common “operator” password valid across an entire network, commands are deniable. To prevent this, we can use digital signatures where the user must enter their personal password to sign a message, validating that it was in fact them and not someone who knew some common code. This is valuable in auditing; if a problem occurs as a result of a command, we want to know who did it.

Protocol Security Support (in practice)

Advances are slow to come as security generally not a priority except in applications where the automation is considered a critical system. Manufacturers often do not support more advanced security when its available.

OPC (legacy, non-UA)

OPC is a DCOM-based protocol and as such inherits support for authentication. In fact this support is on by default and is a frequent problem when integrating these systems. Typically the client authenticates with Windows credentials that must be valid on the server. Access rights are broken up as: local access, remote access, launch and activation. As far as I know, you can’t selectively provide access to only some tags; if the user is authenticated by the system they can do anything to it.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – Yes.
  • Undeniable – Depends?

OPC UA has much improved building automation security, though it is still quite new. It’s looking like it will be the most security-conscious protocol on the market.

Modbus

Modbus is an extremely basic protocol and does not support authentication of any kind. If you are on the network, you can read or write anything you like. Some memory addresses are read only, but they are read only for everyone.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – No.
  • Undeniable – No.

BACnet

BACnet is a relatively advanced protocol.makes its lack of even authentication puzzling. All modern front-ends have segmented user access permission levels, and yet this modern protocol allows the network to be a free-for-all. If you’re on the network you can override points at priority levels above the front end. Read or write access is handled by the end device and is not on a per-user basis.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – No.
  • Undeniable – No.

LONworks

LON does support authentication for the sender of messages though I have yet to see it in practice. The sender of LON requests has a 48 bit authentication key. When a device sees an authenticated request from the sender it sends back a 64 bit random number as a challenge to the sender. The sender uses their key to modify this random number and sends it back, providing some manner of authentication.

This at least protects against arbitrary clients commanding field devices as would be the case with Modbus or BACnet.

  • Secrecy – No.
  • Integrity – No.
  • Authentication – Yes.
  • Undeniable – No, the key is shared.

Conclusion

Building automation security is still in its infancy. Protocols are not built with security in mind, though the new OPC UA protocol looks promising. When you need to secure an automation system, look to defense in depth rather than relying on protocol security. Protect access to your network, lock up your controllers, and segregate your network.

Visual RS-485 Grounding and Termination Differences

RS-485 bus grounding and termination seems like a magical rite at times. Some people get lazy with the details because buses often work even when done incorrectly. I had some untwisted, shielded AWG22 lying around, so I used it to show a quick comparison between bus wiring in a controlled environment. I used a single MS/TP BACnet controller and 120 Ω termination resistors. The cable is of inconsequential length and I don’t have any weird equipment near my desk, so the electromagnetic noise should be minimal.

Shielded, Grounded Reference, No End-of-line Termination

This is with a single controller and a tiny cable length. The only effects we see with a lack of end-of-line termination is higher amplitude signals (~7V peak to peak) and capacitance in the cable resisting voltage changes as you can see at any of the “corners” that get rounded off.

Shielded, No Reference, Terminated Shielded

Grounded Reference, Terminated

The terminated version of the same waveform is shown above in two versions: with RS-485 grounding and no reference. Peak-to-peak amplitude is about 4.5 V. Strangely, it looks like there might be more noise in the example with the grounded reference which is not all that implausible considering there are no big noise sources around besides the ancient electrical system in this building. It’s not a big enough difference to take seriously, though.

Note that the waveform is inverted – that’s just because I had the probes the other way. Sorry about that!

Please note that this is a simple look on a desk in a residential area. All of the important things to do with RS-485 buses really show up in larger systems with real electromagnetic noise. The lack of noise in these captures is not similar to real-world applications. I wanted to have this as a baseline for reference. If you want to see what happens when you don’t ground your bus when you need to, check out this post!

Modbus Is Hard, Then Tedious

Modbus is one of the simpliest protocols you’ll find in the field. It’s the closest protocol “to the metal” of the chip on a controller since it often just directly exposes the controller’s memory. There’s no point discovery here; there are no point names or descriptions to help you integrate. All you have are datasheets and fat-finger data entry. There are hardly even data types.

The labour involved with Modbus integration can be high since point configuration is so tedious. When working with a Modbus controller it is very important to have accurate documentation of its:

  • addressing convention,
  • register layout,
  • and data formatting descriptions.

Some controllers start their addresses at 1, others start at 0. Your front end will have its own convention that you’ll have to match up accordingly (off by one errors are pretty common). The register layout will show you the addresses of the data you want. The data formatting descriptions will tell you about the data’s “endianness” and data types; the modbus protocol is just a binary data storage and retrieval protocol, it doesn’t know what a integer or a floating-point number is and it’s up to you to interpret the results.

Modbus can be seen using RS232, RS422, RS485, or IP for connectivity. People frequently make the mistake of assuming one of the above when they hear a device is Modbus. If it’s RS422 and you didn’t plan for that, you’re going to have a bad time.

When getting documentation for the above from a vendor, make sure you get them to help you interpret it! Most communication cards that come with equipment are generic and apply to multiple applications, which means that much of the data available on those data sheets is meanless for your particular application. They should be able to go over the list and highlight points that actually will mean something in your case.

Once you’ve set up a modbus integration it’s robust and fast. Adding new points shouldn’t be too hard either. It’s the initial stages of poring over data sheets and the trial and error of data types that’s the hard part. Once you’ve integrated two systems over modbus once, you should be able to replicate that integration anywhere with relative ease depending on the tools you’re using.

I’ll update this post with links to more detail, especially around data types which are the trickiest part. I’m working on some straightforward real-world examples that should make it a bit clearer.