Part 3
This is the final article in our three-part series on SCADA networks. Part 1 covered SCADA network equipment, and Part 2 covered SCADA networking protocols and basics. Let's wrap up the series with general SCADA networking and troubleshooting.
What is TCP/IP, what is it used for?
TCP/IP is a combination of two separate protocols: Transmission Control Protocol (TCP) and Internet Protocol (IP). It has to do with the transmission of data back and forth between networked entities.
- IP is responsible for obtaining the correct IP address to send data to. It tells data packets where to go and how to get there.
- TCP is then responsible for delivering the data to the right address. It checks data packets for errors and tries to resubmit the data for transmission if any errors prevent delivery.
As we touched on in Part 2, a protocol is a standardized set of rules that allows two or more networked entities to communicate. In order to send data back and forth successfully, the sender and receiver must agree on the protocol—the rules.
Before TCP/IP, computers from different vendors had difficulty communicating with one another. Each vendor had its own rules. The U.S. Department of Defense developed TCP/IP as a clear, universal standard that allows computers from different vendors to communicate.
In the TCP/IP model, the different communication tasks are divided into four levels or layers. Data must travel through all four layers before it reaches its destination. Each individual layer has its own responsibilities and deals with different protocols. One of the main advantages of this stack model is that it allows you to isolate network communication problems by layer and work in a methodical way to troubleshoot issues.
We touched on the four TCP/IP layers briefly in Part 2. Here they are for review:
- Application layer: Allows the user to interact with the application. This layer handles high-level protocols, including HTTP, HTTPS, RDP, SSH, etc.
- Transport layer: Responsible for flow control, reliability and correction of data. This layer defines what kind of connection needs to be made, like TCP or UDP.
- Internet layer: This layer is responsible for sending data packets. It assigns IP addresses for the traffic source and destination. Routers operate at this level.
- Network access layer: Also known as the physical layer. Responsible for transmitting data between devices on the same network. This layer deals with Ethernet protocol and defines how data should be sent physically through the network. It ensures that data is sent to the correct device using the MAC address—a 48-bit number that's unique to each networked device. Network switches operate at this level.
What are some of the common locations in a network where data originates from?
Data originates from applications. In the case of a solar PV site, there are two ends to that application path—the field devices that capture data, and the SCADA applications that store that data and provide control.
The field devices (inverters, trackers, MET station sensors, etc.) have sensors that gather information. The SCADA system pulls in that data, where other applications receive it. The historian logs and stores the data on local or cloud-based servers, which allows operators and stakeholders to look at the historical data from the plant. This can be useful for troubleshooting. The HMI displays the data and allows operators to give commands to the plant controller.
What is the difference between a Switch, a Router and a Firewall?
In Part 1, we defined switches, routers and firewalls as key pieces of network hardware. Now, let's look at them in terms of the TCP/IP stack model.
Switches work at the lowest and most basic TCP/IP layer—the network access/physical layer. When two devices on the same local network (LAN) want to communicate via Ethernet, the switch creates a point-to-point connection and prevents data collision. It does this by checking the MAC address on each device that's connected to each port of the switch (MAC Table). That's it. Switches never go further than this basic physical layer.
Routers work on the next layer up—the Internet layer—which deals with IP addresses. They send data not between different devices, but between different networks. Each port on a router has an IP address that acts as a gateway to the network it's connected to.
When a router receives data based on source and destination IP, it investigates further. It checks the source and destination IP against its routing table, and routes the data to the interface that has the fastest and closest path to the destination network.
Routers can also provide access control. An access control list (ACL) prevents access to the network based on source/destination IP addresses and application port numbers monitoring. This is called stateless access control because it's simple—it doesn’t do much investigation. The router simply filters traffic based on the source and destination IP and port.
Firewalls operate on a higher level than routers. Modern firewalls do everything routers do, but they have more applications, processing power and security capabilities. They do more investigation into the data and can monitor, filter or block traffic based on more advanced parameters than just source and destination IP and port. This is called stateful access control.
SCADA Network Troubleshooting
How do you know if your devices aren't communicating?
SCADA system alarms, displayed on the HMI screen, alert you when devices stop communicating on the network. This "device down" alarm may be depicted a number of different ways depending on your SCADA alarming software—it might be a link, a notification bubble, or an animation that changes from green to red.
What steps do you follow to troubleshoot a communication failure?
It helps to gather some basic information before starting the troubleshooting process. How long has the device been offline? If the device was never up, there might be something wrong with the logic embedded in it. If the device was up but now has stopped communicating, there might be a network-related or device-related issue. This information may not be 100% accurate (operators may not know all the details) but it can provide some helpful background.
There is no one set sequence of steps that works for absolutely every troubleshooting situation. However, the overall process is the same—to try to narrow down the possibilities in the shortest, quickest way possible. SCADA troubleshooting ability grows with experience and knowledge of the system.
Here are the typical steps to troubleshoot a communication problem:
- "Ping" the device. This is done by opening a command prompt window (or DOS prompt) and testing—for example, a command line would be: Ping 10.10.10.1. If the ping returns a successful ping you know the device is still on the network but not communicating with SCADA. At that point, you can run Modscan and test for data. This test may reveal exception errors or failed protocol connections that are preventing the networked device from communicating with SCADA.
- If the ping is not successful, you know the device isn't connected to the network. You can then try pinging other nearby devices. If the other devices aren't pingable, either, you know you have a larger problem, like a loop issue. If they are pingable, you can narrow it down to that one device. You can log into the firewall and try pinging from there, to see if there's something blocking communication. You can try pinging from individual interfaces in the firewall (network gateways). The goal is to try to isolate the connectivity issue.
- If you're only dealing with one device that's down, you can investigate the device configuration. There may be some network mask misconfiguration, wrong gateway or even a mistyped/duplicated IP address on the device's interface.
- If that all checks out and your remote troubleshooting options have been exhausted, it's time to get an onsite tech to look at the device. The problem could be simple as a loose cable connection, or as complex as a failing device. The tech should physically inspect the cabling, connections, device hardware, power and ports.
Learn More About SCADA Networks
This concludes our article series on SCADA networking. We hope you've found it helpful and valuable!
If you are a solar industry professional who wants to learn more about SCADA networks, we invite you to attend our quarterly Solar PV Operations Training sessions. SCADA networks are covered in-depth on Day 2 of the 3-day training. The training is system agnostic, meaning it can be applied to any SCADA system platform.