Saved by the ‘Shark’ – Leveraging a Valuable Tool

Fifteen years ago Gerald Combs released a little network protocol analyzer called Wireshark (then called Ethereal). At the time it only dissected five protocols and only ran on Linux and Solaris. He decided to share it with the world and released it as open source software.

Immediately after the release he started receiving code from people around the world. They had problems similar to his and were able to modify the little analyzer to suit their needs. They were also kind enough to contribute those modifications back. Those contributions haven’t stopped to this day and Wireshark has grown into a mature, feature-rich, award-winning network analysis tool. People around the world use it to troubleshoot networks, develop software and protocols, and to learn about networking.

Wireshark is an open-source packet analyzer, often used for network troubleshooting, analysis, software and communications protocol development, and education. In my experience, it’s an invaluable tool for developers, testers, techops, and network engineers alike. After 15 years of continuous development and contributions from people across the world, Wireshark now supports over 1,000 protocols and 140,000 different protocol fields. And those numbers keep growing.

So, what makes Wireshark so special? With Wireshark, your network is no longer a black box. This is important because our daily lives depend on networks operating efficiently, reliably and securely. Wireshark lets you peer into your network and see how it operates at a low level. It allows engineers to give meaning and context to the bits and bytes that make up network packets.

Wireshark in action

At Merchant Warehouse, we’ve used Wireshark and tools like it (eg. Fiddler) to great success. Our technology stack is largely a Service Oriented Architecture. Through Genius, Genius EX, Transport.Web, and our numerous Point of Sale integrations, our customers talk to our Application Programming Interface. These, in turn, talk to a number of other services in our datacenter, and ultimately make a call to one of our Payment Processors or Platform Partners, such as LevelUp or the ISIS Mobile Wallet.

Most recently, we encountered a problem with a new Payment Processor that we were integrating. We had developed a plugin that connected our payment gateway with a payment processor. This new plugin translates ISO 8583 (the international standard for financial transaction card originated messages) style messages into a vendor-specific SOAP format. In our User Acceptance Testing (UAT) environment, everything worked as we’d expected. However, when we went to roll our changes out to production, things got complicated. Some of the transactions would complete successfully. Others would seem to get stuck, and eventually the transaction would time out without processing.

Since we had a working UAT environment, we decided to run an A/B test and see what the differences were. Using Wireshark, we were able to verify that ISO 8583 messages were arriving at our plugin in Production, but that no SOAP messages were being sent to our partner. We set up Wireshark to capture the network packets in both our UAT and Production environments and found that the faulty packets were 4 bytes shorter in Production. This quickly led to us identifying and fixing the root cause - configuration drift that had occurred between the two environments.

In the very near future, we’ll be using tools like Chef and Phoenix Servers to completely eliminate & prevent configuration drift between our environments. But in the interim, Wireshark turned what would have been a gnarly debugging problem into just a couple minutes of investigation.