Reading Time: 6 minutes

TL;DR Many network protocols are undocumented or proprietary, which makes traffic analysis, auditing, and malware investigation harder when there is no specification to work from. Protocol reverse engineering (PRE) tries to recover message formats directly from network traces, but existing static tools tend to rely on fixed assumptions, struggle with binary or mixed-format protocols, and do not scale well to large captures. In our paper at the 11th IEEE European Symposium on Security and Privacy (EuroS&P 2026), “MARISSA: Efficient Inference of Network Protocols using Similarity Digest Clustering and Multiple Sequence Algorithms”, we present MARISSA, an automated tool that infers message formats from PCAP files. MARISSA combines similarity digest clustering with multiple sequence alignment to identify field boundaries, works on both textual and binary protocols, and reduces runtime by factors of 2x to 15x compared to the tools we tested. Interested? Keep reading this briefing or grab the code on GitHub.

Why Protocol Reverse Engineering Is Hard

Understanding the structure of a network protocol is useful for traffic inspection, intrusion detection, and forensic analysis. Widely used protocols such as HTTP, DNS, and TLS are documented and standardized, but many others are not: custom schemes in embedded systems, legacy enterprise software, and malware command-and-control channels often lack public specifications. In those cases, an analyst has to recover the message format from captured traffic.

PRE approaches fall into two groups. Dynamic methods instrument the program binary to observe how it builds and parses messages, which gives good results but needs access to a running system and the ability to modify its execution environment. That requirement limits their use in IoT or malware scenarios where the endpoint is not available. Static methods work only from captured traffic, so they are non-intrusive and broadly applicable, but they have their own limits: many rely on recognizable delimiters or keywords, assume prior knowledge of the format, or become slow as the trace grows.

We designed MARISSA to address two goals at the same time: infer message structure from raw, unknown traffic, and scale to large and diverse captures within reasonable runtimes.

The Solution: Similarity Digests Plus Sequence Alignment

MARISSA is a static, protocol-agnostic tool. It does not assume delimiters or format regularity, which lets it handle both textual and binary protocols. It builds on two ideas.

The first is to cluster messages using similarity digest algorithms (SDAs) rather than full byte-by-byte comparison. SDAs produce compact fingerprints that reflect the structural similarity of byte sequences, which keeps cost down while preserving structural information. We evaluated TLSH and ssdeep, and adopted ssdeep as the default clustering backend because it gave the most consistent results on our datasets.

The second is to infer field boundaries with multiple sequence alignment (MSA), a technique from bioinformatics that aligns many sequences at once to reveal regions that stay fixed (static headers) and regions that vary (payloads). To reuse mature MSA libraries, MARISSA transcodes each protocol message from hexadecimal into a compatible character encoding (hex digits A-F are kept, digits 0-9 map to letters G-P), a reversible and loss-less step that preserves message length and byte alignment.

Inside the Pipeline

MARISSA takes a PCAP file as input and produces an inferred message format as output. The workflow has three phases.

Figure 1: The MARISSA pipeline, detailing the pre-processing, processing, and post-processing phases.

Pre-processing. Raw traffic is cleaned and normalized: message extraction, protocol filtering and header stripping (down to the protocol payload), deduplication of retransmitted packets, prefix sanitization to strip static “magic number” prefixes that would otherwise bias the digests, and stream separation by MAC address to keep different communication flows apart. The output is a set of clean, role-grouped message payloads.

Processing. For each stream, MARISSA computes pairwise similarity digest scores into a distance matrix, clusters the messages with OPTICS (a density-based algorithm that handles clusters of varying shape and outliers better than k-means), and aligns the messages in each cluster with MSA. We integrated several genomics MSA tools (MAFFT, Clustal Omega, FAMSA, and MUSCLE) and selected MAFFT as the default for its balance of accuracy and speed. A lightweight cluster-merging step then groups message variants that share the same type identifier, reducing over-segmentation.

Post-processing. The aligned messages are analyzed to classify each field as static (fixed length and content), dynamic (fixed length, variable content, such as a sequence number or timestamp), or variable (variable in both length and content). The result is a structured description of each message type, exported as CSV, which can feed documentation, interoperability testing, fuzzing, or emulation.

Key Findings

We evaluated MARISSA on nine real-world protocols spanning textual and binary formats: DHCP, DNS, FTP, Modbus, ICMP, MQTT, NTP, S7comm, and SMB, with datasets of 100, 500, and 1000 messages. We measured clustering quality and field inference accuracy (precision, accuracy, recall, and F1) against ground truth from tshark dissectors, and compared MARISSA with four existing tools: NEMETYL, NetPlier, BinaryInferno, and MDIplier.

  • Broader protocol coverage. MARISSA processed all nine protocols across all dataset sizes. In our runs, NEMETYL completed only 3 of the 9 protocols, and NetPlier and MDIplier did not handle FTP traffic.
  • Higher inference quality. Across most protocols and dataset sizes, MARISSA produced the best clustering and field inference results among the tools tested. Clustering stayed stable regardless of dataset size, while field inference improved with more messages, which reflects the extra structural redundancy in larger datasets.
  • Lower runtime. MARISSA reduced execution time by factors ranging from 2x to 15x compared to the other tools. We attribute this to the lightweight similarity digest comparisons and the use of optimized MSA libraries.

Case Study: Reverse Engineering ARPChat

To show practical utility, we applied MARISSA to ARPChat, a custom chat application that communicates over the ARP protocol. Its non-standard transport and minimal structure make it a good test case for inferring an unknown format.

Figure 2: Message structure inferred from the ARPChat protocol. D(X) denotes a fixed-length dynamic field of X bytes, and V denotes a variable-length field.

MARISSA recovered the message layout: a static 3-byte prefix (uwu) that separates ARPChat traffic from standard ARP, followed by a 1-byte type identifier (00 chat message, 01 connection request, 02 presence update, 03 disconnect), and the corresponding dynamic and variable fields. Using only this inferred format, we replicated the application’s key features through custom scripts (sending messages, spoofing user presence, and joining chats) without access to the original client.

The case study also shows an honest limit. MARISSA identified three structural groups rather than four, because two byte positions that encode a sequence number and a message count were never updated due to a bug in ARPChat, so they looked static in every captured message. This is a reminder that inference quality depends on the quality of the captured traces.

Takeaways for Reverse Engineers

  • MARISSA works without delimiters, keywords, or prior format knowledge, so it applies to binary and proprietary protocols where keyword-based tools tend to fail.
  • Its output is a structured, byte-level field map per message type, ready to use as input for fuzzers, emulators, or interoperability testing.
  • As a static tool, it operates on captured traffic and does not need access to a live system. It cannot analyze encrypted payloads (TLS, SSH, HTTPS), which stay opaque to any static approach.
  • Trace quality matters. Mixed-protocol captures should be filtered to a single protocol first, and malformed or buggy applications can blur fields that would otherwise be distinct.

Ongoing Work

We plan to extend MARISSA with semantic inference to label fields with domain-specific meanings (usernames, timestamps, commands), support for encrypted traffic through decryption modules or controlled instrumentation, better handling of mixed-traffic captures, and integration into broader workflows such as protocol fuzzing, automated test generation, and IDS rule derivation. To support reproducibility, we release the source code together with the containerization and evaluation scripts used in the paper.

Funding Acknowledgments

This research was supported in part by grant PID2023-151467OA-I00 (CRAPER), funded by MICIU/AEI/10.13039/501100011033 and by ERDF/EU, by grant TED2021-131115A-I00 (MIMFA), funded by MICIU/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR, by grant Proyecto Estratégico Ciberseguridad EINA UNIZAR, funded by the Spanish National Cybersecurity Institute (INCIBE) and the European Union NextGenerationEU/PRTR, by grant Programa de Proyectos Estratégicos de Grupos de Investigación (DisCo research group, ref. T21-23R), funded by the University, Industry and Innovation Department of the Aragonese Government.

And that’s all, folks! In the meantime, we encourage you to read the full paper at EuroS&P 2026, explore and contribute to the tool on GitHub, or reach out to us if you’re interested in collaborating on protocol reverse engineering, traffic analysis, or message format inference. Thanks for reading, and stay secure!

Declaration of Generative AI Technologies in the Writing Process

During the preparation of this post, the authors used Claude (Claude Opus 4.8 model) to improve readability and language. After using this tool, the authors reviewed and edited the content as necessary and take full responsibility for the content of this publication.