How extraction works

Most malware does not arrive as a bare executable. It arrives inside something: a ZIP, an ISO, a 7z, a Word document, an installer. The unpacker is what gets the scanner to the bytes it has to match.

exav’s extraction lives in exav-unpack, a standalone crate with no dependency on the scanning engine, #![forbid(unsafe_code)], and usable on its own as a safe extraction library (see Archive extraction).

The parity principle

If the software on the target’s machine can open a container and exav cannot, that container is an evasion vector: the payload lands intact and the scanner said nothing useful about it.

An attacker picks the delivery format, so exav targets everything real extractors open (7-Zip, WinRAR, WinZip, The Unarchiver, Windows Explorer, macOS Archive Utility, libarchive), not the formats seen in recent samples.

Supported formats tracks this as a matrix: every container and codec those extractors open, exav’s status against each, and the gap list of what is still missing.

Every input is extracted through a seekable source: a file, an HTTP range reader, or a stream buffered first (see Streaming & memory).

The shape of the unpacker

exav finds the format, then hands each member to the scanner one at a time, all under one set of limits (the Budget). A container is something that yields members, and nesting is the same step applied to what comes out.

detect() names the format, from the start of the object, its end, or a search of it. (A magic at the start does not always settle it: a DMG whose data opens with a bzip2 or xz run, as macOS writes compressed images, starts with that run’s magic, and its trailer is what names it a disk image.) walk() then hands the members to a visitor one at a time. A format whose decoder reads forward (ZIP, tar, CAB, ISO, the single-stream compressors, …) produces each member as a reader that decodes as it is read, so neither the container nor the member is resident. A format whose decoder needs random access over the whole container (7z, RAR, OLE, PDF, CHM, the virtual disks and filesystem images, …) is read whole, bounded before it is allocated, up to --max-object-bytes; past it the container is LIMITS-EXCEEDED. 7z’s members still stream once the container is in memory. Which it is is a property of the decoder, not of how exav was invoked; Supported formats lists both.

Past the walk both are the same: a member carries the same Budget as its parent, and scanning it re-enters detect(). So nesting needs no special case, and the extraction budget bounds an archive tree rather than each container in it.

A member’s metadata comes before its content, and a visitor that leaves a member unread costs nothing for it: a ZIP is listed from its central directory, and a tar’s headers are read in turn with the data between them skipped, so listing decodes nothing.

Bounded, budget before allocation

Every extraction runs under one Budget, reserved before a member is read rather than checked after. The defaults below are the scanner’s, with the flag that sets each; the crate’s, the command’s and the WebAssembly package’s are under the safety model.

Limit	Bounds	Default
output bytes	what formats decoded whole produce across the tree	1 GiB, lowered to fit `--max-process-bytes`
per-member size	a member decoded whole (a streamed member is bounded by the ratio and matcher budgets)	`--max-object-bytes` 256M
compression ratio	output ÷ input, the decompression-bomb guard	1000
file count	members across the archive and everything nested	`--max-members` 100000
recursion depth	archives inside archives	`--max-unpack-depth` 16
matcher bytes	bytes fed to the matcher across the tree, a CPU bound	`--max-matcher-bytes` 10G
emulation steps	instructions the PE unpacking emulator runs across the tree	`--max-pe-emulation-steps` 1,000,000,000

Reserving first keeps the peak bounded: a “1 GB from 4 KB” member never allocates 1 GB and then gets rejected.

exav-unpack itself never writes to disk. The scanner may write a member too large to hold to a temporary file (see Streaming & memory). That file is created with O_EXCL, mode 0600, under a name exav chooses, never the member’s path, so zip-slip, path traversal and symlink attacks still have nowhere to write.

Containers have two indexes

A ZIP lists its members twice: a local file header before each member’s data, and a central directory at the end. Every normal reader trusts the central directory, because that is what the specification says is authoritative.

So a member can be hidden: leave it out of the central directory and keep its local header and data in the file. Plenty of extractors still write it out.

exav does dual indexing: after the central-directory pass it scans the raw bytes for local headers the directory did not cover, and extracts those too. The same path is the salvage route when the central directory is corrupt, forged or truncated. On a ZIP read from its source rather than held in memory, the search covers at most 16 MiB of unclaimed bytes, and a search cut short there is reported, not taken as complete.

PK\x03\x04 is four bytes and occurs by chance in ordinary binaries, so a credibility check decides what counts as a member: version at most 6.3, no reserved flag bits, a non-empty path within the length limit and free of NULs, and a known compression method or, failing that, a readable UTF-8 name. A header with an unknown method and a clean name is still a member, reported unscannable, because a packer can stamp a nonsense method on a member to hide it from readers that check the method.

ISO images have the same shape: a CD image carries several volume descriptors, each with its own directory tree over the same sectors. A malicious ISO lists its payload only in the Joliet tree and leaves the primary tree empty, so a reader that parses the primary one sees an empty disc. exav walks every tree.

Recovering what a naive reader would skip

Deferred sizes. A streaming ZIP writer sets a flag and puts the member’s sizes in a data descriptor after the data, so the local header says neither where the member ends nor how much it decodes to. exav takes both from the descriptor whose compressed size is the distance back to the data’s start: one with its PK\x07\x08 marker, or one without it (the marker is optional) that ends where the next header starts. Its sizes are 8 bytes each when the local header has a ZIP64 field. With no descriptor, a member whose codec marks its own end is read up to the next header and decoded to that end; a stored member would swallow the following headers into its content, so it is reported instead.

Truncated streams. A gzip cut short mid-DEFLATE is common, and zcat recovers everything before the cut, so exav scans the salvaged prefix too. Nesting works: a truncated gzip(tar(...)) decompresses the gzip, walks the intact tar entries, and scans each one.

If nothing matches, the verdict is OK. exav is not a file-integrity validator: a scan is PARTIAL only when bytes present in the file were not read. When a stream runs out of input at the end of the file, every byte that exists was decoded and scanned, and the missing tail is absent rather than hidden. The same holds for a bzip2, Zstandard, xz or lzip file cut short, and for the last member of a ZIP, NSIS, EGG, ALZ, XAR, SWF, HWP 3.0 or UPX file cut short, or the last grain or cluster of a VMDK or qcow2 image. bzip2 and Zstandard decode by block, so the block the cut falls in gives nothing: its content went with the cut. Indexed formats are different: a damaged ZIP index can leave member data present but never enumerated, which is why dual indexing exists. A 7z archive keeps its index at its end, so one cut short cannot be listed and is UNSCANNABLE.

A stream damaged part way is different: the compressed bytes after the damage are present but were never decoded, so a clean prefix is UNSCANNABLE. What decoded before the damage is still scanned, so a detection in it is FOUND. A checksum that fails after a full decode is not damage of that kind: everything was decoded and scanned. In a DMG, the files in the part of a compressed run that decoded before the damage are scanned, and the image is UNSCANNABLE unless something is found: a decoder can run on past damage before it notices, so those bytes may not be the disk’s.

A member exav cannot read is reported

When extraction cannot produce a member’s content (an unsupported codec, encryption without a working password, a size deferred beyond recovery, an extent running past the end of the file), exav emits a metadata-only member carrying the name, size and reason, and the scan resolves to UNSCANNABLE or PASSWORD-PROTECTED. A credible header means those bytes are a member the target will extract, so leaving it out would be a clean verdict on something nobody looked at.

A ZIP member found only by its local header and cut off by the end of the file, as in a download that stopped or the first part of a split ZIP scanned on its own, follows the rule above: the bytes present are decoded and scanned, and the cut does not make the scan PARTIAL. One whose stream stops early with more of the file after it is reported as not whole, since with no central directory nothing says its rest is not in those bytes. So is one whose decoded size is not the size its header declares.

The metadata is still matched: .cdb container signatures match on member name, size, encryption flag and position, so a fake invoice.pdf.exe inside an archive is caught by name even when its content is unreadable.

Encryption

exav decrypts ZIP, 7z, DMG, PDF, Office and garbled ARJ members; the schemes are listed in Supported formats. Office documents are tried with VelvetSweatshop, Excel’s built-in default password, and ZIPs with a short list of passwords malware distribution uses (infected, virus, …), so neither needs configuration. --passwords and ClamAV .pwdb databases add your own.

An archive that stays encrypted is still a signal: “this member is encrypted” can be matched by .cdb signatures, and --partial-as password-protected=found turns it into a detection. An encrypted ZIP appended to a picture or document (a polyglot that archive tools open by its trailing directory) is reported PASSWORD-PROTECTED too, once its directory is found consistent.

Hostile input

The extractor’s entire input is attacker-controlled, so:

#![forbid(unsafe_code)], compiler-enforced, across exav’s own decoders (for third-party ones, see Dependencies).
Release builds enable overflow-checks, so a wrapping size calculation traps instead of bypassing a limit.
Each container walk and each file runs under catch_unwind: a decoder panic makes that container UNSCANNABLE, and a panic elsewhere is ERROR for that file, not an aborted batch.
The decoders are tested on wasm32, where usize is 32-bit and overflows on parsed offsets that a 64-bit host would hide show up. It is also the build the WASM sandbox ships.
A read error from the source is reported, never taken for the end of the container.
A member whose checksum fails after a full decode is scanned: a wrong CRC is a corrupt archive, not a reason to stop scanning, and the members (or bzip2 blocks, or lzip members) after it are decoded too. The exceptions are encrypted members, whose CRC confirms the password; a solid RAR member decoded after an earlier one in its group failed, which is UNSCANNABLE when its CRC disagrees; and an xz block failing its check, which its decoder reports as damage.

Validating a decoder

Every decoder is checked against a reference implementation’s output, never only against an encoder written next to it: a round trip through a matching encoder proves that the two agree, and passes when both misread the format the same way. The .Z (LZW compress) decoder once passed such a round trip while decoding real compress output to garbage after a few hundred bytes.

So each decoder gets an external oracle:

Format	Oracle
`.Z`	ncompress 5.0, built from source
CAB Quantum	`cabextract` / libmspack, over 107 generated streams
7z BCJ2	official 7-Zip 25.01 (`7zz x`)
UDF, VHDX, QCOW2, VMDK	`7zz x`, `qemu-img convert -O raw`
NTFS	`ntfscat`
LZ4	`lz4 -dc`
ARC	`arc` 5.21q and `nomarch`
Office OOXML/XLS crypto	msoffcrypto-tool reference vectors
ZIP PPMd	archives written by 7-Zip 25.01; for a model that runs out of memory, the `ppmd-rust` encoder, which exav’s decoder is ported from
bzip2	the `bzip2` crate’s encoder, a port of libbzip2
ARJ	a real archive, and mutations of it for each failure case

A subtly wrong decoder produces plausible bytes rather than an error, and a scan does not stop at a checksum mismatch (see above), so the tests compare what a decoder gives with the files that went in. Where the format carries one (RAR CRC-32, WIM SHA-1, ARC CRC-16, UPX Adler-32), the archive’s own checksum is checked when verification is on.

A new decoder should come with an oracle; not having one is why several formats in the gap list stay closed.