Skip to content
exav

How scanning works

How the scanner works inside. This is the high-level picture; implementation detail lives in the repository’s docs/, and how the crates fit together is in Technical architecture.

Input

Section titled “Input”

Every input is scanned through a seekable source (Read + Seek): a local file, an HTTP range reader, or a buffered stream. Stdin, INSTREAM and ICAP bodies are buffered first (in memory, then in a temporary file), because container formats need to seek. scan_seekable hands the input to the one pipeline every object takes, the input and whatever is unpacked, decoded or carved out of it, at any size and depth. See Streaming & memory for what is held in memory and what is read as it is needed.

How a scan flows

Section titled “How a scan flows”
How a scan flows An object enters identify(), which types it from its start, its end, or a search of it. An archive has its members walked one at a time under a single shared Budget, then its own bytes matched. Anything else has its own bytes matched first, then what is inside it analysed: a packed or installer executable's payload, carved images, decoded payloads, heuristics. Everything found inside re-enters identify(), bounded at --max-unpack-depth of 16. Both paths end in OK, FOUND, or PARTIAL with its category: LIMITS-EXCEEDED, UNSCANNABLE or PASSWORD-PROTECTED; an input that cannot be read, even part way, is ERROR. object identify() archive anything else OK · FOUND · PARTIAL · ERROR PARTIAL: LIMITS-EXCEEDED · UNSCANNABLE · PASSWORD-PROTECTED start · end · a search members one at a time, sharing one Budget; then its own bytes its own bytes; then what is inside: payloads, carved images, heuristics not fully examined is PARTIAL, never OK; input unreadable is ERROR everything found inside re-enters identify() · --max-unpack-depth (16)

A file is OK, FOUND, or PARTIAL when it could not be fully examined, with a category saying why: a limit (LIMITS-EXCEEDED), content with no decoder (UNSCANNABLE) or encryption (PASSWORD-PROTECTED). Anything that stops a scan early is PARTIAL, never OK. ERROR is exav failing rather than a fact about the file: a path it cannot open, a database that will not load, or a source that fails part way through a scan, whose unread bytes are not taken for the end of the file (see Verdicts & exit codes).

File type comes from the content, not the extension: an executable renamed .jpg is still scanned as an executable. The budgets every extraction runs under are in How extraction works.

Signature types and runtime structures

Section titled “Signature types and runtime structures”

Different ClamAV signature types compile to different runtime structures rather than one matcher:

Source Runtime structure Matching
.ndb / .db bodies + .ldb body subsignatures an index of literal anchors, one per body, each tagged with its (target type, case) partition; a body pinned near a fixed offset is kept out of it the index is looked up at each position of the object, and a hit fans out to every body sharing that anchor; each candidate is then verified (wildcards / gaps / nibbles / alternation / offset / nocase). A pinned body is checked where it can start
.ldb PCRE subsignatures regexes compiled lazily, gated by the trigger expression linear-time regex; patterns with lookaround, backreferences or atomic groups use a backtracking engine under a step bound
.hsb / .hdb a size-keyed hash table whole-file digest lookup
.mdb / .msb a section-hash table per-PE-section digest lookup
.cdb container-metadata matchers matched on archive members (name/size/encryption/position)
.imp a size-constrained import-hash map PE imphash lookup
.fdb an imphash map and a list of TLSH digests imphash lookup; TLSH distance to each digest, under the signature’s threshold
.cbc a sandboxed bytecode interpreter (no JIT) a program runs on an object when its trigger matches; a hook program on every file of its type
.yar / .yara a native YARA engine near-full YARA, no runtime codegen

The other formats (icons, certificates, file-type magic, allowlists, ignore lists, phishing and passwords) are listed in Signatures. So the engine is an anchor index, hash tables, lazy regexes and interpreters. Most of the memory is the compiled bodies, the index and the hash tables; see the prebuilt database.

How a body signature is matched

Section titled “How a body signature is matched”

A body is hex with wildcards (??, nibbles such as a?), gaps ({10-}) and alternations, optionally tied to an offset: 0: (the start), EOF-n, EP+n (the entry point) or Sn+n (section n). A ::f modifier asks for a match on word boundaries.

From anchor hit to detection An object's bytes go through the partitions whose target fits the file type. One sweep looks up the anchor index at each position, on the bytes folded to lowercase, and reports every anchor hit as a group and an offset. Each hit fans out to every body sharing the anchor, and each is verified around the hit. A verified .ndb body is a detection. A verified .ldb subsignature is counted, and the logical expressions of the .ldb signatures whose subsignatures hit are evaluated after the sweep; a PCRE subsignature runs only when its trigger holds. object partitions, by (target, case) one sweep over the anchor index fan out, then verify each body .ndb body verified .ldb subsignature verified logical expression its bytes, read once only those whose target fits the file type run every anchor at once: hits as (group, offset) every body sharing the anchor: prefix back, suffix forward, gaps as intervals, offset, ::f word bounds a detection; a normal scan stops here, --all-matches keeps verifying counted, with the offset of its first match (byte comparisons are taken from there) after the sweep, only for the signatures whose subsignatures hit; a PCRE runs if its trigger holds

An anchor is a literal run each body must contain; everything else in the body (wildcards, nibbles, alternations, gaps, the offset) is checked only where an anchor hit. Every anchor is in one index, tagged with its target type and case, and a lookup reports only the partitions that can matter for the object’s type, so a PE never verifies an HTML-only body. An anchor is indexed by the bytes of it the signature set shares least: the whole anchor at one to three bytes, four of them at four or five, and past that two overlapping six-byte windows, of which only the one at an even position is looked up. Filters small enough to stay in cache turn away most positions before the tables are read. Every anchor is looked up on the bytes folded to lowercase, and a case-sensitive one is then compared exactly; the object is copied lowercased only when a case-insensitive body has to be verified. The object is read once, whatever the number of signatures.

A body whose offset pins it to a window of starts at most 4096 bytes wide, counted from the start, the end, the entry point or a section (0:, EP+0:, 100,50:), is not indexed: it is checked at the places it can start, and costs nothing elsewhere in the object.

Verification never backtracks: a body is a sequence of elements, and the positions it can have reached after each one are kept as a set of intervals, so a {10-} gap widens an interval instead of multiplying attempts. Backtracking is left to PCRE subsignatures with lookaround, backreferences or atomic groups (\R and possessive quantifiers among them), which the linear engine cannot run: they run on a backtracking engine under a step bound. For lookaround, a linear prefilter (the pattern with its lookarounds removed, which matches at least as much) rules most objects out first.

How far a check reads around a hit

Section titled “How far a check reads around a hit”
How far a check reads around an anchor hit An object drawn as a strip of 64 KiB blocks. The sweep reads it in order, a chunk at a time, looking up the anchors that straddle a chunk seam from the bytes on each side. At an anchor hit, verification reads the body's prefix backward from the anchor and its suffix forward, as far as the body needs; a gap with no upper bound can reach the end of the object. A block no longer in the cache is read again. The cache bounds what is held, not how far a check reaches. THE OBJECT, IN 64 KIB BLOCKS anchor sweep: in order, a chunk at a time; an anchor across a seam is looked up there, so nothing is read twice prefix: read backward from the anchor suffix: read forward a {n-} gap can reach the end of the object not in the cache: read again Offsets are checked against where the match starts: EOF-n everywhere, EP+n and Sn+n only on an object held in memory. A ::f body is checked against the byte on each side. Held: 8 MiB of 64 KiB blocks (the sweep reads 1 MiB chunks), least recently used dropped first.

Neither the sweep nor a check has a window to fall out of. The sweep reads the object once, in order, and looks up the few positions before a seam from the bytes on both sides of it, so an anchor split across a seam is found without re-reading the chunk. A check asks for the bytes it needs at their offset, before or after the hit, from memory or, past --max-object-bytes, from the block cache.

Over an object not held in memory, regular expressions (PCRE subsignatures, YARA strings) run as lazy DFAs stepped through it, forward to find where a match ends and backward from there to find where it starts. A PCRE with lookaround runs its lookaround-free superset that way: when that finds nothing, the PCRE cannot match either; when it finds something, the scan is LIMITS-EXCEEDED unless something else is found. The same holds for a PCRE with a backreference or an atomic group whose trigger holds.

Memory and spill files

Section titled “Memory and spill files”

An object larger than --max-object-bytes is read through a block cache rather than held, and a member or text view too large to hold is written to a spill file the host provides. The library never writes to disk itself. Streaming & memory shows what reads through the cache, which checks need an object whole, and where spill files fit.