Skip to content
exav

Streaming & memory

Every file gets the same scan whatever its size, except the few checks that parse it whole (below):

  • every .ndb form: wildcards, gaps, anchored offsets, file-type targets;
  • .ldb logical signatures, with their regexes and byte comparisons;
  • the normalised views of HTML, text and scripts that those signatures are written against;
  • YARA rules, bytecode programs, whole-file hashes;
  • archives decoded as they are read, embedded files, heuristics.

What changes with size is where the file is while that runs.

Up to --max-object-bytes: in memory

Section titled “Up to --max-object-bytes: in memory”

A file up to --max-object-bytes (256 MiB by default) is read into memory, and so is an HTTP source that size. Memory is roughly the file’s size, plus a lowercase copy when a case-insensitive signature has to be checked and, for text, one normalised copy at a time. The deobfuscated JavaScript view stops at 32 MiB; a script whose view is longer is reported LIMITS-EXCEEDED unless something is found, though its raw bytes are still scanned in full.

Past it: through a block cache

Section titled “Past it: through a block cache”

A larger file is read through a cache of 64 KiB blocks, of which at most 8 MiB are held, least recently used dropped first. So are an HTTP source of that size and a spill file (below). Verifying a match can still look anywhere in the file: the cache bounds what is held at once, not how far a check may reach, and a block that was dropped is read again when it is needed. The scan is the same one, with the same results, only slower, since the file is read a few times over: once for format detection’s search, carving and the digests (when a hash signature has its size), once for the signature sweep (every anchor at once), twice more for YARA’s prefilter when rules are loaded, and again in the parts a check needs.

One source, read through a block cache A file or an HTTP source past --max-object-bytes, or a spill file, is read through a block cache of 64 KiB blocks holding 8 MiB. Every check asks it for bytes at an offset: the signature sweep a chunk at a time in order, verification and the container walk at any offset, the regexes stepped through, the hashes, typing's search and carving in one shared pass. The checks that parse an object whole run only when it fits within --max-object-bytes. SOURCES WHAT READS IT file HTTP range spill file BlockCache signature sweep verification regexes hashes · typing · carving container walk whole-object checks past --max-object-bytes past the limit too a large member or view 64 KiB blocks, 8 MiB held, least recently used dropped; bytes at any offset, or in 1 MiB chunks in order chunks in order, one pass windows at any offset lazy DFAs stepped through one shared pass; start and end headers and members, by offset held whole only: PE, RAR, OLE

A few things need the file whole and are not done past the limit:

  • the structure of a PE: entry-point and section offsets, section hashes, imports, icons, the Authenticode signature, UPX and packer unpacking;
  • YARA’s pe, elf and dotnet modules;
  • a PCRE subsignature with a lookaround or a backreference, and the fuzzy_img image hash;
  • containers whose format is read whole (7z, RAR, OLE, PDF, the virtual-disk and filesystem images and others; see Supported formats).

A file for which one of these applied is reported LIMITS-EXCEEDED, naming --max-object-bytes, unless something is found. Raise the limit to have them run, at the cost of memory.

The normalised views of a large text file are as large as the file, so they go to a spill file.

Archive members

Section titled “Archive members”

Archives decoded as they are read (ZIP, tar, CAB, ISO/UDF, DMG, LHA, ar, cpio, the single-stream compressors, self-extracting executables) are walked member by member at any size and any depth. Such a member is held in memory up to the same limit, and past it written to a temporary file and scanned from there. With --spill-dir off, or past --max-spill-bytes, it is not scanned and the file is LIMITS-EXCEEDED. 7z members stream the same way once the container is read.

A member that has to be decoded whole (an encrypted ZIP member, a file inside a DMG, a member of a format read whole) is LIMITS-EXCEEDED past --max-object-bytes.

Stdin, INSTREAM and ICAP bodies

Section titled “Stdin, INSTREAM and ICAP bodies”

Container formats need to seek (a ZIP’s directory is at its end), so a stream is buffered before it is scanned: in memory up to --spill-threshold-bytes (16 MiB), then in a temporary file up to --max-spill-bytes (2 GiB). The buffered stream then goes through the same scan as a file. A stream past --max-input-bytes or the spill ceiling is scanned as far as it was held and is LIMITS-EXCEEDED unless that finds something, as a file past --max-input-bytes is. See buffering a stream.

Spill files

Section titled “Spill files”
The two uses of spill files Before a scan, a stream (stdin, INSTREAM, ICAP) is held in memory up to --spill-threshold-bytes and then in a temporary file up to --max-spill-bytes, and becomes a seekable source for the scan. Inside a scan, a streamed archive member that decodes past --max-object-bytes, and the normalised text views of a large text file, are written to spill files and read back through a block cache. A stream past its budget, or past the threshold with --spill-dir off, has what was held scanned; with --spill-dir off, such a member or view is not scanned. Either way a scan that finds nothing is LIMITS-EXCEEDED. BEFORE THE SCAN: A STREAM INSIDE THE SCAN: WHAT IT MAKES stdin · INSTREAM RAM temp file the scan member past the limit text views, large file spill file scanned from there ICAP bodies --spill-threshold-bytes --max-spill-bytes a seekable source decoded past --max-object-bytes HTML, text, JS normalised written as it is made through a block cache Past --max-spill-bytes, or past the threshold with --spill-dir off, what was held is scanned and the stream is LIMITS-EXCEEDED unless that finds something. --max-total-spill-bytes bounds every spill file of a process together.

The library never writes to disk itself: the host hands a scan somewhere to spill (the exav binary passes its --spill-dir files and budgets). Only a member decoded as it is read can spill; one decoded whole is bounded by --max-object-bytes. Without a spill, a member that decodes past --max-object-bytes is not scanned, the normalised views of a text file that large are skipped, and a scan that found nothing is LIMITS-EXCEEDED rather than OK.

The database, not the file, is what uses memory

Section titled “The database, not the file, is what uses memory”

Building the signature index from raw ClamAV databases briefly takes more memory than the index itself. A prebuilt .exavdb is built once on a capable host and loads at about its final size.

Tuning

Section titled “Tuning”

--max-object-bytes (256M) is the most memory one object may use, a file, a decompressed member, an LZ window or a decrypted blob; several such buffers are alive at once across nesting levels, so it does not bound the total. What formats decoded whole hold across one top-level file is charged against 1G, and --max-process-bytes keeps both inside the memory a scan gets. Every bound, and which one to change, is in Limits and tuning; a worked example is in Sizing a server.