Discovery
Discovery decides which files count as documents, and it is deliberately narrow. Files with
the exact lowercase suffix .md or .markdown are structured-markdown; .mdx files are
structured-mdx. Six exact extensionless basenames, README, CONTRIBUTING, CHANGELOG,
SECURITY, SUPPORT, and CODE_OF_CONDUCT, are extensionless-markdown and use the
Markdown adapter. .cursorrules and llms.txt are plain-advisory: they are scanned by an
adapter that extracts no references, unless a policy include names one exactly and binds it
to a grammar, which then reads it; a tree or suffix include keeps the built-in class.
llms.txt is Markdown by its own proposal, so binding it to markdown checks its links. .adoc and .asciidoc are structured-asciidoc, and .rst is
structured-rst. .txt stays off both lists: the suffix says nothing about what is inside.
One file in the tree does say. A Sphinx conf.py names the suffixes that project reads, and
Django’s names .txt, so a file carrying a declared suffix under the directory holding that
conf.py is structured-rst and reads under the reStructuredText grammar. Nothing else moves:
the same suffix anywhere outside that directory is a text file, because a suffix one site
declares says nothing about a file beside it. What a documentation router
serves holds the two declaration forms that are read and what is declined.
A repository policy include still binds a suffix no configuration declares. .ipynb and .org
are unparsed-markup: this engine has no parser for a notebook or
for Org markup, so those files are discovered and counted as unsupported-document-format
and their content is never read, the same honest count an unbound policy include reaches.
Reports retain unparsed-markup on either snapshot side, including files under excluded trees.
Notebook Markdown has measured reference yield, but its decoded cell spans are not physical JSON
spans and the current report cannot name a cell; the
notebook measurement records why that
format remains visible but unparsed. Quarto’s .qmd suffix remains outside the document set,
while MyST’s .md files receive only the CommonMark/GFM semantics their suffix promises. The
adjacent-format measurement records
why MyST, Quarto, and Org need renderer-aware adapters rather than suffix aliases.
An include may instead bind one of the five built-in adapters, which reads the named path or
tree under that grammar without installing another parser. One grammar answers per path per
evaluation: the candidate policy’s bindings, or the base’s when the candidate carries none,
and dropping a binding while keeping the include is policy weakening under
policy/include-binding-removed. Every
other file is a possible reference target, not a built-in document. These rows come directly from the
classifier.
Nine directory names are always skipped, wherever they appear in a path. The list is the
scanner’s own constant, checked in CI, and --explain-scope prints the same one:
node_modules vendor third_party dist build .next target test tests
The names are fixed: no configuration adds one or takes one away. A repository policy can
still readmit coverage underneath a skipped name, because policy adds coverage and never
removes it: a document include admits its one exact path, a plain tree include admits the
whole subtree, and a suffixed tree include admits only its exact tail. That is the monorepo
lever. A package legitimately named build, dist,
test, or tests keeps its prose scanned through one tree include in
the repository policy, and there is no mechanism in the other direction:
Amiss always reads the whole repository, so a monorepo cannot scope a run down to one
package. Skipping is
visible in both directions: skipped documents still show up in the report’s counts, as
excluded. This repository relies on the rule itself: its vendored parser test corpus lives
under corpus/third_party/ exactly so that fixture files full of deliberately broken links
are never read as prose.
test and tests joined the list on measurement. Across fifteen public repositories, 391 of
3,934 discovered documents sat under one of those names and carried 400 references between
them, about one apiece, while producing 45 of the missing rows, every one a deliberately
broken fixture. Two were prose: the READMEs explaining pydantic’s own test suites. Prettier
is the case that made it a defect rather than noise, since seven intentionally malformed MDX
fixtures under tests/format/ refused its entire run; it scans now, and the first thing it
reports is a real break in its contributing guide.
Eleven paths through the classifier:
docs/guide.md structured-markdown scanned
site/page.mdx structured-mdx scanned
README extensionless-markdown scanned
llms.txt plain-advisory scanned, nothing extracted
docs/guide.adoc structured-asciidoc scanned
docs/guide.rst structured-rst scanned
docs/faq.txt structured-rst the docs/conf.py above it declares .txt
notes/plan.txt not a document no conf.py above it declares that suffix
notes/plan.org unparsed-markup counted, never read
vendor/lib/README.md excluded the vendor component is in the closed set
src/parser.rs not a document a reference target only
A file under a declaring root that is not prose is read all the same. Django’s
docs/requirements.txt is 72 bytes of package names: it is discovered, parsed under the
reStructuredText grammar, and extracts nothing, which costs the bytes it holds and reports
no claim. That is also what Sphinx does with it, since Sphinx reads every file under its
source directory carrying a source suffix and then says the file is in no toctree. Reading
the declaration rather than the file is the whole rule: a suffix no conf.py names stays
outside the document set however much prose it holds.
Markdown and MDX recognize frontmatter only at byte zero, optionally after one UTF-8 BOM.
The first complete line must be exactly --- or +++; the closing line repeats it, except
that --- also permits .... A recognized region is opaque to the document grammar and may
contain at most 65,536 bytes, excluding the BOM. An opener without a permitted closer, or a
closer past that bound, remains ordinary document text. The published
frontmatter vectors execute this boundary,
including LF, CRLF, bare CR, BOM, and exact-limit cases, through the production recognizer in
the frontmatter test.
A reference definition whose decoded label begins with exact lowercase amiss: is a
reserved governed claim. Entity and escape decoding happens before that test; case is not
folded. Every reserved definition node contributes its exact source digest, including a
losing normalized duplicate, and only the first normalized definition controls whether a
consumer becomes an ordinary reference. A governed definition spelling the closed value
grammar is evaluated as a claim, described in Claims. Everything else in the
namespace on the candidate side is an unsupported capability boundary: the run ends
incomplete with exit 2. A base-only definition does neither. The
governed-definition vectors
drive extraction, source hashing, candidate-only grouping, and report construction in the
governed test,
including one refusal vector per clause of the value grammar.
A file the engine tried to read and could not is unsupported too, and the run keeps going.
Bytes that do not decode as the suffix promises, an HTML comment the MDX grammar rejects, a
tab-indented list inside a JSX block that markdown-rs rejects though @mdx-js/mdx accepts it, a
document past one of its ceilings: the document is counted, its row carries undecodable-document
for bytes, unparsable-document for markup the parser refuses, or resource-ceiling-crossed, and
every other document in the repository is still reported. A
45 KB link destination past raw-link-destination-bytes costs less, only that one reference,
which is declined as oversized-destination. The two are apart because the fix is: one is an encoding,
the other is the source, and <!-- --> in an MDX document is {/* */} written wrong. The
exception is a page a Docusaurus site holds, since Docusaurus reads that comment by default
where MDX alone refuses it. There the comment the grammar rejected is read as a comment, every
other byte keeping its line and column, and a link written inside it is not read. Superset’s
documentation is the case: 129 pages open with a licence header in one, and all of them are
read. Two undecodable translations used to cost a
2,619-document site its whole report at exit 2, and that is the failure this class replaced.
Nothing is skipped quietly: the human output names each such file, the
unsupported-document-format finding records that its references went unchecked, and a
repository policy protecting that path still fails the run through coverage-reduced.
Every count is reported: discovered, scanned, unsupported, excluded, unlinked. The last count is
zero unless trusted complete site-build evidence supplies a source root, navigation manifest,
rendered entrypoints, and the source documents reachable through the completed HTML link graph.
unlinked-document then names a scanned structured document inside that root which is neither the
manifest nor reachable. Outbound-reference counts never stand in for navigation evidence. The exact
predicate is in the document finding evaluator.
Paths are treated as bytes. Amiss does not fold case and does not normalize Unicode,
because Git addresses files by exact bytes, and a checker that guesses two names are
equivalent will eventually insist that two different files are the same file. A name whose
bytes are not valid UTF-8 is still a name: the entry is classified by the same suffix
rules, scanned, and reported, with its path written as a bytes object holding the raw
bytes as an array of byte values, since JSON text cannot carry them directly. Only a name outside
the path grammar itself, one containing a backslash or a NUL byte, or a bare . or ..
segment, is refused. That refusal is never quiet: the run stops as incomplete, the error
is recorded as UNREPRESENTABLE_PATH with the exact bytes in path_bytes, and the
exit is 2. Dropping such an entry silently would be the worst bug this tool could have:
the report would come back green with a document missing from it, and a missing row is
the one defect no reader can notice.
Both commit-tree and staged-index discovery emit document rows strictly increasing and
unique by those raw path bytes. That ordering is load-bearing: exact document queries and
policy-inventory checks use binary search over it, and two-sided report construction
merge-joins the ordered sides. The
discovery ordering test
pins the Git directory-boundary ordering against both snapshot modes, the report test
pins interleaved base and candidate rows, and the amiss-scan pipeline benchmark
tracks lookup and merge cost as the row count grows.