retrieval.project_loader

discover_files / load_documents / load_chunks / load_chunk_documents / load_ast_chunk_documents over a project root. load_chunk_documents is the production loader most retrievers index over — it returns one chunk-granularity Document per span (docid = "{path}:{start}-{end}"). load_ast_chunk_documents is the AST-boundary analog TreeSitterRetriever indexes over: chunks at tree-sitter node boundaries when a file's language is supported, carrying an enclosing function/class breadcrumb (context), and falling back to the line-based chunker per file otherwise.

Load documents/chunks from an invoking project's directory tree.

Stdlib-only (os, pathlib, fnmatch) — no network libraries — so the default retrieval pipeline stays fully offline. Discovers text-like files under a project root, skipping VCS/dependency/build directories and files that look like secrets by name.

A file is eligible for discovery if its suffix is in DEFAULT_EXTENSIONS or its basename case-insensitively matches DEFAULT_INCLUDE_BASENAMES (well-known extensionless project files such as Dockerfile, Makefile, LICENSE); either way it must still pass the exclude-glob and size checks.

The filename deny-list (DEFAULT_EXCLUDE_GLOBS) is a best-effort guard against accidentally indexing credential files by name — it is not content scanning. A file named notes.txt containing an API key will still be indexed; do not rely on this module for secret detection.

discover_files(root, *, extensions=DEFAULT_EXTENSIONS, exclude_dirs=DEFAULT_EXCLUDE_DIRS, exclude_globs=DEFAULT_EXCLUDE_GLOBS, include_basenames=DEFAULT_INCLUDE_BASENAMES, max_bytes=MAX_FILE_BYTES)

Walk root and return sorted, deterministic list of eligible file paths.

Prunes exclude_dirs (and any dir ending in .egg-info) in-place during os.walk so excluded subtrees are never descended into. A file is eligible if its extension is in extensions or its basename case-insensitively matches include_basenames, and it also passes the filename deny-list and size checks. Symlinks are not followed.

Source code in engine/retrieval/project_loader.py

def discover_files(
    root: "os.PathLike[str] | str",
    *,
    extensions: frozenset = DEFAULT_EXTENSIONS,
    exclude_dirs: frozenset = DEFAULT_EXCLUDE_DIRS,
    exclude_globs: Iterable[str] = DEFAULT_EXCLUDE_GLOBS,
    include_basenames: frozenset = DEFAULT_INCLUDE_BASENAMES,
    max_bytes: int = MAX_FILE_BYTES,
) -> List[Path]:
    """Walk *root* and return sorted, deterministic list of eligible file paths.

    Prunes ``exclude_dirs`` (and any dir ending in ``.egg-info``) in-place
    during ``os.walk`` so excluded subtrees are never descended into. A file
    is eligible if its extension is in *extensions* or its basename
    case-insensitively matches *include_basenames*, and it also passes the
    filename deny-list and size checks. Symlinks are not followed.
    """
    root_path = Path(root)
    found: List[Path] = []

    for dirpath, dirnames, filenames in os.walk(root_path, followlinks=False):
        dirnames[:] = [d for d in dirnames if not _is_excluded_dir(d, exclude_dirs)]

        for filename in filenames:
            file_path = Path(dirpath) / filename
            if _is_eligible_file(
                file_path, extensions, include_basenames, exclude_globs, max_bytes
            ):
                found.append(file_path)

    return sorted(found)

load_ast_chunk_documents(root, *, policy=None, **kw)

Discover files under root and return one AST-boundary chunk- granularity Document per chunk (see retrieval.ast_chunker).

For each file, resolves a tree-sitter language from its suffix (via language_for_path); when a language is found, chunks with chunk_code at AST node boundaries (budget policy.ast_max_chars), carrying a breadcrumb context (enclosing function/class path, e.g. "Bar.baz") on each Document. Files with no mapped language, or whose AST chunking returns no chunks (e.g. a severely broken parse), fall back to the line-based chunk_document — same as load_chunk_documents, including policy's target_chars_for bucket — with an empty context. policy defaults to DEFAULT_POLICY (reproducing today's behavior); **kw is forwarded only to discover_files.

A missing tree-sitter-language-pack install surfaces as a RuntimeError (raised by chunk_code) and is not caught here: it is the signal an all-mode index run uses to skip this strategy (see retrieval.cli._index_all).

Source code in engine/retrieval/project_loader.py

def load_ast_chunk_documents(
    root: "os.PathLike[str] | str",
    *,
    policy: Optional[ChunkingPolicy] = None,
    **kw,
) -> List[Document]:
    """Discover files under *root* and return one AST-boundary chunk-
    granularity ``Document`` per chunk (see ``retrieval.ast_chunker``).

    For each file, resolves a tree-sitter language from its suffix (via
    ``language_for_path``); when a language is found, chunks with
    ``chunk_code`` at AST node boundaries (budget ``policy.ast_max_chars``),
    carrying a breadcrumb ``context`` (enclosing function/class path, e.g.
    ``"Bar.baz"``) on each Document. Files with no mapped language, or whose
    AST chunking returns no chunks (e.g. a severely broken parse), fall back
    to the line-based ``chunk_document`` — same as ``load_chunk_documents``,
    including *policy*'s ``target_chars_for`` bucket — with an empty
    ``context``. *policy* defaults to ``DEFAULT_POLICY`` (reproducing
    today's behavior); **kw is forwarded only to ``discover_files``.

    A missing ``tree-sitter-language-pack`` install surfaces as a
    ``RuntimeError`` (raised by ``chunk_code``) and is *not* caught here: it
    is the signal an all-mode index run uses to skip this strategy (see
    ``retrieval.cli._index_all``).
    """
    resolved_policy = policy or DEFAULT_POLICY
    documents: List[Document] = []
    for document in load_documents(root, **kw):
        language = language_for_path(document.docid)
        chunks = (
            chunk_code(document.docid, document.text, language, resolved_policy.ast_max_chars)
            if language
            else []
        )
        if not chunks:
            target_chars = resolved_policy.target_chars_for(document.docid)
            chunks = chunk_document(document.docid, document.text, target_chars)
        for chunk in chunks:
            docid = f"{document.docid}:{chunk.start_line}-{chunk.end_line}"
            documents.append(
                Document(
                    docid=docid,
                    text=chunk.text,
                    url=document.url,
                    source_path=document.docid,
                    start_line=chunk.start_line,
                    end_line=chunk.end_line,
                    context=getattr(chunk, "context", ""),
                )
            )
    return documents

load_chunk_documents(root, *, policy=None, **kw)

Discover files under root, chunk each one, and return one chunk- granularity Document per chunk.

Each returned Document's docid is "{path}:{start}-{end}" (the file's relative POSIX path plus its 1-based line span), with source_path/start_line/end_line set from the chunk's span — this is what the production retrievers (see retrieval/retrievers.py) index so search results can point a coding agent at an exact file:line location. A file with only blank content yields no chunks and therefore no Documents. policy (default DEFAULT_POLICY, reproducing today's flat 400-char chunking) picks each file's target_chars bucket by suffix; **kw is forwarded only to discover_files.

Source code in engine/retrieval/project_loader.py

def load_chunk_documents(
    root: "os.PathLike[str] | str",
    *,
    policy: Optional[ChunkingPolicy] = None,
    **kw,
) -> List[Document]:
    """Discover files under *root*, chunk each one, and return one chunk-
    granularity ``Document`` per chunk.

    Each returned ``Document``'s ``docid`` is ``"{path}:{start}-{end}"``
    (the file's relative POSIX path plus its 1-based line span), with
    ``source_path``/``start_line``/``end_line`` set from the chunk's span —
    this is what the production retrievers (see ``retrieval/retrievers.py``)
    index so search results can point a coding agent at an exact
    ``file:line`` location. A file with only blank content yields no chunks
    and therefore no Documents. *policy* (default ``DEFAULT_POLICY``,
    reproducing today's flat 400-char chunking) picks each file's
    ``target_chars`` bucket by suffix; **kw is forwarded only to
    ``discover_files``.
    """
    resolved_policy = policy or DEFAULT_POLICY
    documents: List[Document] = []
    for document in load_documents(root, **kw):
        target_chars = resolved_policy.target_chars_for(document.docid)
        for chunk in chunk_document(document.docid, document.text, target_chars):
            docid = f"{document.docid}:{chunk.start_line}-{chunk.end_line}"
            documents.append(
                Document(
                    docid=docid,
                    text=chunk.text,
                    url=document.url,
                    source_path=document.docid,
                    start_line=chunk.start_line,
                    end_line=chunk.end_line,
                )
            )
    return documents

load_chunks(root, *, policy=None, **kw)

Discover files under root, chunk each one, and return a flat list.

Each file's relative POSIX path is used as doc_id when chunking, so chunk ids and doc_ids trace back to a real project path. policy (default DEFAULT_POLICY, reproducing today's flat 400-char chunking) picks each file's target_chars bucket by suffix; **kw is forwarded only to discover_files.

Source code in engine/retrieval/project_loader.py

def load_chunks(
    root: "os.PathLike[str] | str",
    *,
    policy: Optional[ChunkingPolicy] = None,
    **kw,
) -> List[Chunk]:
    """Discover files under *root*, chunk each one, and return a flat list.

    Each file's relative POSIX path is used as ``doc_id`` when chunking, so
    chunk ids and doc_ids trace back to a real project path. *policy*
    (default ``DEFAULT_POLICY``, reproducing today's flat 400-char chunking)
    picks each file's ``target_chars`` bucket by suffix; **kw is forwarded
    only to ``discover_files``.
    """
    resolved_policy = policy or DEFAULT_POLICY
    chunks: List[Chunk] = []
    for document in load_documents(root, **kw):
        target_chars = resolved_policy.target_chars_for(document.docid)
        chunks.extend(chunk_document(document.docid, document.text, target_chars))
    return chunks

load_documents(root, **kw)

Discover files under root and return one Document per readable file.

docid is the file's POSIX-style path relative to root; url is left blank (no network association). Files that fail to decode as text are skipped.

Source code in engine/retrieval/project_loader.py

def load_documents(
    root: "os.PathLike[str] | str",
    **kw,
) -> List[Document]:
    """Discover files under *root* and return one Document per readable file.

    ``docid`` is the file's POSIX-style path relative to *root*; ``url`` is
    left blank (no network association). Files that fail to decode as text
    are skipped.
    """
    root_path = Path(root)
    documents: List[Document] = []

    for file_path in discover_files(root_path, **kw):
        text = read_text_safe(file_path)
        if text is None:
            continue
        docid = file_path.relative_to(root_path).as_posix()
        documents.append(Document(docid=docid, text=text, url=""))

    return documents

read_text_safe(path)

Read path as UTF-8 text, returning None if it looks binary or unreadable.

Returns None when the first 1024 bytes contain a null byte, the content fails UTF-8 decoding, or the file cannot be opened/read.

Source code in engine/retrieval/project_loader.py

def read_text_safe(path: "os.PathLike[str] | str") -> Optional[str]:
    """Read *path* as UTF-8 text, returning None if it looks binary or unreadable.

    Returns None when the first 1024 bytes contain a null byte, the content
    fails UTF-8 decoding, or the file cannot be opened/read.
    """
    try:
        with open(path, "rb") as fh:
            head = fh.read(1024)
        if b"\x00" in head:
            return None
        with open(path, "rb") as fh:
            raw = fh.read()
        return raw.decode("utf-8")
    except (OSError, UnicodeDecodeError):
        return None

Keyword-only overrides

kwarg	Default	Purpose
extensions	DEFAULT_EXTENSIONS	Allowed file suffixes (e.g. frozenset({'.md'}) to index only Markdown)
exclude_dirs	DEFAULT_EXCLUDE_DIRS	Directory names pruned during traversal (e.g. add 'docs-archive')
exclude_globs	DEFAULT_EXCLUDE_GLOBS	Filename deny-list globs (secret-looking names)
include_basenames	DEFAULT_INCLUDE_BASENAMES	Extensionless basenames allowed regardless of extensions (e.g. add 'justfile')
max_bytes	MAX_FILE_BYTES (1 MB)	Per-file size cap

See customize indexing for runnable examples of each override.