Source Analyse¶
The Source Analyse module is a powerful component of Sphinx-CodeLinks that extracts documentation-related content from source code comments. It provides both CLI and API interfaces for flexible integration into documentation workflows.
Key Capabilities:
Extract Sphinx-Needs ID references from source code comments
Process custom one-line comment patterns for rapid documentation
Extract multi-line needs: a need’s fields and its body written across the lines of a comment
Generate structured JSON output for further processing
Support for multiple programming language comment styles
Overview¶
Source Analyse works by parsing source code files and identifying specially marked comments that contain documentation information. This enables developers to embed documentation directly in their source code while maintaining clean separation between code and documentation.
The module supports three primary extraction modes:
Sphinx-Needs ID References - Links between code and requirements/specifications
One-line Needs - Simplified syntax for creating documentation needs
Multi-line Needs - A need’s fields and its body, in reStructuredText or Markdown, written across the lines of a comment
Supported Content Types¶
Sphinx-Needs ID References¶
Extract references to Sphinx-Needs items directly from source code comments, enabling traceability between code implementations and requirements.
One-line Needs¶
Use simplified comment patterns to define Sphinx-Needs items without complex RST syntax. See OneLineCommentStyle for detailed information.
Multi-line Needs¶
Write a whole need in a comment – its type, title and options, and a body in reStructuredText or Markdown – between an @need line and an @endneed line. See Multi-line Needs.
Limitations¶
Current Limitations:
Language Support: C/C++ (
//,/* */), C# (//,/* */,///), Python (#), YAML (#), Rust (//,/* */,///), Go (//,/* */), JSONC (//,/* */) and Bash (#) comment styles are supportedSingle Comment Style: Each analysis run processes only one comment style at a time
Extraction Examples¶
The following examples are configured with the analyse configuration,
Sphinx-Needs ID References¶
Below is an example of a C++ source file containing need ID references and the corresponding JSON output from the analyse.
#include <iostream>
// @need-ids: need_001, need_002, need_003, need_004
void dummy_func1(){
//...
}
// @need-ids: need_003
int main() {
std::cout << "Starting demo_1..." << std::endl;
dummy_func1();
std::cout << "Demo_1 finished." << std::endl;
return 0;
}
[
{
"filepath": "tests/data/need_id_refs/dummy_1.cpp",
"remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/fa5a9129d60203355ae9fe4a725246a88522c60c/tests/data/need_id_refs/dummy_1.cpp#L3",
"source_map": {
"start": { "row": 2, "column": 14 },
"end": { "row": 2, "column": 52 }
},
"tagged_scope": "void dummy_func1(){\n //...\n }",
"need_ids": ["need_001", "need_002", "need_003", "need_004"],
"marker": "@need-ids:",
"type": "need-id-refs"
},
{
"filepath": "tests/data/need_id_refs/dummy_1.cpp",
"remote_url": "https://github.com/useblocks/sphinx-codelinks/blob/fa5a9129d60203355ae9fe4a725246a88522c60c/tests/data/need_id_refs/dummy_1.cpp#L8",
"source_map": {
"start": { "row": 7, "column": 14 },
"end": { "row": 7, "column": 22 }
},
"tagged_scope": "int main() {\n std::cout << \"Starting demo_1...\" << std::endl;\n dummy_func1();\n std::cout << \"Demo_1 finished.\" << std::endl;\n return 0;\n }",
"need_ids": ["need_003"],
"marker": "@need-ids:",
"type": "need-id-refs"
}
]
Output Structure:
filepath- Path to the source file containing the referenceremote_url- URL to the source code in the remote repositorysource_map- Location information (row/column) of the markertagged_scope- The code scope associated with the markerneed_ids- List of referenced need IDsmarker- The marker string used for identificationtype- Type of extraction (“need-id-refs”)
Need ID references in the build¶
The src-trace directive attaches the @need-ids: references in the files it analyses to the needs they name, during the build – the need may be defined in any document, before or after the directive. Each referenced need gets a list in its project’s ref_url_field (default code_url), one entry per reference, in source order:
"REQ_001": {
"code_url": [
"https://github.com/org/repo/blob/<commit>/src/refs.cpp#L1",
"https://github.com/org/repo/blob/<commit>/src/refs.cpp#L3"
]
}
The entries are the project’s
remote_url_patternfilled in for the marker’s line when set_remote_url is on, else the local link (aslocal-urlgives it: the source file copied into the build output, relative to the need’s document). On the need’s card a remote entry renders as a link namedsrc/refs.cpp#L1; a value of the local shape (<file>.<ext>#L<line>) links to the generated source page. Outside a git repository, or in one without a commit yet, there is no remote URL (#2045): the local link is used when local URLs are on, else nothing is attached. With local URLs only, a file referenced by@need-ids:is copied into the output and gets a source page, as a file with a one-line need is (see Global Options for when the copies and pages are written).A need nothing references carries
null, which is removed before schema validation, so a schema withunevaluatedProperties: falsenever sees the field on it. A schema withunevaluatedProperties: falseon a type that code references must declarecode_url, or every referenced need of that type fails validation.An unknown id warns at the source line:
src/refs.cpp:5: WARNING: @need-ids reference to unknown need 'NOSUCH_ID' [codelinks.need_id_ref].suppress_warnings = ["codelinks.need_id_ref"]silences it. Each project with attached references also reportscodelinks [<project>]: N references attached, M unknown; one marker naming an unknown id warns once, even when two fields cover its file.A comment that starts with a configured marker is a reference and never a one-line need, even on the default one-line style, whose start sequence
@would match// @need-ids: A, Btoo.Precedence. The references replace any value the need’s own directive or a
needs_global_optionsdefault gave the field (ubCode’s rule); a user’s needextend overrides them, as the references are attached after every need is read and before anyneedextendis applied.:+code_url:appends to the attached list, but on a need nothing references it fails the build until sphinx-needs #2038 is fixed – set the field (:code_url:) instead. The need is not marked as modified.Overlapping scans attach each reference once: two directives over the same file, or two projects whose source directories overlap. When those two projects’ patterns differ, the reference keeps the URL of the first project by name (ubCode keeps the last one written). Files under different roots are different files, each kept:
src/main.cppin two repositories gives two references. Two projects naming different fields fill each its own field. A one-line need two such scans find is created once (Overlapping scopes).Incremental builds. A file added to, removed from or edited in a directive’s scope re-reads the document hosting the directive on the next build, for its references as for the needs it defines; the cost is one directory walk per scope per build, with no parsing: roughly 0.1 s per 2,000 discovered files on an Apple M2 Pro laptop, whatever their size – the walk is the directive’s own discovery plus a
statper file – while parsing them costs tens of times more (2,000 200-line C++ files: ~0.1 s of walk against ~9 s of analysis). No build output is ever traced: the build’s own output and doctree directories are skipped, and every builder writes an.ignorefile (*) at the root of its output and doctree directories, so withgitignore = true(the default) nothing any builder writes – the extension’s source copies, Sphinx’s_downloads/copies of a traced source, another builder’s whole tree – is traced, wherever the output sits: in_build/, besidedocs/inbuild/<builder>, or inside a traced source directory, whose other files are traced as usual. Withgitignore = falseno ignore file is read, so the directory containing the output and doctree directories is skipped too when it lies inside the documentation source directory (_build/in the Makefile layout), and an output directory elsewhere insidesrc_dirneeds anexcluderule. A need whose references changed has its card rewritten, whichever document it is in: that document is written again, not read again. Aneedtable(or any other filter) in a third document that selects on the field is not rewritten – the cross-document classneedextendhas today.
Migrating from analyse → write rst → include: remove the .. include:: of the generated file (and the steps that generate it); the references now arrive by themselves, in code_url rather than in the remote-url (or remote_url) field the generated needextend directives set, and as a list rather than one comma-joined string. A project that keeps including the generated file gets both. If you declared the field yourself for that route (needs_fields), remove the declaration: the extension registers it, and a declaration of your own warns – the extension’s codelinks.config warning names the cure – and, as a string field, fails schema validation on every referenced need.
Differences from ubCode, which reads the same ref_url_field: ubCode stores one string, the last reference written, where this is a list of every reference; and ubCode drops a reference to an unknown need silently, where this warns. ubCode currently drops every need whose code_url is a list when it imports a Sphinx-built needs.json, in a project whose codelinks configuration registers the field (useblocks/ubcode#3838).
Projects without a src-trace directive¶
A project that no src-trace directive traces still has its @need-ids: references attached, as in ubCode: its whole source directory (src_dir of source_discover) is discovered and analysed, and the references reach the needs they name exactly as a directive’s do – the same field, URLs, warnings and codelinks [<project>]: N references attached, M unknown line. The gate is the directive’s own, and there is no switch of its own: local or remote URLs on, references extracted, at least one marker, and a ref_url_field that is not "". A project that at least one directive traces is left to its directives.
No need is created. One-line needs and RST blocks in such a project’s files are not rendered – there is no directive, and so no document, to hold them. The project’s line counts them:
codelinks [src]: 12 files, 30 references, 2 one-line needs not created (no src-trace directive).When it is analysed. Every build walks the project’s source directory and compares the files, their modification times and sizes with the last analysis; it analyses the project again only when they differ or the configuration changed. The cost of an unchanged build is that walk, with no parsing (see Incremental builds above for what the walk costs). A build that analysed the project again rewrites the cards of the needs whose references changed – or, when none did, writes the root document – so that Sphinx keeps the result and the next build analyses nothing.
Errors do not stop the build. A missing source directory, or a discovery or analysis failure, warns once per project and build (
codelinks.need_id_ref), and that project’s references are not attached. A file whose target lies outside the source directory is skipped with acodelinks.outside_src_dirwarning on every build, since the directory is walked every build – a directive’s scope warns only when its document is read.
Multi-line Needs¶
A multi-line need is one need written across the lines of a comment: an open line with the need’s type and title, its options, a body in a declared markup, and a close line.
Note
Status. sphinx-codelinks extracts and reports these needs (codelinks analyse, below); the
src-trace directive renders them in a following release, once Sphinx-Needs ships the step
that parses a need’s content in its declared markup (sphinx-needs PR …). Until then a
src-trace directive creates no need from them.
#include <stdbool.h>
/**
* @need req: Login must be rate limited
* :id: REQ_LOGIN_1
* :links: SPEC_AUTH, SPEC_LOCK
* :status: open
*
* After **five** failed attempts the account is locked;
* see :need:`SPEC_AUTH`.
* @endneed
*/
bool login(const char *user, const char *password) {
return false;
}
# @need[md] impl: Exponential back-off
# :id: IMPL_BACKOFF
# :links: REQ_LOGIN_1
#
# Doubles the delay after each failure, see {need}`REQ_LOGIN_1`.
# @endneed
def backoff(attempt: int) -> float: ...
The first block’s body is reStructuredText, the project default; the second declares Markdown
(MyST) with @need[md]. The body is written in its declared markup and is parsed, when the need
is rendered, by Sphinx-Needs – so roles and links into the documentation work as in a .. req::
directive’s content. The rest of the block is this tool’s own grammar, neither RST nor MyST.
The grammar¶
After the comment’s prefixes are stripped (see below), a block is:
@need[<markup>] <type>: <title> the open line
:<key>: <value> option lines, immediately after
<continuation> a line indented deeper than its option line continues the value
<body> the first line that is neither an option nor a continuation starts the body
@endneed the close line
The open line starts with the open word (
@need), optionally followed by a markup tag in brackets, then whitespace, the need type and a colon, then the title – the rest of the line, which may be empty. The open word counts only when[, whitespace or the line end follows it:@need-ids:and@needleopen nothing. The type is a need type’s name (letters, digits,_and-); it is checked when the need is rendered, not here. Write the whitespace:@need[md]req: Tand@need req:Tare refused.The markup tag is looked up in the project’s
markupstable, which gives the need’sdoctype(rst→.rstandmd→.mdby default); without a tag the project’sdefault_markupapplies (rst). The tag is matched exactly as written:[RST],[ md ]and an empty[]are unknown tags.Options are
:key: valuelines right after the open line. They may sit at any indentation – at the open line’s, as MyST writes a directive’s options, or indented under it as in RST. Their values are kept as written, as directive strings::links: SPEC_AUTH, SPEC_LOCKis the string"SPEC_AUTH, SPEC_LOCK", converted with the project’s field definitions when the need is rendered, as a directive’s options are. Whitespace must follow the key (:id:R1is body text). A line indented deeper than its option line continues the value, joined with one space, so separate the options from the body with a blank line: an indented first body line written directly after an option is read as that option’s continuation.The body is every line after the options up to the close line: dedented by its common indentation (a TAB counts as one character), never re-flowed, with leading and trailing blank lines trimmed; it may be indented or not. A comment line holding only its prefix (
//,#, or a lone*in a block whose lines carry the*leader) inside it is a blank body line.The close line holds the close word (
@endneed) alone, surrounding whitespace allowed. To write the close word at the start of a body line, escape it: a body line starting\@endneed– after its indentation, so inside an indented literal block too – is emitted as@endneed, its indentation kept. Nothing else is escaped.
One need per block: the open word starting a body line is body text, and is noted with a warning.
Precedence over one-line needs and references¶
The lines of a block belong to it alone: they are never read as one-line needs or @need-ids:
references, whatever they hold. The default one-line start sequence @ is a prefix of @need,
and this rule is what keeps them apart: an open line such as @need req: Title, with a comma, or a
body line such as @param a, b, would otherwise be a one-line need. A refused one-line form
(@need req: T @endneed) hides its own line too; an open line without a close hides nothing. A
one-line marker or a reference on a line outside the block – in the same comment or another – is
found as before.
With multi-line needs switched on, a one-line need whose title starts with the open word, such as
// @need to handle overflow, IMPL_3, impl, is still a one-line need but also draws a
multiline_need_unterminated warning: reword the title, or choose another open word.
Malformed blocks¶
Each problem is an analyser warning at the source file and line; codelinks analyse prints them
as Analyse warning in <file>:<line> - <kind>: <message>.
Kind |
When, and what happens |
|---|---|
|
The close word ends the open line (after whitespace): skipped, and the line is not read as a
one-line need either (a one-line need is written with the one-line marker).
|
|
No close line before the comment run ends: skipped, and the rest of the run is not read for blocks. |
|
The open word starts a line that is not |
|
The markup tag is not in |
|
An option key given twice, or one naming a key of the record itself ( |
|
The open word starts a body line: it stays body text. |
The record¶
codelinks analyse writes each multi-line need to marked_content.json as one record of type
multiline-need. For the C example above, in a repository on GitHub:
{
"filepath": "/home/me/repo/src/auth/login.c",
"remote_url": "https://github.com/useblocks/demo/blob/4f2a9c1e0b7d3a5f6e8c9b0a1d2e3f4a5b6c7d8e/src/auth/login.c#L4",
"source_map": {
"start": { "row": 3, "column": 3 },
"end": { "row": 10, "column": 11 }
},
"tagged_scope": "bool login(const char *user, const char *password) {\n return false;\n}",
"need": {
"type": "req",
"title": "Login must be rate limited",
"id": "REQ_LOGIN_1",
"links": "SPEC_AUTH, SPEC_LOCK",
"status": "open",
"content": "After **five** failed attempts the account is locked;\nsee :need:`SPEC_AUTH`.",
"doctype": ".rst"
},
"markup": "rst",
"source": {
"project": "src",
"path": "src/auth/login.c",
"root": "git",
"commit": "4f2a9c1e0b7d3a5f6e8c9b0a1d2e3f4a5b6c7d8e",
"start": { "line": 4, "col": 3 },
"end": { "line": 11, "col": 11 },
"content_start": { "line": 9, "col": 3 },
"option_lines": { "id": 5, "links": 6, "status": 7 },
"scope": { "kind": "function_definition", "start": 13, "end": 15 }
},
"type": "multiline-need"
}
need– the type, the title, the options in the order written with their values as written,content(the body,\n-joined) anddoctype.markup– the markup tag as resolved.source– the codelinks project; the file, relative to the git root (root: "git") or else to the project’s source directory (root: "src_dir"), POSIX; the commit; the open line’s start, the close line’s end and the first body line’s start (nullwithout a body), each with a 1-basedlineand a 0-basedcol; each option’s line; and the code scope the block is attached to (nullunder the libclang engine).filepath,remote_url,source_mapandtagged_scopeare the fields everymarked_content.jsonrecord has;source_mapspans the open line to the close line, 0-based.
codelinks write rst ignores these records. The configuration is
get_multiline_needs and analyse.multiline_needs.
One-line Needs¶
One-line Needs provide a simplified syntax for creating Sphinx-Needs items directly in source code comments without requiring full RST syntax.
For comprehensive information about one-line needs configuration and usage, see OneLineCommentStyle.
Basic Example:
// @Function Implementation, IMPL_001, impl, [REQ_001, REQ_002]
This single comment line creates a complete Sphinx-Needs item equivalent to:
.. impl:: Function Implementation
:id: IMPL_001
:links: REQ_001, REQ_002
Overlapping scopes¶
When src-trace directives’ scopes overlap – one tracing src/ and another one file in it, or two projects whose source directories nest – a one-line need they both find is created once, by its first owner: the directive Sphinx reads first (documents in name order, directives in document order). The id is the marker’s, or the one Sphinx-Needs generates from the type and title when the one-line style has no id field. Whichever definition is read first keeps the id – codelinks never takes an existing id over, and a hand-written or imported need read later is refused by Sphinx-Needs itself (needs.create_need, or needs.import_need for an imported one). Every other directive finding it skips it with one warning at the marker’s line, naming both documents – src/impl.cpp:1: WARNING: one-line need 'IMPL_1' is already defined in document 'page1': not created again by the src-trace directive in 'page2' (narrow one directive's scope) [codelinks.duplicate_need] – and its codelinks [<project>] line adds , 1 skipped (already defined). A document that skipped a need is read again when the owner’s document changes or is removed, so the need moves to it instead of vanishing (an incremental build leaves an existing owner in place until it is read again; two documents read in the same build give the earlier one, as a fresh build does). With -j N the two directives may be read by different workers, which cannot see each other’s needs: Sphinx-Needs then keeps the first merged and warns needs.duplicate_id, and when that directive later stops tracing the line, the other document takes the need over only when it is next read – until then the need is missing from needs.json, and that page keeps its old card.
Preprocessor-Aware C/C++ Extraction (libclang)¶
By default, Source Analyse uses a tree-sitter parser that extracts every comment,
regardless of the C preprocessor. For C/C++ projects that rely on conditional compilation
(#ifdef VARIANT_A …), this means needs from all branches are extracted — even
branches that are never compiled.
The optional libclang engine addresses this. When an
analyse.preprocessor table is configured and comment_type
is "cpp", each file is parsed as a real translation unit and comments inside inactive
preprocessor branches are dropped. Active needs keep their original line numbers — no
source transformation is performed.
Important
The libclang engine requires an optional dependency:
pip install 'sphinx-codelinks[libclang]'. The wheel bundles the native library, so
no compiler is required on the user’s machine.
How files are selected and parsed¶
File discovery is unchanged — SourceDiscover still decides which
files are candidates. A compile_commands.json database only determines how each
discovered file is parsed:
File |
How it is parsed |
|---|---|
Listed in |
Parsed with the exact flags the compiler used for that translation unit. |
A compiled source ( |
Skipped — assumed to be excluded from the build (e.g. another platform). |
A header ( |
Parsed standalone using |
Any file, when no database is found |
Parsed with |
Header files¶
A compile_commands.json only ever lists compiled translation units (.cpp files);
headers are pulled in via #include and never appear as entries. Sphinx-CodeLinks
therefore parses each discovered header standalone, using the defines and
includes you configure. Include guards resolve correctly, and #ifdef branches are
evaluated against your defines.
Note
Because headers are parsed standalone, they see only the global defines —
not the per-file -D flags from compile_commands.json. To extract a
particular variant’s needs from headers, mirror that variant into defines. Treat
one analysis run as one variant: set defines to the variant you want, and both
sources and headers evaluate their conditions consistently.
Example¶
// include/feature.hpp
#ifndef FEATURE_HPP
#define FEATURE_HPP
// @Always available, IMPL_BASE, impl, [REQ_BASE]
void base();
#ifdef VARIANT_A
// @Variant A only, IMPL_VAR_A, impl, [REQ_A]
void variant_a();
#endif
#endif
With defines = ["VARIANT_A"] both IMPL_BASE and IMPL_VAR_A are extracted.
With defines = [] only IMPL_BASE is extracted — the #ifdef VARIANT_A block is
inactive, so its need is dropped. The include guard (#ifndef FEATURE_HPP) is always
active when the header is parsed on its own, so IMPL_BASE is never suppressed by it.
Variant handling is the caller’s job¶
Which preprocessor branches are active — and therefore which need markers are in
scope — is decided entirely by the -D macros libclang sees (from
compile_commands.json for a compiled source, or from defines for headers
and the standalone path). Sphinx-CodeLinks does not model, generate, or
reconcile build variants itself.
For the result to be meaningful, the macros fed to the extractor and the macros
that select the variant in the code must come from the same source of truth —
whatever drives your variant management (Kconfig, pure::variants, a home-grown
generator). Generate the compile_commands.json / defines for one variant
from that source, run the analysis once, and the extracted markers are exactly
the ones active in that variant. Keeping the inputs consistent is the caller’s
responsibility.
Limitations¶
The command line of a compile_commands.json entry is parsed for Clang/GCC-style
flags (-D, -I, -std= …). MSVC cl.exe slash-style flags (/D,
/I, /std: …) are not recognised, so a database generated for the MSVC
compiler driver yields no defines or include dirs. Generate the database with
clang / clang-cl (which emit Clang-style flags) for use with this engine.
Comment runs and prefixes¶
A block lives in one comment run: one block comment (
/* … */,/** … */,/*! … */), one Python string statement (a docstring, or any bare string statement), or consecutive line comments (//,///,//!,#) on consecutive rows with the same delimiter and only whitespace before them on their rows. A code line, or a change of delimiter, ends a run; a comment after code on its row neither starts nor continues one.The comment prefixes are stripped by the comment’s kind, never by searching the text, so every line of a block keeps its source line:
Comment
Stripped from each line
//,///,//!,#the delimiter, then one space if there is one
/* … */,/** … */,/*! … */first, a first row holding only the opener (with any number of stars, as in a
/*****banner) and a last row holding only the closer (***/) are dropped; then the delimiters; then a*leader (*and one space, or a lone*), only when every non-blank line between the opener’s row and the closer’s carries one – so a plain block’s*emphasis*is kept. A separator line of stars only (*****) counts as carrying the leader and is read as a blank line. Give every line the leader or none: when one line lacks it, an open word behind a*is reported (multiline_need_header) and no need is produced. So is an RST bullet starting* @needin a plain block’s prose outside any need: write such a bullet with-, or give the block leadersPython string statement
the quotes, and the lines’ common indentation, as
inspect.cleandocremoves itUnder the libclang engine what precedes a comment on its row is read from the row itself; a block in an inactive preprocessor branch is not extracted.