Code Snippet Testing

[bmorelli25]

Summary

Support the ability to test code snippets.

See https://github.com/elastic/elasticsearch/tree/main/docs#test-code-snippets for more information.

Related issue that describes functionality in Asciidoc system: elastic/docs#2867

Partially related to #7.

[leemthompo]

Re https://elastic.slack.com/archives/CJE3WCFEH/p1741162012377709?thread_ts=1741124580.043689&cid=CJE3WCFEH

I guess we'll need snippet testing in the elasticsearch repo and docs-content at the very least. Elasticsearch should be back online as soon as the test comments are restored.

[lcawl]

There's a thread about some automated testing that was occurring in the ECK docs too here: https://elastic.slack.com/archives/C05PP2LEC1X/p1741273388324539

[leemthompo]

Current state of play for elasticsearch snippets:

• @colleenmcginnis wrote a script to restore the comments to the files in the elasticsearch repo
• @georgewallace has a PR to add those back to the docs: add es code comments for testing elasticsearch#124283

To restore the existing snippet testing framework to the elasticsearch repo, we'll need the @elastic/es-delivery team to update the parsing logic to parse .md files using the new comment syntax.

As @szabosteve mentioned elsewhere this probably just requires some minimal updates to the MDX logic added for a previous docs migration test.

[brianseeders]

I've started work on making the markdown docs work with snippet testing for Elasticsearch using the markdown from elastic/elasticsearch#124283. I'm not familiar with how this stuff works, so am learning as I go.

I discovered an issue related to the fact that the docs content was also re-organized as part of the markdown conversion.

https://github.com/elastic/elasticsearch/blob/v9.0.0-beta1/docs/reference/vectors/vector-functions.asciidoc?plain=1#L73

See this TESTSETUP comment. TESTSETUP has to be used on the first snippet in a file, like it is here, to denote setup for all of the snippets in the file.

https://github.com/elastic/elasticsearch/blob/d0f24003a84eb8139144feab44b0a1ca46d74104/docs/reference/query-languages/query-dsl-script-score-query.md?plain=1#L373

This is the new location of that content. It's been merged with other content/snippets in a new file. This results in the following error message when trying to run the tests:

TESTSETUP: wasn't first. TESTSETUP can only be used in the first snippet of a document.

So, moving these snippets around is basically like taking test cases out of their test suites and putting them into other ones, it doesn't really work.

Not sure what the solution is or just how big the impact is at the moment, just wanted to get a note here as it's close to the end of my day.

[georgewallace]

@bmorelli25 adding a link to the thread where we are left at now. https://elastic.slack.com/archives/CJE3WCFEH/p1750707001785249?thread_ts=1750327877.680069&cid=CJE3WCFEH. Brian did a bunch of work to try to get this working but with the drift with the migration and the conversion there were multiple issues that made it hard to determine the source of the problem.

[bmorelli25]

cc @KOTungseth for visibility

[cotti]

AI triage

Automated investigation against the current codebase. A human still needs to confirm before any code change.

Original concern

Support snippet testing of code blocks in our docs (parity with the asciidoc-era // TESTSETUP, // TEST[continued], // CONSOLE, // NOTCONSOLE, // TESTRESPONSE, … directives) now that source has moved to MyST Markdown.

Findings

• The work is not in docs-builder, by design. Snippet test directives live as HTML comments in the source .md files. docs-builder already parses and strips them via the explicit comment block handler at src/Elastic.Markdown/Myst/Comments/MultipleLineCommentBlockParser.cs:14-61. The downstream test runner reads the raw .md, not the rendered HTML.
• Current code-block argument schema is intentionally minimal: src/Elastic.Markdown/Myst/CodeBlocks/CodeBlockArguments.cs:12 only accepts callouts and subs. There is no console/notconsole/testsetup/continued/skip argument, and adding test semantics here would be the wrong abstraction (docs-builder would have to take a position on something only Elasticsearch's test runner understands).
• A search across the repo for TESTSETUP|TESTRESPONSE|CONSOLE returns zero hits — there is no test-runner integration code on the docs-builder side, and no test coverage for these markers either.
• Active work lives in elastic/elasticsearch#124283 (restoring the test comments to the converted .md files, pushed to v9.5.0 as of Apr 2026) and requires @elastic/es-delivery to update the Gradle parser at build-tools-internal/.../MdxSnippetParser.java to read MyST Markdown. Both are tracked in the issue thread.
• Known blocker surfaced by @brianseeders: the docs reorg merged previously-separate files, so existing TESTSETUP markers are no longer on the first snippet of a file (which the Gradle test runner requires). This is a content-side problem, not a parser-side problem.

Verdict

Needs stakeholder input

The actionable work is happening — and should keep happening — outside docs-builder. The remaining open questions are about scope and content placement, not about implementing a new feature here.

Proposal

Open questions, grouped by audience:

Product

• Should this issue live in docs-builder at all, or should it move to elastic/elasticsearch (and docs-content / cloud-on-k8s as additional consumers) where the actual work is happening? If docs-builder is just the venue because the snippet syntax debate happens here, that's fine, but the title implies an implementation owed by this repo.
• Are non-Elasticsearch products (Kibana, ECK, Cloud) committed to the same snippet-test contract? If yes, the comment-marker vocabulary should be agreed cross-team once; if no, this stays an Elasticsearch-only concern.

Writer

• @brianseeders's TESTSETUP wasn't first finding means the reorg invalidated some test groupings. Are writers expected to refactor content so every test setup ends up on the first snippet again, or do we need a new "file-scoped setup" marker that doesn't care about position?
• Confirm the agreed comment syntax: HTML comments (<!-- TESTSETUP -->) only, or do we also need to support the historical line-prefix form somewhere?

Engineering

• If we ever do want docs-builder to participate (e.g. surface a "test status" badge on code blocks, validate that referenced setups exist, etc.), that becomes a separate feature with its own issue. Out of scope for the existing thread.

Notes

This issue has been open for 18+ months and is effectively a tracker for cross-repo work. Suggested follow-up: once elastic/elasticsearch#124283 merges and the Gradle parser is updated, close this and replace it with a narrowly-scoped docs-builder issue only if a concrete docs-builder feature is requested (e.g. linting that TESTSETUP appears first in file). Otherwise close as moved.