Skip to content

Add local RAG harness: indexer, retriever, prompt injection and docs#2

Merged
muzaffermahi merged 2 commits into
masterfrom
codex/create-rag-coding-harness-izk58h
May 29, 2026
Merged

Add local RAG harness: indexer, retriever, prompt injection and docs#2
muzaffermahi merged 2 commits into
masterfrom
codex/create-rag-coding-harness-izk58h

Conversation

@muzaffermahi

Copy link
Copy Markdown
Owner

Motivation

  • Provide a local Retrieval-Augmented Generation (RAG) workflow so SmallCode can inject similar code snippets from indexed GitHub repos into model context without external embedding services.
  • Offer an easy way to build and maintain a local snippet index and expose it to the TUI so models get concrete examples for coding/debugging tasks.

Description

  • Add a lightweight local RAG index and embedding implementation in src/rag/index_store.js with tokenized hashed embeddings, cosine search, and upsert/save/load support.
  • Add a retriever in src/rag/retriever.js that plans query variants, runs iterative searches, formats top hits as fenced-code blocks for prompt injection, and exposes a Google fallback URL when confidence is low.
  • Add a GitHub shallow-clone scraper and chunker in src/rag/github_scraper.js and a CLI indexer bin/rag-index.js (installed as smallcode-rag-index) to build .smallcode/rag/index.json from a .smallcode/rag/repos.json repo list.
  • Integrate RAG into the TUI by loading RagRetriever in bin/smallcode.js and injecting getRagContext into both the system prompt and the dynamic context path; update prompts to include RAG context when available.
  • Add user-facing docs docs/rag-harness.md, README additions about quick run and indexing, package changes (package.json/package-lock.json) to expose the new bin and rag:index script, and unit tests in test/rag.test.js covering index ranking, query planning, and prompt formatting.

Testing

  • Added unit tests in test/rag.test.js that exercise RagIndexStore, RagRetriever, and planQuery.
  • Ran the repository test script npm test (which runs node --test test/*.test.js) and the new RAG tests passed.
  • Exercised the indexer locally using the new script npm run rag:index / smallcode-rag-index to verify index creation and CLI output.

Codex Task

@muzaffermahi
muzaffermahi marked this pull request as ready for review May 29, 2026 13:16
@muzaffermahi
muzaffermahi merged commit c9d0db7 into master May 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant