[BUG] HNSW - Cannot return the results in a contigious 2D array. Probably ef or M is too small

[Davidyz]

Describe the bug
sometimes the query produces this error for the chromadb server:

RuntimeError: Cannot return the results in a contigious 2D array. Probably ef or M is too small

To Reproduce
See chroma-core/chroma#2620

Expected behavior

No error

Platform information:

• If the issue is about the CLI, attach the output of pipx runpip vectorcode freeze:

• If the issue is about the neovim plugin, attach the neovim version you're using:

System Information:

For Mac users, please also mention whether you're using intel or apple silicon devices.

• OS: Linux, MacOS, Windows...

Additional context
maybe related: #17
upstream bug tracker: chroma-core/chroma#3486

[l00sed]

I'm still getting this error when running:

vectorcode init
vectorcode vectorise . -r
vectorcode query "How to configure reranker model"

Host at 127.0.0.1:8000 is unavailable. VectorCode will start its own Chromadb at a random port.
Starting querying...
Traceback (most recent call last):
  File ".../site-packages/chromadb/api/base_http_client.py", line 99, in _raise_chroma_error
    resp.raise_for_status()
  File ".../site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://127.0.0.1:57800/api/v2/tenants/default_tenant/databases/default_database/collections/1950930b-a4f5-4068-a9f3-5668e4bbe3eb/query'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../site-packages/vectorcode/main.py", line 78, in async_main
    return_val = await query(final_configs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/vectorcode/subcommands/query/__init__.py", line 102, in query
    for path in await get_query_result_files(collection, configs):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/vectorcode/subcommands/query/__init__.py", line 46, in get_query_result_files
    results = await collection.query(
              ^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/chromadb/api/models/AsyncCollection.py", line 215, in query
    query_results = await self._client._query(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/chromadb/telemetry/opentelemetry/__init__.py", line 134, in async_wrapper
    return await f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/chromadb/api/async_fastapi.py", line 585, in _query
    resp_json = await self._make_request(
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/chromadb/api/async_fastapi.py", line 132, in _make_request
    BaseHTTPClient._raise_chroma_error(response)
  File ".../site-packages/chromadb/api/base_http_client.py", line 103, in _raise_chroma_error
    raise Exception(f"{resp.text} (trace ID: {trace_id})")
Exception: {"error":"RuntimeError('Cannot return the results in a contigious 2D array. Probably ef or M is too small')"} (trace ID: 0)

MacOS M1

[Davidyz]

hi @l00sed , this is an opening issue in the upstream chromadb repo and there's very little I can do in the scope of this repo. The PR that closed this issue added the option to configure the hnsw parameters from the json config, which may help avoiding this error. I've never been able to reproduce this error steadily, and from my experience if you do vectorcode drop and vectorise the repo again, this error usually goes away.

Sorry for the inconvenience.

[l00sed]

That worked, thanks. I think I'm having difficulty with larger codebases. It's working better on a folder with < 100 files. I also had better luck using the dedicated Dockerized ChromaDB— lets you know what's going on in the logs and gives you a better idea of memory consumption. It seemed like there were some issues with memory when I was indexing >1000 files.

docker run -d --rm --name chromadb -p 8000:8000 -v ./chroma:/chroma/chroma -e IS_PERSISTENT=TRUE -e ANONYMIZED_TELEMETRY=TRUE chromadb/chroma:0.6.3

[rieger-jared]

@Davidyz I'm still running into this even after dropping and revectorising. I think the issue might be with repos with large number of files. The process works fine on say 48 markdown files however if I then vectorise 1700 TSX files doing a query will result in the same error.

what's the work around here or are we blocked until this is fixed?

[Davidyz]

@rieger-jared atm the best I can suggest, apart from re-vectorising the collection, is to play around with the hnsw settings (you can configure it in the JSON config). See chroma cookbook for available options and explanations.

The thing is, when I encountered this error, re-vectorising always worked, which means that I can't steadily reproduce this error (I suspect the same is happening for the chroma devs, too). Whatever I try to fix this problem, there's no way for me to know for sure that it's been fixed.

[rieger-jared]

@Davidyz thanks for the response. Out of curiosity, what's the largest number of files you've tried to index for a single collection? Maybe that's unrelated, but that seems to be the only way I can think of reproducing.

[Davidyz]

@Davidyz thanks for the response. Out of curiosity, what's the largest number of files you've tried to index for a single collection? Maybe that's unrelated, but that seems to be the only way I can think of reproducing.

personally, no more than a few hundreds. I'm not sure if this is related because I've encountered this problem on the VectorCode repo too, which contains only ~60 files.

[rieger-jared]

@Davidyz so I think I have a decent reproduction case that illustrates my use case at work. I think this tool is quite promising but this is really blocking me from being able to use it.

reproducing

The Kubernetes project illustrates a good example of the codebase size that most codebases at decently sized company would have. I suppose that most devs around the world work on similar size codebases.

First checkout the Kubernetes repo

git clone git@github.com:kubernetes/kubernetes.git
cd kubernetes

Run vectorcode init

vectorcode init

Since the entire repository is quite large, let's just focus on the vendor folder which has between 4k - 5k go files, but suffices to reproduce the error.

vectorcode vectorise vendor/**/*.go

Now run a basic query for services

vectorcode query "services"

This should result in the error shown in the ticket. Since this is a fresh init, this really shouldn't happen. @Davidyz running this example doesn't take too long, ~5 - 10 minutes. Would you please mind trying this out and seeing if you have the same issue? This was performed against chroma chromadb/chroma:0.6.3. I will try v1 and see if there's the same issue.

[Davidyz]

That's awesome. I'll give it a try myself and see if I can reproduce it using plain chroma (if I successfully reproduce this with chroma, I'll forward this to their repo).

EDIT: managed to trigger the exception in vectorcode, and it looks like setting M to 64 in hnsw settings solved this error. I'll consider setting a bigger M by default. I'll move on to try to reproduce in plain chroma.