image captions using blip. by gsaluja9 · Pull Request #204 · aperture-data/workflows

gsaluja9 · 2025-09-15T20:33:29Z

Adds auto generation of image captions using BLIP.
https://huggingface.co/docs/transformers/main/en/model_doc/blip#transformers.BlipForConditionalGeneration

TODO:

~~Add tests~~ : Adding a validation at build time with a basic script.

Add docs

ad-claw000

This workflow looks like a great addition. LGTM!

ad-claw000

Great addition! However, I noticed a few issues that should be addressed before this is merged:

Hardcoded batch_size: In apps/caption-image/app/images.py, self.batch_size = 32 is hardcoded inside FindImageQueryGenerator.__init__. It ignores the batch_size parameter passed from the CLI in caption_images.py. You should pass batch_size into the generator constructor and use it there.
Pagination logic during updates: The query generator fetches batches using batch_id: idx while filtering on wf_caption_image == None. Because the response handler updates these entities and removes the None condition, the total number of matching images changes dynamically. Depending on how ApertureDB evaluates batch_id, this could lead to skipping images (e.g. batch 1 shifts into batch 0's place after batch 0 is updated). A safer approach might be to not rely on batch_id and instead repeatedly request the first N items (e.g., limit), or use a stable identifier for pagination.
Module-level Model Loading: The AutoProcessor and BlipForConditionalGeneration are loaded at the module level in images.py. This means they are loaded into memory as soon as the module is imported, even if just running --help. Consider lazy-loading them inside the class or function, or at least only when the command is actually executed.

Please let me know when these are updated!

- Pass batch_size from CLI down to QueryGenerator - Replace batch_id pagination with limit to handle dynamic properties - Lazy-load AutoProcessor and Blip model to improve startup time

ad-claw000 · 2026-05-24T18:14:24Z

I've pushed a commit to address the review comments:

batch_size is now correctly propagated from the CLI to the query generator.
Switched from batch_id pagination to using limit since wf_caption_image gets updated dynamically.
The BLIP model and processor are now lazy-loaded on demand to speed up script initialization (e.g. when just running --help).

Let me know if this looks good to go!

Copilot

Pull request overview

This PR introduces a new caption-image workflow that auto-generates image captions using the HuggingFace BLIP model, adds container/build plumbing for it, and adds devcontainer configurations to run workflows against a local ApertureDB stack.

Changes:

Added a new apps/caption-image workflow that finds uncaptioned images and writes captions back to ApertureDB.
Integrated the new app into CI builds and added a Docker build-time “warmup/validation” step.
Added devcontainer configs (compose + scripts) to spin up a local ApertureDB + Lenz + WebUI stack for multiple workflows.

Reviewed changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
workflows-devcontainer.code-workspace	Workspace config to open workflows repo alongside `../app`.
postinstall.sh	Devcontainer post-create script to configure `adb` and install shell completion.
initcommand.sh	Devcontainer init script (build base image + generate `.env`).
configuration_params.py	Emits `ADB_PORT` based on platform (mac vs others).
base/docker/scripts/sitecustomize.py	Improves global exception hook (docstring + avoids shadowing `type`).
apps/caption-image/requirements.txt	Adds Python dependency on `transformers`.
apps/caption-image/README.md	Documents the new caption-image workflow and usage.
apps/caption-image/Dockerfile	Builds caption-image image; installs torch/torchvision + transformers; runs validation.
apps/caption-image/app/warmup_validate.py	Build-time BLIP warmup + caption assertion.
apps/caption-image/app/images.py	QueryGenerator implementation: fetches images, runs BLIP, updates captions.
apps/caption-image/app/caption_images.py	Typer-based entrypoint for running the caption workflow.
apps/caption-image/app/app.sh	Container entrypoint script that runs `caption_images.py`.
.vscode/launch.json	Debug configuration for Python.
.gitignore	Ignores `aperturedb/` directories (local dev data).
.github/workflows/main.yml	Adds `caption-image` to the CI build matrix.
.devcontainer/dataset-ingestion/docker-compose.yml	Local stack compose file for dataset-ingestion devcontainer.
.devcontainer/dataset-ingestion/devcontainer.json	VS Code devcontainer definition for dataset-ingestion.
.devcontainer/crawl-website/docker-compose.yml	Local stack compose file for crawl-website devcontainer.
.devcontainer/crawl-website/devcontainer.json	VS Code devcontainer definition for crawl-website.
.devcontainer/caption-image/docker-compose.yml	Local stack compose file for caption-image devcontainer.
.devcontainer/caption-image/devcontainer.json	VS Code devcontainer definition for caption-image.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Moved configuration_params.py to .devcontainer - Updated initcommand.sh to loop over all devcontainers - Provided ADB_PORT default in docker-compose.yml files - Updated images.py to correctly use batching, add PyTorch inference context, fix missing DONE state, handle execution query errors - Fixes to warmup_validate.py to avoid external network request - Replaced PIP commands with requirements.txt - Addressed logging and env var issues in caption_images.py

Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.

- Add threading.Lock to get_model_and_processor lazy init - Remove unused desc_blobs variable - Change PRELOAD_MODEL default to false in Dockerfile - Replace torchvision with pillow in requirements.txt - Fix WORKFLOW_VERSION quoting in initcommand.sh - Add caption-image service to docker-compose.yml

…of batch_id

ad-claw000 · 2026-05-25T02:10:27Z

Replaced the batch_id pagination logic with a stable identifier (fetching all uncaptioned _uniqueids upfront) to prevent skipping images as their _done status is updated. This addresses point 2 from @ad-claw000's review. See commit 7725c94.

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 12 comments.

- Use count query + server side batch in FindImageQueryGenerator - Add validation for batch_size > 0 - Switch to RGB and handle decode exceptions - Simplify uniqueids-captions zipping to skip failed - Update log levels to check WF_LOG_LEVEL - Add PRELOAD_MODEL flag to docker-compose.yml - Add proper evaluation and no_grad to warmup_validate - Clean up docs to reflect python None - Add set -euo pipefail to bash scripts

ad-claw000 · 2026-05-29T04:23:46Z

I've restored the stable identifier for pagination to prevent skipping images (using _uniqueid). This addresses the feedback from the CHANGES_REQUESTED review regarding pagination skipping images. See commit fa1f7eb.

Copilot

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 4 comments.

- Use batching in FindImage to avoid memory explosion (resolves #3322118023) - Fix double-checked locking thread-safety issue (resolves #3322118041) - Differentiate and abort on transient/system errors (resolves #3322118049)

Copilot

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 4 comments.

…or handling - Implement standard RUN_ONCE and SLEEPING_TIME loop in app.sh, and add status_tools.py progress reporting. - Add RUN_ONCE to caption-image service in docker-compose.yml for consistency. - Handle len(uniqueids) != len(r_blobs) mismatch by marking images as failed instead of silently looping. - Handle per-image generation failures by appending to failed list instead of raising RuntimeError and aborting the batch.

Copilot

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.

…kipping images during updates

ad-claw000 · 2026-05-29T16:45:19Z

I have addressed the review feedback by restoring the stable identifier (_uniqueid) for pagination to ensure no images are skipped during updates. The code has been committed and pushed in 6f871d3.

Addresses the review comment to capture return values and handle non-zero status for update failures.

Copilot

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 3 comments.

…kipping images during updates

ad-claw000 · 2026-06-01T13:50:41Z

I have restored the stable identifier for pagination to prevent skipping images (using _uniqueid). This addresses the feedback from the CHANGES_REQUESTED review regarding pagination skipping images. See commit 14bc6e6.

Copilot

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 1 comment.

…model download

Copilot

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 6 comments.

+    volumes:
+      - ./aperturedb/certificate:/cert


+    volumes:
+      - ./aperturedb/certificate:/etc/lenz/certificate


+    volumes:
+      - ./aperturedb/db:/aperturedb/db
+      - ./aperturedb/logs:/aperturedb/logs


+    volumes:
+      - ./aperturedb/certificate:/etc/nginx/certificate


+  aperturedb:
+    image: aperturedata/aperturedb-community:latest


+      aperturedb:
+        condition: service_started


image captions usign blip.

38084c4

gsaluja9 requested review from bovlb and drewaogle September 15, 2025 22:42

gsaluja9 marked this pull request as ready for review September 17, 2025 13:52

Adding devcontainers. (#208)

2dfb0b8

gsaluja9 requested a review from luisremis September 17, 2025 18:22

stray file.

e3b5994

bovlb reviewed Sep 18, 2025

View reviewed changes

gsaluja9 and others added 2 commits September 19, 2025 09:29

Some review feedback

4a3651f

Merge branch 'main' into image_captions

5516ccd

ad-claw000 approved these changes May 19, 2026

View reviewed changes

ad-claw000 requested changes May 21, 2026

View reviewed changes

luisremis assigned ad-claw000 May 24, 2026

Address review comments for image captions

c1baf46

- Pass batch_size from CLI down to QueryGenerator - Replace batch_id pagination with limit to handle dynamic properties - Lazy-load AutoProcessor and Blip model to improve startup time

Copilot AI review requested due to automatic review settings May 24, 2026 18:14

Copilot started reviewing on behalf of ad-claw000 May 24, 2026 18:14 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

ad-claw000 added 2 commits May 24, 2026 20:21

merge: resolve conflicts with main

324e1be

Copilot AI review requested due to automatic review settings May 24, 2026 20:25

Copilot started reviewing on behalf of ad-claw000 May 24, 2026 20:25 View session

Copilot AI reviewed May 24, 2026

View reviewed changes

ad-claw000 added 2 commits May 24, 2026 22:12

Fix pagination logic to use a stable identifier (unique IDs) instead …

7725c94

…of batch_id

Copilot AI review requested due to automatic review settings May 25, 2026 02:10

Copilot started reviewing on behalf of ad-claw000 May 25, 2026 02:10 View session

Copilot AI reviewed May 25, 2026

View reviewed changes

Copilot started reviewing on behalf of ad-claw000 May 29, 2026 04:23 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py

Comment thread apps/caption-image/app/images.py

Comment thread apps/caption-image/app/images.py

Comment thread docker-compose.yml

ad-claw000 added 2 commits May 29, 2026 05:40

fix(caption-image): address review comments

72801e2

- Use batching in FindImage to avoid memory explosion (resolves #3322118023) - Fix double-checked locking thread-safety issue (resolves #3322118041) - Differentiate and abort on transient/system errors (resolves #3322118049)

test(mcp-server): wait for aperturedb to be healthy before seeding

9b6b44f

Copilot AI review requested due to automatic review settings May 29, 2026 08:59

Copilot started reviewing on behalf of ad-claw000 May 29, 2026 08:59 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

Comment thread apps/caption-image/app/app.sh Outdated

Comment thread apps/caption-image/app/images.py

Comment thread apps/caption-image/app/images.py

Comment thread docker-compose.yml

ad-claw000 added 2 commits May 29, 2026 12:19

Merge remote-tracking branch 'origin/main' into image_captions

a238452

Copilot AI review requested due to automatic review settings May 29, 2026 12:21

Copilot started reviewing on behalf of ad-claw000 May 29, 2026 12:22 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

fix(caption-image): use stable identifier for pagination to prevent s…

6f871d3

…kipping images during updates

fix(caption-image): handle return status from query_fail execution

cc0afc4

Addresses the review comment to capture return values and handle non-zero status for update failures.

Copilot AI review requested due to automatic review settings May 30, 2026 13:30

Copilot started reviewing on behalf of ad-claw000 May 30, 2026 13:30 View session

Copilot AI reviewed May 30, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py

Comment thread apps/caption-image/test.sh

Comment thread apps/caption-image/app/images.py Outdated

OpenClaw Bot added 2 commits June 1, 2026 02:09

fix: use server-side batching and remove trailing whitespace

f828c65

fix(caption-image): use stable identifier for pagination to prevent s…

14bc6e6

…kipping images during updates

Copilot AI review requested due to automatic review settings June 1, 2026 13:50

Copilot started reviewing on behalf of ad-claw000 June 1, 2026 13:51 View session

Copilot AI reviewed Jun 1, 2026

View reviewed changes

Comment thread apps/caption-image/app/images.py Outdated

OpenClaw Bot added 2 commits June 2, 2026 08:46

fix(caption-image): use ApertureDB count and batching for pagination

8159fdd

test(mcp-server): increase client timeout to 300s to allow embedding …

f374c67

…model download

Copilot AI review requested due to automatic review settings June 2, 2026 14:49

Copilot started reviewing on behalf of ad-claw000 June 2, 2026 14:49 View session

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Conversation

gsaluja9 commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ad-claw000 left a comment

Choose a reason for hiding this comment

Uh oh!

ad-claw000 left a comment

Choose a reason for hiding this comment

Uh oh!

ad-claw000 commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ad-claw000 commented May 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ad-claw000 commented May 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gsaluja9 commented Sep 15, 2025 •

edited

Loading

ad-claw000 commented May 24, 2026 •

edited

Loading