feat(crawler): reorganize the code and improve URL capture reliability and quality#14
Conversation
… API paths
- Refactor monolithic crawler service into separate routers:
- routers/ for API endpoint definitions
- services/ for business logic services
- Delete legacy single-file services:
- converter_service.py
- crawler_service.py
- docx_service.py
- file_parser_service.py
- pptx_service.py
- template_service.py
- Update API endpoint paths in platform Convex functions:
- /api/v1/fetch-urls → /api/v1/urls/fetch
- /api/v1/discover → /api/v1/urls/discover
- /api/v1/parse/file → /api/v1/{pdf,docx,pptx}/parse
- /api/v1/convert/{src}-to-{fmt} → /api/v1/{pdf,images}/from-{src}
- /api/v1/template/analyze-pptx-upload → /api/v1/pptx/analyze
- /api/v1/document/generate-docx → /api/v1/docx
- /api/v1/template/generate-docx → /api/v1/docx/from-template
- /api/v1/template/generate-pptx → /api/v1/pptx
- Remove crawl_website operation from crawler action (consolidated into discover + fetch workflow)
- Clean up unused imports in config.py
- Remove test-crawler.sh script
- Increase default viewport to 1920x1080 for better desktop screenshots - Change default wait_until from 'networkidle' to 'load' with 60s timeout - Add configurable timeout parameter to URL-to-image and URL-to-PDF requests - Implement fallback strategy: wait for 'domcontentloaded' first, then 'load' - Add cookie consent dialog auto-dismissal for cleaner screenshots - Scroll through page before capture to trigger lazy-loaded images - Add logging for debugging screenshot capture issues
larryro
changed the title
📝 WalkthroughWalkthroughThis PR refactors the crawler service from a monolithic architecture to a modular, router-based design. It removes the monolithic Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Areas requiring extra attention:
Possibly related PRs
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 26
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
services/crawler/app/main.py (2)
38-68: Startup error handling is OK, but prefer stack traces for init failures.
Right now failures log only{e};logger.exception(...)will preserve the traceback (much faster to debug production startup issues).
81-88: Strip whitespace inallow_originsto avoid subtle CORS mismatches.app.add_middleware( CORSMiddleware, - allow_origins=settings.allowed_origins.split(","), + allow_origins=[o.strip() for o in settings.allowed_origins.split(",") if o.strip()], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], )
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro (Legacy)
📒 Files selected for processing (29)
services/crawler/app/config.py(0 hunks)services/crawler/app/converter_service.py(0 hunks)services/crawler/app/main.py(2 hunks)services/crawler/app/models.py(3 hunks)services/crawler/app/routers/__init__.py(1 hunks)services/crawler/app/routers/crawler.py(1 hunks)services/crawler/app/routers/docx.py(1 hunks)services/crawler/app/routers/image.py(1 hunks)services/crawler/app/routers/pdf.py(1 hunks)services/crawler/app/routers/pptx.py(1 hunks)services/crawler/app/services/__init__.py(1 hunks)services/crawler/app/services/base_converter.py(1 hunks)services/crawler/app/services/crawler_service.py(1 hunks)services/crawler/app/services/image_service.py(1 hunks)services/crawler/app/services/pdf_service.py(1 hunks)services/crawler/app/services/pptx_service.py(0 hunks)services/crawler/app/services/template_service.py(1 hunks)services/crawler/test-crawler.sh(0 hunks)services/platform/convex/agent_tools/convex_tools/crawler/helpers/fetch_page_content.ts(1 hunks)services/platform/convex/agent_tools/convex_tools/files/helpers/parse_file.ts(3 hunks)services/platform/convex/agent_tools/convex_tools/workflows/action_reference.ts(0 hunks)services/platform/convex/model/documents/analyze_pptx.ts(1 hunks)services/platform/convex/model/documents/generate_document_helpers.ts(1 hunks)services/platform/convex/model/documents/generate_docx.ts(1 hunks)services/platform/convex/model/documents/generate_docx_from_template.ts(1 hunks)services/platform/convex/model/documents/generate_pptx.ts(1 hunks)services/platform/convex/predefined_workflows/website_scan.ts(1 hunks)services/platform/convex/workflow/actions/crawler/crawler_action.ts(3 hunks)services/platform/convex/workflow/actions/crawler/helpers/types.ts(1 hunks)
💤 Files with no reviewable changes (5)
- services/crawler/app/services/pptx_service.py
- services/crawler/test-crawler.sh
- services/crawler/app/converter_service.py
- services/platform/convex/agent_tools/convex_tools/workflows/action_reference.ts
- services/crawler/app/config.py
🧰 Additional context used
📓 Path-based instructions (3)
**/*
📄 CodeRabbit inference engine (.cursor/rules/workspace_rules.mdc)
Use English only for ALL user-facing content including UI components, labels, buttons, dialogs, forms, toast messages, error messages, success messages, comments, documentation, README files, variable names, function names, and type names
Files:
services/platform/convex/predefined_workflows/website_scan.tsservices/platform/convex/model/documents/generate_pptx.tsservices/platform/convex/agent_tools/convex_tools/crawler/helpers/fetch_page_content.tsservices/crawler/app/services/template_service.pyservices/platform/convex/model/documents/generate_docx.tsservices/platform/convex/model/documents/generate_docx_from_template.tsservices/crawler/app/routers/crawler.pyservices/crawler/app/routers/pdf.pyservices/platform/convex/model/documents/generate_document_helpers.tsservices/platform/convex/agent_tools/convex_tools/files/helpers/parse_file.tsservices/crawler/app/routers/pptx.pyservices/platform/convex/workflow/actions/crawler/helpers/types.tsservices/crawler/app/services/crawler_service.pyservices/crawler/app/services/pdf_service.pyservices/crawler/app/services/image_service.pyservices/crawler/app/routers/__init__.pyservices/platform/convex/workflow/actions/crawler/crawler_action.tsservices/crawler/app/services/__init__.pyservices/crawler/app/routers/image.pyservices/crawler/app/routers/docx.pyservices/crawler/app/services/base_converter.pyservices/crawler/app/main.pyservices/crawler/app/models.pyservices/platform/convex/model/documents/analyze_pptx.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.cursor/rules/workspace_rules.mdc)
**/*.{ts,tsx,js,jsx}: Use Vercel AI SDK with OpenAI - import from 'ai' and '@ai-sdk/openai', never use raw OpenAI SDK or OpenRouter
Never hallucinate API keys - always use environment variables and existing .env configuration
Use camelCase for function names (e.g.,getUserData)
Use SCREAMING_SNAKE_CASE for constants (e.g.,API_BASE_URL,MAX_RETRIES)
Use feature flags with enums (TypeScript) or const objects (JavaScript) with UPPERCASE_WITH_UNDERSCORE naming
Implement error handling with try-catch pattern: check for result.error and display descriptive toast messages using result.error as title
Files:
services/platform/convex/predefined_workflows/website_scan.tsservices/platform/convex/model/documents/generate_pptx.tsservices/platform/convex/agent_tools/convex_tools/crawler/helpers/fetch_page_content.tsservices/platform/convex/model/documents/generate_docx.tsservices/platform/convex/model/documents/generate_docx_from_template.tsservices/platform/convex/model/documents/generate_document_helpers.tsservices/platform/convex/agent_tools/convex_tools/files/helpers/parse_file.tsservices/platform/convex/workflow/actions/crawler/helpers/types.tsservices/platform/convex/workflow/actions/crawler/crawler_action.tsservices/platform/convex/model/documents/analyze_pptx.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/workspace_rules.mdc)
**/*.{ts,tsx}: Use kebab-case for file names (e.g.,user-profile.tsx)
Use PascalCase for component names (e.g.,UserProfile)
Use descriptive messages as toast title (never generic 'Error'), with optional description for additional context only
Follow component structure: 'use client' directive, imports, interface Props, hooks, effects, event handlers, then render
Prioritize data fetching methods in order: Server Actions (preferred), Route Handlers (when needed), Client-side (minimal use)
Use React.memo for expensive components to optimize performance
Use Next.js Image component for all images instead of native img tags
Use dynamic imports for code splitting
Files:
services/platform/convex/predefined_workflows/website_scan.tsservices/platform/convex/model/documents/generate_pptx.tsservices/platform/convex/agent_tools/convex_tools/crawler/helpers/fetch_page_content.tsservices/platform/convex/model/documents/generate_docx.tsservices/platform/convex/model/documents/generate_docx_from_template.tsservices/platform/convex/model/documents/generate_document_helpers.tsservices/platform/convex/agent_tools/convex_tools/files/helpers/parse_file.tsservices/platform/convex/workflow/actions/crawler/helpers/types.tsservices/platform/convex/workflow/actions/crawler/crawler_action.tsservices/platform/convex/model/documents/analyze_pptx.ts
🧠 Learnings (14)
📚 Learning: 2025-07-20T08:40:24.693Z
Learnt from: CR
Repo: talecorp/poc PR: 0
File: .cursor/rules/ai.mdc:0-0
Timestamp: 2025-07-20T08:40:24.693Z
Learning: Applies to **/app/api/**/*.ts : For text generation, use the `generateObject` or similar functions from the AI SDK
Applied to files:
services/platform/convex/model/documents/generate_pptx.tsservices/platform/convex/model/documents/generate_docx.tsservices/platform/convex/model/documents/generate_docx_from_template.tsservices/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-12-02T08:13:51.424Z
Learnt from: CR
Repo: tale-project/tale PR: 0
File: .cursor/rules/workspace_rules.mdc:0-0
Timestamp: 2025-12-02T08:13:51.424Z
Learning: Applies to services/*/convex/*.ts : Thin wrapper API modules (like `services/platform/convex/documents.ts`) may export multiple Convex functions as wrappers that delegate to model helpers, but must not contain business logic and must only perform argument/return validation and delegation
Applied to files:
services/platform/convex/model/documents/generate_pptx.tsservices/platform/convex/model/documents/generate_docx_from_template.tsservices/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-10-03T11:34:20.628Z
Learnt from: CR
Repo: talecorp/poc2 PR: 0
File: .cursor/rules/convex_rules.mdc:0-0
Timestamp: 2025-10-03T11:34:20.628Z
Learning: Applies to convex/**/*.{ts,js} : Use ctx.storage.getUrl() for signed URLs; do not use deprecated ctx.storage.getMetadata
Applied to files:
services/platform/convex/agent_tools/convex_tools/crawler/helpers/fetch_page_content.ts
📚 Learning: 2025-07-20T08:40:24.693Z
Learnt from: CR
Repo: talecorp/poc PR: 0
File: .cursor/rules/ai.mdc:0-0
Timestamp: 2025-07-20T08:40:24.693Z
Learning: Applies to **/lib/**/*.ts : For text generation, use the `generateObject` or similar functions from the AI SDK
Applied to files:
services/platform/convex/model/documents/generate_docx_from_template.tsservices/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-07-20T08:40:24.693Z
Learnt from: CR
Repo: talecorp/poc PR: 0
File: .cursor/rules/ai.mdc:0-0
Timestamp: 2025-07-20T08:40:24.693Z
Learning: Applies to **/actions/*.ts : For text generation, use the `generateObject` or similar functions from the AI SDK
Applied to files:
services/platform/convex/model/documents/generate_docx_from_template.ts
📚 Learning: 2025-11-30T12:29:39.745Z
Learnt from: CR
Repo: tale-project/poc2 PR: 0
File: .cursor/rules/workspace_rules.mdc:0-0
Timestamp: 2025-11-30T12:29:39.745Z
Learning: Applies to services/**/convex/*.ts : Thin wrapper API modules in services may export multiple Convex functions as thin wrappers that delegate to model helpers, must use snake_case file names and camelCase export names, and must not contain business logic
Applied to files:
services/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-10-03T11:34:20.628Z
Learnt from: CR
Repo: talecorp/poc2 PR: 0
File: .cursor/rules/convex_rules.mdc:0-0
Timestamp: 2025-10-03T11:34:20.628Z
Learning: Applies to convex/**/*.ts : Use Id helper type from ./_generated/dataModel to type document ids (e.g., Id<'users'>)
Applied to files:
services/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-12-02T08:13:24.290Z
Learnt from: CR
Repo: tale-project/tale PR: 0
File: .cursor/rules/convex_rules.mdc:0-0
Timestamp: 2025-12-02T08:13:24.290Z
Learning: Applies to convex/**/convex/*.{ts,tsx} : Thin wrapper API modules (e.g., `services/platform/convex/documents.ts`) may export multiple Convex functions as thin wrappers around model-layer helpers. These files must not contain business logic, should only validate args/returns and delegate to the model layer, must use snake_case for file names and camelCase for each exported function.
Applied to files:
services/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-11-30T03:53:00.316Z
Learnt from: CR
Repo: tale-project/poc2 PR: 0
File: .cursor/rules/convex_rules.mdc:0-0
Timestamp: 2025-11-30T03:53:00.316Z
Learning: Applies to convex/**/!(model)/*.ts : Exception: Thin wrapper API modules (e.g., `services/platform/convex/documents.ts`) may export multiple Convex functions if they only validate args/returns and delegate to the model layer; must use snake_case for file names and camelCase for function names
Applied to files:
services/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-12-02T08:13:51.424Z
Learnt from: CR
Repo: tale-project/tale PR: 0
File: .cursor/rules/workspace_rules.mdc:0-0
Timestamp: 2025-12-02T08:13:51.424Z
Learning: Applies to convex/**/*.ts : Split existing Convex modules that export multiple functions into separate files, each exporting exactly one function, except for thin wrapper modules
Applied to files:
services/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-11-30T12:29:39.745Z
Learnt from: CR
Repo: tale-project/poc2 PR: 0
File: .cursor/rules/workspace_rules.mdc:0-0
Timestamp: 2025-11-30T12:29:39.745Z
Learning: Applies to convex/*.ts : Split existing Convex modules that export multiple functions into separate files with one function per file, except for explicitly designated thin wrapper modules
Applied to files:
services/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-10-03T11:34:20.628Z
Learnt from: CR
Repo: talecorp/poc2 PR: 0
File: .cursor/rules/convex_rules.mdc:0-0
Timestamp: 2025-10-03T11:34:20.628Z
Learning: Applies to convex/**/*.{ts,js} : Always use the new Convex function syntax (query/mutation/action with args/returns/handler)
Applied to files:
services/platform/convex/model/documents/generate_document_helpers.ts
📚 Learning: 2025-12-02T08:13:24.290Z
Learnt from: CR
Repo: tale-project/tale PR: 0
File: .cursor/rules/convex_rules.mdc:0-0
Timestamp: 2025-12-02T08:13:24.290Z
Learning: Applies to convex/**/*.{ts,tsx} : Convex uses file-based routing: a public function in `convex/example.ts` named `f` has reference `api.example.f`; a private function named `g` has reference `internal.example.g`.
Applied to files:
services/platform/convex/agent_tools/convex_tools/files/helpers/parse_file.ts
📚 Learning: 2025-11-30T03:53:00.316Z
Learnt from: CR
Repo: tale-project/poc2 PR: 0
File: .cursor/rules/convex_rules.mdc:0-0
Timestamp: 2025-11-30T03:53:00.316Z
Learning: Applies to convex/**/*.ts : Convex uses file-based routing; a public function in `convex/example.ts` named `f` has reference `api.example.f`, and a private function `g` has reference `internal.example.g`
Applied to files:
services/platform/convex/agent_tools/convex_tools/files/helpers/parse_file.ts
🧬 Code graph analysis (7)
services/crawler/app/routers/pdf.py (4)
services/crawler/app/services/pdf_service.py (1)
get_pdf_service(119-124)services/crawler/app/services/file_parser_service.py (2)
FileParserService(17-147)parse_pdf(20-52)services/crawler/app/routers/docx.py (1)
get_file_parser_service(25-30)services/crawler/app/routers/pptx.py (1)
get_file_parser_service(25-30)
services/platform/convex/model/documents/generate_document_helpers.ts (2)
services/platform/convex/model/documents/types.ts (2)
DocumentSourceType(113-113)DocumentOutputFormat(115-115)services/platform/convex/model/documents/index.ts (2)
DocumentSourceType(17-17)DocumentOutputFormat(18-18)
services/crawler/app/services/image_service.py (2)
services/platform/convex/model/documents/types.ts (1)
WaitUntilType(137-137)services/crawler/app/services/base_converter.py (5)
BaseConverterService(116-194)_get_page(163-167)_wrap_html(169-171)_wait_for_twemoji(173-177)markdown_to_html(179-194)
services/crawler/app/routers/image.py (3)
services/crawler/app/models.py (3)
MarkdownToImageRequest(111-116)HtmlToImageRequest(128-134)UrlToImageRequest(146-153)services/crawler/app/services/image_service.py (3)
get_image_service(233-238)html_to_image(93-136)url_to_image(147-226)services/crawler/app/services/base_converter.py (1)
initialize(125-146)
services/crawler/app/routers/docx.py (6)
services/crawler/app/models.py (3)
GenerateDocxRequest(247-250)GenerateDocxResponse(253-259)ParseFileResponse(265-276)services/crawler/app/services/template_service.py (3)
get_template_service(117-122)generate_docx(86-95)generate_docx_from_template(97-110)services/crawler/app/services/file_parser_service.py (2)
FileParserService(17-147)parse_docx(54-88)services/crawler/app/routers/pdf.py (1)
get_file_parser_service(24-29)services/crawler/app/routers/pptx.py (1)
get_file_parser_service(25-30)services/crawler/app/services/docx_service.py (2)
generate_docx(47-93)generate_docx_from_template(172-209)
services/crawler/app/main.py (4)
services/crawler/app/models.py (1)
HealthResponse(68-75)services/crawler/app/services/crawler_service.py (1)
get_crawler_service(381-386)services/crawler/app/services/pdf_service.py (1)
get_pdf_service(119-124)services/crawler/app/services/image_service.py (1)
get_image_service(233-238)
services/crawler/app/models.py (1)
services/platform/convex/model/documents/types.ts (1)
WaitUntilType(137-137)
🪛 Ruff (0.14.8)
services/crawler/app/routers/crawler.py
68-68: Do not catch blind exception: Exception
(BLE001)
70-73: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
72-72: Use explicit conversion flag
Replace with conversion flag
(RUF010)
124-124: Do not catch blind exception: Exception
(BLE001)
126-129: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
128-128: Use explicit conversion flag
Replace with conversion flag
(RUF010)
147-150: Consider moving this statement to an else block
(TRY300)
152-152: Do not catch blind exception: Exception
(BLE001)
154-157: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
156-156: Use explicit conversion flag
Replace with conversion flag
(RUF010)
services/crawler/app/routers/pdf.py
66-66: Do not catch blind exception: Exception
(BLE001)
68-71: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
70-70: Use explicit conversion flag
Replace with conversion flag
(RUF010)
109-109: Do not catch blind exception: Exception
(BLE001)
111-114: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
113-113: Use explicit conversion flag
Replace with conversion flag
(RUF010)
152-152: Do not catch blind exception: Exception
(BLE001)
154-157: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
156-156: Use explicit conversion flag
Replace with conversion flag
(RUF010)
162-162: Do not perform function call File in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
179-182: Abstract raise to an inner function
(TRY301)
193-193: Do not catch blind exception: Exception
(BLE001)
198-198: Use explicit conversion flag
Replace with conversion flag
(RUF010)
services/crawler/app/routers/pptx.py
35-35: Do not perform function call File in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
53-56: Abstract raise to an inner function
(TRY301)
67-67: Do not catch blind exception: Exception
(BLE001)
71-71: Use explicit conversion flag
Replace with conversion flag
(RUF010)
79-79: Do not perform function call File in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
118-121: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
120-120: Use explicit conversion flag
Replace with conversion flag
(RUF010)
129-132: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
131-131: Use explicit conversion flag
Replace with conversion flag
(RUF010)
139-139: Do not catch blind exception: Exception
(BLE001)
140-143: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
142-142: Use explicit conversion flag
Replace with conversion flag
(RUF010)
163-163: Do not catch blind exception: Exception
(BLE001)
167-167: Use explicit conversion flag
Replace with conversion flag
(RUF010)
173-173: Do not perform function call File in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
190-193: Abstract raise to an inner function
(TRY301)
204-204: Do not catch blind exception: Exception
(BLE001)
209-209: Use explicit conversion flag
Replace with conversion flag
(RUF010)
services/crawler/app/services/pdf_service.py
19-19: Boolean-typed positional argument in function definition
(FBT001)
19-19: Boolean default positional argument in function definition
(FBT002)
21-21: Boolean-typed positional argument in function definition
(FBT001)
21-21: Boolean default positional argument in function definition
(FBT002)
26-26: Boolean-typed positional argument in function definition
(FBT001)
26-26: Boolean default positional argument in function definition
(FBT002)
63-63: Missing type annotation for **pdf_options
(ANN003)
74-74: Missing type annotation for **pdf_options
(ANN003)
services/crawler/app/services/image_service.py
66-67: try-except-continue detected, consider logging the exception
(S112)
66-66: Do not catch blind exception: Exception
(BLE001)
96-96: Boolean-typed positional argument in function definition
(FBT001)
96-96: Boolean default positional argument in function definition
(FBT002)
99-99: Boolean-typed positional argument in function definition
(FBT001)
99-99: Boolean default positional argument in function definition
(FBT002)
141-141: Missing type annotation for **image_options
(ANN003)
153-153: Boolean-typed positional argument in function definition
(FBT001)
153-153: Boolean default positional argument in function definition
(FBT002)
221-221: Consider moving this statement to an else block
(TRY300)
services/crawler/app/routers/__init__.py
18-24: __all__ is not sorted
Apply an isort-style sorting to __all__
(RUF022)
services/crawler/app/services/__init__.py
24-39: __all__ is not sorted
Apply an isort-style sorting to __all__
(RUF022)
services/crawler/app/routers/image.py
54-54: Do not catch blind exception: Exception
(BLE001)
56-59: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
58-58: Use explicit conversion flag
Replace with conversion flag
(RUF010)
98-98: Do not catch blind exception: Exception
(BLE001)
100-103: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
102-102: Use explicit conversion flag
Replace with conversion flag
(RUF010)
143-143: Do not catch blind exception: Exception
(BLE001)
145-148: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
147-147: Use explicit conversion flag
Replace with conversion flag
(RUF010)
services/crawler/app/routers/docx.py
82-82: Do not catch blind exception: Exception
(BLE001)
86-86: Use explicit conversion flag
Replace with conversion flag
(RUF010)
93-93: Do not perform function call File in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
126-129: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
128-128: Use explicit conversion flag
Replace with conversion flag
(RUF010)
136-136: Do not catch blind exception: Exception
(BLE001)
137-140: Within an except clause, raise exceptions with raise ... from err or raise ... from None to distinguish them from errors in exception handling
(B904)
139-139: Use explicit conversion flag
Replace with conversion flag
(RUF010)
166-166: Do not catch blind exception: Exception
(BLE001)
170-170: Use explicit conversion flag
Replace with conversion flag
(RUF010)
176-176: Do not perform function call File in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable
(B008)
193-196: Abstract raise to an inner function
(TRY301)
207-207: Do not catch blind exception: Exception
(BLE001)
212-212: Use explicit conversion flag
Replace with conversion flag
(RUF010)
services/crawler/app/services/base_converter.py
119-119: Missing return type annotation for special method __init__
Add return type annotation: None
(ANN204)
173-173: Missing return type annotation for private function _wait_for_twemoji
Add return type annotation: None
(ANN202)
services/crawler/app/main.py
16-16: Import from collections.abc instead: AsyncGenerator
Import from collections.abc
(UP035)
services/crawler/app/models.py
74-74: Boolean positional value in function call
(FBT003)
75-75: Boolean positional value in function call
(FBT003)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Build Docker Image
🔇 Additional comments (25)
services/platform/convex/predefined_workflows/website_scan.ts (1)
10-11: LGTM! Documentation updated to reflect new API paths.The documentation comments have been correctly updated to reference the new crawler API endpoints (
/api/v1/urls/discoverand/api/v1/urls/fetch), aligning with the router-based refactor across the codebase.services/platform/convex/model/documents/generate_docx_from_template.ts (1)
43-43: Verify that the template endpoint is implemented.The endpoint has been updated to
/api/v1/docx/from-template. Ensure the backend implements this endpoint for template-based DOCX generation.services/crawler/app/services/template_service.py (1)
17-18: No action required. All imported modules and symbols exist and are properly exported:
pptx_service.pyexportsPptxServiceclass andget_pptx_service()functiondocx_service.pyexportsDocxServiceclass andget_docx_service()functionservices/platform/convex/model/documents/generate_document_helpers.ts (1)
22-32: All format-specific conversion endpoints are properly implemented.Verification confirms the backend routers implement all required combinations:
- PDF:
/from-markdown(line 32),/from-html(line 74),/from-url(line 117)- Images:
/from-markdown(line 19),/from-html(line 62),/from-url(line 106)The endpoint construction in
getEndpointPath()correctly aligns with the backend router paths.services/platform/convex/model/documents/generate_pptx.ts (1)
80-80: PPTX generation endpoint is properly implemented and configured.The backend router at
services/crawler/app/routers/pptx.pyconfirms the POST/api/v1/pptxendpoint is correctly implemented. The endpoint accepts JSON slide content with optional template and branding parameters, delegates totemplate_service.generate_pptx_from_content(), and returns the generated PPTX as base64 with proper error handling.services/platform/convex/model/documents/analyze_pptx.ts (1)
84-84: PPTX analysis endpoint is properly implemented.The
/api/v1/pptx/analyzeendpoint exists inservices/crawler/app/routers/pptx.py(line 33) with full multipart form upload support viaUploadFile = File(...). The endpoint correctly handles file upload, validates input, calls the template service, and returnsAnalyzePptxResponsewith proper error handling.services/platform/convex/model/documents/generate_docx.ts (1)
60-60: Backend endpoint confirmed:/api/v1/docxis implemented with proper request/response models.The endpoint exists at
services/crawler/app/routers/docx.pywith a POST handler that acceptsGenerateDocxRequestand returnsGenerateDocxResponse. The response includessuccess,file_base64,file_size, and optionalerrorfields, with proper error handling in place.services/platform/convex/agent_tools/convex_tools/crawler/helpers/fetch_page_content.ts (1)
91-91: Verify that the backend crawler service implements the/api/v1/urls/fetchendpoint with the correct request/response contract.The TypeScript client at line 91 calls
${crawlerServiceUrl}/api/v1/urls/fetch. Confirm the crawler router implements this endpoint and that the request/response format matches what the client expects.services/platform/convex/workflow/actions/crawler/helpers/types.ts (1)
1-14: LGTM!The operation union type is correctly updated to reflect only the supported operations (
discover_urls|fetch_urls). The interface is well-structured with clear optional parameters for each operation type.services/crawler/app/services/crawler_service.py (1)
191-195: Good improvement to exception handling.Narrowing the exception handler to specific types (
JSONDecodeError,TypeError) is a best practice - it prevents silently swallowing unexpected errors while still handling the expected failure modes of JSON parsing.services/platform/convex/agent_tools/convex_tools/files/helpers/parse_file.ts (1)
63-84: Good integration of endpoint routing with improved logging.The endpoint selection logic is cleanly integrated, and including
endpointin the debug log will help with troubleshooting routing issues.services/crawler/app/services/__init__.py (1)
1-22: Well-structured package initializer.Good documentation of the service layer components and clean centralized exports. This aligns well with the modular router-based architecture.
services/platform/convex/workflow/actions/crawler/crawler_action.ts (3)
16-20: Clean simplification of the action interface.The removal of
crawl_websitefrom the operation union and description aligns with the modular architecture refactor. The remaining operations are well-documented.
134-141: Endpoint registration verified.The fetch endpoint path
/api/v1/urls/fetchis correctly registered in the backend crawler router atservices/crawler/app/routers/crawler.py(line 76:@router.post("/fetch")), combined with the APIRouter prefix/api/v1/urls(line 18), and included in the FastAPI app atservices/crawler/app/main.py(line 92).
85-92: Endpoint verified and correctly registered.The
/api/v1/urls/discoverendpoint is properly registered in the backend crawler router atservices/crawler/app/routers/crawler.py(line 21) with the APIRouter prefix/api/v1/urls. The fetch call matches the backend endpoint definition and request/response types.services/crawler/app/services/base_converter.py (1)
119-124: LGTM on the class initialization and thread-safe setup.The double-checked locking pattern in
initialize()correctly prevents race conditions during concurrent initialization attempts. The_lockensures only one browser instance is created.services/crawler/app/models.py (2)
74-75: LGTM on the health response model updates.The new
pdf_service_initializedandimage_service_initializedfields properly reflect the new modular service architecture. The default values ofFalseare appropriate since services start uninitialized.
142-143: Good defaults for reliability improvements.Changing
wait_untilfrom"networkidle"to"load"with explicit timeout fields improves reliability for sites with slow external resources. The 60-second default timeout is reasonable for URL capture operations.Also applies to: 150-153
services/crawler/app/routers/__init__.py (1)
1-24: Router aggregation looks clean and maintainable.
The explicit__all__and aliases make router registration/imports straightforward.services/crawler/app/services/pdf_service.py (1)
115-125: Singleton accessor is straightforward.
If you ever run single-process async with high concurrency, consider a tiny lock to avoid double-init, but it’s usually benign here.services/crawler/app/main.py (1)
99-111: Health endpoint implementation is correct and does not have the cited issue.The services use a proper lazy initialization pattern: constructors (init) only initialize cheap flags and None values, while expensive resource allocation (Playwright browser launch) happens in separate async
initialize()methods. The health endpoint calls the service getters to retrieve cached singleton instances and reports theirinitializedflags—it does not trigger any resource allocation. Subsequent calls to/healthafter the first request are very cheap.Likely an incorrect or invalid review comment.
services/crawler/app/services/image_service.py (4)
1-13: LGTM!Imports are appropriate and the module docstring clearly describes the service's purpose.
138-145: LGTM!Clean delegation pattern. Consider adding type hints to
**image_optionsfor better IDE support and documentation (e.g., usingTypedDictorUnpack), but this is optional.
186-206: Good approach for triggering lazy-loaded content.The scrolling logic and subsequent wait time effectively trigger lazy-loaded images before capturing the full-page screenshot.
229-238: LGTM!Standard singleton pattern. The actual async initialization is safely protected by the lock in
BaseConverterService._get_page().
- Replace f-string logging with logger.exception for better tracebacks - Avoid exposing internal error details in HTTP responses - Use `from None` or `from e` for proper exception chaining - Centralize get_file_parser_service() in file_parser_service module - Use model_dump() instead of manual dict conversion in docx router - Add input validation for DOCX content JSON structure - Use AsyncGenerator from collections.abc instead of typing module - Add HttpUrl validation for /check endpoint URL parameter - Handle empty strings in CORS allowed_origins parsing - Add proper type annotations for **kwargs and methods - Sort __all__ exports alphabetically for consistency - Improve CSS injection in PdfService with robust HTML parsing - Handle 'commit' and 'networkidle' wait strategies in ImageService - Change set_content wait_until from 'networkidle' to 'load' for PDFs
When the index scoring loop breaks early (e.g., due to missing intermediate fields), conditions for later index fields were not being added to post-filter. Now uses a Set to track which fields were actually satisfied by the index and ensures all remaining conditions are properly added to post-filter. Addresses CodeRabbit review comment #14. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When the index scoring loop breaks early (e.g., due to missing intermediate fields), conditions for later index fields were not being added to post-filter. Now uses a Set to track which fields were actually satisfied by the index and ensures all remaining conditions are properly added to post-filter. Addresses CodeRabbit review comment #14. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 participant
Summary by CodeRabbit
Refactored Crawler API Architecture
New Features
Bug Fixes
Deprecations
crawl_websiteoperation; usediscover_urlsandfetch_urlsseparately instead./api/v1/convert/*→/api/v1/{service}/from-*;/api/v1/fetch-urls→/api/v1/urls/fetch;/api/v1/discover→/api/v1/urls/discover.Configuration Updates
✏️ Tip: You can customize this high-level summary in your review settings.