feat: Dynamic SEO/crawler endpoints

[hhvrc]

No description provided.

[Copilot]

Pull request overview

Adds dynamic crawler/metadata endpoints (robots/sitemap/llms) driven by environment flags and auto-discovered public routes.

Changes:

• Replace static robots.txt with a runtime robots.txt endpoint that can deny crawlers and optionally advertise a sitemap.
• Add new sitemap.xml and llms.txt endpoints.
• Introduce route-discovery (public-routes) and boolean flag parsing (isTruthy) utilities.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
static/robots.txt	Removes the static robots file so the new dynamic endpoint can take effect.
src/routes/(meta)/robots.txt/+server.ts	Implements dynamic robots.txt with env-controlled deny + sitemap link.
src/routes/(meta)/sitemap.xml/+server.ts	Implements a generated sitemap based on discovered public routes.
src/routes/(meta)/llms.txt/+server.ts	Implements an llms.txt document generated from public/app routes + env metadata.
src/routes/(meta)/+layout.ts	Adds route-group config (prerender off) for meta endpoints.
src/lib/utils/public-routes.ts	Discovers routes via import.meta.glob and exposes public/authenticated route lists.
src/lib/utils/parse.ts	Adds isTruthy helper for env-var style boolean parsing.
src/lib/utils/index.ts	Re-exports the new parse utility.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.