Robots & AI Crawler Guide

AI-Readable Content Resources

The following resources are optimized for AI assistants, LLMs, and answer engines:

/llm.txt - Plain text version (recommended for LLMs)
/llm.html - HTML version with structured data
/ai.txt - Plain text version (alternate)
/ai.html - HTML version with structured data (alternate)

For AI Tools: Use /llm.txt or /ai.txt for the most efficient parsing. Use the HTML versions for structured data and schema markup.

Static HTML Pages (No JavaScript Required)

All key site content is available as static HTML for crawlers and AI tools:

/home.html - Homepage
/features.html - Features
/pricing.html - Pricing
/docs.html - Documentation
/company/about.html - About
/company/manifesto.html - Manifesto
/company/careers.html - Careers

Product Pages (Static HTML)

Foundation (Free Tier)

/product/templates.html - Templates: Create reusable standards from AWS
/product/deployments.html - Deployments: Deploy services from templates
/product/cost-monitoring.html - Cost Monitoring: AWS spend by service and template
/product/drift-prevention.html - Drift Detection: Detect infrastructure divergence

Accelerate ($499/month)

/product/environments.html - Environments: Unlimited, cloned, and ephemeral
/product/autoscaling.html - Autoscaling: Safe, template-aware scaling
/product/observability.html - Observability: Metrics, alerts, and health
/product/policies.html - Policies & Guardrails: Enforce standards automatically

Robots.txt Directives

# Robots.txt for Reploy
# https://reploylabs.com

# AI-Readable Content Available At:
# /llm.txt - Plain text version (recommended for LLMs)
# /llm.html - HTML version with structured data
# /ai.txt - Plain text version (alternate)
# /ai.html - HTML version with structured data (alternate)

User-agent: *
Allow: /

# AI Crawlers - Explicitly Allowed
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

# Search Engine Crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: facebookexternalhit
Allow: /

User-agent: LinkedInBot
Allow: /

# Sitemap
Sitemap: https://reploylabs.com/sitemap.xml

About Reploy

Reploy is the platform layer for teams on AWS. We interpret your AWS resources and automatically provide environments, service maps, autoscaling, observability, and cost controls — all running inside your existing account.

Learn more at reploylabs.com