Generate llms.txt files for your website. Like robots.txt for search engines, llms.txt helps AI assistants understand and reference your content accurately.
1 credit per page · Smart filtering · Clean markdown output
# example.com
> Your AI-friendly site index
- [Getting Started](https://example.com/docs/start)
Quick start guide for new users
- [API Reference](https://example.com/docs/api)
Complete API documentation
- [Tutorials](https://example.com/tutorials)
Step-by-step guides
A new standard for making websites AI-friendly
A structured index of your site's pages with titles, URLs, and descriptions. Think of it as a sitemap designed for AI.
Complete content of your site in clean, readable markdown. Perfect for AI context windows and RAG applications.
Generate your llms.txt files in four simple steps
Provide your website URL and optional path filter
Smart crawler finds all relevant pages automatically
Choose which pages to include in the output
Download llms.txt and llms-full.txt files
Why your site needs llms.txt
Enable AI coding assistants to accurately reference your API docs, SDKs, and libraries.
Example:
Generate llms.txt for your API documentation so GitHub Copilot and Cursor can help developers use your SDK correctly.
Create training data for retrieval-augmented generation applications and AI chatbots.
Example:
Export your help center as llms-full.txt and import into vector databases to power internal AI assistants.
Future-proof your site for AI search engines and LLM-powered discovery.
Example:
Make your blog and marketing content discoverable by AI-first search tools and content aggregators.
Everything you need to make your site AI-ready
Automatically crawls your site using a headless browser to find all relevant pages, including JavaScript-rendered content.
Focus on specific sections like /docs or /blog. Perfect for large sites where you only need certain content.
Uses Mozilla Readability to extract main content, then converts to clean GitHub-flavored Markdown.
Generates both llms.txt (index with titles & descriptions) and llms-full.txt (complete content).
Perfect for importing into vector databases like Pinecone, Weaviate, or ChromaDB for AI applications.
Set URL limits, toggle image inclusion, and review/select pages before scraping.
Both help machines understand your site, but for different purposes
| Aspect | robots.txt | llms.txt |
|---|---|---|
| Purpose | Crawling rules | Content access |
| Audience | Search engines | AI assistants |
| Content | Allow/Disallow rules | Titles, URLs, content |
| Format | Simple directives | Markdown |
| Goal | Control indexing | Enable AI understanding |
| Era | Web 2.0 | AI-First Web |
Pay only for pages you actually scrape. Discovery is free.
credit per page
scraped
Free
URL Discovery
Free
Page Selection
1 credit
Per Page Scraped