{"id":"3ca6389b-48ed-4f09-b1dd-360fba893849","kind":"products","code":"UB-D8015","name":"Crawler.sh","nameBasis":"original-brand","summary":"[Product Introduction] Crawler.sh is a fast, local-first web crawler and AEO/SEO analysis tool built on Rust. It provides command line (CLI) and native desktop application versions, and all data processing is completed on the user's local machine, ensuring data privacy and processing speed. This tool is designed to replace bloated enterprise-level software and slow cloud services to help users efficiently complete website auditing, link checking, and data scraping. [Product Features] Extremely fast site crawling: Supports custom concurrency and depth, and can complete crawling and status code detection of thousands of pages in seconds. Markdown content extraction: Automatically convert web page text into clean Markdown format for easy archiving or as contextual data for AI agents. Automated SEO analysis: Built-in 16 automatic checks to quickly identify technical SEO issues such as missing titles, duplicate meta descriptions, and dead links. Flexible data export: Supports real-time streaming output and can export results to JSON, CSV, TXT, or standard Sitemap XML.","description":"[Product Introduction] Crawler.sh is a fast, local-first web crawler and AEO/SEO analysis tool built on Rust. It provides command line (CLI) and native desktop application versions, and all data processing is completed on the user's local machine, ensuring data privacy and processing speed. This tool is designed to replace bloated enterprise-level software and slow cloud services to help users efficiently complete website auditing, link checking, and data scraping. [Product Features] Extremely fast site crawling: Supports custom concurrency and depth, and can complete crawling and status code detection of thousands of pages in seconds. Markdown content extraction: Automatically convert web page text into clean Markdown format for easy archiving or as contextual data for AI agents. Automated SEO analysis: Built-in 16 automatic checks to quickly identify technical SEO issues such as missing titles, duplicate meta descriptions, and dead links. Flexible data export: Supports real-time streaming output and can export results to JSON, CSV, TXT, or standard Sitemap XML.","country":null,"category":"Productivity","originalName":"Crawler.sh","eventDate":null,"website":null,"sourceUpdatedAt":"2026-11-03T21:00:17.124Z","englishUpdatedAt":"2026-11-04T01:06:52.172Z","reviewedAt":"2026-11-04T01:06:52.172Z","snapshotAt":"2026-11-04T00:48:01.362Z","sourceHash":"a44ac2fb01b51159043bac45b48e1d89d3bd94f65a4a745e7fceca672c5b7fc2","companyId":"a4ac7773-cb95-40cb-b59b-5b1eaf3b5bc9","foundedYear":null,"logoUrl":null,"originalText":"【产品介绍】Crawler.sh 是一款基于 Rust 构建的快速、本地优先的网页爬虫及 AEO/SEO 分析工具。它提供命令行（CLI）和原生桌面应用版本，所有数据处理均在用户本地机器完成，确保了数据隐私与处理速度。该工具旨在替代臃肿的企业级软件与缓慢的云服务，帮助用户高效完成网站审计、链接检查及数据抓取工作。【产品功能】极速站点爬取：支持自定义并发与深度，可在数秒内完成数千页面的爬取与状态码检测。Markdown 内容提取：自动将网页正文转换为干净的 Markdown 格式，便于归档或作为 AI Agent 的上下文数据。SEO 自动化分析：内置 16 项自动检查，快速识别标题缺失、元描述重复及死链等技术性 SEO 问题。灵活数据导出：支持实时流式输出，可将结果导出为 JSON、CSV、TXT 或标准 Sitemap XML。","sources":[],"related":[{"id":"a4ac7773-cb95-40cb-b59b-5b1eaf3b5bc9","kind":"companies","code":"UB-C11893","name":"Crawler.sh","nameBasis":"source-english","summary":"Local Markdown Extractor for AI Training and RAG","description":"Turn any website into clean, RAG-ready Markdown. Runs locally, renders JavaScript, respects robots.txt. No headless Chrome, no per-page fees.","country":null,"category":null,"originalName":"Crawler.sh","eventDate":null,"website":"https://crawler.sh/","sourceUpdatedAt":"2026-10-26T00:00:00.000Z","englishUpdatedAt":"2026-11-01T02:03:53.856Z","reviewedAt":"2026-11-01T02:03:53.856Z","snapshotAt":"2026-11-01T00:40:31.077Z","sourceHash":"4e6fbd7228da19e975458c599996b42943962384737298445375061bf311bb6d","companyId":null,"foundedYear":null,"logoUrl":null,"path":"/companies/crawler-sh"}],"path":"/products/crawler-sh","structured":{"city":null,"country":null,"location":null,"role":null,"category":"Productivity","company":{"name":"Crawler.sh","path":"/companies/crawler-sh"}},"definition":"Crawler.sh is a product in the Productivity category from Crawler.sh.","summaryBasis":"translated","thin":false,"canonical":"https://uniquebase.ai/products/crawler-sh"}