How do I install the TheCrawler MCP server?

You can install the TheCrawler MCP server with one click using McpMux. Download the free McpMux desktop app from mcpmux.com/download, open it, find TheCrawler in the server registry, and click Install. McpMux automatically configures the server for your AI client — whether it's Cursor, Claude Desktop, VS Code, ChatGPT, Windsurf, or JetBrains.

Which AI clients support the TheCrawler MCP server?

The TheCrawler MCP server works with all MCP-compatible AI clients including Cursor, Claude Desktop, Claude Code, VS Code (GitHub Copilot), ChatGPT, Windsurf, JetBrains IDEs, Zed, Cline, Gemini CLI, and Amazon Q Developer. McpMux acts as a universal gateway — configure TheCrawler once and all your AI clients can use it.

Back to Registry

TheCrawler MCP Server

Community

Contributed by manchittlab

Crawl public web pages, extract clean markdown, and run diagnostic-first extraction contracts with required-field and missing-field evidence for agent workflows.

Install with McpMux Source Code Docs

About the TheCrawler MCP Server

The TheCrawler MCP server is a local (stdio) Model Context Protocol server available in the McpMux registry. Crawl public web pages, extract clean markdown, and run diagnostic-first extraction contracts with required-field and missing-field evidence for agent workflows. This is a community-contributed MCP server by manchittlab.

Install the TheCrawler MCP server with one click using McpMux. It works with Cursor, Claude Desktop, Claude Code, VS Code, ChatGPT, Windsurf, JetBrains, and any MCP-compatible AI client. No authentication required — install and start using it immediately.

Transport Configuration

1234567891011121314151617181920212223{
  "type": "stdio",
  "command": "node",
  "args": [
    "${input:THECRAWLER_ENGINE_DIR}/dist/mcp.js"
  ],
  "env": {
    "NODE_OPTIONS": "--use-system-ca"
  },
  "metadata": {
    "inputs": [
      {
        "id": "THECRAWLER_ENGINE_DIR",
        "label": "TheCrawler engine directory",
        "description": "Path to the engine folder from a GitHub source checkout. Setup: git clone https://github.com/manchittlab/TheCrawler.git && cd TheCrawler/engine && npm install && npm run build",
        "type": "directory_path",
        "required": true,
        "secret": false,
        "placeholder": "C:\\Users\\you\\TheCrawler\\engine or /home/you/TheCrawler/engine"
      }
    ]
  }
}

Details

TransportLocal (stdio)

Hostinglocal

AuthNone required

Capabilities

Tools

Links

Repository Homepage Documentation

Supported AI Clients

The TheCrawler MCP server works with all MCP-compatible AI clients through McpMux:

CursorClaude DesktopClaude CodeVS CodeChatGPTWindsurfJetBrainsZedClineGemini CLIAmazon Q

Related MCP Servers

Brave Search (Docker)

Search the web using the Brave Search API via Docker container. Supports web search, news search, and local POI search with privacy-focused results.

Brave Search (npx)

Search the web using the Brave Search API via npx. Supports web search, news search, and local POI search with privacy-focused results.

Firecrawl (npx)

Scrape, crawl, search, and extract structured data from websites. Converts web pages to clean markdown with JavaScript rendering support.

Perplexity (npx)

AI-powered search using Perplexity models. Provides web search, deep research, and reasoning capabilities with real-time web context.

Exa Search (Remote)

AI-powered web search that understands meaning, not just keywords. Hosted remote MCP server, no local setup required.

Exa Search (npx)

AI-powered web search that understands meaning, not just keywords. Find similar pages, search by content type, and get clean text from any URL via npx.

Install TheCrawler with McpMux

One-click install from the McpMux desktop app. Auto-configures for Cursor, Claude, VS Code, ChatGPT, Windsurf, JetBrains, and any MCP-compatible client.

Install