Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

MCP Documentation Scraper Template

A complete Python template for building Model Context Protocol (MCP) servers with documentation scraping, vector search, and flexible deployment options.

Features

  • 🔍 Dual Search Modes: Semantic search (ML-powered) or keyword search (lightweight)
  • 🤖 Multi-Provider Embeddings: HuggingFace (local), OpenAI, or Azure OpenAI
  • 📚 Web Scraping: Configurable documentation crawler with depth control
  • 💾 Vector Storage: ChromaDB with persistent storage
  • ⏰ Auto-Updates: Scheduled documentation refresh
  • 🔌 Dual Transport: stdio (local MCP) or HTTP/SSE (remote access)
  • 🐳 Docker Ready: Containerized deployment with profiles
  • ☸️ Kubernetes Ready: Production deployment with autoscaling

Quick Start

Option 1: stdio Mode (Local MCP - Default)

For: VS Code, Claude Desktop integration on your local machine

  1. Configure environment:

    cp .env.example .env
    # Edit .env and set your DOCS_URLS
    # Keep TRANSPORT=stdio (or leave it out, stdio is default)
  2. Start container:

    docker-compose up -d

    This starts mcp-server-template container (no ports exposed).

  3. Configure MCP client (VS Code or Claude Desktop):

    {
      "mcpServers": {
        "docs-scraper": {
          "command": "docker",
          "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
        }
      }
    }

Option 2: HTTP Mode (Remote Access)

For: Testing API, remote access, or preparing for Kubernetes deployment

  1. Configure environment for HTTP:

    cp .env.example .env
    # Edit .env and set:
    # TRANSPORT=http
    # DOCS_URLS=https://your-docs-site.com/
  2. Start container with HTTP profile:

    docker-compose --profile http up -d

    This starts mcp-server-template-http container on port 3003.

  3. Test the server:

    # Check health
    curl http://localhost:3003/health
    
    # MCP clients connect to:
    # http://localhost:3003/sse

Switching Between Modes

Stop current mode:

docker-compose down

Start stdio mode:

docker-compose up -d

Start HTTP mode:

docker-compose --profile http up -d

View logs:

# stdio mode
docker logs mcp-server-template -f

# HTTP mode
docker logs mcp-server-template-http -f

Configuration

Key environment variables (see .env.example for full list):

  • TRANSPORT: Transport mode - stdio (local MCP) or http (remote/Kubernetes)
  • DOCS_URLS: Documentation URLs to scrape (comma-separated for multiple sites)
    • Single: DOCS_URLS=https://docs.example.com/
    • Multiple: DOCS_URLS=https://docs.example.com/,https://api.example.com/docs/,https://guides.example.com/
  • SWAGGER_URLS: Swagger/OpenAPI JSON URLs (comma-separated, optional)
  • USE_EMBEDDINGS: Enable semantic search (true) or keyword search (false)
  • EMBEDDING_PROVIDER: huggingface, openai, or azure
  • CRAWL_MAX_DEPTH: Maximum crawl depth (0-3, recommended 2)
  • AUTO_UPDATE_ENABLED: Enable scheduled documentation updates (true/false)
  • HTTP_PORT: Port for HTTP mode (default: 3000, mapped to 3003 on host)

Transport Modes

stdio Mode (Local MCP)

Best for: Local development, VS Code, Claude Desktop

This server uses stdio transport for direct MCP client communication via stdin/stdout pipes.

VS Code (settings.json):

{
  "mcp.servers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "docs-scraper": {
      "command": "docker",
      "args": ["exec", "-i", "mcp-server-template", "python", "-m", "mcp_server_template.server"]
    }
  }
}

HTTP Mode (Remote Access)

Best for: Remote access, multi-user, Kubernetes deployments

Set TRANSPORT=http in .env and use the HTTP profile:

docker-compose --profile http up -d

Endpoints:

  • GET /health - Health check
  • GET /sse - MCP Server-Sent Events endpoint

MCP Client Configuration (for remote HTTP clients):

{
  "mcpServers": {
    "docs-scraper": {
      "url": "http://localhost:3003/sse",
      "transport": "sse"
    }
  }
}

Kubernetes Deployment

See k8s-deployment.yaml for production Kubernetes deployment with:

  • Horizontal Pod Autoscaling
  • Persistent Volume Claims for ChromaDB
  • Ingress configuration
  • Health checks and readiness probes

Customization

This is a template - fork and customize for your documentation sources:

  1. Update DOCS_URLS in .env
  2. Customize scraping in src/mcp_server_template/documentation_scraper.py
  3. Add custom tools in src/mcp_server_template/server.py
  4. Adjust chunking and search parameters as needed

Requirements

  • Python 3.12+
  • Docker & Docker Compose (recommended)
  • 500MB+ memory for semantic search (50MB for keyword-only)

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Contributors

Languages