Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

All notable changes to this project will be documented in this file.

## [3.0.0] - 2026-08-12

### Removed
- **BREAKING**: Removed the `discover` tool and its Bright Data Discover API integration. `discover` was part of `base_tools`, so it was included in every tool group, and it was also in the default tool set served when neither `GROUPS` nor `TOOLS` is configured; it is no longer exposed by the server in any configuration. For research workflows that relied on relevance-ranked discovery, use `search_engine` (or `search_engine_batch`) to find sources and `scrape_as_markdown` (or `scrape_batch`) to read them.

## [2.11.1] - 2026-07-27

### Fixed
Expand Down
28 changes: 11 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
<a href="#quick-start">Quick Start</a> •
<a href="#pricing-and-free-tier">Pricing</a> •
<a href="#use-cases">Use Cases</a> •
<a href="#tools-reference-69-tools">Tools</a> •
<a href="#tools-reference-68-tools">Tools</a> •
<a href="#agent-skills">Agent Skills</a> •
<a href="#documentation">Docs</a> •
<a href="#support">Support</a>
Expand All @@ -32,7 +32,7 @@

## Overview

The Bright Data MCP server gives AI agents real-time access to public web data. It exposes **69 tools** covering:
The Bright Data MCP server gives AI agents real-time access to public web data. It exposes **68 tools** covering:

- **Web search** — Google, Bing, and Yandex results as structured data
- **Page scraping** — any URL as Markdown or HTML, with bot detection, CAPTCHA solving, and proxy rotation handled automatically on every request
Expand Down Expand Up @@ -262,7 +262,6 @@ Answer questions using live web data instead of training data. Search, then read
|------|-------|
| Search the web for current information | `search_engine`, `search_engine_batch` |
| Read a specific page as clean Markdown | `scrape_as_markdown`, `scrape_batch` |
| Find the most relevant sources for a research question, ranked by AI relevance score | `discover` |

Example prompts: "What's Tesla's current stock price?", "Get today's weather forecast for New York", "Find the most cited sources on EU AI regulation from the last 6 months".

Expand All @@ -289,7 +288,7 @@ Build competitor profiles from live data: funding, headcount, hiring, customer r
| Company pages, employees, job postings | `web_data_linkedin_company_profile`, `web_data_linkedin_job_listings` |
| Customer sentiment | `web_data_google_maps_reviews`, `web_data_facebook_company_reviews`, app store review tools |
| Competitor pricing pages | `scrape_as_markdown`, `scrape_batch` |
| Market discovery | `search_engine_batch`, `discover` |
| Market discovery | `search_engine_batch` |

Example prompt: "Analyze Notion as a competitor: pricing, funding, hiring focus, and what customers complain about".

Expand Down Expand Up @@ -352,7 +351,6 @@ Gather source material from many pages at once, filtered by recency and relevanc
| Task | Tools |
|------|-------|
| Collect multiple sources in one call | `scrape_batch` (up to 10 URLs) |
| Find sources by topic with date filtering | `discover` with `start_date` / `end_date` |
| News and finance data | `web_data_yahoo_finance_business`, `search_engine` with news queries |

---
Expand All @@ -361,11 +359,10 @@ Gather source material from many pages at once, filtered by recency and relevanc

| Capability | Bright Data MCP | Typical web MCP servers |
|------------|-----------------|------------------------|
| Total tools | 69 | 2–10 |
| Total tools | 68 | 2–10 |
| Platform-specific structured JSON extractors | 45 tools across e-commerce, social, business, finance, travel, app stores | Rare; generic scraping only |
| Unblocking (bot detection bypass, CAPTCHA solving, proxy rotation) | Built into every request | Usually none; blocked on protected sites |
| Search engines | Google, Bing, Yandex | Usually one |
| AI-relevance-ranked search with intent | Yes (`discover`) | Not offered |
| Browser automation | 13 tools, remote browser, no local setup | Limited or none |
| LLM response collection (ChatGPT, Grok, Perplexity) | Yes | Not offered |
| Package registry data (npm, PyPI) | Yes | Not offered |
Expand All @@ -381,7 +378,7 @@ Tools are organized into groups so you only load what you need. Fewer tools mean

- `GROUPS` enables tool bundles. Comma-separated: `GROUPS="ecommerce,browser"` (local) or `&groups=ecommerce,browser` (hosted URL)
- `TOOLS` adds individual tools on top: `TOOLS="extract,scrape_as_html"`
- Base tools are always enabled: `search_engine`, `search_engine_batch`, `scrape_as_markdown`, `scrape_batch`, `discover`
- Base tools are always enabled: `search_engine`, `search_engine_batch`, `scrape_as_markdown`, `scrape_batch`
- Group ID `custom` is reserved; use `TOOLS` for individual picks

| Group ID | Contents | Tool count |
Expand Down Expand Up @@ -437,13 +434,12 @@ Coding agent setup (Claude Code / Cursor / Windsurf) — npm and PyPI package da

---

## Tools Reference (69 Tools)
## Tools Reference (68 Tools)

### Which tool to use

- **Known URL, need the content:** `scrape_as_markdown`. Multiple URLs (up to 10): `scrape_batch`
- **Need to find information:** `search_engine`. Multiple queries (up to 10): `search_engine_batch`
- **Deep research or RAG, need relevance-ranked sources:** `discover` with an `intent`
- **Page is on a supported platform (Amazon, LinkedIn, TikTok, etc.):** use the matching `web_data_*` tool — returns clean JSON, faster and more reliable than scraping the same page
- **Structured JSON from an unsupported page:** `extract`
- **Raw HTML:** `scrape_as_html`
Expand All @@ -459,15 +455,14 @@ Notes that apply to all `web_data_*` tools:
- If a `web_data_*` call fails, `scrape_as_markdown` works on the same URL as a fallback

<details>
<summary><b>Search and Scraping — 8 tools</b></summary>
<summary><b>Search and Scraping — 7 tools</b></summary>

| Tool | Description | Group |
|------|-------------|-------|
| `search_engine` | Search Google, Bing, or Yandex. Google returns JSON (URL, title, description); Bing and Yandex return Markdown. Paginate with the `cursor` parameter | always enabled |
| `search_engine_batch` | Up to 10 search queries in one call | always enabled |
| `scrape_as_markdown` | Any URL as Markdown. Bot protection and CAPTCHA handled automatically | always enabled |
| `scrape_batch` | Up to 10 URLs in one call; returns an array of URL/content pairs in Markdown | always enabled |
| `discover` | AI-relevance-ranked web search. Returns scored results (title, description, URL, relevance score). Supports intent-based ranking, geo-targeting, date filtering, keyword filtering | always enabled |
| `scrape_as_html` | Any URL as raw HTML | `advanced_scraping` |
| `extract` | Scrape a page and convert it to structured JSON using AI, with an optional custom extraction prompt | `advanced_scraping` |
| `session_stats` | Tool usage counts for the current session | `advanced_scraping` |
Expand Down Expand Up @@ -639,11 +634,10 @@ worse results.

1. Need search results? → `search_engine` (single) or `search_engine_batch` (up to 10 queries). ALWAYS instead of WebSearch.
2. Need content from a URL? → `scrape_as_markdown` (single) or `scrape_batch` (up to 10 URLs). ALWAYS instead of WebFetch. Works on ALL websites.
3. Need relevance-ranked deep research? → `discover` with an `intent`.
4. Page on a supported platform AND the `web_data_*` tool is available? → use it. Cleaner JSON, faster, more reliable than scraping.
5. Need raw HTML? → `scrape_as_html` (advanced_scraping group).
6. Need AI-extracted JSON from an arbitrary page? → `extract` (advanced_scraping group).
7. Need interaction (click, type, scroll)? → `scraping_browser_*` tools (browser group), always snapshot before acting on refs.
3. Page on a supported platform AND the `web_data_*` tool is available? → use it. Cleaner JSON, faster, more reliable than scraping.
4. Need raw HTML? → `scrape_as_html` (advanced_scraping group).
5. Need AI-extracted JSON from an arbitrary page? → `extract` (advanced_scraping group).
6. Need interaction (click, type, scroll)? → `scraping_browser_*` tools (browser group), always snapshot before acting on refs.

## Parameter Guardrails (Critical)

Expand Down
1 change: 0 additions & 1 deletion assets/Tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
|scrape_batch|Scrape up to 10 webpages in one request and return an array of URL/content pairs in Markdown format.|
|scrape_as_html|Scrape a single webpage with advanced extraction and return the HTML response body. Handles sites protected by bot detection or CAPTCHA.|
|extract|Scrape a webpage as Markdown and convert it to structured JSON using AI sampling, with an optional custom extraction prompt.|
|discover|Search the web and rank results by AI-driven relevance. Returns scored results with title, description, URL, and relevance score. Supports intent-based ranking, geo-targeting, date filtering, and keyword filtering.|
|session_stats|Report how many times each tool has been called during the current MCP session.|
|web_data_amazon_product|Quickly read structured Amazon product data. Requires a valid product URL containing /dp/. Often faster and more reliable than scraping.|
|web_data_amazon_product_reviews|Quickly read structured Amazon product review data. Requires a valid product URL containing /dp/. Often faster and more reliable than scraping.|
Expand Down
1 change: 0 additions & 1 deletion manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@
{"name": "scrape_batch", "description": "Scrape multiple webpage URLs with advanced options for content extraction and get back the results in Markdown. This tool can unlock any webpage even if it uses bot detection or CAPTCHA. Processes up to 10 URLs."},
{"name": "scrape_as_html", "description": "Scrape a single webpage URL with advanced options for content extraction and get back the results in HTML. This tool can unlock any webpage even if it uses bot detection or CAPTCHA."},
{"name": "extract", "description": "Scrape a webpage and extract structured data as JSON. First scrapes the page as markdown, then uses AI sampling to convert it to structured JSON format. This tool can unlock any webpage even if it uses bot detection or CAPTCHA."},
{"name": "discover", "description": "Search the web and rank results by AI-driven relevance. Returns scored results with title, description, and URL. Supports intent-based ranking, geo-targeting, date filtering, and keyword filtering."},
{"name": "session_stats", "description": "Tell the user about the tool usage during this session"},
{"name": "web_data_amazon_product", "description": "Quickly read structured amazon product data. Requires a valid product URL with /dp/ in it. This can be a cache lookup, so it can be more reliable than scraping."},
{"name": "web_data_amazon_product_reviews", "description": "Quickly read structured amazon product review data. Requires a valid product URL with /dp/ in it. This can be a cache lookup, so it can be more reliable than scraping."},
Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@brightdata/mcp",
"version": "2.11.1",
"version": "3.0.0",
"description": "An MCP interface into the Bright Data toolset",
"type": "module",
"main": "./server.js",
Expand Down
121 changes: 1 addition & 120 deletions server.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ const base_timeout = process.env.BASE_TIMEOUT
const base_max_retries = Math.min(
parseInt(process.env.BASE_MAX_RETRIES || '0', 10), 3);
const pro_mode_tools = ['search_engine', 'scrape_as_markdown',
'search_engine_batch', 'scrape_batch', 'discover'];
'search_engine_batch', 'scrape_batch'];
const tool_groups = process.env.GROUPS ?
process.env.GROUPS.split(',').map(g=>g.trim().toLowerCase())
.filter(Boolean) : [];
Expand Down Expand Up @@ -486,125 +486,6 @@ addTool({
}),
});

addTool({
name: 'discover',
description: 'Search the web and rank results by AI-driven relevance. '
+'Returns scored results with title, description, and URL. Supports '
+'intent-based ranking, geo-targeting, date filtering, and keyword '
+'filtering.',
annotations: {
title: 'Discover',
readOnlyHint: true,
openWorldHint: true,
},
parameters: z.object({
query: z.string().describe('The search query'),
intent: z.string().optional().describe('Describes the specific goal '
+'of the search to help the AI evaluate and rank result relevance.'
+'If not provided, the query string is used as the intent'),
country: z.string().length(2).optional()
.describe('2-letter ISO country code for localized results '
+'(e.g., "US", "GB", "DE")'),
city: z.string().optional()
.describe('City for localized results (e.g., "New York", '
+'"Berlin")'),
language: z.string().optional()
.describe('Language code (e.g., "en", "es", "fr")'),
num_results: z.number().int().optional()
.describe('Exact number of search results to return'),
filter_keywords: z.array(z.string()).optional()
.describe('Keywords that must appear in search results'),
remove_duplicates: z.boolean().optional()
.describe('Remove duplicate results (default: true)'),
start_date: z.string().optional()
.describe('Only content updated from this date (YYYY-MM-DD)'),
end_date: z.string().optional()
.describe('Only content updated until this date (YYYY-MM-DD)'),
}),
execute: tool_fn('discover', async(data, ctx)=>{
let body = {query: data.query, format: 'json'};
if (data.intent)
body.intent = data.intent;
if (data.country)
body.country = data.country;
if (data.city)
body.city = data.city;
if (data.language)
body.language = data.language;
if (data.num_results)
body.num_results = data.num_results;
if (data.filter_keywords)
body.filter_keywords = data.filter_keywords;
if (data.remove_duplicates===false)
body.remove_duplicates = false;
if (data.start_date)
body.start_date = data.start_date;
if (data.end_date)
body.end_date = data.end_date;
let trigger_response = await axios({
url: 'https://api.brightdata.com/discover',
method: 'POST',
data: body,
headers: {
...api_headers(ctx.clientName, 'discover'),
'Content-Type': 'application/json',
},
});
let task_id = trigger_response.data?.task_id;
if (!task_id)
throw new Error('No task_id returned from discover request');
console.error(`[discover] triggered with task ID: ${task_id}`);
let max_attempts = polling_timeout;
let attempts = 0;
while (attempts<max_attempts)
{
try {
if (ctx && ctx.reportProgress)
{
await ctx.reportProgress({
progress: attempts,
total: max_attempts,
message: `Polling for discover results (attempt `
+`${attempts+1}/${max_attempts})`,
});
}
let poll_response = await axios({
url: 'https://api.brightdata.com/discover',
params: {task_id},
method: 'GET',
headers: api_headers(ctx.clientName, 'discover'),
});
if (poll_response.data?.status==='processing')
{
console.error(`[discover] still processing, polling `
+`again (attempt ${attempts+1}/${max_attempts})`);
attempts++;
await new Promise(resolve=>setTimeout(resolve, 1000));
continue;
}
console.error(`[discover] results received after `
+`${attempts+1} attempts`);
let results = poll_response.data?.results || [];
results = results.map(r=>({
link: r.link,
title: r.title,
description: r.description,
relevance_score: r.relevance_score,
}));
return JSON.stringify(results);
} catch(e){
console.error(`[discover] polling error: ${e.message}`);
if (e.response?.status===400)
throw e;
attempts++;
await new Promise(resolve=>setTimeout(resolve, 1000));
}
}
throw new Error(`Timeout after ${max_attempts} seconds waiting `
+`for discover results`);
}),
});

const SEARCHABLE_DATASETS_DESC = [
'Supported dataset_id values:',
'- gd_l1viktl72bvl7bjuj0: LinkedIn people profiles',
Expand Down
2 changes: 1 addition & 1 deletion tool_groups.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
'use strict'; /*jslint node:true es9:true*/

const base_tools = ['search_engine', 'scrape_as_markdown', 'discover'];
const base_tools = ['search_engine', 'scrape_as_markdown'];

export const GROUPS = {
ECOMMERCE: {
Expand Down