Build Your Own Search Result Scraper with Markdown Output Using FastAPI, SearXNG, and Browserless
>Learn how to build your own search result scraper with FastAPI, SearXNG, and Browserless, and return results in Markdown format using a proxy.
🎧 Listen — ~7 min
Ready · Build Your Own Search Result Scr
Today, I'm excited to share with you a detailed guide on how to build your own search result scraper that returns results in Markdown format. We'll be using FastAPI, SearXNG, and Browserless, and we'll run everything in Docker containers.
This tutorial is perfect for early-stage students or anyone interested in web scraping and data extraction. By the end of this guide, you'll have a working application that can fetch search results, scrape content, and convert it into Markdown format, all while using a proxy.
Table of Contents
- Services We'll Use
- Purpose of Scraping
- Prerequisites
- Docker Setup
- Manual Setup
- Writing the Code
- Running SearXNG and Browserless in Docker
- Using Proxies
- Full Source Code
- Enjoyed the Post?
Services We'll Use
- FastAPI: A modern, fast (high-performance), web framework for building APIs with Python 3.6+.
- SearXNG: A free internet metasearch engine which aggregates results from various search services and databases.
- Browserless: A web browser automation service that allows you to scrape web pages without dealing with the browser directly.
Purpose of Scraping
Web scraping allows you to extract useful information from websites and use it for various purposes like data analysis, content aggregation, and more. In this tutorial, we'll focus on scraping search results and converting them into Markdown format for easy readability and integration with other tools.
Prerequisites
Before we begin, make sure you have the following installed:
- Python 3.11
- Virtualenv
You can install the prerequisites using the following commands:
1sudo apt-get update
2sudo apt-get install python3.11
3
4pip install virtualenvDocker Setup
You can use Docker to simplify the setup process. Follow these steps:
-
Clone the repository:
sh1git clone https://github.com/essamamdani/search-result-scraper-markdown.git 2cd search-result-scraper-markdown -
Run Docker Compose:
sh1docker compose up --build
With this setup, if you change the .env or main.py file, you no longer need to restart Docker. Changes will be reloaded automatically.
Manual Setup
Follow these steps for manual setup:
-
Clone the repository:
sh1git clone https://github.com/essamamdani/search-result-scraper-markdown.git 2cd search-result-scraper-markdown -
Create and activate virtual environment:
sh1virtualenv venv 2source venv/bin/activate -
Install dependencies:
sh1pip install -r requirements.txt -
Create a .env file in the root directory with the following content:
bash1SEARXNG_URL=http://localhost:8888 2BROWSERLESS_URL=http://localhost:3000 3TOKEN=b7a7ad74da294fa39ed75c01cfe4e41b 4PROXY_PROTOCOL=http 5PROXY_URL=us.premium-residential.geonode.com 6PROXY_USERNAME=geonode_OglxV49yXfxxxx 7PROXY_PASSWORD=xxxxx-ef6a-42a6-98fa-cc9e86cc0628 8PROXY_PORT=9000 9REQUEST_TIMEOUT=300
Writing the Code
Here's the complete code for our FastAPI application:
.env File
1SEARXNG_URL=http://localhost:8888
2BROWSERLESS_URL=http://localhost:3000
3TOKEN=b7a7ad74da294fa39ed75c01cfe4e41b
4PROXY_PROTOCOL=http
5PROXY_URL=us.premium-residential.geonode.com
6PROXY_USERNAME=geonode_OglxV49yXfxxxx
7PROXY_PASSWORD=xxxxx-ef6a-42a6-98fa-cc9e86cc0628
8PROXY_PORT=9000
9REQUEST_TIMEOUT=300Explanation of Each Variable
- SEARXNG_URL: This is the URL where your SearXNG service is running. In this setup, it's running locally on port 8888.
- BROWSERLESS_URL: This is the URL where your Browserless service is running. In this setup, it's running locally on port 3000.
- TOKEN: This is a placeholder for any API token that might be required by your services. In this specific example, it's not actively used but can be kept for future use or services that require authentication.
- PROXY_PROTOCOL: The protocol used by your proxy service. Typically, this will be either
httporhttps. - PROXY_URL: The URL or IP address of your proxy service provider. Here, we're using a Geonode proxy.
- PROXY_USERNAME: The username for authenticating with your proxy service. This is specific to your Geonode account.
- PROXY_PASSWORD: The password for authenticating with your proxy service. This is specific to your Geonode account.
- PROXY_PORT: The port number on which your proxy service is running. Common ports include
8080and9000. - REQUEST_TIMEOUT: The timeout duration for HTTP requests, specified in seconds. This helps ensure your application doesn't hang indefinitely while waiting for a response.
Why Use a .env File?
- Security: Keeps sensitive information like API keys, tokens, and passwords out of your codebase.
- Configuration: Allows easy configuration changes without modifying the code.
- Environment-Specific Settings: Easily switch configurations between different environments (development, testing, production) by changing the .env file.
How to Use the .env File
- Create the .env file: In the root of your project directory, create a file named
.env. - Add your variables: Copy the variables listed above into your .env file, replacing the example values with your actual values.
- Load the .env file in your code: Use
python-dotenvto load these variables into your application.
main.py
1import os
2from dotenv import load_dotenv
3import httpx
4from fastapi import FastAPI, Query
5
6from fastapi.responses import JSONResponse, PlainTextResponse
7from markdownify import markdownify as md
8from bs4 import BeautifulSoup, Comment
9import json
10
11load_dotenv()
12
13SEARXNG_URL = os.getenv('SEARXNG_URL')
14BROWSERLESS_URL = os.getenv('BROWSERLESS_URL')
15TOKEN = os.getenv('TOKEN')
16PROXY_PROTOCOL = os.getenv('PROXY_PROTOCOL', 'http')
17PROXY_URL = os.getenv('PROXY_URL')
18PROXY_USERNAME = os.getenv('PROXY_USERNAME')
19PROXY_PASSWORD = os.getenv('PROXY_PASSWORD')
20PROXY_PORT = os.getenv('PROXY_PORT')
21REQUEST_TIMEOUT = int(os.getenv('REQUEST_TIMEOUT', '30'))
22
23domains_only_for_browserless = ["twitter", "x", "facebook"] # Add more domains here
24
25app = FastAPI()
26
27HEADERS = {
28 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
29}
30
31def fetch_normal_content(url, proxies):
32 try:
33 response = httpx.get(url, headers=HEADERS, timeout=REQUEST_TIMEOUT, proxies=proxies)
34 response.raise_for_status()
35 return response.text
36 except httpx.RequestError as e:
37 print(f"An error occurred while requesting {url}: {e}")
38 except httpx.HTTPStatusError as e:
39 print(f"HTTP error occurred: {e}")
40 return None
41
42def fetch_browserless_content(url, proxies):
43 try:
44 browserless_url = f"{BROWSERLESS_URL}/content"
45
46 params = {}
47 if TOKEN:
48 params['token'] = TOKEN
49
50 proxy_url = f"{PROXY_PROTOCOL}://{PROXY_URL}:{PROXY_PORT}" if PROXY_URL and PROXY_PORT else None
51 if proxy_url:
52 params['--proxy-server'] = proxy_url
53
54 browserless_data = {
55 "url": url,
56 "rejectResourceTypes": ["image"],
57 "rejectRequestPattern": ["/^.*\\.(css)/"],
58 "gotoOptions": {"waitUntil": "networkidle2"},
59 "bestAttempt": True
60 }
61 if PROXY_USERNAME and PROXY_PASSWORD:
62 browserless_data["authenticate"] = {
63 "username": PROXY_USERNAME,
64 "password": PROXY_PASSWORD
65 }
66
67 headers = {
68 'Cache-Control': 'no-cache',
69 'Content-Type': 'application/json'
70 }
71
72 response = httpx.post(browserless_url, params=params, headers=headers, data=json.dumps(browserless_data), timeout=REQUEST_TIMEOUT*2)
73
74 response.raise_for_status()
75 return response.text
76 except httpx.RequestError as e:
77 print(f"An error occurred while requesting Browserless for {url}: {e}")
78 except httpx.HTTPStatusError as e:
79 print(f"HTTP error occurred with Browserless: {e}")
80 return None
81
82def fetch_content(url):
83 proxies = None
84 if PROXY_URL and PROXY_USERNAME and PROXY_PORT:
85 proxies = {
86 "http://": f"{PROXY_PROTOCOL}://{PROXY_USERNAME}:{PROXY_PASSWORD}@{PROXY_URL}:{PROXY_PORT}",
87 "https://": f"{PROXY_PROTOCOL}://{PROXY_USERNAME}:{PROXY_PASSWORD}@{PROXY_URL}:{PROXY_PORT}"
88 }
89 print(f"Using proxy {proxies}")
90 if any(domain in url for domain in domains_only_for_browserless):
91 content = fetch_browserless_content(url, proxies)
92 else:
93 content = fetch_normal_content(url, proxies)
94 if content is None:
95 content = fetch_browserless_content(url, proxies)
96
97 return content
98
99def clean_html(html):
100 soup = BeautifulSoup(html, 'html.parser')
101
102 for script_or_style in soup(["script", "style", "header", "footer", "noscript", "form", "input", "textarea", "select", "option", "button", "svg", "iframe", "object", "embed", "applet"]):
103 script_or_style.decompose()
104
105 for tag in soup.find_all(True):
106 tag.attrs = {key: value for key, value in tag.attrs.items() if key not in ['class', 'id', 'style']}
107
108 for comment in soup.find_all(string=lambda text: isinstance(text, Comment)):
109 comment.extract()
110
111 return str(soup)
112
113def parse_html_to_markdown(html, url):
114 cleaned_html = clean_html(html)
115 markdown_content = md(cleaned_html)
116 return {
117 "title": BeautifulSoup(html, 'html.parser').title.string if BeautifulSoup(html, 'html.parser').title else 'No title',
118 "url": url,
119 "md": clean_markdown(markdown_content)
120 }
121
122def clean_markdown(markdown):
123 lines = markdown.split('\n')
124 cleaned_lines = [line.strip() for line in lines if line.strip()]
125 return '\n'.join(cleaned_lines)
126
127def search(query: str, num_results: int) -> list:
128 searxng_query_url = f"{SEARXNG_URL}/search?q={query}&categories=general&format=json"
129 try:
130 response = httpx.get(searxng_query_url, headers=HEADERS, timeout=REQUEST_TIMEOUT)
131 response.raise_for_status()
132 except httpx.RequestError as e:
133 return [{"error": f"Search query failed with error: {e}"}]
134 except httpx.HTTPStatusError as e:
135 return [{"error": f"Search query failed with HTTP error: {e}"}]
136
137 search_results = response.json()
138 results = []
139
140 for result in search_results["results"][:num_results]:
141 url = result["url"]
142 title = result["title"]
143 html_content = fetch_content(url)
144 if html_content:
145 markdown_data = parse_html_to_markdown(html_content, url)
146 results.append({
147 "title": title,
148 "url": url,
149 "markdown_content": (
150 f"Title: {markdown_data['title']}\n\n"
151 f"URL Source: {markdown_data['url']}\n\n"
152 f"Markdown Content:\n{markdown_data['md']}"
153 )
154 })
155
156 return results
157
158@app.get("/", response_class=JSONResponse)
159def get_search_results(q: str = Query(..., description="Search query"), num_results: int = Query(5, description="Number of results")):
160 result_list = search(q, num_results)
161 return result_list
162
163@app.get("/r/{url:path}", response_class=PlainTextResponse)
164def fetch_url(url: str):
165 html_content = fetch_content(url)
166 if html_content:
167 markdown_data = parse_html_to_markdown(html_content, url)
168 response_text = (
169 f"Title: {markdown_data['title']}\n\n"
170 f"URL Source: {markdown_data['url']}\n\n"
171 f"Markdown Content:\n{markdown_data['md']}"
172 )
173 return PlainTextResponse(response_text)
174 return PlainTextResponse("Failed to retrieve content")
175
176if __name__ == "__main__":
177 import uvicorn
178 uvicorn.run(app, host="0.0.0.0", port=8000)Running SearXNG and Browserless in Docker
Next, let's set up and run the SearXNG and Browserless services using Docker. Create a shell script run-services.sh with the following content:
1#!/bin/bash
2
3docker run -d --name searxng -p 8888:8888 searxng/searxng
4
5docker run -d --name browserless -p 3000:3000 browserless/chrome
6
7echo "SearXNG is running at http://localhost:8888"
8echo "Browserless is running at http://localhost:3000"Make the script executable and run it:
1chmod +x run-services.sh
2./run-services.shThis script will pull and run the SearXNG and Browserless Docker images, making them accessible on your local machine.
Using Proxies
In this tutorial, I'm using Geonode proxies to scrape content. You can use my Geonode affiliate link to get started with their proxy services.
Full Source Code
You can find the full source code for this project on my GitHub repository.
Enjoyed the Post?
I hope you found this tutorial helpful! Feel free to reach out if you have any questions or need further assistance. Happy scraping!
Enjoyed the post? Follow my blog at essamamdani.com for more tutorials and insights.
⚡ AI Engineering Briefing
AI model releases, practical tutorials, developer tools, and engineering research. Unsubscribe anytime.
Comments
Comments are temporarily unavailable while the public discussion space is being configured.