LIVE
Back to journal
7 min read

Build Your Own Search Result Scraper with Markdown Output Using FastAPI, SearXNG, and Browserless

>Learn how to build your own search result scraper with FastAPI, SearXNG, and Browserless, and return results in Markdown format using a proxy.

ShareXLinkedIn

🎧 Listen — ~7 min

Ready · Build Your Own Search Result Scr

0:00 / 7:00
Build Your Own Search Result Scraper with Markdown Output Using FastAPI, SearXNG, and Browserless
Verified by Essa Mamdani

Today, I'm excited to share with you a detailed guide on how to build your own search result scraper that returns results in Markdown format. We'll be using FastAPI, SearXNG, and Browserless, and we'll run everything in Docker containers.

This tutorial is perfect for early-stage students or anyone interested in web scraping and data extraction. By the end of this guide, you'll have a working application that can fetch search results, scrape content, and convert it into Markdown format, all while using a proxy.

Table of Contents

  1. Services We'll Use
  2. Purpose of Scraping
  3. Prerequisites
  4. Docker Setup
  5. Manual Setup
  6. Writing the Code
  7. Running SearXNG and Browserless in Docker
  8. Using Proxies
  9. Full Source Code
  10. Enjoyed the Post?

Services We'll Use

  • FastAPI: A modern, fast (high-performance), web framework for building APIs with Python 3.6+.
  • SearXNG: A free internet metasearch engine which aggregates results from various search services and databases.
  • Browserless: A web browser automation service that allows you to scrape web pages without dealing with the browser directly.

Purpose of Scraping

Web scraping allows you to extract useful information from websites and use it for various purposes like data analysis, content aggregation, and more. In this tutorial, we'll focus on scraping search results and converting them into Markdown format for easy readability and integration with other tools.

Prerequisites

Before we begin, make sure you have the following installed:

  • Python 3.11
  • Virtualenv

You can install the prerequisites using the following commands:

sh
1sudo apt-get update
2sudo apt-get install python3.11
3
4pip install virtualenv

Docker Setup

You can use Docker to simplify the setup process. Follow these steps:

  1. Clone the repository:

    sh
    1git clone https://github.com/essamamdani/search-result-scraper-markdown.git
    2cd search-result-scraper-markdown
  2. Run Docker Compose:

    sh
    1docker compose up --build

With this setup, if you change the .env or main.py file, you no longer need to restart Docker. Changes will be reloaded automatically.

Manual Setup

Follow these steps for manual setup:

  1. Clone the repository:

    sh
    1git clone https://github.com/essamamdani/search-result-scraper-markdown.git
    2cd search-result-scraper-markdown
  2. Create and activate virtual environment:

    sh
    1virtualenv venv
    2source venv/bin/activate
  3. Install dependencies:

    sh
    1pip install -r requirements.txt
  4. Create a .env file in the root directory with the following content:

    bash
    1SEARXNG_URL=http://localhost:8888
    2BROWSERLESS_URL=http://localhost:3000
    3TOKEN=b7a7ad74da294fa39ed75c01cfe4e41b
    4PROXY_PROTOCOL=http
    5PROXY_URL=us.premium-residential.geonode.com
    6PROXY_USERNAME=geonode_OglxV49yXfxxxx
    7PROXY_PASSWORD=xxxxx-ef6a-42a6-98fa-cc9e86cc0628
    8PROXY_PORT=9000
    9REQUEST_TIMEOUT=300

Writing the Code

Here's the complete code for our FastAPI application:

.env File

env
1SEARXNG_URL=http://localhost:8888
2BROWSERLESS_URL=http://localhost:3000
3TOKEN=b7a7ad74da294fa39ed75c01cfe4e41b
4PROXY_PROTOCOL=http
5PROXY_URL=us.premium-residential.geonode.com
6PROXY_USERNAME=geonode_OglxV49yXfxxxx
7PROXY_PASSWORD=xxxxx-ef6a-42a6-98fa-cc9e86cc0628
8PROXY_PORT=9000
9REQUEST_TIMEOUT=300

Explanation of Each Variable

  • SEARXNG_URL: This is the URL where your SearXNG service is running. In this setup, it's running locally on port 8888.
  • BROWSERLESS_URL: This is the URL where your Browserless service is running. In this setup, it's running locally on port 3000.
  • TOKEN: This is a placeholder for any API token that might be required by your services. In this specific example, it's not actively used but can be kept for future use or services that require authentication.
  • PROXY_PROTOCOL: The protocol used by your proxy service. Typically, this will be either http or https.
  • PROXY_URL: The URL or IP address of your proxy service provider. Here, we're using a Geonode proxy.
  • PROXY_USERNAME: The username for authenticating with your proxy service. This is specific to your Geonode account.
  • PROXY_PASSWORD: The password for authenticating with your proxy service. This is specific to your Geonode account.
  • PROXY_PORT: The port number on which your proxy service is running. Common ports include 8080 and 9000.
  • REQUEST_TIMEOUT: The timeout duration for HTTP requests, specified in seconds. This helps ensure your application doesn't hang indefinitely while waiting for a response.

Why Use a .env File?

  1. Security: Keeps sensitive information like API keys, tokens, and passwords out of your codebase.
  2. Configuration: Allows easy configuration changes without modifying the code.
  3. Environment-Specific Settings: Easily switch configurations between different environments (development, testing, production) by changing the .env file.

How to Use the .env File

  1. Create the .env file: In the root of your project directory, create a file named .env.
  2. Add your variables: Copy the variables listed above into your .env file, replacing the example values with your actual values.
  3. Load the .env file in your code: Use python-dotenv to load these variables into your application.

main.py

python
1import os
2from dotenv import load_dotenv
3import httpx
4from fastapi import FastAPI, Query
5
6from fastapi.responses import JSONResponse, PlainTextResponse
7from markdownify import markdownify as md
8from bs4 import BeautifulSoup, Comment
9import json
10
11load_dotenv()
12
13SEARXNG_URL = os.getenv('SEARXNG_URL')
14BROWSERLESS_URL = os.getenv('BROWSERLESS_URL')
15TOKEN = os.getenv('TOKEN')
16PROXY_PROTOCOL = os.getenv('PROXY_PROTOCOL', 'http')
17PROXY_URL = os.getenv('PROXY_URL')
18PROXY_USERNAME = os.getenv('PROXY_USERNAME')
19PROXY_PASSWORD = os.getenv('PROXY_PASSWORD')
20PROXY_PORT = os.getenv('PROXY_PORT')
21REQUEST_TIMEOUT = int(os.getenv('REQUEST_TIMEOUT', '30'))
22
23domains_only_for_browserless = ["twitter", "x", "facebook"] # Add more domains here
24
25app = FastAPI()
26
27HEADERS = {
28    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
29}
30
31def fetch_normal_content(url, proxies):
32    try:
33        response = httpx.get(url, headers=HEADERS, timeout=REQUEST_TIMEOUT, proxies=proxies)
34        response.raise_for_status()
35        return response.text
36    except httpx.RequestError as e:
37        print(f"An error occurred while requesting {url}: {e}")
38    except httpx.HTTPStatusError as e:
39        print(f"HTTP error occurred: {e}")
40    return None
41
42def fetch_browserless_content(url, proxies):
43    try:
44        browserless_url = f"{BROWSERLESS_URL}/content"
45        
46        params = {}
47        if TOKEN:
48            params['token'] = TOKEN
49            
50        proxy_url = f"{PROXY_PROTOCOL}://{PROXY_URL}:{PROXY_PORT}" if PROXY_URL and PROXY_PORT else None
51        if proxy_url:
52            params['--proxy-server'] = proxy_url
53            
54        browserless_data = {
55            "url": url,
56            "rejectResourceTypes": ["image"],
57            "rejectRequestPattern": ["/^.*\\.(css)/"],
58            "gotoOptions": {"waitUntil": "networkidle2"},
59            "bestAttempt": True
60        }
61        if PROXY_USERNAME and PROXY_PASSWORD:
62            browserless_data["authenticate"] = {
63                "username": PROXY_USERNAME,
64                "password": PROXY_PASSWORD
65            }
66            
67        headers = {
68            'Cache-Control': 'no-cache',
69            'Content-Type': 'application/json'
70        }
71        
72        response = httpx.post(browserless_url, params=params, headers=headers, data=json.dumps(browserless_data), timeout=REQUEST_TIMEOUT*2)
73
74        response.raise_for_status()
75        return response.text
76    except httpx.RequestError as e:
77        print(f"An error occurred while requesting Browserless for {url}: {e}")
78    except httpx.HTTPStatusError as e:
79        print(f"HTTP error occurred with Browserless: {e}")
80    return None
81
82def fetch_content(url):
83    proxies = None
84    if PROXY_URL and PROXY_USERNAME and PROXY_PORT:
85        proxies = {
86            "http://": f"{PROXY_PROTOCOL}://{PROXY_USERNAME}:{PROXY_PASSWORD}@{PROXY_URL}:{PROXY_PORT}",
87            "https://": f"{PROXY_PROTOCOL}://{PROXY_USERNAME}:{PROXY_PASSWORD}@{PROXY_URL}:{PROXY_PORT}"
88        }
89        print(f"Using proxy {proxies}")
90    if any(domain in url for domain in domains_only_for_browserless):
91        content = fetch_browserless_content(url, proxies)
92    else:
93        content = fetch_normal_content(url, proxies)
94        if content is None:
95            content = fetch_browserless_content(url, proxies)
96
97    return content
98
99def clean_html(html):
100    soup = BeautifulSoup(html, 'html.parser')
101    
102    for script_or_style in soup(["script", "style", "header", "footer", "noscript", "form", "input", "textarea", "select", "option", "button", "svg", "iframe", "object", "embed", "applet"]):
103        script_or_style.decompose()
104    
105    for tag in soup.find_all(True):
106        tag.attrs = {key: value for key, value in tag.attrs.items() if key not in ['class', 'id', 'style']}
107    
108    for comment in soup.find_all(string=lambda text: isinstance(text, Comment)):
109        comment.extract()
110    
111    return str(soup)
112
113def parse_html_to_markdown(html, url):
114    cleaned_html = clean_html(html)
115    markdown_content = md(cleaned_html)
116    return {
117        "title": BeautifulSoup(html, 'html.parser').title.string if BeautifulSoup(html, 'html.parser').title else 'No title',
118        "url": url,
119        "md": clean_markdown(markdown_content)
120    }
121
122def clean_markdown(markdown):
123    lines = markdown.split('\n')
124    cleaned_lines = [line.strip() for line in lines if line.strip()]
125    return '\n'.join(cleaned_lines)
126
127def search(query: str, num_results: int) -> list:
128    searxng_query_url = f"{SEARXNG_URL}/search?q={query}&categories=general&format=json"
129    try:
130        response = httpx.get(searxng_query_url, headers=HEADERS, timeout=REQUEST_TIMEOUT)
131        response.raise_for_status()
132    except httpx.RequestError as e:
133        return [{"error": f"Search query failed with error: {e}"}]
134    except httpx.HTTPStatusError as e:
135        return [{"error": f"Search query failed with HTTP error: {e}"}]
136
137    search_results = response.json()
138    results = []
139    
140    for result in search_results["results"][:num_results]:
141        url = result["url"]
142        title = result["title"]
143        html_content = fetch_content(url)
144        if html_content:
145            markdown_data = parse_html_to_markdown(html_content, url)
146            results.append({
147                "title": title,
148                "url": url,
149                "markdown_content": (
150                f"Title: {markdown_data['title']}\n\n"
151                f"URL Source: {markdown_data['url']}\n\n"
152                f"Markdown Content:\n{markdown_data['md']}"
153            )
154            })
155    
156    return results
157
158@app.get("/", response_class=JSONResponse)
159def get_search_results(q: str = Query(..., description="Search query"), num_results: int = Query(5, description="Number of results")):
160    result_list = search(q, num_results)
161    return result_list
162
163@app.get("/r/{url:path}", response_class=PlainTextResponse)
164def fetch_url(url: str):
165    html_content = fetch_content(url)
166    if html_content:
167        markdown_data = parse_html_to_markdown(html_content, url)
168        response_text = (
169            f"Title: {markdown_data['title']}\n\n"
170            f"URL Source: {markdown_data['url']}\n\n"
171            f"Markdown Content:\n{markdown_data['md']}"
172        )
173        return PlainTextResponse(response_text)
174    return PlainTextResponse("Failed to retrieve content")
175
176if __name__ == "__main__":
177    import uvicorn
178    uvicorn.run(app, host="0.0.0.0", port=8000)

Running SearXNG and Browserless in Docker

Next, let's set up and run the SearXNG and Browserless services using Docker. Create a shell script run-services.sh with the following content:

sh
1#!/bin/bash
2
3docker run -d --name searxng -p 8888:8888 searxng/searxng
4
5docker run -d --name browserless -p 3000:3000 browserless/chrome
6
7echo "SearXNG is running at http://localhost:8888"
8echo "Browserless is running at http://localhost:3000"

Make the script executable and run it:

sh
1chmod +x run-services.sh
2./run-services.sh

This script will pull and run the SearXNG and Browserless Docker images, making them accessible on your local machine.

Using Proxies

In this tutorial, I'm using Geonode proxies to scrape content. You can use my Geonode affiliate link to get started with their proxy services.

Full Source Code

You can find the full source code for this project on my GitHub repository.

Enjoyed the Post?

I hope you found this tutorial helpful! Feel free to reach out if you have any questions or need further assistance. Happy scraping!

Enjoyed the post? Follow my blog at essamamdani.com for more tutorials and insights.

ShareXLinkedIn

⚡ AI Engineering Briefing

AI model releases, practical tutorials, developer tools, and engineering research. Unsubscribe anytime.

Comments

Comments are temporarily unavailable while the public discussion space is being configured.