Bulk Check HTTP Status Codes with Python (Script Built with AI)

You can check the HTTP status codes of thousands of URLs with a short Python script that reads a list from a text file and saves the results to CSV. For each URL it records the code of the first response (e.g. 301), the final code and URL after redirects, and the number of hops. Below you’ll find tested code, instructions for installing Python on Windows, macOS and Linux, and the story of how I wrote the first version with the help of ChatGPT.

I’ve always wanted to learn to code well enough to build a website or an app as an MVP – a Minimum Viable Product (the smallest working version of a product) – in an evening or two.

An MVP, or Minimum Viable Product, is a basic version of a product that contains only the most important features. The point of an MVP is to test an idea on the market, gather user feedback and validate assumptions before investing in the full version of the product. That way you can make changes quickly and adapt the product to what customers need.

For the most part it was also about building tools to help me with creating content, analysing websites and plenty of other tasks. Whatever the goal, coding has always been a skill I wanted to master.

Not that I never tried. At one point I tinkered quite a lot with PHP. It came naturally, because it was an easy way to bring the HTML and CSS I’d prepared to life. None of it was particularly complex, though. For some reason, object-oriented programming with classes was beyond me. Years went by, and instead of PHP I was getting to grips with yet more CMSs, plugin settings and volumes of SEO documentation. I also had a brief fling with Ruby. Somewhere on my shelf I still have books on the language and the Ruby on Rails framework!

The urge to learn to code never faded, but I was spending all my time on SEO. Going through more and more data, documentation and tools, I noticed that Python was increasingly being used for working with data, content and analysis. So I started trying a few things in it, but I knew from the start that with my lack of time I wouldn’t get very far.

Programming and writing code with AI

And this story probably wouldn’t have had a happy ending if it weren’t for language models and tools such as ChatGPT or Ollama. Suddenly it turned out that if you have an idea and break it down into small steps, you can get working code that does what the MVP needs. Today you can do the same with any AI assistant, e.g. ChatGPT, Claude or Gemini. You still have to read and test the generated code, though. As you’ll see below, the first version of my script had bugs that aren’t obvious at first glance.

And simple tools? For example: you need to check the status of 20,000 URLs. You want to know whether they work (200), redirect (301) or maybe return a 4xx or 5xx error. With AI’s help you can put together a simple Python script to run on your own computer in an evening, often faster.

Need to build a website crawler that isn’t limited to Google Sheets? Or one that can analyse a site built with dynamic frameworks like Angular? An AI assistant will give you a starting point that you then refine and test.

How to prompt AI for code

I’ll show you how I do it. You can always add more features or ask the AI to extend or change your app. Also remember it’s better to ask for small things many times than to expect the AI to hand you a finished, multi-file application in one go.

In my experience it works well to state in the prompt what you want to achieve, what data you’ll provide and how, and what data you expect back and in what form. To illustrate this, I’ll show you step by step how I put together a simple Python script that helped me analyse more than 20,000 URLs in an online shop.

The brief was simple. I had a list of over 20,000 URLs taken from an online shop. I wanted to check which URL returned a 200 (OK), which was unavailable and returned a 404, and which redirected elsewhere (301). The shop ran on IdoSell (formerly IAI-Shop), a Polish SaaS e-commerce platform, so after a certain number of requests a 429 code – Too Many Requests – could appear, and further requests would be blocked by the server. My IP had been added to an allowlist, so in theory I could send any number of requests without being ‘throttled’.

Here’s how my collaboration with ChatGPT went

First I had to define my needs, requirements and expectations. I decided myself that I wanted to use Python because, as I mentioned, I want to learn it – and it’s simply well suited to this task.

Here’s my prompt from 2024 (translated from Polish):

Prepare a Python script that carries out the following steps in order:

1. Take a URL from the urls.txt file
2. Choose the first proxy that hasn't been used recently
3. Connect to the page and check the response header (2xx, 3xx, 4xx, 5xx)
4. Once you get a valid code (other than 429), add it to an SQL database. Save the data using the schema ID; Connection date; URL; response code
5. If a 3xx response code occurs, add an extra field in SQL after "response code" called "target URL" and put the URL the redirect points to there
6. If a URL returns a 429 error or another timeout-related error, flag those URLs accordingly. If the error repeats for 10 URLs in a row, wait 60 seconds before carrying on.
7. After finishing the connection for one URL, take the next URL, choose the next available PROXY and continue from step 3.

In the console, show the percentage progress based on all URLs, the URL currently being processed, the proxy in use and the time elapsed since the code was started.

Why did I want the script to work like this?

First, keeping more than 20,000 URLs in the script file itself is bad for code readability. Second, with the URLs in a separate file you can feed the script any list and it works universally.

Second thing: I figured I could add proxy support to my script.

A proxy is an intermediary between a user’s device and the internet. When you use a proxy, your requests first go to an intermediary server, which then forwards them to the target server. The responses also pass through the proxy before they reach you. A proxy can be used to increase privacy, get around regional blocks or improve security.

Why a proxy in such a simple script? With that many URLs and the server’s restrictions, protecting myself against a single IP address being blocked seemed sensible. I also hoped that spreading requests across several IP addresses would speed the whole thing up, since rate limits are usually counted per IP.

At one request per second, 20,000 URLs is about 20,000 seconds – roughly 5.5 hours. A shorter pause or a few parallel requests cut that time many times over, but only if the server can take it and doesn’t start responding with 429.

I’ll let you in on a secret, though: when running this script, especially on a shop platform with limits, I ended up dropping the proxies.

Back to the prompt itself – the third thing was saving the data. I’d already been through scripts that generated HTML documents with a table and the relevant data. But those were for much smaller lists – 500 URLs at most.

With many thousands of URLs, I decided it would be best to keep the collected data in a database. Instead of setting up MySQL or PostgreSQL, I used SQLite – an SQL database stored in a single file.

Finally, I asked for a safeguard that would pause the script for 60 seconds after 10 failed requests (e.g. ones returning a 429). That was meant to give the server time to ‘cool down’.

As a result, ChatGPT gave me this code (the 2024 version – don’t use it, the fixed version is below):

import requests
import sqlite3
import time
from datetime import datetime

# Path to the file with URLs and proxies
url_file = 'urls.txt'
proxy_file = 'proxies.txt'

# Initialise the SQLite database
conn = sqlite3.connect('responses.db')
c = conn.cursor()

# Create the table if it doesn't exist
c.execute('''CREATE TABLE IF NOT EXISTS responses (
    id INTEGER PRIMARY KEY,
    connection_date TEXT,
    url TEXT,
    response_code INTEGER,
    target_url TEXT
)''')
conn.commit()

# Function to load URLs from a file
def load_urls(file_path):
    with open(file_path, 'r') as f:
        return [line.strip() for line in f.readlines()]

# Function to load proxies from a file
def load_proxies(file_path):
    with open(file_path, 'r') as f:
        return [line.strip() for line in f.readlines()]

# Function to choose an unused proxy
def select_proxy(proxy_list, last_used_proxy):
    for proxy in proxy_list:
        if proxy != last_used_proxy:
            return proxy
    return proxy_list[0]

# Function to check a URL using a proxy
def check_url(url, proxy):
    try:
        response = requests.get(url, proxies={'http': proxy, 'https': proxy}, timeout=10)
        return response.status_code, response.url if response.history else None
    except requests.exceptions.RequestException as e:
        return None, None

# Function to add a record to the database
def add_to_db(url, status_code, target_url=None):
    connection_date = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    c.execute('INSERT INTO responses (connection_date, url, response_code, target_url) VALUES (?, ?, ?, ?)',
              (connection_date, url, status_code, target_url))
    conn.commit()

# Function that runs the process
def process_urls(urls, proxies):
    last_used_proxy = None
    timeout_count = 0
    total_urls = len(urls)
    start_time = time.time()

    for i, url in enumerate(urls):
        # Choose an unused proxy
        proxy = select_proxy(proxies, last_used_proxy)
        last_used_proxy = proxy

        # Check the URL
        status_code, target_url = check_url(url, proxy)

        # Logic for errors and timeouts
        if status_code == 429 or status_code is None:
            timeout_count += 1
            if timeout_count >= 10:
                print(f'10 errors in a row. Waiting 60 seconds.')
                time.sleep(60)
                timeout_count = 0
        else:
            timeout_count = 0

            # Add the result to the database
            if status_code and status_code != 429:
                if 300 <= status_code < 400 and target_url:
                    add_to_db(url, status_code, target_url)
                else:
                    add_to_db(url, status_code)

        # Percentage and time calculations
        elapsed_time = time.time() - start_time
        progress = (i + 1) / total_urls * 100
        print(f'Progress: {progress:.2f}%, URL: {url}, Proxy: {proxy}, Time: {elapsed_time:.2f} s')

# Main call
urls = load_urls(url_file)
proxies = load_proxies(proxy_file)
process_urls(urls, proxies)

# Close the database connection
conn.close()

To be clear: this script was 100% generated by ChatGPT and it’s a first draft. When I read it carefully some time later, I found bugs in it that show nicely why you have to check code from AI:

• Redirects would never have been saved. The requests library follows redirects by default, so status_code is the code of the last response (e.g. 200), not the 301. The condition for 3xx codes could never be met. You can see it in the screenshot above: nothing but 200 codes and ‘Docelowy URL’ (target URL): None.
• Proxy rotation didn’t work. In practice the select_proxy function only switches between the first two addresses on the list.
• A PHP file isn’t a proxy as far as the requests library is concerned. The proxies= parameter expects an HTTP proxy server, not the URL of a script with a ?url= parameter.
• Errors vanished without a trace. URLs that timed out or returned 429 never made it into the database, so the results didn’t show what couldn’t be checked.

On top of that there was a separate proxy file. I asked the AI to write it in PHP so that I could upload it to ordinary servers running Apache, LiteSpeed or Nginx. Here’s the prompt I wrote (translated from Polish):

I need you to prepare PHP code for a proxy file that I'll put on several HTTP servers. The proxy file will be used by a Python script to connect to the URLs listed in the urls.txt file.
I have to use a proxy because the site I'm connecting to limits automated requests to 1 per IP address every 2 seconds or so. I'll use about 5 different servers for this.
I need the proxy server to fetch only the page's response code or other data requested by me/the Python script.

Not the most elaborate prompt, I know. But I needed to know roughly what the code for such a file might look like. Here’s what ChatGPT suggested:

<?php
// Function to send an HTTP request to the given URL
function fetch_url($target_url) {
    $ch = curl_init();

    // Set cURL options
    curl_setopt($ch, CURLOPT_URL, $target_url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // Receive the response as a string
    curl_setopt($ch, CURLOPT_HEADER, true);         // Fetch the response headers
    curl_setopt($ch, CURLOPT_NOBODY, true);         // Headers only, no response body
    curl_setopt($ch, CURLOPT_TIMEOUT, 10);          // Connection timeout

    // Execute the request
    $response = curl_exec($ch);

    // Check whether an error occurred
    if(curl_errno($ch)) {
        return ['error' => 'Error: ' . curl_error($ch)];
    }

    // Get the HTTP response code
    $http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);

    // Close the cURL connection
    curl_close($ch);

    // Return only the HTTP response code
    return ['http_code' => $http_code];
}

// Check whether the URL was passed via GET
if (isset($_GET['url'])) {
    $target_url = $_GET['url'];

    // Call the function that fetches the response code
    $result = fetch_url($target_url);

    // Return the result as JSON
    header('Content-Type: application/json');
    echo json_encode($result);
} else {
    // If no URL was given, send an error message
    echo json_encode(['error' => 'No URL. Provide the ?url= parameter']);
}
?>

Note that a PHP file like this, exposed publicly without any protection, lets anyone send requests to any address from your server. If you do use one, restrict access (e.g. with a secret key in a parameter or a list of allowed IPs) and delete it when you’re done. In practice, most sites don’t need a proxy at all. A sensible request rate is enough.

Without delays and with many threads, it’s very easy to overload your own server. From a single computer it isn’t a DDoS in the strict sense, but on small hosting the effect can be similar: the site slows down or stops responding to real users.

DDoS (Distributed Denial of Service) is a type of cyberattack in which many infected devices simultaneously send a huge number of requests to a single server or website. The goal is to overload the system, slowing it down or bringing it down completely, so the services become unavailable to real users.

Python script for checking HTTP status codes (2026 version)

Below you’ll find the new version, which I rewrote and tested in October 2026 on Python 3.14 and requests 2.34. The code uses only the standard library and requests, which officially supports Python 3.10 to 3.15, so it should also run without any problems on the brand-new Python 3.15. I dropped the proxies and the SQLite database. The results go into a CSV file you can open in Excel or Google Sheets. The script:

• reads URLs from the urls.txt file, skipping blank lines, comments and duplicates,
• sends requests with its own User-Agent header and a timeout, so it won’t hang on a dead server,
• records the code of the first response (e.g. 301), the final code and URL, the number of redirects and the whole chain,
• detects redirect loops (more than 10 hops),
• after a 429 it waits (according to the Retry-After header, if the server sends one) and retries the request up to 3 times,
• checks several URLs in parallel (ThreadPoolExecutor, 3 threads by default, 10 at most),
• saves every result as it goes. If you stop it with Ctrl+C, the data collected so far stays in the file.

Save the code below as status_check.py:

"""Bulk check HTTP status codes for a list of URLs.

Usage:
    python status_check.py                       # reads urls.txt, saves results.csv
    python status_check.py -i list.txt -o report.csv -w 2 -d 1
"""
import argparse
import csv
import threading
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime

import requests

# Change to your own name and URL - the server admin will know who's asking.
USER_AGENT = "Mozilla/5.0 (compatible; StatusCheck/1.0; +https://your-domain.com/)"
TIMEOUT = (5, 15)       # (connect timeout, read timeout) in seconds
MAX_REDIRECTS = 10      # more hops = a loop or a configuration error
MAX_RETRIES_429 = 3     # how many times to retry a request after a 429

_local = threading.local()


def get_session():
    """Each thread gets its own session (requests.Session isn't fully thread-safe)."""
    if not hasattr(_local, "session"):
        s = requests.Session()
        s.headers.update({"User-Agent": USER_AGENT})
        s.max_redirects = MAX_REDIRECTS
        _local.session = s
    return _local.session


def load_urls(path):
    """Reads URLs (one per line), skips blank lines, comments (#) and duplicates."""
    seen, urls = set(), []
    with open(path, encoding="utf-8-sig") as f:
        for line in f:
            url = line.strip()
            if url and not url.startswith("#") and url not in seen:
                seen.add(url)
                urls.append(url)
    return urls


def check_url(url, delay):
    """Returns the first response code, the final code and URL, and the number of redirects."""
    result = {"url": url}
    session = get_session()
    for attempt in range(MAX_RETRIES_429 + 1):
        result.update({
            "first_status": "", "final_status": "", "final_url": "", "redirects": 0,
            "chain": "", "error": "", "time_ms": "",
            "checked_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
        })
        if delay:
            time.sleep(delay)
        start = time.perf_counter()
        try:
            # GET instead of HEAD - some servers handle HEAD badly.
            # stream=True: we fetch the headers without downloading the whole page.
            with session.get(url, timeout=TIMEOUT, allow_redirects=True, stream=True) as r:
                hops = r.history  # 3xx responses on the way to the final URL
                result.update({
                    "first_status": hops[0].status_code if hops else r.status_code,
                    "final_status": r.status_code,
                    "final_url": r.url,
                    "redirects": len(hops),
                    "chain": " > ".join(f"{h.url} ({h.status_code})" for h in hops + [r]) if hops else "",
                    "time_ms": round((time.perf_counter() - start) * 1000),
                })
                if r.status_code == 429 and attempt < MAX_RETRIES_429:
                    # 429 = too many requests. Wait as long as the server says (Retry-After) or progressively longer.
                    retry_after = r.headers.get("Retry-After", "")
                    wait = int(retry_after) if retry_after.isdigit() else 10 * (attempt + 1)
                    time.sleep(min(wait, 120))
                    continue
        except requests.exceptions.TooManyRedirects:
            result["error"] = f"More than {MAX_REDIRECTS} redirects (loop?)"
        except requests.exceptions.Timeout:
            result["error"] = "Timeout"
        except requests.exceptions.RequestException as e:
            result["error"] = type(e).__name__
        return result
    return result


def main():
    p = argparse.ArgumentParser(description="Bulk check HTTP status codes")
    p.add_argument("-i", "--input", default="urls.txt", help="file with URLs (default: urls.txt)")
    p.add_argument("-o", "--output", default="results.csv", help="CSV file for the results")
    p.add_argument("-w", "--workers", type=int, default=3, help="number of parallel threads (1-10)")
    p.add_argument("-d", "--delay", type=float, default=0.5, help="pause before each request in a thread [s]")
    args = p.parse_args()

    workers = max(1, min(args.workers, 10))  # upper limit, so we don't overload the server
    urls = load_urls(args.input)
    total = len(urls)
    print(f"URLs to check: {total}, threads: {workers}, pause: {args.delay} s")

    fields = ["url", "first_status", "final_status", "final_url", "redirects",
              "chain", "error", "time_ms", "checked_at"]
    start = time.time()
    # utf-8-sig + semicolon: the file opens correctly in Excel with Polish regional settings and in Google Sheets
    with open(args.output, "w", newline="", encoding="utf-8-sig") as f:
        writer = csv.DictWriter(f, fieldnames=fields, delimiter=";")
        writer.writeheader()
        pool = ThreadPoolExecutor(max_workers=workers)
        futures = [pool.submit(check_url, u, args.delay) for u in urls]
        try:
            for done, fut in enumerate(as_completed(futures), 1):
                row = fut.result()
                writer.writerow(row)
                f.flush()  # every result goes straight to disk
                status = row["error"] or f'{row["first_status"]} -> {row["final_status"]}'
                print(f"[{done}/{total} {done / total:.0%}] {status} {row['url']}")
        except KeyboardInterrupt:
            print("\nStopped (Ctrl+C). The results saved so far stay in the CSV file.")
            pool.shutdown(wait=False, cancel_futures=True)
            return
        pool.shutdown()

    print(f"Done in {time.time() - start:.1f} s. Results: {args.output}")


if __name__ == "__main__":
    main()

Before the first run, change the USER_AGENT value in the code to your own name and URL. That way the server admin knows who is sending the requests. The script only fetches the response headers (stream=True) rather than whole pages, which saves bandwidth and time.

How many threads should you use? The defaults (3 threads, a 0.5 s pause in each thread) give at most about 6 requests per second – fewer in practice, because the server’s response time adds up too. On SaaS shop platforms and weaker hosting, start more cautiously, e.g. -w 2 -d 1, and only speed up once no 429 codes or timeouts appear in the results. A 429 means the server is asking you to slow down, and the Retry-After header tells you how many seconds to wait. Use this to check your own sites, or sites whose owner has agreed to the analysis.

How to run a Python script on your own computer

First you need Python. At the time of this update (13 October 2026), the latest stable version on python.org is Python 3.15.0, released on 9 October 2026. The 3.14 series (most recently 3.14.8 from 30 September) is still supported too, while Python 3.10 no longer gets fixes. Install the latest stable version python.org offers – the script in this post doesn’t need any new language features, just Python 3.10 or newer.

---

Installing Python on Windows

Since version 3.14, the recommended way to install Python on Windows is the Python install manager. The classic .exe installer is deprecated and, according to the documentation, won’t be released for Python 3.16 and later.

Step 1: Download Python

• Go to python.org/downloads and download the Python install manager, or install it from the Microsoft Store (search for ‘Python Install Manager’).
• Run the downloaded file and install the manager.

Step 2: Installation

• Open a terminal (PowerShell or Command Prompt: Win + R, type cmd and press Enter).
• Type py install 3.15 (or py install default to get the default version). py list shows the installed versions.
• On the first install, the manager may ask whether to add a directory to the PATH variable. It’s optional: the py and python commands work without it.

Step 3: Check the installation

• In the terminal, type: python --version
• If you see a Python version number, the installation was successful.

---

Installing Python on macOS

Step 1: Check which Python is installed

• Open Terminal (Cmd + Space, type ‘Terminal’, press Enter) and type: python3 --version
• Even if you see a version number, it may be an older Python supplied by Apple with the Xcode tools. Don’t remove it, because the system uses it, but install a current version for your own scripts.

Step 2: Install from python.org or via Homebrew

• Option 1: download the .pkg installer from python.org and go through the wizard. At the end, in the Applications → Python 3.15 folder (the number depends on the version you installed), double-click Install Certificates.command. Without it, HTTPS connections from the script may fail with a certificate error.
• Option 2: if you use Homebrew (a popular package manager for macOS), install it with the command from brew.sh:

  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• Once Homebrew is installed, type: brew install python

Step 3: Check the installation

• Open a new terminal window and type: python3 --version
• You should see the number of the version you’ve just installed.

---

Installing Python on Linux (Debian, Ubuntu)

Step 1: Check which Python version is installed

• Open a terminal (Ctrl + Alt + T) and type:

  python3 --version

• Most Linux distributions already come with Python 3 installed.

Step 2: Install Python (if it’s missing)

• Use the package manager. The python3-venv package will come in handy for creating a virtual environment (more on that in a moment):

  sudo apt update
  sudo apt install python3 python3-venv

Step 3: Check the installation

• In the terminal, type: python3 --version. You should see a Python 3 version. The version in your distribution’s repository is often older than the one on python.org, but it’s more than enough for this script.

---

Creating and running Python scripts

A Python script is a text file with the .py extension that contains Python code. You can create it in any text editor.

Step 1: Choose where to save the file

• You can put the .py file in any folder on your computer.
• To make life easier, create a separate folder for your scripts, e.g.:
• Windows: C:\MyScripts
• macOS: /Users/username/MyScripts, Linux: /home/username/MyScripts

Step 2: Create the file

• Open a text editor (I recommend VS Code – more on it below).
• In a new file, type some simple code, e.g.:

  print("Hello, world!")

• Save the file with the .py extension, e.g. as hello.py, in the folder you chose.

Step 3: Open a terminal and go to the folder

• Windows: press Win + R, type cmd and press Enter (or open PowerShell).
• macOS: Cmd + Space, type ‘Terminal’. Linux: Ctrl + Alt + T.

In the terminal, go to the folder where you saved the script:

  cd C:\MyScripts      # Windows
  cd ~/MyScripts       # macOS/Linux

Step 4: Run the script

  python hello.py     # Windows
  python3 hello.py    # macOS/Linux

If everything works, you’ll see the message Hello, world! in the terminal.

Choosing a text editor

You can write Python scripts in all sorts of editors, from simple ones to full-blown IDEs. In my view, the best choice is:

Visual Studio Code (VS Code):

A free, very popular editor with support for extensions, including Python support.
Installation: VS Code
To make writing Python easier, install the Python extension from the Marketplace.

How to use my script to check your URLs

These instructions are for the new version of the script in this post (status_check.py). All you need is two files in one folder: the script and the list of URLs.

Running the URL check script

1. Create a virtual environment and install the requests library

The script uses the requests library, which isn’t part of the standard Python installation. The safest way is to install it in a virtual environment (venv) – a separate folder with libraries for a given project. Homebrew on macOS and newer versions of Debian and Ubuntu block installing packages into the system Python and show an externally-managed-environment error instead. A venv gets around this. In the script’s folder, type:

# Windows
python -m venv .venv
.venv\Scripts\python -m pip install requests

# macOS / Linux
python3 -m venv .venv
.venv/bin/python -m pip install requests

2. Prepare the URL file (urls.txt)

Create a urls.txt file in the same folder as the script and put the URLs to check in it, one per line. There can be 100, 1,000 or 50,000 of them. The more URLs, the longer the analysis takes. Lines starting with # are skipped.

https://example.com/category/
https://example.com/category/subpage1
https://example.com/category/subpage2
https://example.com/blog/blog-post

3. Run the script

.venv\Scripts\python status_check.py

or on macOS/Linux:

.venv/bin/python status_check.py

You can change the settings with parameters, e.g. -i list.txt (input file), -o report.csv (output file), -w 2 (number of threads) and -d 1 (pause in seconds). --help shows the full list.

4. Wait for the script to finish. Don’t close the terminal or put the computer to sleep. The console shows the progress, e.g. [3/5 60%] 301 -> 200 http://semurai.pl/.

5. The script’s output

A results.csv file will appear in the script’s folder (UTF-8 encoding, semicolon as the separator). You can open it in Excel, import it into Google Sheets or load it into your own tool. If your Excel expects commas, change delimiter=";" in the code to delimiter=",". Here’s what the result of my test on a few URLs looked like:

url	first_status	final_status	final_url	redirects	error
https://semurai.pl/	200	200	https://semurai.pl/	0	–
https://semurai.pl/nie-istnieje-test/	404	404	https://semurai.pl/nie-istnieje-test/	0	–
http://semurai.pl/	301	200	https://semurai.pl/	1	–
http://www.semurai.pl/akademia/blad-404-410-4xx	301	200	https://semurai.pl/akademia/blad-404-410-4xx/	2	–
https://nie-istnieje-domena-xyz123.pl/	–	–	–	0	ConnectionError

The CSV file also has the columns chain (the whole redirect chain, e.g. http://www.semurai.pl/... (301) > https://semurai.pl/... (301) > https://semurai.pl/.../ (200)), time_ms (response time) and checked_at (when the URL was checked).

How to read the results

• first_status 3xx, final_status 200: the URL redirects to a working page. Check whether the redirect is permanent (301/308) or temporary (302/307). There’s more on this in my post on 301, 302, 307 and 308 redirects.
• redirects greater than 1: a redirect chain, as in my test (http + www → https without www → URL with a trailing slash). It’s worth pointing internal links straight at the final URL.
• final_status 404 or 410: the page doesn’t exist. Decide whether to redirect it to an equivalent or leave the error. My post on 404, 410 and other 4xx errors will help.
• 5xx, Timeout, ConnectionError: a problem on the server or network side (I explain what the individual codes mean in my post on 5xx server errors). Check these URLs again after a while before drawing conclusions.
• 429: the server has rate-limited you. Run the script again for those URLs with fewer threads and a longer pause.

A list like this is only the beginning. If after a shop migration or a change of URL structure the results show hundreds of redirects and 404 errors and you don’t know where to start, I can sort it out as part of an SEO audit.

The original version of the script (2024). Below I’m leaving the packages with the first version: with and without proxy support. They contain the bugs described above (among other things, they don’t save redirect codes), so treat them as archive material. For real work, use the code from the 2026 version section.

You can process the CSV data however you like: load it into Google Sheets, feed it into your own app or build an HTML view with a readable table. An AI assistant can help you write the code for those tasks too. Here’s what my request to export data from the SQLite database to CSV looked like in the first version of the script:

What else can you do with Python and ChatGPT?

Pretty much anything you want – that’s programming for you. For example, I wrote myself a simple tool to pull selected data from specific websites. That gave me a lot of valuable tabular data, which let me build a website with unique content.

You can also build a simple crawler and collect information about any site, track changes on a page, check its status on a schedule, connect to various APIs and much more. It all depends on your needs, your resources and how well you understand how the technology and programming work.

Recently, with practically no input from me, an .htaccess file with the required 301 redirects for an online shop pretty much generated itself! I only supplied the URLs from the sitemap and the new URLs from the production site, added the patterns to follow, and in the blink of an eye I had a ready, working .htaccess file. Instead of matching every URL by hand, I built myself a tool that did most of the work for me. All that was left for me was to verify the file and check it was correct – and that’s a step you shouldn’t skip.

Finally: don’t be afraid to experiment! Sometimes a few lines of code can save you a ton of time.

FAQ: checking HTTP status codes with Python

Do I need to know how to code to use this script?

No. Just install Python, copy the code into a file and run it in the terminal following the instructions in this post. If you want to change something, e.g. add a column with the X-Robots-Tag header, you can ask an AI assistant to do it. Test the modified script on a few URLs before running it on the whole list.

Why does the script show two response codes?

Because with a redirect the server first responds with a 3xx code, and only the target URL returns a 200 (or an error). The first_status column shows what the checked URL returns, and final_status shows where the user and the search engine bot finally end up.

Which Python version does the script run on?

Python 3.10 or newer – that’s what the current requests library (2.34.2) requires. I tested the script on Python 3.14. It doesn’t use any new language features, and requests already officially supports Python 3.15 (released on 9 October 2026), so it should work on the new version without any changes too.

How many URLs can I check at once?

There’s no limit on the length of the list. The limit is the pace the server can handle. With the default settings (3 threads, a 0.5 s pause) and server responses of around 0.3 s, checking 20,000 URLs takes roughly an hour and a half. If 429 codes or timeouts appear, reduce the number of threads and lengthen the pause.

Can I use a ready-made tool instead of a script?

Yes. SEO crawlers such as Screaming Frog can also check response codes for a list of URLs. A script comes in handy when you want full control over the pace and the output format, want to run checks on a schedule, or don’t have a licence for a paid tool.