FlareSolverr is a tool that helps users evade Cloudflare challenges. Usually, FlareSolverr is used in web scraping projects as bots visit hundreds or thousands of websites, so these challenges are quite frequent.
It’s built upon Selenium and UndetectedChromeDriver and acts like a proxy server. You send POST requests to the FlareSolverr server with a payload that indicates various features, such as the type of HTTP request, the URL, and timeout time.
While you could build a custom solution that would do the same thing, FlareSolverr works right out of the box with minimal tinkering required, so it’s a great addition to any web scraping project.
How FlareSolverr Works
FlareSolverr, when installed on a machine, creates a server that is constantly running in the background. While no requests are being sent, the FlareSolverr proxy is completely idle.
When a user sends a POST request, however, the Flaresolverr server picks up the request and initiates a Selenium session that runs Undetected ChromeDriver. FlareSolverr will then visit the website described in the payload.
If it’s presented with a Cloudflare challenge, FlareSolverr will attempt to solve it (or wait until timeout). Once the Cloudflare challenge is solved, the HTML and the cookies are sent back to the user.
There’s one Cloudflare challenge FlareSolverr cannot solve – CAPTCHAs. There are no good software-based solvers right now, so Flaresolverr will completely fail upon these Cloudflare challenges.
Advantages and Disadvantages of Using FlareSolverr
FlareSolverr is great in that it works right out of the box. If you’re spinning up a quick web scraping project that intends to work with Cloudflare websites (of which there are many), it’s the perfect addition.
While the benefit is major, there are some things to consider. First, there’ll be major performance overheads if you intend to send a lot of requests to the proxy server. It’ll start a headless browser session for each one, which can quickly run up performance costs.
As such, FlareSolverr is not the best if you scrape at an enormous scale. Additionally, while the out-of-the-box solution is a boon if you’re running a small project, it’ll become a hindrance if you need to customize it.
So, FlareSolverr is perfect if you need something quick and effective for decently-sized projects. But as your web scraping goals grow, you’ll need to create your own solution.
Finally, FlareSolverr used to be able to solve every Cloudflare challenge. Unfortunately, CAPTCHAs have become much more difficult, making them impossible (currently) to solve natively within FlareSolverr. So, you’ll have to look for another solution to solve that particular Cloudflare challenge.
Installing FlareSolverr
You can install FlareSolverr on Windows, Linux, or through Docker. FlareSolverr’s developers recommend using Docker, however, other versions work just as fine.
Installing FlareSolverr on Windows
One of the easiest ways to get FlareSolverr on your Windows machine is to use the precompiled binaries . You can use the source code and compile it yourself. However, it’s intended to be run on a Linux machine, so it may not be as effective.
Once you download the ZIP file, extract it anywhere on your machine. Then, open the folder and execute “Flaresolverr.exe”.
If required, allow the proxy server through your Windows firewall. If everything is successful, you’ll get a Terminal window stating that the test was successful and that FlareSolverr is being served on a particular IP address.
Installing FlareSolverr on Linux
While the developers recommend using Docker for all installations, you can install Flaresolverr on Linux without it.
Start by launching the terminal and creating a folder for the FlareSolverr installation.
mkdir FlareSolverrcd FlareSolverrThen, use the “wget” command to download the tar.gz file from the Github repository .
Once that’s completed, use the tar command to extract the files:
tar -xzf flaresolverr_linux_x64.tar.gzMove to the newly created folder:
cd flaresolverrYou can now execute FlareSolverr through a command:
./flaresolverrLike with Windows, the proxy server should initiate and provide you with an IP address that will serve as our endpoint.
Installing FlareSolverr on Docker
Docker is the recommended way to install FlareSolverr. You’ll need Docker first, which you can install on Windows, Mac, and Linux .
Once installed, you’ll have to pull the FlareSolverr image into your Docker container. Run the following command in the terminal window:
docker pull flaresolverr/flaresolverrThen execute:
docker run flaresolverr/flaresolverrYour terminal window should populate with FlareSolverr details and the proxy server IP address.
Alternatively, if you’re using Windows, you can use the search bar to find the FlareSolverr container and run it.
You can also use Docker Compose to clone the repository and run the executable.
Configuring FlareSolverr
There are many settings you can tinker with for FlareSolverr that may help you solve Cloudflare protection challenges. You can do so by either setting the command at execution or by changing environment variables.
You can change your FlareSolverr timeout time by starting it with a specific argument:
docker run -d --name flaresolverr -p 8191:8191 -e BROWSER_TIMEOUT=60000 flaresolverr/flaresolverrYou can perform the same process to change the time zone of the Docker instance:
docker run -d --name flaresolverr -p 8191:8191 -e TZ=America/New_York flaresolverr/flaresolverrIf you want to make more permanent changes, it’s best to set environment variables in your Docker container. For that you’ll need to create a “docker-compose.yml” and set these settings:
version: '3'services: flaresolverr: image: flaresolverr/flaresolverr container_name: flaresolverr ports: - "8191:8191" environment: - TZ=America/New_York # Set the time zone (optional) restart: unless-stoppedYou’ll then have to use the terminal, go to the location of the file and run this command:
docker-compose up -dFlareSolverr Usage
Once you configure FlareSolverr to your liking, you’ll need to start sending requests to the FlareSolverr instance. While you can use cURL to do so, Python is our preferred way to send HTTP requests.
You’ll usually need two libraries: Requests and json. Json arrives inbuilt with Python, but you’ll need to install Requests separately.
Open up Terminal and run:
pip install requestsOnce that’s set up, we can start sending requests to the FlareSolverr endpoint. Make sure FlareSolverr is running before you attempt to execute the code below:
import requestsimport json# Flaresolverr endpointurl = 'http://localhost:8191/v1'# Request payloaddata = { "cmd": "request.get", "url": "https://iproyal.com", "maxTimeout": 60000 # 60 seconds}# Headersheaders = { 'Content-Type': 'application/json'}# Send POST request to Flaresolverrresponse = requests.post(url, data=json.dumps(data), headers=headers)# Print the response contentprint(response.text)We start by importing the aforementioned libraries and create an object that stores our FlareSolverr endpoint IP address. If the one listed in the code doesn’t work, try copying and pasting the IP address provided in the Terminal when you started the FlareSolverr instance.
FlareSolverr needs a payload so it understands what to do. As such, we create a dictionary with several key and value pairs:
- We start by setting the command to “request.get”, so the FlareSolverr instance knows to send a GET HTTP request.
- We also provide a URL where we want to send the GET request.
- Finally, we set a custom timeout period.
Additionally, we’ve included some headers and set them to the json file format. It ensures that FlareSolverr sends and retrieves the correct data format, which reduces error rates.
Finally, we send a POST request to our FlareSolverr instance and print out the response.
You may also want to change some other settings in FlareSolverr. User agents , for example, are important when evading various Cloudflare protection challenges.
import requestsimport json# Flaresolverr endpointurl = 'http://localhost:8191/v1'# Request payload, including a custom User-Agentdata = { "cmd": "request.get", "url": "https://example.com", "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36", "maxTimeout": 60000 # 60 seconds}# Headers for the POST requestheaders = { 'Content-Type': 'application/json'}# Send POST request to Flaresolverrresponse = requests.post(url, data=json.dumps(data), headers=headers)# Print the response contentprint(response.text)Our code is almost identical. However, the data dictionary now includes a custom FlareSolverr user agent. Setting these can help reduce the likelihood of running into various Cloudflare challenges. You should be careful, however, as improperly set user agents can increase the likelihood of Cloudflare challenges or even block your access to a website entirely.
Conclusion
You should integrate FlareSolverr in any web scraping project where you don’t need a lot of customization and won’t be running millions of requests. Bypassing Cloudflare protected websites becomes a lot easier with FlareSolverr as it works perfectly right out of the box.
Building a custom solution similar to FlareSolverr will become important once you want to squeeze every bit of performance out of your web scraping project, however. FlareSolverr is a great starting point but will rarely be enough for business-grade web scraping.
Finally, if the only Cloudflare challenge you’re having issues with is CAPTCHAs, FlareSolverr won’t be a lot of help. As it currently stands for a couple of years, FlareSolverr cannot solve CAPTCHAs on its own.
