Skip to main content

Hello ! This is my blog powered by Known. I post articles and links about coding, FOSS, but not only, in French (mostly) or English (if I did not find anything related before).

phyks.me

github.com/phyks

phyks@phyks.me

 

Filtering ads with your Raspberry Pi

13 min read

TL;DR: Please have a look at the benchmark section below, to be aware of the limitations of this particular setup and decide whether to spend some time putting it in place or not.

I recently came across this Pi-Hole project that claims to be "a black hole for Internet advertisements" (thanks nicofrand for making me discover this!). The idea was really attractive: having a simple Raspberry Pi on the network doing all the ad filtering for the whole network, rather than having to maintain a separate uBlock Origin install on each and every computer of the network. It was also particularly attractive as having such ad blocker on a smartphone requires a rooted device. Plus there was a really nice web interface to control the whole ad blocking device.

While looking at it more in depth, I realized it was actually very limited:

  1. First, it was built around specific software and was doing some magical stuff using these software. It was really painful to get away from them. Basically, it uses dnsmasq to expose a DNS service, a standard hosts file to block the hosts serving ads, and a lighthttpd webserver. Problem is I already have a DNS resolver (unbound) on this Raspberry Pi, and a web server (nginx). I did not want to spend a lot of time trying to integrate it in my existing setup if finally it was not that powerful, so I decided to look at it in detail before installing.
  2. Second issue was that it relies on dnsmasq. dnsmasq is a simple program that allow you to answer DNS queries by using the hosts defined in /etc/hosts and to forward every other requests to another DNS server (typically your ISP DNS server). Pi-Hole lets you configure two DNS servers to forward queries to, default one being 8.8.8.8 (Google :/). I already have a resolver on this Raspberry Pi and I do want to do the resolution myself, especially since my ISP DNS servers lies, and I do not want to use public DNS server on another network. So I had to hack on Pi-Hole to do some DNS resolution. About these issues, I'd like to point to two very interesting articles from Bortzmeyer: this one about Google DNS (in French) and this one about having your own DNS resolver (same). Also, being a DNS resolver, it may be cumbersome to disable it temporarily to load some website that absolutely requires the ads to be loaded.
  3. Last issue was that contrary to uBlock which filters at the requests level (and even sometimes at the HTML level), the fact that it is basically an alternative DNS resolver means you can only filter at the domain level. That is, you either whitelist (default) or blacklist a domain which is serving ads or malware, but you cannot differentiate different paths for a given domain. While browsing Libération website, I can see uBlock is blocking queries such as http://s3.amazonaws.com/files.wrapper.theadtech.com/native/placements/liberation.fr/pconfig?r=5a6f5d98b4d608. Such queries cannot be blocked by Pi-Hole without blacklisting the whole Amazon S3 network.

Given these facts, I remembered Privoxy which can be used as a filtering proxy, in a way similar to uBlock. Given that it is a proxy, it can filter in details, just as uBlock do and you can very easily disable it (simply disable the proxy). Plus, almost any devices offers you a proxy setting, so it should work both on my Android phones and computers. In this article I describe how I set up a Pi-Hole alternative based on Unbound (to have my own DNS resolver and block some things at the domain level) coupled with a Privoxy proxy to filter out ads.

Limitations: So, contrary to Pi-Hole, the setup described here will be able to remove ads in a similar way to uBlock/AdBlock. If you go through the whole article till the end, it will also have the element hiding features. Being a proxy setting, it will be easy to toggle it on and off (either by Prixovy toggling features or by manually turning off proxy on your device), if required. However, be aware of the remaining limitations with regards to HTTPS streams (section 4.15). As AdBlock/uBlock runs in the browser, it can filter ads in HTTPS streams as well, Privoxy will not be as efficient without HTTPS interception (which is generally not a good idea). However, it should perform rather well in the vast majority of situations (also note that AdBlock for rooted Android devices is also a proxy, so for them, it will not change anything).

I assume you already have a running Raspberry Pi with some basic install. Typically, see this previous article if this is not the case

Set up a DNS resolver

Let's install a DNS resolver on the Raspberry Pi, to answer DNS queries on the network. I am installing unbound and configuring Unbound here.

$ sudo apt-get install unbound
$ curl -o /etc/unbound/root.hints https://www.internic.net/domain/named.cache
$ # (Optional) Set crontask to download root.hints file every six months

The last curl command is used to fetch the root hints, to query hosts that are not cached. See this section of the ArchWiki page for more infos.

Then, you can create a basic configuration file for unbound.

$ cat /etc/unbound/unbound.conf.d/local.conf
server:
    username: "unbound"
    interface: 0.0.0.0  # Listen on all interfaces
    root-hints: "/etc/unbound/root.hints"
    access-control: 192.168.0.0/8 allow  # Access-control, see "Example" section in https://www.unbound.net/documentation/unbound.conf.html

Then, enable and start Unbound service at startup:

$ sudo systemctl start unbound && sudo systemctl enable unbound

You can now change the resolver to be used on your Raspberry Pi and on your whole network. To change it on your Raspberry Pi, have a look at this wiki page (for Raspbian). To set it as the default DNS resolver on your network, have a look at your router configuration, and set the DNS resolvers address to the address of your Raspberry Pi. Don't forget to open the ports in your firewall (53 tcp and udp).

To check everything is working fine, you can use dig (from dnsutils package on Debian-based distributions). Typically, dig google.fr should give you some results (in the ANSWER SECTION) and the IP address in the SERVER line should be the one of your Raspberry Pi.

Note: At this point, we should emphasize that having an open DNS resolver (that is, a DNS resolver that can answer to anyone) can be a security risk especially since some DDoS attacks use it. Then, you should make sure that your Raspberry Pi DNS server is only accessible from your local network, and that no third-party has access to it. This should be done through the access-control line in the above configuration, but this can also be enforced by the firewall running on your Raspberry Pi and the firewall on your router (typically, most routers provided by your ISP block any incoming connections, check this).

Block some domains based on hosts

Now, we would like unbound to block some domains that are known to serve ads and malware, in a similar way as Pi-Hole does. For this purpose, we will use unbound-block-hosts script to import hosts files into Unbound configuration. Basically, for every such domain, Unbound will return 127.0.0.1.

unbound-block-hosts is designed with the Dan Pollock's hosts file in mind, whereas I wanted to be able to import any host file in Unbound. Here is a forked and patched version for this purpose (very ugly patch, as I am not fluent in Perl :/).

We will create an includes dir in the Unbound configuration directory (mkdir /etc/unbound/includes/), and include the rules in the main configuration by appending include: "/etc/unbound/includes/*.conf" to the /etc/unbound/unbound.conf.d/local.conf previously created.

Now, you can run ./unbound-block-hosts --url="SOME_URL" --file=/etc/unbound/includes/FOOBAR-blocking.conf to generate a matching configuration for a given hosts list. Typically, I have a script doing:

#/bin/sh
set -e

cd "$(dirname "$0")"

echo "Fetch Malware domains list and append to unbound"
./unbound-block-hosts --url="http://www.malwaredomainlist.com/hostslist/hosts.txt" --file=/etc/unbound/includes/malwaredomainlist-blocking.conf --address="YOUR_RASPBERRY_PI_IP"
echo "Fetch Yoyo ad servers list and append to unbound"
curl "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=unbound;showintro=0&mimetype=plaintext" > /etc/unbound/includes/yoyoadservers-blocking.conf

systemctl reload unbound

which is crontask-ed to run every day. Default address is 127.0.0.1 which means the local host for the client machine. I do not want to have too many 404 on my local webservers, so I'd rather put the IP address of the Raspberry Pi and have a webserver answering a 404 on it.

Install a webserver

As simple as

sudo apt-get install nginx

Install and configure Privoxy

Now, you can install Privoxy:

sudo apt-get install privoxy

The default configuration should be mostly ok. You can look at the /etc/privoxy/config file to adapt it to your needs (the file is really an example of well documented config file). Two options you might be interesting in changing are the debug option (to enable logging, which is disabled by default) and listen-addr. You will want to set the latter to:

listen-address  127.0.0.1:8118
listen-address  YOUR_RASPBERRY_PI_IP:8118

so that the Prixovy proxy is accessible from the rest of your LAN. As always, do not forget to configure your firewall to let the Privoxy connections pass through. At this point, you should try to set the proxy in your browser's preferences and check that everything is working fine. You should be able to browse to any web page, but the proxy will not do anything else for the moment.

Note: At this point, we should emphasize that having such a proxy is a security risk, as anyone having access to your proxy can browse the web with your IP address (and you may be held liable for anything illegal done with it). Then, you should make sure that your Raspberry Pi Privoxy is only accessible from your local network, and that no third-party has access to it. This should be enforced by the firewall running on your Raspberry Pi and the firewall on your router (typically, most routers provided by your ISP block any incoming connections, check this).

Privoxy, as installed by the Raspbian package, enables a couple of filters out of the box. As we will be translating Adblock rules into Privoxy rules, we can disable them. Edit the /etc/privoxy/match-all.action file to get something like this:

#############################################################################
# Id: match-all.action,v
#
# This file contains the actions that are applied to all requests and
# may be overruled later on by other actions files. Less experienced
# users should only edit this file through the actions file editor.
#
#############################################################################
{ \
+change-x-forwarded-for{block} \
+client-header-tagger{css-requests} \
+client-header-tagger{image-requests} \
+filter{refresh-tags} \
+filter{webbugs} \
+filter{jumping-windows} \
+filter{ie-exploits} \
+hide-from-header{block} \
+hide-referrer{conditional-block} \
}
/ # Match all URLs

In particular, I disabled the filters img-reorder (which is really intensive for the Raspberry Pi, and takes a few hundreds of milliseconds to process a regular page) and banners-by-size as we will be importing Adblock rules which should give better results. deanimate-gifs and session-cookies-only is a matter of taste (respectively it prevents animated GIFs by replacing them by their last frame and only allowing temporary cookies).

Block ads using Privoxy

We will now be importing adBlock rules into privoxy. One way to do it is to use this Haskell script.

To install it directly on your Raspberry Pi, provided you have a recent Raspberry Pi:

This was not my case, so I set up a builder on my server to run daily. Some rules are provided by the author and my builds are available here.

The way to set up the resulting files into Privoxy is very well detailed on the page of the project.

Note: My builds are made with the Element Hiding feature and example.com as the domainCSS parameter. You should replace any occurrence of example.com by the FQDN or IP adrdess of your Raspberry Pi when importing it. Please, do not put too much load on my hosted builds and consider hosting your owns.

Benchmark

All the tests were made with a Raspberry Pi of the first model with 512MB of RAM (model 1B). The Raspberry Pi has a wired access to internet (100MB/s port only on the Raspberry Pi). My laptop is wired as well (gigabit ethernet). Home connection to internet is a fiber access (923 Mbps download, 250 Mbps upload, as reported by DSLReports).

Testing the DNS server

Without the DNS server,

$ # Using my ISP resolver
$ % dig @192.168.0.254 example.com
...
;; Query time: 7 msec
;; SERVER: 192.168.0.254#53(192.168.0.254)

$ # Using Google DNS
$ dig @8.8.8.8 example.com
...
;; Query time: 5 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)

$ # Using the DNS resolver on my Raspberry Pi
$ dig @192.168.0.1 example.com
...
;; Query time: 505 msec
;; SERVER: 192.168.0.1#53(192.168.0.1)

$ # Using it another time, now that the domain is in cache
$ dig @192.168.0.1 example.com
...
;; Query time: 5 msec
;; SERVER: 192.168.0.1#53(192.168.0.1)

These are typical times, the value is typically the one obtained as average of a few runs.

We can see that there is some overhead when first accessing a domain, as the Pi has to do the full DNS resolution. Afterwards, the domain is kept in cache and it is as fast to use the DNS server from the Pi as it is to use any other one.

Testing the Privoxy setup

Now, let us focus on the performances of the Privoxy on the Raspberry Pi. I tested it with a few websites, and results were roughly the same. Here is a detailed example of Liberation's website, a French journal. This example is interesting as my µBlock setup on my laptop blocks 23 different things when I don't use the DNS nor the Privoxy proxy. It is also an interesting example as out of the 23 blocked contents, only 14 of them could be blocked by DNS (with the setup described above).

The main issue here is that Privoxy is very long to process the page with all the filters, and it is way too heavy for my low power Raspberry Pi first model.

The main HTML document for this page takes 7 seconds to load when passing through the proxy, mainly due to the processing time. When reloading the page, it only takes 400ms as it is already in cache. As a comparison, it takes only 24ms when loading it directly.

The complete setup looks equivalent to the µBlock setup on my laptop.

I don't have a more recent version of the Raspberry Pi (typically Raspberry Pi 3) to test what the performances are on such a more powerful system. If you can try it, let me know, I am curious about the way it handles the load, and I could publish an edit to this article.

 

Doing low cost telepresence (for under $200)

8 min read

With a friend, we recently started a project of building a project of low cost telepresence robot (sorry, link in French only) at our local hackerspace.

The goal is to build a robot that could be used to move around a room remotely, and stream audio and video in both directions. Our target budget is $200. We got a first working version (although it does not yet stream audio), and it is time for some explanations on the setup and how to build your own =) All the instructions, code and necessary stuff can be found at our git repo.

Screen capture


3D model

Basic idea

When taking part in a group meeting remotely, using some videoconference solution, it is often frustrating not being able to move around the room on the other side. This prevents us from having parallel discussions, and if the remote microphone is poor quality, we often do not hear clearly everybody speaking. Plus, someone speaking may be hidden by another speaker and many other such problems happen.

The goal was then to find a solution to do videoconferences (streaming both audio and video in both directions) and be able to move on the other side, to be able to see everyone and to come closer to the current speaker. Commercial solutions exist but they are really expensive (a few thousands dollars). We wanted to have the same basic features for $200, and it seems we almost achieved it!

Bill of Materials

The whole system is built around a Raspberry Pi and a PiCamera, which offer decent performances at a very fair price. The rest is really basic DIY stuff.

Here is the complete bill of materials:

Total: $140

Notes:

  • We had to use a Raspberry Pi model 2 for the nice performance boost on this model. Even more important is the increased number of GPIOs on this model, with 2 usable hardware PWMs (provided that you don't use the integrated sound card output). This is useful to control the two wheels with hardware PWM and have a precise control of the move. The camera holder can be safely controlled with a software PWM and we did not experience any troubles doing so.
  • You can easily replace those parts by equivalent ones as long as you keep in mind that the battery pack should be able to provide enough current for the raspberry pi and the servos. We used standard USB battery packs for simplicity and user friendliness. However, they are more expensive than standard modelling lithium batteries and provide less current in general.
  • We had to use two battery packs. Indeed, the peak current due to the servos starting was too excessive for the battery pack and it was crashing the raspberry pi. Using two separate alimentation lines for the raspberry pi and the servos, we no longer have this problem and this solution is easier than tweaking the alimentation line until the raspberry pi stops freezing (which it may never do).

For the next version, we plan to add:

Total with these parts: $228

Notes:

  • We used an HDMI screen as the official RaspberryPi screen uses most of the GPIOs pins, which we need. We decided to use bluetooth speakers as the integrated sound card was not usable as we were using the two hardware PWM lines for motion. This way, we have a speaker with a built-in microphone, which smaller than having the two of them separately.
  • The USB bluetooth adapter is impressively expensive, but it is the only one we found at the moment which we were sure would be compatible with Linux without any problems. Plus others adapters we found were not much cheaper.
  • The total budget is $223 without shipping. It is a bit over the initial budget goal, but we can easily lower it to $200. Indeed, we did not especially look for the cheaper parts. In particular, we bought the servos from Adafruit and I think we can find some servos for less (especially the camera holder servo, which can be a micro servo at $5 and should be enough). The bluetooth adapter is quite expensive as well and we could find a cheaper one I think. Budget shrinkage will be our next goal, once we have everything working.

Building the robot

All the necessary stuff is in our git repo (or its github mirror, both should be kept in sync). The repo contains three main directories: - blueprints which are the models of the robot. - disty which is the main server code on the Raspberry Pi. - webview which is the web controller served by the Raspberry Pi.

First of all, you should cut the parts and print the 3D parts in the blueprints dir. eps files in this directory are ready to cut files whereas svg files should be the same ones in easily editable format. You should laser cut the top and bottom files. picam_case_* files are the camera case we used,

You should 3D print:

  • the picam_case_* files for the camera case (licensed under CC BY SA).
  • teleprez.blend is the complete CAO model of the robot in Blender.
  • camera_servo_holder.stl is the plastic part to hold the camera servo. You need to print it once. wheel_servo_holder.stl is the plastic part to hold the servos for the wheels. You need four of them.

Assembling your Disty robot should be straightforward and easy to do if you look at the following pictures :) Use two ball transfer units to stabilize the robot and lock them with some rubber band (or anything better than that). Adjust tightly the height of the wheels so that the two wheels and the ball transfer units touch the ground.

Disty

Disty

Disty

GPIO pinout for the connection can be found at https://raw.githubusercontent.com/hackEns/Disty/master/blueprints/gpio.png.

GPIO pinout

For the electrical wiring, we used a standard USB-Micro USB cable to power the Raspberry Pi from one battery (located below the robot, to add weight on the ball transfer units and ensure contact is made with the surface). On the other battery, we just cut a USB - Micro USB cable to plug into it and connect the servos directly through a piece of breadboard to the battery. We had to use two batteries to prevent the draw from the servos to reboot the Raspberry Pi.

Here you are, you have a working Disty!

Running it

This may not be super user-friendly at the moment, we hope to improve this in the future.

Download any Linux image you want for your Raspberry Pi. Install uv4l and the uv4l-webrtc component. Enable the camera and ensure you can take pictures from the command line (there is a lot of doc) about this on the web.

Then, clone the Git repo somewhere on your Raspberry Pi. You should build the main disty code (which is the serverside code). This code will handle the control of the servos (emit PWMs etc) and listen on UDP port 4242 for instructions sent from the webview. Instructions to build it are located in the associated README. You will need cmake and a system-wide install of wiringpi to build the code.

You can then start the robot. Start by launching the disty program (as root as you need access to the GPIOs), ./disty, and then start the webview, ./run.py as root also as it serves the webview on port 80, which is below 1024 and owned by root. If you have ZeroConf on your Raspberry Pi (or a decent router), you can go to http://disty (or whatever hostname is set on your Raspberry Pi) to get the webview. Else, use the IP address instead. Webview usage should be almost straightforward.

It should work out of the box on your local LAN. If you are behind a NAT, it will need some black magic (which is implemented but may not be sufficient) to connect the remote user and Disty camera. In any case, you need to be able to access the webview (disty port 80) from the remote side.

Contributing!

All contributions and feedbacks are more than welcomed!

All the source code we wrote is under a beer-ware license, under otherwise specified.

* --------------------------------------------------------------------------------
* "THE BEER-WARE LICENSE" (Revision 42):
* Phyks and Élie wrote this file. As long as you retain this notice you
* can do whatever you want with this stuff (and you can also do whatever you want
* with this stuff without retaining it, but that's not cool...). If we meet some
* day, and you think this stuff is worth it, you can buy us a beer
* in return.
*                                                                       hackEns
* ---------------------------------------------------------------------------------

If you need a more legally valid license, you can consider Disty to be under an MIT license.

Some sources of inspiration and documentation

 

Controlling servomotors on a Raspberry Pi

5 min read

EDIT: This article might not be very beginner-friendly. If you think it could be worth more explanations, feel free to let me know.

For a project (documentation to be added soon) of low-cost telepresence robot, I had to handle three servomotors with a Raspberry Pi. I chose a Raspberry Pi board as it is very cheap and has a decent camera which can be easily used to stream video at a good resolution and framerate, without using too much CPU. Then, I wanted to control everything with the Raspberry Pi, including the servomotors. Here comes the troubles…

First of all, I had to control three servomotors: two continuous rotation servos for the wheels, and an extra standard servo for the camera orientation. First revisions of the RaspberryPi only have one hard PWM, shared with the audio circuit, which makes it really difficult to use for this purpose. But Raspberry Pi 2 has two hardware PWMs (one is shared with the audio circuit, preventing you from using the onboard audio output). That is enough for my use case, as I have two continuous rotation servos which have to be precise (and will be controlled via hard PWM) and an extra servo for the camera orientation which can be easily controlled in soft PWM.

My code can be found here. The important files are the header of Servo class, the actual Servo class and a file containing some extra constants.

Hard PWM

Next problem is to find a way to control them easily. On an Arduino, you just have to include the Servo library and write angles to your servos. On the Raspberry Pi, on the contrary, there are no such things. WiringPi is a nice C library to handle GPIOs (including PWM) but there are no default failsafe settings to use, and one has to play a lot with the different configuration commands.

Most of the servos expect pulses of variable width, sent on a 20ms basis. Each 20ms, it samples the wave to get the width of the pulse. The base PWM frequency on the Raspberry Pi is 19.2MHz, and you can set a clock prescaler (clock) and a number of samples (range). The base frequency of your PWM signal will then be given by the formula (as 50Hz = 20ms):

19.2MHz / (clock * range) = 50Hz

The stop position is for 1.5ms pulses in general, for a continuous rotation servo. Then, we also want that there exists a valid int such that we could write it to stop the servo. pwmWrite(pin, STOP_VALUE) should stop the servo, with STOP_VALUE being an int. This is important as servos are symmetric with respect to this position and a slight offset will make the calibration of the servo speeds really difficult.

Good results were obtained on my robot with a clock prescaler of 200 and a range of 1920. Then, the stop value was 141 and the full forward value was 155 (value to write to make the servo turn at full speed). This gives me 14 different speeds for the servo, which is more than enough in my case. If you want finer control, you should play with the clock prescaler and range values.

Soft PWM

Then comes the control of the camera orientation using Software PWM. I explored two possibilities: ServoBlaster which is not compatible out of the box with the Raspberry Pi 2 (but a solution exists) and the software PWM library from wiringPi. Both gives equivalent results, including jitter (a bit less jitter for ServoBlaster, but still too much for a stable camera orientation). Then, I used wiringPi which was easier to integrate in my code. I found a dirty hack to fix servo jitter, as explained below.

First, the base frequency of the software PWM is 100Hz, but we need 50Hz for the servo control. Then, we have to softPwmCreate with a range of 200. We can then write values to the softPwm, and it will most likely work, at least if there are not much processes running.

My problem was that once I started the camera capture, my software pwm got preempted a lot, and this was giving a lot of jitter. This was really funny because when I was using autocompletion in the shell for instance, my servo had some jitter that I could hear. I tried to play with nice without success and could not get rid of the jitter.

However, I finally found a really hacky way of getting rid of the jitter (may not be applicable in your case): if I stop sending PWM signal to the servo, it holds still. Plus, if I send a 0 value on the PWM, this is an out-of-range value for the servo and he just ignores it, holding still as well. Then, the solution was easy. I just had to send the soft PWM signal to the servo, wait a bit until he has moved (based on the average speed in the datasheet, I know how long I have to wait) and then send 0 on the software PWM. This way, the servo holds still, and there is no more jittering.

Hope this helps :)

 
 

Filtering ads with your Raspberry Pi

13 min read

TL;DR: Please have a look at the benchmark section below, to be aware of the limitations of this particular setup and decide whether to spend some time putting it in place or not.

I recently came across this Pi-Hole project that claims to be "a black hole for Internet advertisements" (thanks nicofrand for making me discover this!). The idea was really attractive: having a simple Raspberry Pi on the network doing all the ad filtering for the whole network, rather than having to maintain a separate uBlock Origin install on each and every computers of the network. It was also particularly attractive as having such ad blocker on a smartphone requires a rooted device. Plus there was a really nice web interface to control the whole ad blocking device.

While looking at it more in depth, I realized it was actually very limited:

  1. First, it was built around specific softwares and was doing some magical stuff using these softwares. It was really painful to get away from them. Basically, it uses dnsmasq to expose a DNS service, a standard hosts file to block the hosts serving ads, and a lighthttpd webserver. Problem is I already have a DNS resolver (unbound) on this Raspberry Pi, and a web server (nginx). I did not want to spend a lot of time trying to integrate it in my existing setup if finally it was not that powerful, so I decided to look at it in details before installing.
  2. Second issue was that it relies on dnsmasq. dnsmasq is a simple program that allow you to answer DNS queries by using the hosts defined in /etc/hosts and to forward every other requests to another DNS server (typically your ISP DNS server). Pi-Hole lets you configure two DNS servers to forward queries to, default one being 8.8.8.8 (Google :/). I already have a resolver on this Raspberry-Pi and I do want to do the resolution myself, especially since my ISP DNS servers lies, and I do not want to use public DNS server on another network. So I had to hack on Pi-Hole to do some DNS resolution. About these issues, I'd like to point to two very interesting articles from Bortzmeyer: this one about Google DNS (in French) and this one about having your own DNS resolver (same). Also, being a DNS resolver, it may be cumbersome to disable it temporarily to load some website that absolutely requires the ads to be loaded.
  3. Last issue was that contrary to uBlock which filters at the requests level (and even sometimes at the HTML level), the fact that is basically an alternative DNS resolver means you can only filter at the domain level. That is, you either whitelist (default) or blacklist a domain which is serving ads or malwares, but you cannot differentiate different paths for a given domain. While browsing Libération website, I can see uBlock is blocking queries such as http://s3.amazonaws.com/files.wrapper.theadtech.com/native/placements/liberation.fr/pconfig?r=5a6f5d98b4d608. Such queries cannot be blocked by Pi-Hole without blacklisting the whole Amazon S3 network.

Given these facts, I remembered Privoxy which can be used as a filtering proxy, in a way similar to uBlock. Given that it is a proxy, it can filter in details, just as uBlock do and you can very easily disable it (simply disable the proxy). Plus, almost any devices offer you a proxy setting, so it should work both on my Android phones and computers. In this article I describe how I set up a Pi-Hole alternative based on Unbound (to have my own DNS resolver and block some things at the domain level) coupled with a Privoxy proxy to filter out ads.

Limitations: So, contrary to Pi-Hole, the setup described here will be able to remove ads in a similar way to uBlock/AdBlock. If you go through the whole article till the end, it will also have the element hiding features. Being a proxy setting, it will be easy to toggle it on and off (either by Prixovy toggling features or by manually turning off proxy on your device), if required. However, be aware of the remaining limitations with regards to HTTPS streams (section 4.15). As AdBlock/uBlock runs in the browser, it can filter ads in HTTPS streams as well, Privoxy will not be as efficient without HTTPS interception (which is generally not a good idea). However, it should perform rather well in the vast majority of situations (also note that AdBlock for rooted Android devices is also a proxy, so for them, it will not change anything).

I assume you already have a running Raspberry Pi with some basic install. Typically, see this previous article if this is not the case

Set up a DNS resolver

Let's install a DNS resolver on the Raspberry Pi, to answer DNS queries on the network. I am installing unbound and configuring Unbound here.

$ sudo apt-get install unbound $ curl -o /etc/unbound/root.hints https://www.internic.net/domain/named.cache $ # (Optional) Set crontask to download root.hints file every six months

The last curl command is used to fetch the root hints, to query hosts that are not cached. See this section of the ArchWiki page for more infos.

Then, you can create a basic configuration file for unbound.

$ cat /etc/unbound/unbound.conf.d/local.conf server: username: "unbound" interface: 0.0.0.0 # Listen on all interfaces root-hints: "/etc/unbound/root.hints" access-control: 192.168.0.0/8 allow # Access-control, see "Example" section in https://www.unbound.net/documentation/unbound.conf.html

Then, enable and start Unbound service at startup:

$ sudo systemctl start unbound && sudo systemctl enable unbound

You can now change the resolver to be used on your Raspberry Pi and on your whole network. To change it on your Raspberry Pi, have a look at this wiki page (for Raspbian). To set it as the default DNS resolver on your network, have a look at your router configuration, and set the DNS resolvers address to the address of your Raspberry Pi. Don't forget to open the ports in your firewall (53 tcp and udp).

To check everything is working fine, you can use dig (from dnsutils package on Debian-based distributions). Typically, dig google.fr should give you some results (in the ANSWER SECTION) and the IP address in the SERVER line should be the one of your Raspberry Pi.

Note: At this point, we should emphasize that having an open DNS resolver (that is, a DNS resolver that can answer to anyone) can be a security risk especially since some DDoS attacks use it. Then, you should make sure that your Raspberry Pi DNS server is only accessible from your local network, and that no third-party has access to it. This should be done through the access-control line in the above configuration, but this can also be enforced by the firewall running on your Raspberry Pi and the firewall on your router (typically, most routers provided by your ISP block any incoming connections, check this).

Block some domains based on hosts

Now, we would like unbound to block some domains that are known to serve ads and malwares, in a similar way as Pi-Hole does. For this purpose, we will use unbound-block-hosts script to import hosts files into Unbound configuration. Basically, for every such domain, Unbound will return 127.0.0.1.

unbound-block-hosts is designed with the Dan Pollock's hosts file in mind, whereas I wanted to be able to import any host file in Unbound. Here is a forked and patched version for this purpose (very ugly patch, as I am not fluent in Perl :/).

We will create an includes dir in the Unbound configuration directory (mkdir /etc/unbound/includes/), and include the rules in the main configuration by appending include: "/etc/unbound/includes/*.conf" to the /etc/unbound/unbound.conf.d/local.conf previously created.

Now, you can run ./unbound-block-hosts --url="SOME_URL" --file=/etc/unbound/includes/FOOBAR-blocking.conf to generate a matching configuration for a given hosts list. Typically, I have a script doing:

```

/bin/sh

set -e

cd "$(dirname "$0")"

echo "Fetch Malware domains list and append to unbound" ./unbound-block-hosts --url="http://www.malwaredomainlist.com/hostslist/hosts.txt" --file=/etc/unbound/includes/malwaredomainlist-blocking.conf --address="YOUR_RASPBERRY_PI_IP" echo "Fetch Yoyo ad servers list and append to unbound" curl "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=unbound;showintro=0&mimetype=plaintext" > /etc/unbound/includes/yoyoadservers-blocking.conf

systemctl reload unbound ```

which is crontask-ed to run every day. Default address is 127.0.0.1 which means the local host for the client machine. I do not want to have too many 404 on my local webservers, so I'd rather put the IP address of the Raspberry Pi and have a webserver answering a 404 on it.

Install a webserver

As simple as

sudo apt-get install nginx

Install and configure Privoxy

Now, you can install Privoxy:

sudo apt-get install privoxy

The default configuration should be mostly ok. You can look at the /etc/privoxy/config file to adapt it to your needs (the file is really an example of well documented config file). Two options you might be interesting in changing are the debug option (to enable logging, which is disabled by default) and listen-addr. You will want to set the latter to:

listen-address 127.0.0.1:8118 listen-address YOUR_RASPBERRY_PI_IP:8118

so that the Prixovy proxy is accessible from the rest of your LAN. As always, do not forget to configure your firewall to let the Privoxy connections pass through. At this point, you should try to set the proxy in your browser's preferences and check that everything is working fine. You should be able to browse to any web page, but the proxy will not do anything else for the moment.

Note: At this point, we should emphasize that having such a proxy is a security risk, as anyone having access to your proxy can browse the web with your IP address (and you may be held liable for anything illegal done with it). Then, you should make sure that your Raspberry Pi Privoxy is only accessible from your local network, and that no third-party has access to it. This should be enforced by the firewall running on your Raspberry Pi and the firewall on your router (typically, most routers provided by your ISP block any incoming connections, check this).

Privoxy, as installed by the Raspbian package, enables a couple of filters out of the box. As we will be translating Adblock rules into Privoxy rules, we can disable them. Edit the /etc/privoxy/match-all.action file to get something like this:

```

Id: match-all.action,v

This file contains the actions that are applied to all requests and

may be overruled later on by other actions files. Less experienced

users should only edit this file through the actions file editor.

{ \ +change-x-forwarded-for{block} \ +client-header-tagger{css-requests} \ +client-header-tagger{image-requests} \ +filter{refresh-tags} \ +filter{webbugs} \ +filter{jumping-windows} \ +filter{ie-exploits} \ +hide-from-header{block} \ +hide-referrer{conditional-block} \ } / # Match all URLs ```

In particular, I disabled the filters img-reorder (which is really intensive for the Raspberry Pi, and takes a few hundreds of milliseconds to process a regular page) and banners-by-size as we will be importing Adblock rules which should give better results. deanimate-gifs and session-cookies-only is a matter of taste (respectively it prevents animated GIFs by replacing them by their last frame and only allowing temporary cookies).

Block ads using Privoxy

We will now be importing adBlock rules into privoxy. One way to do it is to use this Haskell script.

To install it directly on your Raspberry Pi, provided you have a recent Raspberry Pi:

This was not my case, so I set up a builder on my server to run daily. Some rules are provided by the author and my builds are available here.

The way to set up the resulting files into Privoxy is very well detailed on the page of the project.

Note: My builds are made with the Element Hiding feature and example.com as the domainCSS parameter. You should replace any occurrence of example.com by the FQDN or IP adrdess of your Raspberry Pi when importing it. Please, do not put too much load on my hosted builds and consider hosting your owns.

Benchmark

All the tests were made with a Raspberry Pi of the first model with 512MB of RAM (model 1B). The Raspberry Pi has a wired access to internet (100MB/s port only on the Raspberry Pi). My laptop is wired as well (gigabit ethernet). Home connection to internet is a fiber access (923 Mbps download, 250 Mbps upload, as reported by DSLReports).

Testing the DNS server

Without the DNS server,

``` $ # Using my ISP resolver $ % dig @192.168.0.254 example.com ... ;; Query time: 7 msec ;; SERVER: 192.168.0.254#53(192.168.0.254)

$ # Using Google DNS $ dig @8.8.8.8 example.com ... ;; Query time: 5 msec ;; SERVER: 8.8.8.8#53(8.8.8.8)

$ # Using the DNS resolver on my Raspberry Pi $ dig @192.168.0.1 example.com ... ;; Query time: 505 msec ;; SERVER: 192.168.0.1#53(192.168.0.1)

$ # Using it another time, now that the domain is in cache $ dig @192.168.0.1 example.com ... ;; Query time: 5 msec ;; SERVER: 192.168.0.1#53(192.168.0.1) ```

These are typical times, the value is typically the one obtained as average of a few runs.

We can see that there is some overhead when first accessing a domain, as the Pi has to do the full DNS resolution. Afterwards, the domain is kept in cache and it is as fast to use the DNS server from the Pi as it is to use any other one.

Testing the Privoxy setup

Now, let us focus on the performances of the Privoxy on the Raspberry Pi. I tested it with a few websites, and results were roughly the same. Here is a detailed example of Liberation's website, a French journal. This example is interesting as my µBlock setup on my laptop blocks 23 different things when I don't use the DNS nor the Privoxy proxy. It is also an interesting example as out of the 23 blocked contents, only 14 of them could be blocked by DNS (with the setup described above).

The main issue here is that Privoxy is very long to process the page with all the filters, and it is way too heavy for my low power Raspberry Pi first model.

The main HTML document for this page takes 7 seconds to load when passing through the proxy, mainly due to the processing time. When reloading the page, it only takes 400ms as it is already in cache. As a comparison, it takes only 24ms when loading it directly.

The complete setup looks equivalent to the µBlock setup on my laptop.

I don't have a more recent version of the Raspberry Pi (typically Raspberry Pi 3) to test what the performances are on such a more powerful system. If you can try it, let me know, I am curious about the way it handles the load, and I could publish an edit to this article.