Magnus Carlsson

Building for Raspberry Pi using distcc

Building locally on a Raspberry Pi can be slow. The most common way around this is to cross-compile, which is much quicker, but requires more setup.1 There is also a third alternative, distcc, which distributes the compilation to remote computers in a transparent manner. Setting up distcc is somewhat simpler compared to cross-compiling, but we still gain speed compared to local build.

The distcc program can distribute builds across several machines on a network.2 Today we will setup and compile CANlib on an old Raspberry Pi Model B running Rasbian 8.0 using distcc to distribute the compilation onto a Linux computer running Ubuntu 16.04.

Setting up the Host

First we need to set up and configure our Ubuntu host machine.

Install required packages

Using apt we install the required packages distcc and git.

$ sudo apt update
$ sudo apt install distcc
$ sudo apt install git

Fetch and set up the toolchain

The Raspberry Pi Foundation is providing a ready-to-use toolchain in their GitHub repository. Create a directory and fetch the toolchain from GitHub. We are only interested in the latest revision and not the whole history so we use the argument --depth 1 .

$ mkdir ~/rpi_cross
$ cd ~/rpi_cross
$ git clone https://github.com/raspberrypi/tools.git --depth 1

Now we have to make sure that when distcc calls our compiler, it should use our cross compiler. To do so we create symbolic links to the compilers.

$ cd tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin
$ ln -s arm-linux-gnueabihf-c++ c++
$ ln -s arm-linux-gnueabihf-cc cc
$ ln -s arm-linux-gnueabihf-cpp cpp
$ ln -s arm-linux-gnueabihf-g++ g++
$ ln -s arm-linux-gnueabihf-gcc gcc

Configure distcc

We edit the distcc configurations file found in /etc/default/distcc.

  1. Set STARTDISTCC to “true” in order to start the distcc daemon, distccd, at boot.
  2. Change ALLOWEDNETS to allow the network or individual computers to connect. We just allow our Raspberry Pi to connect so we set its ip-address here.
  3. Comment out the row containing ‘LISTENER=”127.0.0.1″‘, otherwise distcc will only listen on the loopback interface.
  4. Set maximum allowed tasks using JOBS. You can get the number of available processors on your computer by using the nproc command. On our machine we wanted to limit the number of jobs to two.

The edited configuration file at /etc/default/distcc will now look like the following (comments omitted):

STARTDISTCC="true"
ALLOWEDNETS="192.168.0.27"
NICE="10"
JOBS="2"
ZEROCONF="false"

Now we need to make sure that distcc finds and uses our toolchain. We do this by including the path to the cross toolchain first in the PATH variable in /etc/init.d/distcc. Our user is named ‘ubu’ on the Ubuntu host so the line becomes:

PATH=/home/ubu/rpi_cross/tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

Reload systemd, since we made changes to the init script, and start distccd.

sudo systemctl daemon-reload
sudo systemctl start distcc

Setting up the Raspberry Pi

We can now set up and configure our Raspberry Pi.

Install required packages

Using apt we first install the required packages. bc is required to be able to execute make prepare on the kernel.

sudo apt update
sudo apt install distcc
sudo apt install bc

Setup and configure distcc

Create symbolic links to distcc.

$ cd /usr/local/bin
$ sudo ln -s /usr/bin/distcc c++
$ sudo ln -s /usr/bin/distcc cc
$ sudo ln -s /usr/bin/distcc cpp
$ sudo ln -s /usr/bin/distcc g++
$ sudo ln -s /usr/bin/distcc gcc

Setup the distcc environment variables. Note that you will have to do this at every login, unless you put them in /etc/profile.

  1. Set the DISTCC_HOSTS environment variable to a space-delimited list of host machines. We just add our single host using 192.168.0.2, which is the IP-address of the host machine.
  2. If you do not want to compile anything locally, set the environment variable “DISTCC_SKIP_LOCAL_RETRY”= to 1
$ export DISTCC_HOSTS="192.168.0.2"
$ export DISTCC_SKIP_LOCAL_RETRY=1

Find correct version of Linux headers

How to find out what version of the Linux headers to use is described in the blog article Building CANlib on Raspberry Pi.

Fetch and build Linux Headers

How to download and extract the kernel source is done in the same way as is described in the blog article Building CANlib on Raspberry Pi.

Now we can setup kernel configuration and build the headers in exactly the same way as is described in the blog article Building CANlib on Raspberry Pi. However, this time we will now use the distcc toolchain and send the compilation job to our Ubuntu host.

$ sudo modprobe configs
$ zcat /proc/config.gz > .config
$ make prepare
$ make modules_prepare

The two make commands now took about two minutes, only one minute less than building locally, but still about 30% reduction in time.

Versions used

The host uses the following versions:

OS                     Ubuntu 16.04 x86_64 with kernel 4.4.0-21-generic 
distcc                 version 3.1 (package version 3.1-6.2)            
raspberrypi/tools.git  rev 6376a50ed059b3c852ce3836981ba5ff07d6b368 

The client, a Raspberry Pi Model B is using the following versions:

OS            Rasbian 8.0 with kernel 4.4.11+              
distcc        version 3.1 (package version 3.1-6.2)        
firmware.git  rev 3b98f7433649e13cf08f54f509d11491c99c4c0b 
linux.git     rev 233755da0e7903fccb41f0b8c14e1da5244b69ec

Footnotes

1 For a description on how to build the kernel for a Raspberry Pi, click here.

2 distcc documentation can be found here.

Feedback