Monthly Archives: December 2015

Cross Building Linux With ELLCC: Part 2 – Raspberry Pi B+

Update: Cross Building Linux With ELLCC: Part 3 – Other Flavors of Pi
In a previous post I tested the waters to see how close ELLCC is to cross compiling the Linux kernel. That post was mostly a theoretical exercise to see what it would take. Over the holidays I decided that I needed to do something a little more concrete. My son, a 10 year old obsessed with Minecraft, got a Raspberry Pi for Christmas. My nefarious plan is to divert his attention away from Minecraft at least long enough to make his Pi and camera module into something he can use to spy on the family. My other nefarious plan is to use ELLCC to build the Linux kernel that we’ll be running on his Pi.

In the previous post I used the source from the LLVMLinux site. This time I started with the kernel source for the Raspberry Pi. The good news is that the source can be built for the Pi. The bad news is that it doesn’t have all the patches need to compile with a clang based compiler. It turned out that the build wasn’t too bad: I had Christmas Eve and Day off and in between family events and Santa preparations I was able to massage the source enough to boot the kernel on our Pi.

To get started I followed the instructions on Linux From Scratch on the Raspberry Pi.
Here are the exact steps I took. First I created a new directory to hold the ELLCC tool chain and the Raspberry Linux kernel source, then I downloaded and untar’d both:

mkdir ecclinux
cd ecclinux
wget http://ellcc.org/releases/ellcc-x86_64-linux-eng-0.1.23.tgz
tar xvfp ellcc-x86_64-linux-eng-0.1.23.tgz
wget https://github.com/raspberrypi/linux/archive/rpi-4.1.y.tar.gz
tar xvfp rpi-4.1.y.tar.gz

I had to change the kernel sources slightly during my build. I saved a patch
file, here’s how to apply it:

patch -p0 < ellcc/libecc/llvmlinux/ecc-rpi-4.1.y.patch

There is a special little 'm' makefile that helps with the build. Copy
it into the top level of the kernel source directory:

cd linux-rpi-4.1.y/
cp ../ellcc/libecc/llvmlinux/makefile .

Now the actual build is easy:

make bcmrpi_defconfig
make

I have my Raspberry Pi on my home network at 192.124.43.98. I also allowed ssh
and scp root access by follwoing the instructions here. This lets me
just copy the new kernel to my Pi:

scp ./arch/arm/boot/zImage root@192.124.43.98:/boot/kernel.img

Now I can ssh to the Pi and reboot:

[~] dev% ssh root@192.124.43.98
root@192.124.43.98's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@raspberrypi:~# reboot
Connection to 192.124.43.98 closed by remote host.
Connection to 192.124.43.98 closed.
[~] dev%

I ping the Pi until it reboots and log in again:

[~] dev% ssh root@192.124.43.98
root@192.124.43.98's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Dec 26 20:27:15 2015 from 192.124.43.2
root@raspberrypi:~# dmesg | less
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1.15 (rich@dev) (ecc 0.1.23 based on clang version 3.8.0 (trunk) (based on LLVM 3.8.0svn)) #1 Sat Dec 26 11:32:33 CST 2015
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
...

Very nice!

Using ELLCC to (Cross) Compile Toybox

Toybox combines many common Linux command line utilities together into a single BSD-licensed executable. I wanted to to see how hard it would be to compile and cross compile it using ELLCC. It turned out to be pretty easy after I added a few Linux specific header files. These are the files I added:

[~/ellcc] dev% ls ~/ellcc/libecc/include/linux
fs.h  if_vlan.h  loop.h  nbd.h  rfkill.h  rtc.h  sched.h  sockios.h  soundcard.h

The needed include files are in the ELLCC source tree now and are available in the 0.1.22 binary release. Here’s all you need to do to build Toybox:

[rich@dev ~]$ mkdir toybox 
[rich@dev ~]$ cd toybox
[rich@dev toybox]$ wget http://landley.net/toybox/downloads/toybox-0.6.1.tar.gz
--2015-12-23 17:02:37--  http://landley.net/toybox/downloads/toybox-0.6.1.tar.gz
Resolving landley.net (landley.net)... 208.113.171.142
Connecting to landley.net (landley.net)|208.113.171.142|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 736371 (719K) [application/x-tar]
Saving to: ‘toybox-0.6.1.tar.gz’

toybox-0.6.1.tar.gz  100%[=====================>] 719.11K   845KB/s   in 0.9s   

2015-12-23 17:02:38 (845 KB/s) - ‘toybox-0.6.1.tar.gz’ saved [736371/736371]

[rich@dev toybox]$ tar xfp toybox-0.6.1.tar.gz 
[rich@dev toybox]$ cd toybox-0.6.1/
[rich@dev toybox-0.6.1]$ CC="/home/rich/ellcc/bin/ecc -target armv6-linux-engeabihf" CFLAGS="-Wno-format-extra-args -Wno-uninitialized" make defconfig toybox
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
        -DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/null Config.in > /dev/null
scripts/make.sh
Generate headers from toys/*/*.c...
generated/newtoys.h Library probe.......
Make generated/config.h from .config.
generated/flags.h generated/globals.h generated/help.h
Compile toybox................................................................................................................................................................strip: Unable to recognise the format of the input file `toybox_unstripped'
strip failed, using unstripped
.

Oops. The strip failed because the standard x86_64 Linux strip doesn’t understand ARM object file formats. We can use ecc-strip:

[rich@dev toybox-0.6.1]$ ~/ellcc/bin/ecc-strip -o toybox toybox_unstripped
[rich@dev toybox-0.6.1]$ file toybox
toybox: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=11e423037c6fb0e4ee10ef5f53a421fca86a81e3, stripped               
[rich@dev toybox-0.6.1]$ size toybox                                             
   text    data     bss     dec     hex filename                                 
 401916    3208   21020  426144   680a0 toybox  

Lest’s send it over to my trusty Raspberry Pi:

                                 
[rich@dev toybox-0.6.1]$ scp toybox rich@192.124.43.98:                                                   
rich@192.124.43.98's password:                                                   
scp: ./toybox: Permission denied                                                 
[rich@dev toybox-0.6.1]$ ssh rich@192.124.43.98
rich@192.124.43.98's password:                                                   
                                                                                 
The programs included with the Debian GNU/Linux system are free software;        
the exact distribution terms for each program are described in the               
individual files in /usr/share/doc/*/copyright.                                  
                                                                                 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent                
permitted by applicable law.                                                     
Last login: Sun Dec 20 19:20:36 2015 from 192.124.43.2   

What commands does Toybox have?

                        
rich@raspberrypi:~$ ./toybox                                                     
acpi base64 basename blkid blockdev bunzip2 bzcat cal cat catv chattr            
chgrp chmod chown chroot chvt cksum clear cmp comm count cp cpio cut             
date df dirname dmesg dos2unix du echo egrep eject env expand factor             
fallocate false fgrep find flock free freeramdisk fstype fsync grep              
groups halt head help hexedit hostid hostname hwclock id ifconfig                
inotifyd insmod install ionice iorenice kill killall killall5 link               
ln login logname losetup ls lsattr lsmod lspci lsusb makedevs md5sum             
mix mkdir mkfifo mknod mkpasswd mkswap mktemp modinfo mount mountpoint           
mv nbd-client nc netcat nice nl nohup nproc nsenter od oneit partprobe           
passwd paste patch pidof pivot_root pmap poweroff printenv printf                
ps pwd pwdx readahead readlink realpath reboot renice reset rev rfkill           
rm rmdir rmmod sed seq setsid sh sha1sum shred sleep sort split stat             
strings su swapoff swapon switch_root sync sysctl tac tail taskset               
tee time timeout touch toysh true truncate tty umount uname uniq unix2dos        
unlink uptime usleep uudecode uuencode vconfig vmstat w wc which who             
whoami xargs xxd yes                                                             
rich@raspberrypi:~$ ./toybox ls -l
total 3112                                                                       
-rwxr-x--x 1 rich rich  141580 2015-12-20 19:22 hello                            
-r-xr-xr-x 1 rich rich 1446476 2015-12-23 21:21 toybox   

Now let’s build for the x86_64:

                        
rich@raspberrypi:~$ logout                                                                           
Connection to 192.124.43.98 closed.                                              
[rich@dev toybox-0.6.1]$ make clean                                              
rm -f kconfig/zconf.hash.c kconfig/zconf.tab.c kconfig/lex.zconf.c kconfig/conf kconfig/mconf                                                                     
rm -rf toybox toybox_unstripped generated change .singleconfig*                  
[rich@dev toybox-0.6.1]$ CC="/home/rich/ellcc/bin/ecc -target x86_64-linux-eng" CFLAGS="-Wno-format-extra-args -Wno-uninitialized" make defconfig toybox     
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \        
        -DPROJECT_NAME=\"ToyBox\"                                                
scripts/genconfig.sh                                                             
kconfig/conf -D /dev/null Config.in > /dev/null                                  
scripts/make.sh                                                                  
Generate headers from toys/*/*.c...                                              
generated/newtoys.h Library probe.......                                         
Make generated/config.h from .config.                                            
generated/flags.h generated/globals.h generated/help.h                           
Compile toybox.................................................................................................................................................................                                                                    
[rich@dev toybox-0.6.1]$ file toybox
toybox: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6d60409b768431310a5ce6085e595b6a98a66d45, stripped                  
[rich@dev toybox-0.6.1]$ size toybox
   text    data     bss     dec     hex filename
 376786    6192   22152  405130   62e8a toybox                                   
[rich@dev toybox-0.6.1]$ 

Cool stuff.

Building clang/LLVM with ELLCC Pre-compiled Binaries

ELLCC has periodic pre-compiled binary releases that can be download and installed on various Linux hosts. A coworker wanted to take advantage of that and use ELLCC to bootstrap clang/LLVM instead of GCC. The steps described here can be used to build clang/LLVM on systems where either no GCC exists, or where the version of GCC that’s installed isn’t new enough to handle clang/LLVM C++11 code base.

An advantage of using the pre-compiled ELLCC binaries is that they are statically linked and can be run on just about any new-ish Linux box (post about 2.16).

Here’s how I build clang/LLVM using ELLCC. First I created a directory to work in:

sh-4.3$ mkdir clang
sh-4.3$ cd clang/

Then I downloaded an ELLCC binary tarball from the download page and un-tared it:

sh-4.3$ tar xvfp ~/Downloads/ellcc-x86_64-linux-eng-0.1.21.tgz

This created the ellcc directory and the subdirectories containing the ELLCC release. Then I got the latest LLVM and clang sources:

sh-4.3$ svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
sh-4.3$ cd llvm/tools/
sh-4.3$ svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
sh-4.3$ cd ../..

My coworker ran into problems because currently ELLCC only supports static linking and parts of clang/LLVM are unconditionally built as shared libraries. Also, there are a few small incompatibilities resulting from the fact that ELLCC uses musl as its standard C library. A small set of patches are needed to make the stock clang/LLVM sources compile with ELLCC. Thus is the patch file: http://ellcc.org/clang-ecc.patch. You can download and apply it like this:

sh-4.3$ cd llvm
sh-4.3$ wget http://ellcc.org/clang-ecc.patch
sh-4.3$ patch -p0 < clang-ecc.patch
sh-4.3$ cd ..

Now we're ready to build:

sh-4.3$ mkdir llvm-build
sh-4.3$ cd llvm-build/
sh-4.3$ CC=/home/rich/ellcc/bin/ecc CXX=/home/rich/ellcc/bin/ecc++ CPP="/home/rich/ellcc/bin/ecc -E" ../llvm-ecc-src/configure --enable-optimized --enable-shared=no
sh-4.3$ make -j 6

Your configure line will be a little different if you're running tcsh: You'll need to set CC, etc. as environment variables with setenv. Also, I use "make -j 6" because I happen to have a 6 core system. You'll want to adjust for your system.

After the build, the bin directory looks like this:

sh-4.3$ ls Release+Asserts/bin/
arcmt-test    lli-child-target  llvm-link            llvm-stress
bugpoint      llvm-ar           llvm-lit             llvm-symbolizer
c-arcmt-test  llvm-as           llvm-lto             llvm-tblgen
c-index-test  llvm-bcanalyzer   llvm-mc              not
clang         llvm-config       llvm-mcmarkup        obj2yaml
clang++       llvm-cov          llvm-nm              opt
clang-check   llvm-c-test       llvm-objdump         sancov
clang-format  llvm-cxxdump      llvm-pdbdump         scan-build
clang-tblgen  llvm-diff         llvm-PerfectShuffle  scan-view
count         llvm-dis          llvm-profdata        verify-uselistorder
diagtool      llvm-dsymutil     llvm-ranlib          yaml2obj
FileCheck     llvm-dwarfdump    llvm-readobj         yaml-bench
fpcmp         llvm-dwp          llvm-rtdyld
llc           llvm-extract      llvm-size
lli           llvm-lib          llvm-split

The clang created has been statically linked:

sh-4.3$ file Release+Asserts/bin/clang
Release+Asserts/bin/clang: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=2465acda3f4acf5c88e0491f984f4f4048b47015, not stripped

If you use that clang to rebuild, the result will use the normal system shared libraries.

When everything is finished building you can do a "make install" to install clang and the LLVM tools in the default location which will be under /usr/local unless you supply a --prefix option to the configure command. You now have a compiled version of clang that uses your system header files and libraries. Unlike ecc, it also can create and use shared libraries and in fact uses them by default.