Author Archives: rich

Cross Building Linux With ELLCC: Part 3 – Other Flavors of Pi

Update: Cross Building Linux With ELLCC: Part 4 – Building Linux on the Mac
In my last post I described using ELLCC to build Linux for the Raspberry Pi B+. In ELLCC version 0.1.24 I added a script to build Linux for both the Pi B+ and the Pi 2. As a bonus, the script also builds a bunch of userland programs

First grab the latest binary release:

[~] dev% wget http://ellcc.org/releases/ellcc-x86_64-linux-eng-0.1.24.tgz
[~] dev% tar xvfp ellcc-x86_64-linux-eng-0.1.24.tgz

Now decide what target to build:

[~] dev% cd ellcc/libecc/ecclinux/
[~/ellcc/libecc/ecclinux] dev% ls
00_rpilinux-4.1.15.patch   bin/    etc/            makefile      sbin/
99_llvmlinux-4.1.15.patch  build*  iproute2.patch  ntfs3g.patch  sinit/
[~/ellcc/libecc/ecclinux] dev% ./build 
usage: ./build [options] target
  options:
    -nokernel           Don't build the Linux kernel
    -clean              Clean up a previous build
  targets:
    help                This message
    bcmrpi              Raspberry Pi    (ARMv6)
    bcm2709             Raspberry Pi 2  (ARMv7)
[~/ellcc/libecc/ecclinux] dev%

Run the build command again, with the board you need as an argument (I’m going to time the build with time):

[~/ellcc/libecc/ecclinux] dev% time ./build bcm2709
... (lots of output removed)
4984.143u 435.744s 22:52.57 394.8%      0+0k 704+4154968io 2pf+0w
[~/ellcc/libecc/ecclinux] dev% 

As SpongeBob would say, “23 minutes later.” When the build is finished, the ecclinux directory looks like this:

[~/ellcc/libecc/ecclinux] dev% ls
00_rpilinux-4.1.15.patch   bin/       etc/            linux-4.1.15/  sbin/
99_llvmlinux-4.1.15.patch  build*     image/          makefile       sinit/
arm-linux-engeabihf/       download/  iproute2.patch  ntfs3g.patch
[~/ellcc/libecc/ecclinux] dev%

There are three new directories: download/ contains tar files that have been downloaded in case you need them again, arm-linux-engeabihf/ is where the build was made for the ARMv7, and image/ is where the binaries were installed. Let’s take a look at image/arm-linux-engeabihf/:

[~/ellcc/libecc/ecclinux] dev% ls image/arm-linux-engeabihf/
bcm2709/  bin/  etc/  lib/  sbin/  usr/  var/
[~/ellcc/libecc/ecclinux] dev% 

The bcm2709/ directory is what will be in the /boot directory in the final Raspberry Pi 2 SD image. It has Pi 2 specific files in it, including the kernel:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% ls bcm2709/
dtbs/  System.map-4.1.15-v7  vmlinuz-4.1.15-v7
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

Here’s what the bin and sbin directories look like:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% ls bin/ sbin/ 
bin/:
[*           ed*           ksh*         nltrigger*   sha256sum*    tty*
basename*    eject*        last*        nohup*       sha512sum*    umount*
cal*         env*          lastlog*     od*          sleep*        uname*
cat*         ex@           link*        pagesize*    smdev*        unexpand*
chattr*      expand*       ln*          passwd*      sort*         uniq*
chgrp*       expr*         logger*      paste*       split*        unlink*
chmod*       fallocate*    login*       pidof*       sponge*       unshare*
chown*       false*        logname*     pivot_root*  stat*         uptime*
chroot*      find*         ls*          printenv*    strings*      uudecode*
chvt*        flock*        lsattr*      printf*      su*           uuencode*
cksum*       fold*         lsmod*       ps*          swaplabel*    uuidgen*
clear*       free*         lsusb*       pwd*         swapoff*      view@
cmp*         freeramdisk*  md5sum*      readahead*   swapon*       vim*
cols*        fsfreeze*     mesg*        readlink*    switch_root*  vimdiff@
comm*        getconf*      mkdir*       renice*      sync*         vimtutor*
cp*          getty*        mkfifo*      respawn*     sysctl*       vtallow*
cron*        grep*         mknod*       rm*          tail*         watch*
ctrlaltdel*  halt*         mkswap*      rmdir*       tar*          wc*
cut*         head*         mktemp*      rmmod*       tee*          which*
date*        hostname*     mount*       run_nldev*   test*         who*
dd*          hwclock*      mountpoint*  rview@       tftp*         whoami*
df*          id*           mv*          rvim@        time*         xargs*
dirname*     insmod*       nice*        sed*         touch*        xxd*
dmesg*       join*         nl*          seq*         tr*           yes*
du*          kill*         nldev*       setsid*      true*
echo*        killall5*     nlmon*       sha1sum*     truncate*

sbin/:
badblocks*   findfs@        mke2fs*            ntfs-3g.secaudit*  resize2fs*
blkid*       fsck*          mkfs.ext2@         ntfs-3g.usermap*   routef*
bridge*      fsck.ext2@     mkfs.ext3@         ntfscat*           routel*
ctstat@      fsck.ext3@     mkfs.ext4@         ntfsclone*         rtacct*
debugfs*     fsck.ext4@     mkfs.ext4dev@      ntfscluster*       rtmon*
dumpe2fs*    fsck.ext4dev@  mkfs.ntfs@         ntfscmp*           rtpr*
e2freefrag*  genl*          mklost+found*      ntfscp*            rtstat@
e2fsck*      ifcfg*         mkntfs*            ntfsfix*           sdhcp*
e2image*     ifstat*        mount.lowntfs-3g@  ntfsinfo*          sinit*
e2label@     ip*            mount.ntfs-3g@     ntfslabel*         ss*
e2undo*      lnstat*        nstat*             ntfsls*            tune2fs*
e4defrag*    logsave*       ntfs-3g*           ntfsresize*        uuidd*
filefrag*    lowntfs-3g*    ntfs-3g.probe*     ntfsundelete*
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

Not bad for 23 minutes of work!

If you want to try the kernel on your Pi 2, you need to copy vmlinuz-4.1.15-v7 to the Rasbian /boot directory as kernel7.img. In my case, my Pi 2 is at 192.124.43.100 so I can do this:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% scp bcm2709/vmlinuz-4.1.15-v7 root@192.124.43.100:/boot/kernel7.img
root@192.124.43.100's password: 
vmlinuz-4.1.15-v7                              100% 4355KB   4.3MB/s   00:00    
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

After a reboot:

[rich@dev ~]$ ssh root@192.124.43.100
root@192.124.43.100's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Jan  1 17:28:43 2016 from 192.124.43.2
root@raspberrypi:~# dmesg | less
[    0.000000] Booting Linux on physical CPU 0xf00
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1.15-v7 (rich@dev) (ecc 0.1.24 based on clang version 3.8.0 (trunk) (based on LLVM 3.8.0svn)) #1 SMP Mon Jan 4 20:45:34 CST 2016

Not bad. To complete the kernel update you’ll also want to update the kernel modules:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% ls lib/modules/4.1.15-v7/
kernel/            modules.builtin      modules.dep.bin  modules.softdep
modules.alias      modules.builtin.bin  modules.devname  modules.symbols
modules.alias.bin  modules.dep          modules.order    modules.symbols.bin
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% scp -r lib/modules/4.1.15-v7/ root@192.124.43.100:/lib/modules
root@192.124.43.100's password: 
Permission denied, please try again.
root@192.124.43.100's password: 
modules.symbols                                100%  146KB 146.2KB/s   00:00    
modules.dep                                    100%  131KB 131.1KB/s   00:00  
...
modules.builtin.bin                            100% 5903     5.8KB/s   00:00    
modules.builtin                                100% 4341     4.2KB/s   00:00    
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

Reboot again and even the mouse and keyboard work.

In Part 4, I’ll show how to make the SD image and use the userland programs that we’ve built. In the mean time you can copy them to the Pi’s SD card and use them.

Cross Building Linux With ELLCC: Part 2 – Raspberry Pi B+

Update: Cross Building Linux With ELLCC: Part 3 – Other Flavors of Pi
In a previous post I tested the waters to see how close ELLCC is to cross compiling the Linux kernel. That post was mostly a theoretical exercise to see what it would take. Over the holidays I decided that I needed to do something a little more concrete. My son, a 10 year old obsessed with Minecraft, got a Raspberry Pi for Christmas. My nefarious plan is to divert his attention away from Minecraft at least long enough to make his Pi and camera module into something he can use to spy on the family. My other nefarious plan is to use ELLCC to build the Linux kernel that we’ll be running on his Pi.

In the previous post I used the source from the LLVMLinux site. This time I started with the kernel source for the Raspberry Pi. The good news is that the source can be built for the Pi. The bad news is that it doesn’t have all the patches need to compile with a clang based compiler. It turned out that the build wasn’t too bad: I had Christmas Eve and Day off and in between family events and Santa preparations I was able to massage the source enough to boot the kernel on our Pi.

To get started I followed the instructions on Linux From Scratch on the Raspberry Pi.
Here are the exact steps I took. First I created a new directory to hold the ELLCC tool chain and the Raspberry Linux kernel source, then I downloaded and untar’d both:

mkdir ecclinux
cd ecclinux
wget http://ellcc.org/releases/ellcc-x86_64-linux-eng-0.1.23.tgz
tar xvfp ellcc-x86_64-linux-eng-0.1.23.tgz
wget https://github.com/raspberrypi/linux/archive/rpi-4.1.y.tar.gz
tar xvfp rpi-4.1.y.tar.gz

I had to change the kernel sources slightly during my build. I saved a patch
file, here’s how to apply it:

patch -p0 < ellcc/libecc/llvmlinux/ecc-rpi-4.1.y.patch

There is a special little 'm' makefile that helps with the build. Copy
it into the top level of the kernel source directory:

cd linux-rpi-4.1.y/
cp ../ellcc/libecc/llvmlinux/makefile .

Now the actual build is easy:

make bcmrpi_defconfig
make

I have my Raspberry Pi on my home network at 192.124.43.98. I also allowed ssh
and scp root access by follwoing the instructions here. This lets me
just copy the new kernel to my Pi:

scp ./arch/arm/boot/zImage root@192.124.43.98:/boot/kernel.img

Now I can ssh to the Pi and reboot:

[~] dev% ssh root@192.124.43.98
root@192.124.43.98's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@raspberrypi:~# reboot
Connection to 192.124.43.98 closed by remote host.
Connection to 192.124.43.98 closed.
[~] dev%

I ping the Pi until it reboots and log in again:

[~] dev% ssh root@192.124.43.98
root@192.124.43.98's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Dec 26 20:27:15 2015 from 192.124.43.2
root@raspberrypi:~# dmesg | less
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1.15 (rich@dev) (ecc 0.1.23 based on clang version 3.8.0 (trunk) (based on LLVM 3.8.0svn)) #1 Sat Dec 26 11:32:33 CST 2015
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
...

Very nice!

Using ELLCC to (Cross) Compile Toybox

Toybox combines many common Linux command line utilities together into a single BSD-licensed executable. I wanted to to see how hard it would be to compile and cross compile it using ELLCC. It turned out to be pretty easy after I added a few Linux specific header files. These are the files I added:

[~/ellcc] dev% ls ~/ellcc/libecc/include/linux
fs.h  if_vlan.h  loop.h  nbd.h  rfkill.h  rtc.h  sched.h  sockios.h  soundcard.h

The needed include files are in the ELLCC source tree now and are available in the 0.1.22 binary release. Here’s all you need to do to build Toybox:

[rich@dev ~]$ mkdir toybox 
[rich@dev ~]$ cd toybox
[rich@dev toybox]$ wget http://landley.net/toybox/downloads/toybox-0.6.1.tar.gz
--2015-12-23 17:02:37--  http://landley.net/toybox/downloads/toybox-0.6.1.tar.gz
Resolving landley.net (landley.net)... 208.113.171.142
Connecting to landley.net (landley.net)|208.113.171.142|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 736371 (719K) [application/x-tar]
Saving to: ‘toybox-0.6.1.tar.gz’

toybox-0.6.1.tar.gz  100%[=====================>] 719.11K   845KB/s   in 0.9s   

2015-12-23 17:02:38 (845 KB/s) - ‘toybox-0.6.1.tar.gz’ saved [736371/736371]

[rich@dev toybox]$ tar xfp toybox-0.6.1.tar.gz 
[rich@dev toybox]$ cd toybox-0.6.1/
[rich@dev toybox-0.6.1]$ CC="/home/rich/ellcc/bin/ecc -target armv6-linux-engeabihf" CFLAGS="-Wno-format-extra-args -Wno-uninitialized" make defconfig toybox
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
        -DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/null Config.in > /dev/null
scripts/make.sh
Generate headers from toys/*/*.c...
generated/newtoys.h Library probe.......
Make generated/config.h from .config.
generated/flags.h generated/globals.h generated/help.h
Compile toybox................................................................................................................................................................strip: Unable to recognise the format of the input file `toybox_unstripped'
strip failed, using unstripped
.

Oops. The strip failed because the standard x86_64 Linux strip doesn’t understand ARM object file formats. We can use ecc-strip:

[rich@dev toybox-0.6.1]$ ~/ellcc/bin/ecc-strip -o toybox toybox_unstripped
[rich@dev toybox-0.6.1]$ file toybox
toybox: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=11e423037c6fb0e4ee10ef5f53a421fca86a81e3, stripped               
[rich@dev toybox-0.6.1]$ size toybox                                             
   text    data     bss     dec     hex filename                                 
 401916    3208   21020  426144   680a0 toybox  

Lest’s send it over to my trusty Raspberry Pi:

                                 
[rich@dev toybox-0.6.1]$ scp toybox rich@192.124.43.98:                                                   
rich@192.124.43.98's password:                                                   
scp: ./toybox: Permission denied                                                 
[rich@dev toybox-0.6.1]$ ssh rich@192.124.43.98
rich@192.124.43.98's password:                                                   
                                                                                 
The programs included with the Debian GNU/Linux system are free software;        
the exact distribution terms for each program are described in the               
individual files in /usr/share/doc/*/copyright.                                  
                                                                                 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent                
permitted by applicable law.                                                     
Last login: Sun Dec 20 19:20:36 2015 from 192.124.43.2   

What commands does Toybox have?

                        
rich@raspberrypi:~$ ./toybox                                                     
acpi base64 basename blkid blockdev bunzip2 bzcat cal cat catv chattr            
chgrp chmod chown chroot chvt cksum clear cmp comm count cp cpio cut             
date df dirname dmesg dos2unix du echo egrep eject env expand factor             
fallocate false fgrep find flock free freeramdisk fstype fsync grep              
groups halt head help hexedit hostid hostname hwclock id ifconfig                
inotifyd insmod install ionice iorenice kill killall killall5 link               
ln login logname losetup ls lsattr lsmod lspci lsusb makedevs md5sum             
mix mkdir mkfifo mknod mkpasswd mkswap mktemp modinfo mount mountpoint           
mv nbd-client nc netcat nice nl nohup nproc nsenter od oneit partprobe           
passwd paste patch pidof pivot_root pmap poweroff printenv printf                
ps pwd pwdx readahead readlink realpath reboot renice reset rev rfkill           
rm rmdir rmmod sed seq setsid sh sha1sum shred sleep sort split stat             
strings su swapoff swapon switch_root sync sysctl tac tail taskset               
tee time timeout touch toysh true truncate tty umount uname uniq unix2dos        
unlink uptime usleep uudecode uuencode vconfig vmstat w wc which who             
whoami xargs xxd yes                                                             
rich@raspberrypi:~$ ./toybox ls -l
total 3112                                                                       
-rwxr-x--x 1 rich rich  141580 2015-12-20 19:22 hello                            
-r-xr-xr-x 1 rich rich 1446476 2015-12-23 21:21 toybox   

Now let’s build for the x86_64:

                        
rich@raspberrypi:~$ logout                                                                           
Connection to 192.124.43.98 closed.                                              
[rich@dev toybox-0.6.1]$ make clean                                              
rm -f kconfig/zconf.hash.c kconfig/zconf.tab.c kconfig/lex.zconf.c kconfig/conf kconfig/mconf                                                                     
rm -rf toybox toybox_unstripped generated change .singleconfig*                  
[rich@dev toybox-0.6.1]$ CC="/home/rich/ellcc/bin/ecc -target x86_64-linux-eng" CFLAGS="-Wno-format-extra-args -Wno-uninitialized" make defconfig toybox     
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \        
        -DPROJECT_NAME=\"ToyBox\"                                                
scripts/genconfig.sh                                                             
kconfig/conf -D /dev/null Config.in > /dev/null                                  
scripts/make.sh                                                                  
Generate headers from toys/*/*.c...                                              
generated/newtoys.h Library probe.......                                         
Make generated/config.h from .config.                                            
generated/flags.h generated/globals.h generated/help.h                           
Compile toybox.................................................................................................................................................................                                                                    
[rich@dev toybox-0.6.1]$ file toybox
toybox: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6d60409b768431310a5ce6085e595b6a98a66d45, stripped                  
[rich@dev toybox-0.6.1]$ size toybox
   text    data     bss     dec     hex filename
 376786    6192   22152  405130   62e8a toybox                                   
[rich@dev toybox-0.6.1]$ 

Cool stuff.

Building clang/LLVM with ELLCC Pre-compiled Binaries

ELLCC has periodic pre-compiled binary releases that can be download and installed on various Linux hosts. A coworker wanted to take advantage of that and use ELLCC to bootstrap clang/LLVM instead of GCC. The steps described here can be used to build clang/LLVM on systems where either no GCC exists, or where the version of GCC that’s installed isn’t new enough to handle clang/LLVM C++11 code base.

An advantage of using the pre-compiled ELLCC binaries is that they are statically linked and can be run on just about any new-ish Linux box (post about 2.16).

Here’s how I build clang/LLVM using ELLCC. First I created a directory to work in:

sh-4.3$ mkdir clang
sh-4.3$ cd clang/

Then I downloaded an ELLCC binary tarball from the download page and un-tared it:

sh-4.3$ tar xvfp ~/Downloads/ellcc-x86_64-linux-eng-0.1.21.tgz

This created the ellcc directory and the subdirectories containing the ELLCC release. Then I got the latest LLVM and clang sources:

sh-4.3$ svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
sh-4.3$ cd llvm/tools/
sh-4.3$ svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
sh-4.3$ cd ../..

My coworker ran into problems because currently ELLCC only supports static linking and parts of clang/LLVM are unconditionally built as shared libraries. Also, there are a few small incompatibilities resulting from the fact that ELLCC uses musl as its standard C library. A small set of patches are needed to make the stock clang/LLVM sources compile with ELLCC. Thus is the patch file: http://ellcc.org/clang-ecc.patch. You can download and apply it like this:

sh-4.3$ cd llvm
sh-4.3$ wget http://ellcc.org/clang-ecc.patch
sh-4.3$ patch -p0 < clang-ecc.patch
sh-4.3$ cd ..

Now we're ready to build:

sh-4.3$ mkdir llvm-build
sh-4.3$ cd llvm-build/
sh-4.3$ CC=/home/rich/ellcc/bin/ecc CXX=/home/rich/ellcc/bin/ecc++ CPP="/home/rich/ellcc/bin/ecc -E" ../llvm-ecc-src/configure --enable-optimized --enable-shared=no
sh-4.3$ make -j 6

Your configure line will be a little different if you're running tcsh: You'll need to set CC, etc. as environment variables with setenv. Also, I use "make -j 6" because I happen to have a 6 core system. You'll want to adjust for your system.

After the build, the bin directory looks like this:

sh-4.3$ ls Release+Asserts/bin/
arcmt-test    lli-child-target  llvm-link            llvm-stress
bugpoint      llvm-ar           llvm-lit             llvm-symbolizer
c-arcmt-test  llvm-as           llvm-lto             llvm-tblgen
c-index-test  llvm-bcanalyzer   llvm-mc              not
clang         llvm-config       llvm-mcmarkup        obj2yaml
clang++       llvm-cov          llvm-nm              opt
clang-check   llvm-c-test       llvm-objdump         sancov
clang-format  llvm-cxxdump      llvm-pdbdump         scan-build
clang-tblgen  llvm-diff         llvm-PerfectShuffle  scan-view
count         llvm-dis          llvm-profdata        verify-uselistorder
diagtool      llvm-dsymutil     llvm-ranlib          yaml2obj
FileCheck     llvm-dwarfdump    llvm-readobj         yaml-bench
fpcmp         llvm-dwp          llvm-rtdyld
llc           llvm-extract      llvm-size
lli           llvm-lib          llvm-split

The clang created has been statically linked:

sh-4.3$ file Release+Asserts/bin/clang
Release+Asserts/bin/clang: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=2465acda3f4acf5c88e0491f984f4f4048b47015, not stripped

If you use that clang to rebuild, the result will use the normal system shared libraries.

When everything is finished building you can do a "make install" to install clang and the LLVM tools in the default location which will be under /usr/local unless you supply a --prefix option to the configure command. You now have a compiled version of clang that uses your system header files and libraries. Unlike ecc, it also can create and use shared libraries and in fact uses them by default.

Cross Building Linux With ELLCC: Part 1

Update: Cross Building Linux With ELLCC: Part 2 – Raspberry Pi B+
ELLCC is a clang/LLVM based cross compilation tool chain. It is available pre-compiled for several different host systems and unlike GCC can generate programs for several targets with one set of tools. This week I thought it would be fun to try to compiler the Linux kernel using ELLCC. In this post I’ll build the patched Linux kernel from the LLVMLinux project git repository.

You need to be using ELLCC version 0.1.18 or ELLCC svn-5503 or later to build Linux. Download one of the ELLCC binary releases or follow the instructions for building ELLCC from source.

Create a directory next to the ELLCC directory. For example if the ELLCC directory is ~/ellcc:

[~] dev% mkdir ~/llvmlinux
[~] dev% cd ~/llvmlinux

We’re going to follow the instructions on the LLVMLinux site to get the pre-patched Linux kernel sources:

[~/llvmlinux] dev% git clone git://git.linuxfoundation.org/llvmlinux/kernel.git

There is a copy of the little ‘m’ makefile modified for ELLCC in the ELLCC distribution. Make a symbolic link to it:

[~/llvmlinux/kernel] dev% ln -s ../../ellcc/libecc/llvmlinux/makefile .

This little “m” makefile is set up to build Linux for the ARM. You can modify it to select another target.

Configure the kernel:

[~/llvmlinux/kernel] dev% make menuconfig

For now, under System Type, choose Dummy Virtual Machine.
Disable the following devices (LLVM bugs submitted):

  • Device Drivers->Serial ATA and Parallel ATA drivers (libata)->OPTI FireStar PATA support (Very Experimental) – LLVM bug 25330
  • Device Drivers->Network device support->Ethernet driver support->Exar Xframe 10Gb Ethernet Adapter – LLVM bug 25332
  • Device Drivers->Sound card support->Advanced Linux Sound Architecture->PCI sound devices->Aztech AZF3328 / PCI168 – LLVM bug 25330
  • Kernel hacking->Build targets in Documentation/ tree

Exit and save the configuration.

Build Linux:

[~/llvmlinux] dev% cd kernel/
[~/llvmlinux/kernel] dev% make
make[1]: Entering directory '/home/rich/llvmlinux/kernel'
  CHK     include/config/kernel.release
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
make[2]: 'include/generated/mach-types.h' is up to date.
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     include/generated/uapi/linux/version.h
  Kernel: arch/arm/boot/Image is ready
  LD      arch/arm/boot/compressed/vmlinux
following symbols must have non local/private scope:
$d
$d
$d
arch/arm/boot/compressed/Makefile:190: recipe for target 'arch/arm/boot/compressed/vmlinux' failed
make[3]: *** [arch/arm/boot/compressed/vmlinux] Error 1
arch/arm/boot/Makefile:52: recipe for target 'arch/arm/boot/compressed/vmlinux' failed
make[2]: *** [arch/arm/boot/compressed/vmlinux] Error 2
arch/arm/Makefile:310: recipe for target 'zImage' failed
make[1]: *** [zImage] Error 2
make[1]: Leaving directory '/home/rich/llvmlinux/kernel'
makefile:71: recipe for target 'all' failed
make: *** [all] Error 2
[~/llvmlinux/kernel] dev%

So the kernel and a bunch of modules get built but subsequent processing fails. In the next part of this post I’ll try to track down what the problem is.

A Little Example of ELLCC Under Windows

As I mentioned previously, it is easy to use ELLCC to compile C and C++ programs under Windows. Now that the latest releases of ELLCC come with the MinGW-w64 header files and libraries, all you have to do is download and install a Windows binary release and you have a full C/C++ development environment including the clang/LLVM based ecc compiler, binutils, and GDB. The tool chain can target Windows as well as various Linux systems as a cross compiler. You can also run ELLCC under Linux to target Windows. In fact, the Windows release has been compiled under Linux.

Here is a simple example of compiling and debugging a simple “hello world” program using the Windows cmd shell:

                                                                            
Microsoft Windows [Version 6.3.9600]                                             
(c) 2013 Microsoft Corporation. All rights reserved.                             
                                                                                 
C:\Users\rich>cd \                                                               
                                                                                 
C:\>cd ellcc                                                                     
                                                                                 
C:\ellcc>bin/ecc examples\hello\main.c -g                                        
'bin' is not recognized as an internal or external command,                      
operable program or batch file.                                                  
                                                                                 
C:\ellcc>bin\ecc examples\hello\main.c -g                                        
                                                                                 
C:\ellcc>bin/ecc-gdb a.exe                                                       
'bin' is not recognized as an internal or external command,                      
operable program or batch file.                                                  
                                                                                 
C:\ellcc>bin\ecc-gdb a.exe                                                       
GNU gdb (GDB) 7.7                                                                
Copyright (C) 2014 Free Software Foundation, Inc.                                
License GPLv3+: GNU GPL version 3 or later     
This is free software: you are free to change and redistribute it.               
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"       
and "show warranty" for details.                                                 
This GDB was configured as "i386-mingw32".                                       
Type "show configuration" for configuration details.                             
For bug reporting instructions, please see:                                      
.                                         
Find the GDB manual and other documentation resources online at:                 
.                                
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.exe...done.
(gdb) break main
Breakpoint 1 at 0x401571: file examples\hello\main.c, line 5.
(gdb) run
Starting program: C:\ellcc\a.exe
[New Thread 2432.0xc38]
warning: Can not parse XML library list; XML support was disabled at compile tim
e

Breakpoint 1, main () at examples\hello\main.c:5
5           printf("hello world\n");
(gdb) list
1       #include <stdio.h>
2
3       int main()
4       {
5           printf("hello world\n");
6       }
(gdb) c
Continuing.
hello world
[Inferior 1 (process 2432) exited normally]
(gdb)

Since I’m a *nix guy I’m more comfortable with a shell, as you can see by my ‘/’ vs. ‘\’ fumbling above. Here is the same thing done using the MSYS command shell:


rich@VirtualWin8-64 MSYS ~
$ cd /c/ellcc

rich@VirtualWin8-64 MSYS /c/ellcc
$ bin/ecc examples/hello/main.c -g

rich@VirtualWin8-64 MSYS /c/ellcc
$ bin/ecc-gdb a.exe
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.exe...done.
(gdb) break main
Breakpoint 1 at 0x401571: file examples/hello\main.c, line 5.
(gdb) run
Starting program: C:\ellcc\a.exe
[New Thread 1936.0x538]
warning: Can not parse XML library list; XML support was disabled at compile time

Breakpoint 1, main () at examples/hello\main.c:5
5           printf("hello world\n");
(gdb) c
Continuing.
hello world
[Inferior 1 (process 1936) exited normally]
(gdb)

Cool stuff.

ELLCC now has 64 bit ARM AArch64 Support

A new binary release of the ELLCC cross compilation tool chain is available.
ELLCC is a pre-packaged set of tools designed to support cross compilation
for a variety of target processors.

ELLCC includes:

  • ecc (a clang/LLVM based C/C++ compiler)
  • binutils
  • GDB

The ELLCC run-time libraries are supplied pre-built for all targets and
are all covered with BSD-like licenses:

  • libc++ The C++ standard library
  • musl The POSIX compliant C library
  • compiler-rt Low level compiler support
  • expat XML parsing
  • libedit Command history
  • ncurses Terminal handling
  • zlib Compression

ELLCC is entirely self hosting.

From the ChangeLog:

version 0.1.11:

* Added aarch64 support.
* Update to musl 1.1.7.
* Update to LLVM r232721.

Binary tarballs are available via FTP or HTTP from the download page.

The tool chain can build programs for ARM, Mips, PowerPC, and x86 Linux and stand-alone targets and x86 Windows systems.

Pre-built binaries are available for ARM, Mips, PowerPC, and x86 Linux host systems as well as x86 Windows and Mac OS X hosts.

All host systems can build programs for all the supported targets.

Available targets are listed here

Building mbed TLS with ELLCC

When I saw the recent announcement that PolarSSL is now owned by ARM and called mbed TLS, I thought it would be interesting to see how easily it could be built using ELLCC.

I downloaded the latest release (mbedtls-1.3.10-gpl.tgz) from the web site on to my x86_64 Linux box. A quick look at the README.txt file showed that mbed TLS can be built with make, cmake, or Microsoft Visual Studio. I opted for make.

It turns out that using ELLCC to cross build this library is very easy. Here is the command line for ARM:

[~/mbedtls] dev% tar xvfpz ~/Downloads/mbedtls-1.3.10-gpl.tgz
[~/mbedtls] dev% cd mbedtls-1.3.10/
[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target arm-linuxlld-engeabihf"

(I happen to have ELLCC installed in my home directory at ~/ellcc.) This single command built the library as well as a test suite. I have binfmt installed on my Linux box, which a cool way of telling my x86_64 Linux kernel how to run foreign executables (in this case ARM binaries). So I decided to try running the test suite.

[~/mbedtls/mbedtls-1.3.10] dev% make check
Running checks (Success if all tests PASSED)
 - test_suite_aes.ecb
   PASSED (77 / 77 tests (0 skipped))

 - test_suite_aes.cbc

[many tests deleted]

 - test_suite_version
   PASSED (5 / 5 tests (0 skipped))

[~/mbedtls/mbedtls-1.3.10] dev% 

All the tests passed. Pretty easy, huh? To prove I’ve really built for ARM, I used the file command:

[~/mbedtls/mbedtls-1.3.10] dev% file tests/test_suite_aes.cbc
tests/test_suite_aes.cbc: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, not stripped
[~/mbedtls/mbedtls-1.3.10] dev%

So I thought, “This is cool, how about using ELLCC’s MinGW64 support to try a build for Windows?”. The answer was only slightly more complicated.

First I tried:

[~/mbedtls/mbedtls-1.3.10] dev% make clean
[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target x86_64-w64-mingw32"

This didn’t work out so well. I got:

...
  CC    x509write_crt.c
  CC    x509write_csr.c
  CC    xtea.c
  AR    libmbedtls.a
ar: creating libmbedtls.a
  RL    libmbedtls.a
  LN    libpolarssl.a -> libmbedtls.a
  CC    aes/aescrypt2.c
../library/libmbedtls.a: error adding symbols: Archive has no index; run ranlib to add one
ecc: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [aes/aescrypt2] Error 1
make: *** [all] Error 2
[~/mbedtls/mbedtls-1.3.10] dev%

Apparently my system ar command doesn’t handles Windows PE object files. No problem. I’ll just tell make to use ELLCC’s ecc-ar, which has been built with PE format support:

[~/mbedtls/mbedtls-1.3.10] dev% make clean
[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target x86_64-w64-mingw32" AR=~/ellcc/bin/ecc-ar

This time, building the test suite failed with errors like:

...
../library/libmbedtls.a(net.o):(.text+0x4ce): undefined reference to `__imp_WSAGetLastError'
../library/libmbedtls.a(net.o):(.text+0x503): undefined reference to `__imp_shutdown'
../library/libmbedtls.a(net.o):(.text+0x513): undefined reference to `__imp_closesocket'
ecc: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [pkey/dh_client] Error 1
make: *** [all] Error 2
[~/mbedtls/mbedtls-1.3.10] dev

I seem to be missing a library needed for linking. I needed to find which library was missing, so I used the trusty nm command to find out:

[~/ellcc/libecc/mingw/x86_64-w64-mingw32/sys-root/mingw/lib] dev% ~/ellcc/bin/ecc-nm -A *.a | grep __imp_closesocket 
/home/rich/ellcc/bin/ecc-nm: libruntimeobject.a: File format not recognized
libws2_32.a:dscdbs00149.o:0000000000000000 I __imp_closesocket
libwsock32.a:divxs00037.o:0000000000000000 I __imp_closesocket
[~/ellcc/libecc/mingw/x86_64-w64-mingw32/sys-root/mingw/lib] dev%

Looks like I have two choices. libws2_32.a sounds newer (grin), so I’ll try that. The tests/Makefile has the line:

LDFLAGS += -L../library -lmbedtls $(SYS_LDFLAGS)

All I have to do is add a define for SYS_LDFLAGS:

[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target x86_64-w64-mingw32" AR=~/ellcc/bin/ecc-ar SYS_LDFLAGS=-lws2_32
...
  Generate      test_suite_rsa.c
  CC            test_suite_rsa.c
  Generate      test_suite_shax.c
  CC            test_suite_shax.c
  Generate      test_suite_x509parse.c
  CC            test_suite_x509parse.c
  Generate      test_suite_x509write.c
  CC            test_suite_x509write.c
  Generate      test_suite_xtea.c
  CC            test_suite_xtea.c
  Generate      test_suite_version.c
  CC            test_suite_version.c
[~/mbedtls/mbedtls-1.3.10] dev% 

Let’s try the file command again.

file tests/test_suite_aes.cbc
tests/test_suite_aes.cbc: PE32+ executable (console) x86-64, for MS Windows
[~/mbedtls/mbedtls-1.3.10] dev%

Perfect. Now the test suite. Since binfmt can handle Windows executables and run them under Wine, things should just work.

[~/mbedtls/mbedtls-1.3.10] dev% make check
...
XTEA Encrypt CBC #4 ............................................... PASS
XTEA Encrypt CBC #5 ............................................... PASS
XTEA Encrypt CBC #6 ............................................... PASS
XTEA Decrypt CBC #1 ............................................... PASS
XTEA Decrypt CBC #2 ............................................... PASS
XTEA Decrypt CBC #3 ............................................... PASS
XTEA Decrypt CBC #4 ............................................... PASS
XTEA Decrypt CBC #5 ............................................... PASS
XTEA Decrypt CBC #6 ............................................... PASS
XTEA Selftest ..................................................... PASS


PASSED (25 / 25 tests (0 skipped))

 - test_suite_version
   Check compiletime library version ................................. PASS
Check runtime library version ..................................... PASS
Check for POLARSSL_VERSION_C ...................................... PASS
Check for POLARSSL_AES_C when already present ..................... PASS
Check for unknown define .......................................... PASS


PASSED (5 / 5 tests (0 skipped))

[~/mbedtls/mbedtls-1.3.10] dev% 

Voila!

Using ELLCC to Build for the Cortex-M3

This weekend I decided to work on getting ELLCC to support the ARM Cortex-M processors. The Cortex-M processors use the Thumb 16 bit instruction set, so adding M support involves adding a configuration file for a member of the Cortex-M family and then building the run-time libraries specifically for the target processor. To start I chose the Cortex-M3. Its configuration file currently looks like this:

[~/ellcc/libecc/config] dev% cat thumb-linux-engeabi
based_on: arm-ellcc-linux
compiler:
options:
- -target arm-ellcc-linux
- -mthumb
- -mcpu=cortex-m3
- -mfpu=none
- -mfloat-abi=softfp
linker:
static_crt1: $R/lib/thumb-linux-engeabi/crt1.o
dynamic_crt1: $R/lib/thumb-linux-engeabi/Scrt1.o
crtbegin: $R/lib/thumb-linux-engeabi/crtbegin.o
crtend: $R/lib/thumb-linux-engeabi/crtend.o
library_paths:
- -L$R/lib/thumb-linux-engeabi

With the config file in place, I attempted to build ELLCC’s standard C library musl for the M. Building a library for a new configuration is pretty easy:

[~/ellcc/libecc] dev% make thumb-linux-engeabi
[~/ellcc/libecc] dev% cd
[~] dev% cd ~/ellcc/libecc
[~/ellcc/libecc] dev% make thumb-linux-engeabi

Well, it isn’t quite that simple. The musl library has several assembly files that are written for the specific target processor. If the assembly language of the desired processor doesn’t match the sources, the sources have to be modified to support the new target. Luckily Thumb instructions are mostly a subset of 32 bit ARM instructions so the changes needed to build musl for the M were fairly minimal.

The next step to support a given processor is to compiler the compiler-rt library. This library contains support functions that handle operations that the processor doesn’t support directly and compiler generated built in functions. In the case of the M processors, this includes things like floating point functions since the M3 doesn’t have a floating point unit. Building compiler-rt was probably the most challenging part of this little project. I had never fully ported the compiler-rt build rules to use the new config files properly so rather than cobbling together something that could handle building for the M, I rewrote the build rules to properly support config files.

The last step is to build the C++ standard libraries, assuming you want C++ support. I ran into one little glitch here. The M doesn’t support 64 bit atomic operations so I had to modify those operations for the M.

Now that the libraries are built, a little experimentation is in order. I decided to try out a simple “hello world”. Here’s the source:

[~/ellcc/examples/hello] dev% cat main.c
#include <stdio.h>

int main()
{
printf("hello world\n");
}

The make files in the example directories can build for multiple targets as long as config files have been set up, so I built for the M:

[~/ellcc/examples/hello] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% ~/ellcc/bin/qemu-arm hello
hello world
[~/ellcc/examples/hello] dev%

Nice! I was curious how the Thumb instruction set affected code size so I did a side by side comparison:

[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
19540 156 1412 21108 5274 hello
[~/ellcc/examples/hello] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
rm -f *.o hello hello.bin hello.log elkconfig.ld
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
27432 156 1412 29000 7148 hello

ELLCC creates static binaries by default, so those sizes include all the standard library bits that the program needs, i.e. printf() and software floating point support functions. It is kind of cool that the M is about 15% smaller than the equivalent ARM code for this example.

I decided to try C++. Here’s the code:

[~/ellcc/examples/hellocpp] dev% cat main.cpp
#include <iostream>

int main()
{
std::cout << "hello world" << std::endl; }

Here are the results:

[~/ellcc/examples/hellocpp] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% ~/ellcc/bin/qemu-arm hellocpp
hello world

And a size comparison:

[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
867540 500 10824 878864 d6910 hellocpp
[~/ellcc/examples/hellocpp] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
rm -f *.o hellocpp hellocpp.bin hellocpp.log elkconfig.ld
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
1412708 500 10824 1424032 15baa0 hellocpp

Cool stuff.

Finally a size comparison of ELLCC's ecc clang/LLVM based compiler compiled for Thumb and ARM:

[~/ellcc] dev% size bin-thumb-linux-engeabi/ecc
text data bss dec hex filename
37889520 33760 63000 37986280 2439fe8 bin-thumb-linux-engeabi/ecc
[~/ellcc] dev% size bin-arm-linux-engeabi/ecc
text data bss dec hex filename
49504456 33760 63000 49601216 2f4dac0 bin-arm-linux-engeabi/ecc

Looks like the Thumb code is about 25% smaller.

Precompiled binaries supporting the Thumb and all the other ELLCC targets are available in the 0.1.10 release available on the Download page.

Using Config Files to use ELLCC+MinGW to Cross Compile for Windows

ELLCC is based on clang/LLVM of course. Previously I mentioned that I was experimenting with YAML based config files for the compiler driver. When I made that post and subsequently, several people had objections to the approach because it potentially required reading a file each time the compiler is involked and that it would make the already complicated driver code even more complicated. It came up again today in the LLVM mailing list and it got me to thinking that maybe another real world example might help.

I used MinGW to cross compile ELLCC for Windows hosts. I started to wonder what an ELLCC config file might look like for using clang to cross compile for Windows instead. I came up with this as a first cut example:

# based_on `i386-ellcc-linux'
# based_on `ellcc-linux'
---
based_on:        ''
global:          
  static_default:  true
compiler:        
  options:         
    - -target i386-ellcc-win32
    # - -no-integrated-as
  c_include_dirs:  
    - /usr/lib64/gcc/i686-w64-mingw32/4.8.4/include
    - /usr/lib64/gcc/i686-w64-mingw32/4.8.4/include-fixed
    - /usr/i686-w64-mingw32/sys-root/mingw/include
  cxx_include_dirs: 
    - '$R/include/c++'
assembler:       
  exe: '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/bin/as'
  output:          
    - '-o $O'
linker:          
  exe: '/usr/libexec/gcc/i686-w64-mingw32/4.8.4/collect2'
  output:          
    - '-o $O'
  start:           
    #- -e _start
  opt_static:      
    - -Bstatic
  opt_rdynamic:    
    - -export-dynamic
  opt_dynamic:     
    - -Bdynamic
  opt_shared:      
    - -shared
  opt_notshared:   
    - -dynamic-linker /usr/libexec/ld.so
  opt_pthread:     
    - -pthread
  options:         
    - -m i386pe
  static_crt1: '/usr/i686-w64-mingw32/sys-root/mingw/lib/../lib/crt2.o'
  dynamic_crt1: '$R/lib/i386-linux-eng/Scrt1.o'
  crtbegin:    '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/crtbegin.o'
  crtend:      '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/crtend.o'
  library_paths:   
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4'
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/lib/../lib'
    - '-L/usr/i686-w64-mingw32/sys-root/mingw/lib/../lib'
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/lib'
    - '-L/usr/i686-w64-mingw32/sys-root/mingw/lib'
  cxx_libraries:   
    - '-lc++'
    - -lm
  profile_libraries: 
    - -lprofile_rt
  c_libraries:     
    - '-lmingw32'
    - '-lgcc'
    - '-lgcc_eh'
    - '-lmoldname'
    - '-lmingwex'
    - '-lmsvcrt'
    - '-ladvapi32'
    - '-lshell32'
    - '-luser32'
    - '-lkernel32'
    - '-lmingw32'
    - '-lgcc'
    - '-lgcc_eh'
    - '-lmoldname'
    - '-lmingwex'
    - '-lmsvcrt'
...

I called it i386-w64-mingw32, and, Hey Presto! a Windows cross compiler on my Linux box.

[~] dev% cat main.c
#include <stdio.h>

int main()
{
    printf("hello world\n");
}
[~] dev% ~/ellcc/bin/ecc -target i386-w64-mingw32 main.c
[~] dev% ./a.exe 
fixme:winediag:start_process Wine-Compholio is a Wine testing version containing experimental patches.
fixme:winediag:start_process Please don't report bugs at winehq.org and use our issue tracker instead:
fixme:winediag:start_process https://github.com/compholio/wine-compholio/issues
hello world
[~] dev% file a.exe
a.exe: PE32 executable (console) Intel 80386, for MS Windows
[~] dev%