Tag Archives: ARM

ELLCC 0.1.28 Released

A new binary release of the ELLCC cross compilation tool chain is available.
ELLCC is a pre-packaged set of tools designed to support cross compilation
for a variety of target processors.

This release has simple instructions for cross building Linux for the
Raspberry Pi and updates several libraries.
This is the last release which will be built with autohell. I hope that I can
figure out how to use cmake soon for the next release.

ELLCC includes:

  • ecc (a clang/LLVM based C/C++ compiler)
  • binutils
  • GDB
  • QEMU (on x86_64 Linux hosts)

The ELLCC run-time libraries are supplied pre-built for all targets and
are all covered with BSD-like licenses:

  • libc++ The C++ standard library
  • musl The POSIX compliant C library
  • compiler-rt Low level compiler support
  • expat XML parsing
  • libedit Command history
  • ncurses Terminal handling
  • zlib Compression
  • curl ULR syntax data transfer
  • libevent Event notification
  • mbed TLS TLS
  • c-ares Asynchronous DNS
  • libssh2 Client side SSH2
  • libmetalink Metalink support
  • openldap Lightweight directory access protocol
  • nanomsg Several common communication patterns

ELLCC is entirely self hosting.

From the ChangeLog:

version 0.1.28:

* Update to curl 7.47.0.
* Update to binutils 2.26.
* Update autoconf files for several libraries.
* Update to LLVM r258477.
* Added missing aarch64 compiler-rt functions: __floatditf, etc.
* Add wireless tools build to ecclinux.
* Update to LLVM r257984.
* Add less build to ecclinux.

Binary tarballs are available from here.

The tool chain can build programs for ARM, Mips, PowerPC, and x86 Linux and
stand-alone targets and x86 Windows systems.

Pre-built binaries are available for ARM, Mips, and x86 Linux host systems
as well as x86_64 Windows and Mac OS X hosts.

All host systems can build programs for all the supported targets.

Available targets are listed here.

From 0 to Raspberry Pi Linux in 45 Minutes

I thought it would be cool to see how long it takes to make a boot-able Raspberry Pi disk using ELLCC. This is a timed experiment. In the end we’ll have an SD card that can be booted on the Pi. First, get a binary ELLCC release:

[~/ellcc-release] dev% mkdir test
[~/ellcc-release] dev% cd test
[~/test] dev% wget http://ellcc.org/releases/ellcc-x86_64-linux-eng-0.1.27.tgz
--2016-01-12 22:00:19--  http://ellcc.org/releases/ellcc-x86_64-linux-eng-0.1.27.tgz
Resolving ellcc.org (ellcc.org)... 174.102.201.124
Connecting to ellcc.org (ellcc.org)|174.102.201.124|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 650187071 (620M) [application/x-gzip]
Saving to: ‘ellcc-x86_64-linux-eng-0.1.27.tgz’

ellcc-x86_64-linux-e 100%[=====================>] 620.07M  11.0MB/s   in 56s    

2016-01-12 22:01:15 (11.1 MB/s) - ‘ellcc-x86_64-linux-eng-0.1.27.tgz’ saved [650187071/650187071]
[~/test] dev% tar xvfp ellcc-x86_64-linux-eng-0.1.27.tgz

Second, check what the options are:

[~/test] dev% cd ellcc/libecc/ecclinux/
[~/test/ellcc/libecc/ecclinux] dev% ./build 
usage: ./build [options] target
  options:
    -packages 'packag1 package2 ...'    Specific packages to build
    -exclude  'packag1 package2 ...'    Specific packages to exclude
    -extras                             Include additional packages
    -clean                              Clean up a previous build
  targets:
    help                                This message
    bcmrpi                              Raspberry Pi    (ARMv6)
    bcm2709                             Raspberry Pi 2  (ARMv7)
  available packages:
    kernel sinit sbase ubase smdev nldev nlmon svc sdhcp loksh
          ntfs3g iproute2 e2fsprogs vim dropbear
  extra packages:
    ellcc
  packages not buildable on the Mac:
    uboot iproute2 e2fsprogs vim

Third, build an image for the Pi 2 that includes the ELLCC tool chain (-extras):

[~/test/ellcc/libecc/ecclinux] dev% ./build -extras bcm2709j
...

Fourth, insert an SD card and install:

[~/test/ellcc/libecc/ecclinux] dev% su
Password: 
[root@dev ecclinux]# ./install
Warning! This command will destroy all data on /dev/sdc.
Continue? [yes/no] yes
...

Put the SD in the Pi and boot. Voila!

Cross Building Linux With ELLCC: Part 5 – Making an SD Card to Boot the Pi.

UPDATE: A more streamlined version of this process is available: From 0 to Raspberry Pi Linux in 45 Minutes

In previous posts I’ve described building the Linux kernel using ELLCC, building Linux for a Raspberry Pi B+, building Linux for the Pi 2, and building Linux on a Mac. In this post we’ll build enough userland and create an SD card that will boot on the Raspberry Pi 2. I just got it to boot this evening and it only gets to the login prompt where I can log in as root, but I think it is pretty cool nonetheless.

I mentioned the ellcc/libecc/ecclinux directory that is in the ELLCC source tree and is included in the ELLCC binary distributions. In this directory, there is a little script that will get sources for the Linux kernel from kernel.org, and a bunch of userland programs from suckless.org and other places. The script will then patch the sources if necessary and build them.

These are the steps I took to create a boot-able SD card. I happened to have a 32 Gig SD card lying around, so I decided to use it. On my Linux box, an SD card shows up as /dev/sdc* so the following commands reflect that. Your system may assign different name. You should verify the name by using df, or something similar, to determine what a newly inserted SD card is.

I’m going to go into a bit of detail here because it is the first time I did this for Linux and I want to keep a detailed record.

UPDATE: I made a little script that populates the SD card. It is in the source repository. The script replaces/enhances the rest of this post.

The first thing I did was to remove the old partitions on the card.

  sudo parted /dev/sdc print
    Model: Generic- SD/MMC (scsi)
    Disk /dev/sdc: 32.1GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags: 

    Number  Start   End     Size    Type     File system  Flags
     1      512B    62.9MB  62.9MB  primary
     2      62.9MB  67.1MB  4209kB  primary

My card had 2 partitions that I got rid of:

  sudo parted /dev/sdc rm 1
    Information: You may need to update /etc/fstab.

  sudo parted /dev/sdc rm 2
    Information: You may need to update /etc/fstab.

  sudo parted /dev/sdc print                                       
    Model: Generic- SD/MMC (scsi)
    Disk /dev/sdc: 32.1GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags: 

    Number  Start  End  Size  Type  File system  Flags

Now that the old partitions are gone, we can make new partitions for the Pi. The Pi boots using two partitions, one is in MS-DOS or FAT format, the other is in Linux ext4 format. The Pi has an interesting boot sequence, which is nicely described on this page, but basically the GPU starts up first and executes a first and second stage boot loader, which is really the GPU’s OS, before the ARM is brought up and the Linux kernel is booted. The FAT partition is used by the GPU and contains all the files it needs. The ext4 partition contains the rest of the Linux userland. The FAT partition os mounted as /boot after the ext4 partition is mounted as /.So the next step is to create these two partitions. First the FAT partition:

sudo parted -a optimal /dev/sdc unit chs mkpart primary 0,130,2 8,40,31
sudo parted /dev/sdc set 1 lba on

Now we can fill the rest of the disk with the ext4 partition:

sudo parted -a optimal /dev/sdc mkpart primary 67.1MB 100%

Now we’ll format the file systems:

  sudo mkfs -t fat /dev/sdc1
    mkfs.fat 3.0.27 (2014-11-12)

Then the ext4 partition:

  sudo mkfs -t ext4 /dev/sdc2
    mke2fs 1.42.12 (29-Aug-2014)
    Creating filesystem with 7820288 4k blocks and 1957888 inodes
    Filesystem UUID: 3823b7f5-66b6-4e9f-a10e-4a6658d6469f
    Superblock backups stored on blocks: 
            32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
            4096000

    Allocating group tables: done                            
    Writing inode tables: done                            
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done   

We have to mark the first partition as the boot partition:

  sudo parted /dev/sdc set 1 lba on
    Information: You may need to update /etc/fstab.

  sudo parted /dev/sdc print
    Model: Generic- SD/MMC (scsi)
    Disk /dev/sdc: 32.1GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags: 

    Number  Start   End     Size    Type     File system  Flags
     1      4194kB  67.1MB  62.9MB  primary  fat16        lba
     2      67.1MB  32.1GB  32.0GB  primary  ext4

Now we can populate the card. First, build everything using the script. If you run the script with no arguments, it will give you a hint about how to run it:

[~] dev% cd ellcc/libecc/ecclinux
[~/ellcc/libecc/ecclinux] dev% ./build 
usage: ./build [options] target
  options:
    -packages 'packag1 package2 ...'    Specific packages to build
    -exclude  'packag1 package2 ...'    Specific packages to exclude
    -clean                              Clean up a previous build
  targets:
    help                                This message
    bcmrpi                              Raspberry Pi    (ARMv6)
    bcm2709                             Raspberry Pi 2  (ARMv7)
  available packages:
    uboot kernel sinit sbase ubase smdev nldev nlmon svc sdhcp loksh
          ntfs3g iproute2 e2fsprogs vim ellcc
  packages not buildable on the Mac:
    uboot iproute2 e2fsprogs vim

In this case I’ll build for the Raspberry Pi 2:

[~/ellcc/libecc/ecclinux] dev% ./build bcm2709
  CHK     include/config/kernel.release
  GEN     ./Makefile
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
  Using /home/rich/ellcc/libecc/ecclinux/linux-4.1.15 as source for kernel
...

The SD card can be populated after the build completes:

 
  mkdir boot root
  sudo mount /dev/sdc1 boot
  sudo mount /dev/sdc2 root
  cp -r image/arm-linux-engeabihf/bcm2709/* boot
  sudo cp boot/vmlinuz-4.1.15-v7 boot/kernel7.img
  cd image/arm-linux-engeabihf/
  tar cvfp - bin etc lib sbin usr var | (cd ../../root; sudo tar xfp -)
  cd cm2709/
  rm -fr *.old
  tar cvfp - * | ( cd ../../../boot ; sudo tar xfp - )
  cd ../../..
  cd etc
  chmod oug+x rc.*
  cp * ../root/etc/
  cd ../sbin
  tar cvfp - * | ( cd ../root/sbin/ ; sudo tar xf - )
  cd ../bin
  chmod oug+x *
  tar cvfp - * | ( cd ../root/bin/ ; sudo tar xf - )
  sudo cp boot.rpi/* boot/
  cd root
  sudo mkdir root proc sys var/run
  sudo mknod dev/tty0 c 4 0
  cd ..
  sudo umount boot root

I popped the SD card into my Pi 2 and finally got the login: prompt. The initial root password is empty.

This has been a busy weekend. I got a boot, the Packers won, and it is late Sunday night and I have to work early tomorrow. I will be making the SD create steps into a script soon, but for now I’ll put up a new binary release of ELLCC, version 0.1.26, that has all the changes if you want to try this out.

Cross Building Linux With ELLCC: Part 4 – Building Linux on the Mac

In a previous post I described cross compiling the Linux kernel and userland for the Raspberry Pi using ELLCC. In this post, I’ll do the same thing on Mac OS X.
I’d like to thank Emmanuel Blot for making this possible. He sent the necessary patches. Thanks Emmanuel!

First we need to make a case insensitive file system:

[~] Richards-Mac-mini% hdiutil create -size 10g -type SPARSEBUNDLE -nospotlight -volname ellcc -fs "Case-sensitive Journaled HFS+" -attach ./ellcc.dmg
/dev/disk1              GUID_partition_scheme           /dev/disk1s1           EFI
/dev/disk1s2            Apple_HFS                       /Volumes/ellcc
created: /Users/rich/ellcc.dmg.sparsebundle

Then get the the latest binary release of ELLCC:

[~] Richards-Mac-mini% cd /Volumes/ellcc/
[/Volumes/ellcc] Richards-Mac-mini% wget http://ellcc.org/releases/ellcc-Mac_OS_X_10.11.2-0.1.25.tgz
--2016-01-06 17:58:38--  http://ellcc.org/releases/ellcc-Mac_OS_X_10.11.2-0.1.25.tgz
Resolving ellcc.org... 174.102.201.124
Connecting to ellcc.org|174.102.201.124|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 417609504 (398M) [application/x-gzip]
Saving to: 'ellcc-Mac_OS_X_10.11.2-0.1.25.tgz'
ellcc-Mac_OS_X_10.11 100%[===================>] 398.26M  11.2MB/s    in 40s
2016-01-06 17:59:18 (10.0 MB/s) - 'ellcc-Mac_OS_X_10.11.2-0.1.25.tgz' saved [417609504/417609504]
[/Volumes/ellcc] Richards-Mac-mini% tar xvfp ellcc-Mac_OS_X_10.11.2-0.1.25.tgz
...

Now we can build. Check for supported boards and choose the one you want:

[/Volumes/ellcc] Richards-Mac-mini% cd ellcc/libecc/ecclinux/
[/Volumes/ellcc/ellcc/libecc/ecclinux] Richards-Mac-mini% ./build 
usage: ./build [options] target
  options:
    -nokernel           Don't build the Linux kernel
    -clean              Clean up a previous build
  targets:
    help                This message
    bcmrpi              Raspberry Pi    (ARMv6)
    bcm2709             Raspberry Pi 2  (ARMv7)
[/Volumes/ellcc/ellcc/libecc/ecclinux] Richards-Mac-mini% ./build bcm2709
...
Generating configuration file for e2fsprogs version 1.42.13
Release date is May, 2015
checking build system type... x86_64-apple-darwin15.2.0
checking host system type... x86_64-apple-darwin15.2.0
checking for gcc... /Volumes/ellcc/ellcc/bin/ecc -target arm-linux-engeabihf
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... configure: error: in `/Volumes/ellcc/ellcc/libecc/ecclinux/arm-linux-engeabihf/e2fsprogs':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
[/Volumes/ellcc/ellcc/libecc/ecclinux] Richards-Mac-mini%

Looks like I have a problem with the userland build, but the kernel built. I’ll look at that tomorrow. It’s late. Nice!

Cross Building Linux With ELLCC: Part 3 – Other Flavors of Pi

Update: Cross Building Linux With ELLCC: Part 4 – Building Linux on the Mac
In my last post I described using ELLCC to build Linux for the Raspberry Pi B+. In ELLCC version 0.1.24 I added a script to build Linux for both the Pi B+ and the Pi 2. As a bonus, the script also builds a bunch of userland programs

First grab the latest binary release:

[~] dev% wget http://ellcc.org/releases/ellcc-x86_64-linux-eng-0.1.24.tgz
[~] dev% tar xvfp ellcc-x86_64-linux-eng-0.1.24.tgz

Now decide what target to build:

[~] dev% cd ellcc/libecc/ecclinux/
[~/ellcc/libecc/ecclinux] dev% ls
00_rpilinux-4.1.15.patch   bin/    etc/            makefile      sbin/
99_llvmlinux-4.1.15.patch  build*  iproute2.patch  ntfs3g.patch  sinit/
[~/ellcc/libecc/ecclinux] dev% ./build 
usage: ./build [options] target
  options:
    -nokernel           Don't build the Linux kernel
    -clean              Clean up a previous build
  targets:
    help                This message
    bcmrpi              Raspberry Pi    (ARMv6)
    bcm2709             Raspberry Pi 2  (ARMv7)
[~/ellcc/libecc/ecclinux] dev%

Run the build command again, with the board you need as an argument (I’m going to time the build with time):

[~/ellcc/libecc/ecclinux] dev% time ./build bcm2709
... (lots of output removed)
4984.143u 435.744s 22:52.57 394.8%      0+0k 704+4154968io 2pf+0w
[~/ellcc/libecc/ecclinux] dev% 

As SpongeBob would say, “23 minutes later.” When the build is finished, the ecclinux directory looks like this:

[~/ellcc/libecc/ecclinux] dev% ls
00_rpilinux-4.1.15.patch   bin/       etc/            linux-4.1.15/  sbin/
99_llvmlinux-4.1.15.patch  build*     image/          makefile       sinit/
arm-linux-engeabihf/       download/  iproute2.patch  ntfs3g.patch
[~/ellcc/libecc/ecclinux] dev%

There are three new directories: download/ contains tar files that have been downloaded in case you need them again, arm-linux-engeabihf/ is where the build was made for the ARMv7, and image/ is where the binaries were installed. Let’s take a look at image/arm-linux-engeabihf/:

[~/ellcc/libecc/ecclinux] dev% ls image/arm-linux-engeabihf/
bcm2709/  bin/  etc/  lib/  sbin/  usr/  var/
[~/ellcc/libecc/ecclinux] dev% 

The bcm2709/ directory is what will be in the /boot directory in the final Raspberry Pi 2 SD image. It has Pi 2 specific files in it, including the kernel:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% ls bcm2709/
dtbs/  System.map-4.1.15-v7  vmlinuz-4.1.15-v7
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

Here’s what the bin and sbin directories look like:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% ls bin/ sbin/ 
bin/:
[*           ed*           ksh*         nltrigger*   sha256sum*    tty*
basename*    eject*        last*        nohup*       sha512sum*    umount*
cal*         env*          lastlog*     od*          sleep*        uname*
cat*         ex@           link*        pagesize*    smdev*        unexpand*
chattr*      expand*       ln*          passwd*      sort*         uniq*
chgrp*       expr*         logger*      paste*       split*        unlink*
chmod*       fallocate*    login*       pidof*       sponge*       unshare*
chown*       false*        logname*     pivot_root*  stat*         uptime*
chroot*      find*         ls*          printenv*    strings*      uudecode*
chvt*        flock*        lsattr*      printf*      su*           uuencode*
cksum*       fold*         lsmod*       ps*          swaplabel*    uuidgen*
clear*       free*         lsusb*       pwd*         swapoff*      view@
cmp*         freeramdisk*  md5sum*      readahead*   swapon*       vim*
cols*        fsfreeze*     mesg*        readlink*    switch_root*  vimdiff@
comm*        getconf*      mkdir*       renice*      sync*         vimtutor*
cp*          getty*        mkfifo*      respawn*     sysctl*       vtallow*
cron*        grep*         mknod*       rm*          tail*         watch*
ctrlaltdel*  halt*         mkswap*      rmdir*       tar*          wc*
cut*         head*         mktemp*      rmmod*       tee*          which*
date*        hostname*     mount*       run_nldev*   test*         who*
dd*          hwclock*      mountpoint*  rview@       tftp*         whoami*
df*          id*           mv*          rvim@        time*         xargs*
dirname*     insmod*       nice*        sed*         touch*        xxd*
dmesg*       join*         nl*          seq*         tr*           yes*
du*          kill*         nldev*       setsid*      true*
echo*        killall5*     nlmon*       sha1sum*     truncate*

sbin/:
badblocks*   findfs@        mke2fs*            ntfs-3g.secaudit*  resize2fs*
blkid*       fsck*          mkfs.ext2@         ntfs-3g.usermap*   routef*
bridge*      fsck.ext2@     mkfs.ext3@         ntfscat*           routel*
ctstat@      fsck.ext3@     mkfs.ext4@         ntfsclone*         rtacct*
debugfs*     fsck.ext4@     mkfs.ext4dev@      ntfscluster*       rtmon*
dumpe2fs*    fsck.ext4dev@  mkfs.ntfs@         ntfscmp*           rtpr*
e2freefrag*  genl*          mklost+found*      ntfscp*            rtstat@
e2fsck*      ifcfg*         mkntfs*            ntfsfix*           sdhcp*
e2image*     ifstat*        mount.lowntfs-3g@  ntfsinfo*          sinit*
e2label@     ip*            mount.ntfs-3g@     ntfslabel*         ss*
e2undo*      lnstat*        nstat*             ntfsls*            tune2fs*
e4defrag*    logsave*       ntfs-3g*           ntfsresize*        uuidd*
filefrag*    lowntfs-3g*    ntfs-3g.probe*     ntfsundelete*
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

Not bad for 23 minutes of work!

If you want to try the kernel on your Pi 2, you need to copy vmlinuz-4.1.15-v7 to the Rasbian /boot directory as kernel7.img. In my case, my Pi 2 is at 192.124.43.100 so I can do this:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% scp bcm2709/vmlinuz-4.1.15-v7 root@192.124.43.100:/boot/kernel7.img
root@192.124.43.100's password: 
vmlinuz-4.1.15-v7                              100% 4355KB   4.3MB/s   00:00    
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

After a reboot:

[rich@dev ~]$ ssh root@192.124.43.100
root@192.124.43.100's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Jan  1 17:28:43 2016 from 192.124.43.2
root@raspberrypi:~# dmesg | less
[    0.000000] Booting Linux on physical CPU 0xf00
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1.15-v7 (rich@dev) (ecc 0.1.24 based on clang version 3.8.0 (trunk) (based on LLVM 3.8.0svn)) #1 SMP Mon Jan 4 20:45:34 CST 2016

Not bad. To complete the kernel update you’ll also want to update the kernel modules:

[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% ls lib/modules/4.1.15-v7/
kernel/            modules.builtin      modules.dep.bin  modules.softdep
modules.alias      modules.builtin.bin  modules.devname  modules.symbols
modules.alias.bin  modules.dep          modules.order    modules.symbols.bin
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev% scp -r lib/modules/4.1.15-v7/ root@192.124.43.100:/lib/modules
root@192.124.43.100's password: 
Permission denied, please try again.
root@192.124.43.100's password: 
modules.symbols                                100%  146KB 146.2KB/s   00:00    
modules.dep                                    100%  131KB 131.1KB/s   00:00  
...
modules.builtin.bin                            100% 5903     5.8KB/s   00:00    
modules.builtin                                100% 4341     4.2KB/s   00:00    
[~/ellcc/libecc/ecclinux/image/arm-linux-engeabihf] dev%

Reboot again and even the mouse and keyboard work.

In Part 4, I’ll show how to make the SD image and use the userland programs that we’ve built. In the mean time you can copy them to the Pi’s SD card and use them.

Cross Building Linux With ELLCC: Part 2 – Raspberry Pi B+

Update: Cross Building Linux With ELLCC: Part 3 – Other Flavors of Pi
In a previous post I tested the waters to see how close ELLCC is to cross compiling the Linux kernel. That post was mostly a theoretical exercise to see what it would take. Over the holidays I decided that I needed to do something a little more concrete. My son, a 10 year old obsessed with Minecraft, got a Raspberry Pi for Christmas. My nefarious plan is to divert his attention away from Minecraft at least long enough to make his Pi and camera module into something he can use to spy on the family. My other nefarious plan is to use ELLCC to build the Linux kernel that we’ll be running on his Pi.

In the previous post I used the source from the LLVMLinux site. This time I started with the kernel source for the Raspberry Pi. The good news is that the source can be built for the Pi. The bad news is that it doesn’t have all the patches need to compile with a clang based compiler. It turned out that the build wasn’t too bad: I had Christmas Eve and Day off and in between family events and Santa preparations I was able to massage the source enough to boot the kernel on our Pi.

To get started I followed the instructions on Linux From Scratch on the Raspberry Pi.
Here are the exact steps I took. First I created a new directory to hold the ELLCC tool chain and the Raspberry Linux kernel source, then I downloaded and untar’d both:

mkdir ecclinux
cd ecclinux
wget http://ellcc.org/releases/ellcc-x86_64-linux-eng-0.1.23.tgz
tar xvfp ellcc-x86_64-linux-eng-0.1.23.tgz
wget https://github.com/raspberrypi/linux/archive/rpi-4.1.y.tar.gz
tar xvfp rpi-4.1.y.tar.gz

I had to change the kernel sources slightly during my build. I saved a patch
file, here’s how to apply it:

patch -p0 < ellcc/libecc/llvmlinux/ecc-rpi-4.1.y.patch

There is a special little 'm' makefile that helps with the build. Copy
it into the top level of the kernel source directory:

cd linux-rpi-4.1.y/
cp ../ellcc/libecc/llvmlinux/makefile .

Now the actual build is easy:

make bcmrpi_defconfig
make

I have my Raspberry Pi on my home network at 192.124.43.98. I also allowed ssh
and scp root access by follwoing the instructions here. This lets me
just copy the new kernel to my Pi:

scp ./arch/arm/boot/zImage root@192.124.43.98:/boot/kernel.img

Now I can ssh to the Pi and reboot:

[~] dev% ssh root@192.124.43.98
root@192.124.43.98's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@raspberrypi:~# reboot
Connection to 192.124.43.98 closed by remote host.
Connection to 192.124.43.98 closed.
[~] dev%

I ping the Pi until it reboots and log in again:

[~] dev% ssh root@192.124.43.98
root@192.124.43.98's password: 

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Dec 26 20:27:15 2015 from 192.124.43.2
root@raspberrypi:~# dmesg | less
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.1.15 (rich@dev) (ecc 0.1.23 based on clang version 3.8.0 (trunk) (based on LLVM 3.8.0svn)) #1 Sat Dec 26 11:32:33 CST 2015
[    0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
...

Very nice!

Using ELLCC to (Cross) Compile Toybox

Toybox combines many common Linux command line utilities together into a single BSD-licensed executable. I wanted to to see how hard it would be to compile and cross compile it using ELLCC. It turned out to be pretty easy after I added a few Linux specific header files. These are the files I added:

[~/ellcc] dev% ls ~/ellcc/libecc/include/linux
fs.h  if_vlan.h  loop.h  nbd.h  rfkill.h  rtc.h  sched.h  sockios.h  soundcard.h

The needed include files are in the ELLCC source tree now and are available in the 0.1.22 binary release. Here’s all you need to do to build Toybox:

[rich@dev ~]$ mkdir toybox 
[rich@dev ~]$ cd toybox
[rich@dev toybox]$ wget http://landley.net/toybox/downloads/toybox-0.6.1.tar.gz
--2015-12-23 17:02:37--  http://landley.net/toybox/downloads/toybox-0.6.1.tar.gz
Resolving landley.net (landley.net)... 208.113.171.142
Connecting to landley.net (landley.net)|208.113.171.142|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 736371 (719K) [application/x-tar]
Saving to: ‘toybox-0.6.1.tar.gz’

toybox-0.6.1.tar.gz  100%[=====================>] 719.11K   845KB/s   in 0.9s   

2015-12-23 17:02:38 (845 KB/s) - ‘toybox-0.6.1.tar.gz’ saved [736371/736371]

[rich@dev toybox]$ tar xfp toybox-0.6.1.tar.gz 
[rich@dev toybox]$ cd toybox-0.6.1/
[rich@dev toybox-0.6.1]$ CC="/home/rich/ellcc/bin/ecc -target armv6-linux-engeabihf" CFLAGS="-Wno-format-extra-args -Wno-uninitialized" make defconfig toybox
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
        -DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/null Config.in > /dev/null
scripts/make.sh
Generate headers from toys/*/*.c...
generated/newtoys.h Library probe.......
Make generated/config.h from .config.
generated/flags.h generated/globals.h generated/help.h
Compile toybox................................................................................................................................................................strip: Unable to recognise the format of the input file `toybox_unstripped'
strip failed, using unstripped
.

Oops. The strip failed because the standard x86_64 Linux strip doesn’t understand ARM object file formats. We can use ecc-strip:

[rich@dev toybox-0.6.1]$ ~/ellcc/bin/ecc-strip -o toybox toybox_unstripped
[rich@dev toybox-0.6.1]$ file toybox
toybox: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=11e423037c6fb0e4ee10ef5f53a421fca86a81e3, stripped               
[rich@dev toybox-0.6.1]$ size toybox                                             
   text    data     bss     dec     hex filename                                 
 401916    3208   21020  426144   680a0 toybox  

Lest’s send it over to my trusty Raspberry Pi:

                                 
[rich@dev toybox-0.6.1]$ scp toybox rich@192.124.43.98:                                                   
rich@192.124.43.98's password:                                                   
scp: ./toybox: Permission denied                                                 
[rich@dev toybox-0.6.1]$ ssh rich@192.124.43.98
rich@192.124.43.98's password:                                                   
                                                                                 
The programs included with the Debian GNU/Linux system are free software;        
the exact distribution terms for each program are described in the               
individual files in /usr/share/doc/*/copyright.                                  
                                                                                 
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent                
permitted by applicable law.                                                     
Last login: Sun Dec 20 19:20:36 2015 from 192.124.43.2   

What commands does Toybox have?

                        
rich@raspberrypi:~$ ./toybox                                                     
acpi base64 basename blkid blockdev bunzip2 bzcat cal cat catv chattr            
chgrp chmod chown chroot chvt cksum clear cmp comm count cp cpio cut             
date df dirname dmesg dos2unix du echo egrep eject env expand factor             
fallocate false fgrep find flock free freeramdisk fstype fsync grep              
groups halt head help hexedit hostid hostname hwclock id ifconfig                
inotifyd insmod install ionice iorenice kill killall killall5 link               
ln login logname losetup ls lsattr lsmod lspci lsusb makedevs md5sum             
mix mkdir mkfifo mknod mkpasswd mkswap mktemp modinfo mount mountpoint           
mv nbd-client nc netcat nice nl nohup nproc nsenter od oneit partprobe           
passwd paste patch pidof pivot_root pmap poweroff printenv printf                
ps pwd pwdx readahead readlink realpath reboot renice reset rev rfkill           
rm rmdir rmmod sed seq setsid sh sha1sum shred sleep sort split stat             
strings su swapoff swapon switch_root sync sysctl tac tail taskset               
tee time timeout touch toysh true truncate tty umount uname uniq unix2dos        
unlink uptime usleep uudecode uuencode vconfig vmstat w wc which who             
whoami xargs xxd yes                                                             
rich@raspberrypi:~$ ./toybox ls -l
total 3112                                                                       
-rwxr-x--x 1 rich rich  141580 2015-12-20 19:22 hello                            
-r-xr-xr-x 1 rich rich 1446476 2015-12-23 21:21 toybox   

Now let’s build for the x86_64:

                        
rich@raspberrypi:~$ logout                                                                           
Connection to 192.124.43.98 closed.                                              
[rich@dev toybox-0.6.1]$ make clean                                              
rm -f kconfig/zconf.hash.c kconfig/zconf.tab.c kconfig/lex.zconf.c kconfig/conf kconfig/mconf                                                                     
rm -rf toybox toybox_unstripped generated change .singleconfig*                  
[rich@dev toybox-0.6.1]$ CC="/home/rich/ellcc/bin/ecc -target x86_64-linux-eng" CFLAGS="-Wno-format-extra-args -Wno-uninitialized" make defconfig toybox     
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \        
        -DPROJECT_NAME=\"ToyBox\"                                                
scripts/genconfig.sh                                                             
kconfig/conf -D /dev/null Config.in > /dev/null                                  
scripts/make.sh                                                                  
Generate headers from toys/*/*.c...                                              
generated/newtoys.h Library probe.......                                         
Make generated/config.h from .config.                                            
generated/flags.h generated/globals.h generated/help.h                           
Compile toybox.................................................................................................................................................................                                                                    
[rich@dev toybox-0.6.1]$ file toybox
toybox: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6d60409b768431310a5ce6085e595b6a98a66d45, stripped                  
[rich@dev toybox-0.6.1]$ size toybox
   text    data     bss     dec     hex filename
 376786    6192   22152  405130   62e8a toybox                                   
[rich@dev toybox-0.6.1]$ 

Cool stuff.

ELLCC now has 64 bit ARM AArch64 Support

A new binary release of the ELLCC cross compilation tool chain is available.
ELLCC is a pre-packaged set of tools designed to support cross compilation
for a variety of target processors.

ELLCC includes:

  • ecc (a clang/LLVM based C/C++ compiler)
  • binutils
  • GDB

The ELLCC run-time libraries are supplied pre-built for all targets and
are all covered with BSD-like licenses:

  • libc++ The C++ standard library
  • musl The POSIX compliant C library
  • compiler-rt Low level compiler support
  • expat XML parsing
  • libedit Command history
  • ncurses Terminal handling
  • zlib Compression

ELLCC is entirely self hosting.

From the ChangeLog:

version 0.1.11:

* Added aarch64 support.
* Update to musl 1.1.7.
* Update to LLVM r232721.

Binary tarballs are available via FTP or HTTP from the download page.

The tool chain can build programs for ARM, Mips, PowerPC, and x86 Linux and stand-alone targets and x86 Windows systems.

Pre-built binaries are available for ARM, Mips, PowerPC, and x86 Linux host systems as well as x86 Windows and Mac OS X hosts.

All host systems can build programs for all the supported targets.

Available targets are listed here

Using ELLCC to Build for the Cortex-M3

This weekend I decided to work on getting ELLCC to support the ARM Cortex-M processors. The Cortex-M processors use the Thumb 16 bit instruction set, so adding M support involves adding a configuration file for a member of the Cortex-M family and then building the run-time libraries specifically for the target processor. To start I chose the Cortex-M3. Its configuration file currently looks like this:

[~/ellcc/libecc/config] dev% cat thumb-linux-engeabi
based_on: arm-ellcc-linux
compiler:
options:
- -target arm-ellcc-linux
- -mthumb
- -mcpu=cortex-m3
- -mfpu=none
- -mfloat-abi=softfp
linker:
static_crt1: $R/lib/thumb-linux-engeabi/crt1.o
dynamic_crt1: $R/lib/thumb-linux-engeabi/Scrt1.o
crtbegin: $R/lib/thumb-linux-engeabi/crtbegin.o
crtend: $R/lib/thumb-linux-engeabi/crtend.o
library_paths:
- -L$R/lib/thumb-linux-engeabi

With the config file in place, I attempted to build ELLCC’s standard C library musl for the M. Building a library for a new configuration is pretty easy:

[~/ellcc/libecc] dev% make thumb-linux-engeabi
[~/ellcc/libecc] dev% cd
[~] dev% cd ~/ellcc/libecc
[~/ellcc/libecc] dev% make thumb-linux-engeabi

Well, it isn’t quite that simple. The musl library has several assembly files that are written for the specific target processor. If the assembly language of the desired processor doesn’t match the sources, the sources have to be modified to support the new target. Luckily Thumb instructions are mostly a subset of 32 bit ARM instructions so the changes needed to build musl for the M were fairly minimal.

The next step to support a given processor is to compiler the compiler-rt library. This library contains support functions that handle operations that the processor doesn’t support directly and compiler generated built in functions. In the case of the M processors, this includes things like floating point functions since the M3 doesn’t have a floating point unit. Building compiler-rt was probably the most challenging part of this little project. I had never fully ported the compiler-rt build rules to use the new config files properly so rather than cobbling together something that could handle building for the M, I rewrote the build rules to properly support config files.

The last step is to build the C++ standard libraries, assuming you want C++ support. I ran into one little glitch here. The M doesn’t support 64 bit atomic operations so I had to modify those operations for the M.

Now that the libraries are built, a little experimentation is in order. I decided to try out a simple “hello world”. Here’s the source:

[~/ellcc/examples/hello] dev% cat main.c
#include <stdio.h>

int main()
{
printf("hello world\n");
}

The make files in the example directories can build for multiple targets as long as config files have been set up, so I built for the M:

[~/ellcc/examples/hello] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% ~/ellcc/bin/qemu-arm hello
hello world
[~/ellcc/examples/hello] dev%

Nice! I was curious how the Thumb instruction set affected code size so I did a side by side comparison:

[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
19540 156 1412 21108 5274 hello
[~/ellcc/examples/hello] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
rm -f *.o hello hello.bin hello.log elkconfig.ld
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
27432 156 1412 29000 7148 hello

ELLCC creates static binaries by default, so those sizes include all the standard library bits that the program needs, i.e. printf() and software floating point support functions. It is kind of cool that the M is about 15% smaller than the equivalent ARM code for this example.

I decided to try C++. Here’s the code:

[~/ellcc/examples/hellocpp] dev% cat main.cpp
#include <iostream>

int main()
{
std::cout << "hello world" << std::endl; }

Here are the results:

[~/ellcc/examples/hellocpp] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% ~/ellcc/bin/qemu-arm hellocpp
hello world

And a size comparison:

[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
867540 500 10824 878864 d6910 hellocpp
[~/ellcc/examples/hellocpp] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
rm -f *.o hellocpp hellocpp.bin hellocpp.log elkconfig.ld
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
1412708 500 10824 1424032 15baa0 hellocpp

Cool stuff.

Finally a size comparison of ELLCC's ecc clang/LLVM based compiler compiled for Thumb and ARM:

[~/ellcc] dev% size bin-thumb-linux-engeabi/ecc
text data bss dec hex filename
37889520 33760 63000 37986280 2439fe8 bin-thumb-linux-engeabi/ecc
[~/ellcc] dev% size bin-arm-linux-engeabi/ecc
text data bss dec hex filename
49504456 33760 63000 49601216 2f4dac0 bin-arm-linux-engeabi/ecc

Looks like the Thumb code is about 25% smaller.

Precompiled binaries supporting the Thumb and all the other ELLCC targets are available in the 0.1.10 release available on the Download page.

Adding Networking Support to ELK: Part 3

Again I’m posting this as a post-in-progress. I was hoping to finish this before real life intrudes tomorrow. Things are going well but tomorrow looks like a real stretch goal.

In Part 1 of this post I wrote about some design goals for ELK networking. In Part 2, I described some of the implementation details that evolved as I integrated the LwIP TCP/IP stack into ELK. In this post, I’ll describe adding an Ethernet driver to ELK’s network implementation.

As in Part 2, I’m writing this as development is progressing, so be please patient if I tend to go off on tangents.

LwIP comes with a skeleton example Ethernet driver in src/netif/ethernetif.c. My ARM development environment is currently QEMU emulating the Versatile Express A9 board, which emulates the LAN 91C111 Ethernet controller. Eventually, I’d like to get ELK networking running on the Raspberry Pi, but since it uses a USB based Ethernet controller (the smsc9512), that will require a bit more work.

OK, enough with Google. Time to get to work. I’ll start or by making a copy of ethernetif.c which I have called lan91c111.c. Now I’ll start filling in the details.

I’ve made some basic changes to the driver and now I’ll add it to the list of ELK source files in sources.mk. I know the first build is going to spit out a bunch of errors, and sure enough it does. I’ll add a few “#if RICH” lines to mark places I have to add functionality. I got through the “#if RICH” part when I realized that I’m missing the whole infrastructure for socket ioctl() calls. I realized that when I started about how to initialize Ethernet devices. So I’m side tracking to add ioctl() support to the network interfaces.

As an aside, it turns out I had to make another change to the LwIP sources. One of the ioctl() calls supported on sockets is named IP_HDRINCL. It turns out that LwIP uses that name at the lower levels to represent a packet that already contains an address. I changed all instances of IP_HDRINCL to IP_HDRINCLUDED in LwIP to get around the naming conflict.

There are two interfaces that LwIP supports to configure Ethernet devices, basically because LwIP has two modes of operation as I mentioned in Part 1. The core of LwIP is meant to be used by a single (or the only) thread. That is the first mode, and is usually reserved for smaller systems where all network activity is done by a single thread. The second mode has a network thread which has a message based interface to all the other threads in the system. ELK uses the second mode, so we have to use the netifapi functions, defined in the lwip/netifapi.h file. I had to to enable this API in the ELK specific lwipopts.h file.

The ioctl() calls for sockets are documented in the netdevice(7) man page. We’ll see how well the LwIP API maps to Linux’s idea of network configuration.

There are a set of ioctl() calls defined that are used to configure network devices as described in netdevice(7). I’ve implemented a fairly large subset of socket ioctl() calls to prepare for actually adding the Ethernet driver. Here is an example of some of them:

/* ELK running as a VM enabled OS.
 */
#define _GNU_SOURCE
#include <sys/ioctl.h>
#include <net/if.h>
#include <net/if_arp.h>
#include <arpa/inet.h>
#include <errno.h>
#include <string.h>

#include <sys/cdefs.h>
#include <stdio.h>

#include "command.h"

int main(int argc, char **argv)
{
  setprogname("elk");
  printf("%s started. Type \"help\" for a list of commands.\n", getprogname());

  int sfd = socket(AF_INET, SOCK_STREAM /*|SOCK_NONBLOCK */, 0);
  if (sfd < 0) {
    printf("socket(AF_INET) failed: %s\n", strerror(errno));
  }

  int s;
  struct ifreq ifreq;
  struct in_addr in_addr;

  // Set the device name for subsequent calls.
  strcpy(ifreq.ifr_name, "ln01");

  // Set the interface IP{ address.
  inet_aton("192.124.43.4", &in_addr);
  ifreq.ifr_addr.sa_family = AF_INET;
  memcpy(ifreq.ifr_addr.sa_data, &in_addr, sizeof(in_addr));
  s = ioctl(sfd, SIOCSIFADDR, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFADDR) failed: %s\n", strerror(errno));
  }

  // Set ine interface netmask.
  inet_aton("255.255.255.0", &in_addr);
  ifreq.ifr_netmask.sa_family = AF_INET;
  memcpy(ifreq.ifr_netmask.sa_data, &in_addr, sizeof(in_addr));
  s = ioctl(sfd, SIOCSIFNETMASK, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFNETMASK) failed: %s\n", strerror(errno));
  }

  // Set the interface MAC address.
  ifreq.ifr_hwaddr.sa_family = ARPHRD_ETHER;
  ifreq.ifr_hwaddr.sa_data[0] = 0x01;
  ifreq.ifr_hwaddr.sa_data[1] = 0x02;
  ifreq.ifr_hwaddr.sa_data[2] = 0x03;
  ifreq.ifr_hwaddr.sa_data[3] = 0x04;
  ifreq.ifr_hwaddr.sa_data[4] = 0x05;
  ifreq.ifr_hwaddr.sa_data[5] = 0x06;
  s = ioctl(sfd, SIOCSIFHWADDR, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFHWADDR) failed: %s\n", strerror(errno));
  }
  printf("Try the command 'inetif'\n");

  // Enter the kernel command processor.
  do_commands(getprogname());
}

Where is what the compiled example produces:

[~/ellcc/examples/elk] dev% make run
Compiling main.c
Linking elk
Running elk
enter 'control-A x' to exit QEMU
audio: Could not init `oss' audio driver
elk started. Type "help" for a list of commands.
Try the command 'inetif'
elk % inetif
0 lo: active flags=137 (0x01)  mtu 0
        inet 127.0.0.1  netmask 255.0.0.0  broadcast 127.255.255.255
1 ln01: active flags=66 (0x32)  mtu 1500
        inet 192.124.43.4  netmask 255.255.255.0  broadcast 192.124.43.255
        ether 01:02:03:04:05:06
elk % 

ln01 is my stubbed driver. It looks like the ioctl() calls have worked. lwip_network.c is the source file implementing ELK's LwIP socket interface. Now it's time to get the driver to do something.

As I started to flesh out the driver I realized that the vexpress-a9 board that QEMU is emulating uses the LAN9118 Ethernet controller, not the SMC91C111 that I thought. That got me thinking. I really don't want to reinvent the wheel each time for all the different Ethernet controllers out there and it is always better to find something that exists and adapt it rather than to write it from scratch. With that in mind, I decided what it would take to use Ethernet drivers borrowed from NetBSD for ELK. As I see it, the first step is to get a NetBSD driver to compile in the ELK environment with as few source changes as possible. Step two is to make some glue code to make the NetBSD driver fit into the LwIP environment.

Yikes! I just spent a harrowing half a day trying to isolate the NetBSD driver for the LAN9118 so that it could "drop in" as a LwIP driver. I went down the path of stubbing out all the include files that were directly and indirectly included and compiling the source file, over and over, adding definitions to the include files as needed. It turns out that the driver has lots of dependencies on the NetBSD kernel, which is not surprising. I've decided to try a different route, but I did stumble on this interesting post about rump kernels, which are basically doing what ELK is doing, but with NetBSD as the basis.

I like the direction this new approach is taking. I'm using an idea from the LwIP wiki to create a driver framework. The driver skeleton has been fleshed out in ethernetif_driver.c and low level support for the LAN9118 is taking shape in lan9118.c. I hope I find some time over the next few days to fill in the reast of the details.