Tag Archives: Cross Compiling

Using ELLCC to Build for the Cortex-M3

This weekend I decided to work on getting ELLCC to support the ARM Cortex-M processors. The Cortex-M processors use the Thumb 16 bit instruction set, so adding M support involves adding a configuration file for a member of the Cortex-M family and then building the run-time libraries specifically for the target processor. To start I chose the Cortex-M3. Its configuration file currently looks like this:

[~/ellcc/libecc/config] dev% cat thumb-linux-engeabi
based_on: arm-ellcc-linux
compiler:
options:
- -target arm-ellcc-linux
- -mthumb
- -mcpu=cortex-m3
- -mfpu=none
- -mfloat-abi=softfp
linker:
static_crt1: $R/lib/thumb-linux-engeabi/crt1.o
dynamic_crt1: $R/lib/thumb-linux-engeabi/Scrt1.o
crtbegin: $R/lib/thumb-linux-engeabi/crtbegin.o
crtend: $R/lib/thumb-linux-engeabi/crtend.o
library_paths:
- -L$R/lib/thumb-linux-engeabi

With the config file in place, I attempted to build ELLCC’s standard C library musl for the M. Building a library for a new configuration is pretty easy:

[~/ellcc/libecc] dev% make thumb-linux-engeabi
[~/ellcc/libecc] dev% cd
[~] dev% cd ~/ellcc/libecc
[~/ellcc/libecc] dev% make thumb-linux-engeabi

Well, it isn’t quite that simple. The musl library has several assembly files that are written for the specific target processor. If the assembly language of the desired processor doesn’t match the sources, the sources have to be modified to support the new target. Luckily Thumb instructions are mostly a subset of 32 bit ARM instructions so the changes needed to build musl for the M were fairly minimal.

The next step to support a given processor is to compiler the compiler-rt library. This library contains support functions that handle operations that the processor doesn’t support directly and compiler generated built in functions. In the case of the M processors, this includes things like floating point functions since the M3 doesn’t have a floating point unit. Building compiler-rt was probably the most challenging part of this little project. I had never fully ported the compiler-rt build rules to use the new config files properly so rather than cobbling together something that could handle building for the M, I rewrote the build rules to properly support config files.

The last step is to build the C++ standard libraries, assuming you want C++ support. I ran into one little glitch here. The M doesn’t support 64 bit atomic operations so I had to modify those operations for the M.

Now that the libraries are built, a little experimentation is in order. I decided to try out a simple “hello world”. Here’s the source:

[~/ellcc/examples/hello] dev% cat main.c
#include <stdio.h>

int main()
{
printf("hello world\n");
}

The make files in the example directories can build for multiple targets as long as config files have been set up, so I built for the M:

[~/ellcc/examples/hello] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% ~/ellcc/bin/qemu-arm hello
hello world
[~/ellcc/examples/hello] dev%

Nice! I was curious how the Thumb instruction set affected code size so I did a side by side comparison:

[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
19540 156 1412 21108 5274 hello
[~/ellcc/examples/hello] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
rm -f *.o hello hello.bin hello.log elkconfig.ld
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
27432 156 1412 29000 7148 hello

ELLCC creates static binaries by default, so those sizes include all the standard library bits that the program needs, i.e. printf() and software floating point support functions. It is kind of cool that the M is about 15% smaller than the equivalent ARM code for this example.

I decided to try C++. Here’s the code:

[~/ellcc/examples/hellocpp] dev% cat main.cpp
#include <iostream>

int main()
{
std::cout << "hello world" << std::endl; }

Here are the results:

[~/ellcc/examples/hellocpp] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% ~/ellcc/bin/qemu-arm hellocpp
hello world

And a size comparison:

[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
867540 500 10824 878864 d6910 hellocpp
[~/ellcc/examples/hellocpp] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
rm -f *.o hellocpp hellocpp.bin hellocpp.log elkconfig.ld
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
1412708 500 10824 1424032 15baa0 hellocpp

Cool stuff.

Finally a size comparison of ELLCC's ecc clang/LLVM based compiler compiled for Thumb and ARM:

[~/ellcc] dev% size bin-thumb-linux-engeabi/ecc
text data bss dec hex filename
37889520 33760 63000 37986280 2439fe8 bin-thumb-linux-engeabi/ecc
[~/ellcc] dev% size bin-arm-linux-engeabi/ecc
text data bss dec hex filename
49504456 33760 63000 49601216 2f4dac0 bin-arm-linux-engeabi/ecc

Looks like the Thumb code is about 25% smaller.

Precompiled binaries supporting the Thumb and all the other ELLCC targets are available in the 0.1.10 release available on the Download page.

Using Config Files to use ELLCC+MinGW to Cross Compile for Windows

ELLCC is based on clang/LLVM of course. Previously I mentioned that I was experimenting with YAML based config files for the compiler driver. When I made that post and subsequently, several people had objections to the approach because it potentially required reading a file each time the compiler is involked and that it would make the already complicated driver code even more complicated. It came up again today in the LLVM mailing list and it got me to thinking that maybe another real world example might help.

I used MinGW to cross compile ELLCC for Windows hosts. I started to wonder what an ELLCC config file might look like for using clang to cross compile for Windows instead. I came up with this as a first cut example:

# based_on `i386-ellcc-linux'
# based_on `ellcc-linux'
---
based_on:        ''
global:          
  static_default:  true
compiler:        
  options:         
    - -target i386-ellcc-win32
    # - -no-integrated-as
  c_include_dirs:  
    - /usr/lib64/gcc/i686-w64-mingw32/4.8.4/include
    - /usr/lib64/gcc/i686-w64-mingw32/4.8.4/include-fixed
    - /usr/i686-w64-mingw32/sys-root/mingw/include
  cxx_include_dirs: 
    - '$R/include/c++'
assembler:       
  exe: '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/bin/as'
  output:          
    - '-o $O'
linker:          
  exe: '/usr/libexec/gcc/i686-w64-mingw32/4.8.4/collect2'
  output:          
    - '-o $O'
  start:           
    #- -e _start
  opt_static:      
    - -Bstatic
  opt_rdynamic:    
    - -export-dynamic
  opt_dynamic:     
    - -Bdynamic
  opt_shared:      
    - -shared
  opt_notshared:   
    - -dynamic-linker /usr/libexec/ld.so
  opt_pthread:     
    - -pthread
  options:         
    - -m i386pe
  static_crt1: '/usr/i686-w64-mingw32/sys-root/mingw/lib/../lib/crt2.o'
  dynamic_crt1: '$R/lib/i386-linux-eng/Scrt1.o'
  crtbegin:    '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/crtbegin.o'
  crtend:      '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/crtend.o'
  library_paths:   
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4'
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/lib/../lib'
    - '-L/usr/i686-w64-mingw32/sys-root/mingw/lib/../lib'
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/lib'
    - '-L/usr/i686-w64-mingw32/sys-root/mingw/lib'
  cxx_libraries:   
    - '-lc++'
    - -lm
  profile_libraries: 
    - -lprofile_rt
  c_libraries:     
    - '-lmingw32'
    - '-lgcc'
    - '-lgcc_eh'
    - '-lmoldname'
    - '-lmingwex'
    - '-lmsvcrt'
    - '-ladvapi32'
    - '-lshell32'
    - '-luser32'
    - '-lkernel32'
    - '-lmingw32'
    - '-lgcc'
    - '-lgcc_eh'
    - '-lmoldname'
    - '-lmingwex'
    - '-lmsvcrt'
...

I called it i386-w64-mingw32, and, Hey Presto! a Windows cross compiler on my Linux box.

[~] dev% cat main.c
#include <stdio.h>

int main()
{
    printf("hello world\n");
}
[~] dev% ~/ellcc/bin/ecc -target i386-w64-mingw32 main.c
[~] dev% ./a.exe 
fixme:winediag:start_process Wine-Compholio is a Wine testing version containing experimental patches.
fixme:winediag:start_process Please don't report bugs at winehq.org and use our issue tracker instead:
fixme:winediag:start_process https://github.com/compholio/wine-compholio/issues
hello world
[~] dev% file a.exe
a.exe: PE32 executable (console) Intel 80386, for MS Windows
[~] dev%

Adding Networking Support to ELK: Part 3

Again I’m posting this as a post-in-progress. I was hoping to finish this before real life intrudes tomorrow. Things are going well but tomorrow looks like a real stretch goal.

In Part 1 of this post I wrote about some design goals for ELK networking. In Part 2, I described some of the implementation details that evolved as I integrated the LwIP TCP/IP stack into ELK. In this post, I’ll describe adding an Ethernet driver to ELK’s network implementation.

As in Part 2, I’m writing this as development is progressing, so be please patient if I tend to go off on tangents.

LwIP comes with a skeleton example Ethernet driver in src/netif/ethernetif.c. My ARM development environment is currently QEMU emulating the Versatile Express A9 board, which emulates the LAN 91C111 Ethernet controller. Eventually, I’d like to get ELK networking running on the Raspberry Pi, but since it uses a USB based Ethernet controller (the smsc9512), that will require a bit more work.

OK, enough with Google. Time to get to work. I’ll start or by making a copy of ethernetif.c which I have called lan91c111.c. Now I’ll start filling in the details.

I’ve made some basic changes to the driver and now I’ll add it to the list of ELK source files in sources.mk. I know the first build is going to spit out a bunch of errors, and sure enough it does. I’ll add a few “#if RICH” lines to mark places I have to add functionality. I got through the “#if RICH” part when I realized that I’m missing the whole infrastructure for socket ioctl() calls. I realized that when I started about how to initialize Ethernet devices. So I’m side tracking to add ioctl() support to the network interfaces.

As an aside, it turns out I had to make another change to the LwIP sources. One of the ioctl() calls supported on sockets is named IP_HDRINCL. It turns out that LwIP uses that name at the lower levels to represent a packet that already contains an address. I changed all instances of IP_HDRINCL to IP_HDRINCLUDED in LwIP to get around the naming conflict.

There are two interfaces that LwIP supports to configure Ethernet devices, basically because LwIP has two modes of operation as I mentioned in Part 1. The core of LwIP is meant to be used by a single (or the only) thread. That is the first mode, and is usually reserved for smaller systems where all network activity is done by a single thread. The second mode has a network thread which has a message based interface to all the other threads in the system. ELK uses the second mode, so we have to use the netifapi functions, defined in the lwip/netifapi.h file. I had to to enable this API in the ELK specific lwipopts.h file.

The ioctl() calls for sockets are documented in the netdevice(7) man page. We’ll see how well the LwIP API maps to Linux’s idea of network configuration.

There are a set of ioctl() calls defined that are used to configure network devices as described in netdevice(7). I’ve implemented a fairly large subset of socket ioctl() calls to prepare for actually adding the Ethernet driver. Here is an example of some of them:

/* ELK running as a VM enabled OS.
 */
#define _GNU_SOURCE
#include <sys/ioctl.h>
#include <net/if.h>
#include <net/if_arp.h>
#include <arpa/inet.h>
#include <errno.h>
#include <string.h>

#include <sys/cdefs.h>
#include <stdio.h>

#include "command.h"

int main(int argc, char **argv)
{
  setprogname("elk");
  printf("%s started. Type \"help\" for a list of commands.\n", getprogname());

  int sfd = socket(AF_INET, SOCK_STREAM /*|SOCK_NONBLOCK */, 0);
  if (sfd < 0) {
    printf("socket(AF_INET) failed: %s\n", strerror(errno));
  }

  int s;
  struct ifreq ifreq;
  struct in_addr in_addr;

  // Set the device name for subsequent calls.
  strcpy(ifreq.ifr_name, "ln01");

  // Set the interface IP{ address.
  inet_aton("192.124.43.4", &in_addr);
  ifreq.ifr_addr.sa_family = AF_INET;
  memcpy(ifreq.ifr_addr.sa_data, &in_addr, sizeof(in_addr));
  s = ioctl(sfd, SIOCSIFADDR, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFADDR) failed: %s\n", strerror(errno));
  }

  // Set ine interface netmask.
  inet_aton("255.255.255.0", &in_addr);
  ifreq.ifr_netmask.sa_family = AF_INET;
  memcpy(ifreq.ifr_netmask.sa_data, &in_addr, sizeof(in_addr));
  s = ioctl(sfd, SIOCSIFNETMASK, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFNETMASK) failed: %s\n", strerror(errno));
  }

  // Set the interface MAC address.
  ifreq.ifr_hwaddr.sa_family = ARPHRD_ETHER;
  ifreq.ifr_hwaddr.sa_data[0] = 0x01;
  ifreq.ifr_hwaddr.sa_data[1] = 0x02;
  ifreq.ifr_hwaddr.sa_data[2] = 0x03;
  ifreq.ifr_hwaddr.sa_data[3] = 0x04;
  ifreq.ifr_hwaddr.sa_data[4] = 0x05;
  ifreq.ifr_hwaddr.sa_data[5] = 0x06;
  s = ioctl(sfd, SIOCSIFHWADDR, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFHWADDR) failed: %s\n", strerror(errno));
  }
  printf("Try the command 'inetif'\n");

  // Enter the kernel command processor.
  do_commands(getprogname());
}

Where is what the compiled example produces:

[~/ellcc/examples/elk] dev% make run
Compiling main.c
Linking elk
Running elk
enter 'control-A x' to exit QEMU
audio: Could not init `oss' audio driver
elk started. Type "help" for a list of commands.
Try the command 'inetif'
elk % inetif
0 lo: active flags=137 (0x01)  mtu 0
        inet 127.0.0.1  netmask 255.0.0.0  broadcast 127.255.255.255
1 ln01: active flags=66 (0x32)  mtu 1500
        inet 192.124.43.4  netmask 255.255.255.0  broadcast 192.124.43.255
        ether 01:02:03:04:05:06
elk % 

ln01 is my stubbed driver. It looks like the ioctl() calls have worked. lwip_network.c is the source file implementing ELK's LwIP socket interface. Now it's time to get the driver to do something.

As I started to flesh out the driver I realized that the vexpress-a9 board that QEMU is emulating uses the LAN9118 Ethernet controller, not the SMC91C111 that I thought. That got me thinking. I really don't want to reinvent the wheel each time for all the different Ethernet controllers out there and it is always better to find something that exists and adapt it rather than to write it from scratch. With that in mind, I decided what it would take to use Ethernet drivers borrowed from NetBSD for ELK. As I see it, the first step is to get a NetBSD driver to compile in the ELK environment with as few source changes as possible. Step two is to make some glue code to make the NetBSD driver fit into the LwIP environment.

Yikes! I just spent a harrowing half a day trying to isolate the NetBSD driver for the LAN9118 so that it could "drop in" as a LwIP driver. I went down the path of stubbing out all the include files that were directly and indirectly included and compiling the source file, over and over, adding definitions to the include files as needed. It turns out that the driver has lots of dependencies on the NetBSD kernel, which is not surprising. I've decided to try a different route, but I did stumble on this interesting post about rump kernels, which are basically doing what ELK is doing, but with NetBSD as the basis.

I like the direction this new approach is taking. I'm using an idea from the LwIP wiki to create a driver framework. The driver skeleton has been fleshed out in ethernetif_driver.c and low level support for the LAN9118 is taking shape in lan9118.c. I hope I find some time over the next few days to fill in the reast of the details.

ELK: Closer to Embedded Linux Without the Linux

In a previous post I gave an update on the development status of ELK, the Embedded Little Kernel. ELK allows you to use the ELLCC tool chain to target bare metal environments. ELK is currently under development, but is available and becoming quite usable for the ARM.

ELK can be configured with a range of functionality, from a very simple “hello world” environment where you take control of everything, to a full MMU enabled virtual memory based system. In all cases, ELK uses the musl C standard library compiled for Linux so ELK can provide a very POSIX-like environment in the bare metal work (i.e. kernel space).

An example of elk in action can be found in the ELLCC source repository in the ELK example directory. You can configure the example to build four configurations:

  • Running from flash with no MMU.
  • Running from flash with virtual memory enabled.
  • Running from RAM with no MMU.
  • Running from RAM with MMU enabled.

The full ELK source code can be found here. Functionality currently supported by ELK:

  • Threading using pthread_create(). Many other thread synchronization functions are available, like POSIX mutexes and semaphores.
  • Virtual file system support, with a RAM, device, and fifo (pipe) file system supported currently.
  • A simple command processor for debugging and system testing.

ELK works by trapping and emulating Linux system calls. The current state of system call support is available on the system call status page.

ELK Status Update

I previously mentioned ELK, an Embedded Little (or Linux) Kernel, that can be used to do bare metal development with ELLCC. The goal of ELK is to use the musl Linux C standard run-time library to provide a POSIX environment on bare metal, i.e. in kernel space. I’ve been focused primarily on the ARM version of ELK as a proof of concept prototype, but I plan to port ELK to all the targets supported by ELLCC.

ELK is able to use the musl library compiled for Linux because it traps the Linux system calls and implements their functionality, or at least enough of their functionality to be useful in a bare metal environment.

I put together a status page that shows the progress I’ve been making on emulating the Linux systems calls in ELK. You can find it here. The source for all of this can be found in the ELLCC source repository. In particular, you can look at the glue that helps makes this work for the ARM, the ARM crt1.S file mentioned on the status page.

Building Lua with ELLCC This Time With a Twist

In a previous post, I used ELLCC to cross build Lua for the ARM running Linux. Subsequently I’ve done a bit of work to make ELLCC more configurable, and also to handle bare metal compilation more easily. I had a little time on my hands tonight so I thought I’d try out the changes with the latest Lua 5.2.3 version.
First, I modified the lua-5.2.3/src/Makefile to add the following around line 27:

# Start of ELLCC definitions.
PLAT= generic
ELLCC= /home/rich/ellcc
ELLCCBIN= $(ELLCC)/bin/
CC= $(ELLCCBIN)ecc
ELLCCPREFIX= $(ELLCCBIN)ecc-
AR= $(ELLCCPREFIX)ar rcu
RANLIB= $(ELLCCPREFIX)ranlib

TARGET= x86_64-linux-eng
MYCFLAGS= -target $(TARGET)
MYLDFLAGS= -target $(TARGET)
# End of ELLCC definitions.

Then I build for my host system, an x86_64 Linux box:

[~/lua-5.2.3/src] dev% make
/home/rich/ellcc/bin/ecc -O2 -Wall -DLUA_COMPAT_ALL  -target x86_64-linux-eng   -c -o lapi.o lapi.c
...
/home/rich/ellcc/bin/ecc -O2 -Wall -DLUA_COMPAT_ALL  -target x86_64-linux-eng   -c -o linit.o linit.c
/home/rich/ellcc/bin/ecc-ar rcu liblua.a lapi.o lcode.o lctype.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o lauxlib.o lbaselib.o lbitlib.o lcorolib.o ldblib.o liolib.o lmathlib.o loslib.o lstrlib.o ltablib.o loadlib.o linit.o 
/home/rich/ellcc/bin/ecc-ranlib liblua.a
/home/rich/ellcc/bin/ecc -O2 -Wall -DLUA_COMPAT_ALL  -target x86_64-linux-eng   -c -o lua.o lua.c
/home/rich/ellcc/bin/ecc -o lua  -target x86_64-linux-eng lua.o liblua.a -lm  
/home/rich/ellcc/bin/ecc -O2 -Wall -DLUA_COMPAT_ALL  -target x86_64-linux-eng   -c -o luac.o luac.c
/home/rich/ellcc/bin/ecc -o luac  -target x86_64-linux-eng luac.o liblua.a -lm  
[~/lua-5.2.3/src] dev% ./lua
Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio
> print ("hello")
hello
> 
[~/lua-5.2.3/src] dev% 

So far so good. How about a cross compile?

[~/lua-5.2.3/src] dev% make clean
rm -f liblua.a lua luac lapi.o lcode.o lctype.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o ltm.o lundump.o lvm.o lzio.o lauxlib.o lbaselib.o lbitlib.o lcorolib.o ldblib.o liolib.o lmathlib.o loslib.o lstrlib.o ltablib.o loadlib.o linit.o  lua.o luac.o
[~/lua-5.2.3/src] dev% make TARGET=arm-linux-engeabi
/home/rich/ellcc/bin/ecc -O2 -Wall -DLUA_COMPAT_ALL  -target arm-linux-engeabi   -c -o lapi.o lapi.c
...
/home/rich/ellcc/bin/ecc -o lua  -target arm-linux-engeabi lua.o liblua.a -lm  
/home/rich/ellcc/bin/ecc -O2 -Wall -DLUA_COMPAT_ALL  -target arm-linux-engeabi   -c -o luac.o luac.c
/home/rich/ellcc/bin/ecc -o luac  -target arm-linux-engeabi luac.o liblua.a -lm  
[~/lua-5.2.3/src] dev% ~/ellcc/bin/qemu-arm lua
Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio
> print ("hello")
hello
> 
[~/lua-5.2.3/src] dev% 

Now for a real test. How about a bare metal test on the ARM?

[~/lua-5.2.3/src] dev% rm lua
[~/lua-5.2.3/src] dev% make TARGET=arm-elk-engeabi
/home/rich/ellcc/bin/ecc -o lua  -target arm-elk-engeabi lua.o liblua.a -lm  
[~/lua-5.2.3/src] dev%  ~/ellcc/bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel lua
audio: Could not init `oss' audio driver
unhandled system call (256) args: 1, 268472376, 1610613215, 1610613023, 1207961673, 1241513744
Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio
> print ("hello")
unhandled system call (175) args: 1, 1241512616, 0, 8, 237792, 0
unhandled system call (174) args: 2, 1241512644, 1241512624, 8, 237792, 0
hello
unhandled system call (174) args: 2, 1241512644, 1241512624, 8, 237792, 0
> 

As described in the previous post, the unhandled system calls are those that haven’t been implemented in ELK yet. In this case, they are

...
#define __NR_rt_sigaction       174
#define __NR_rt_sigprocmask     175
...
#define __NR_set_tid_address    256
...

none of which are needed in my little test. Bare metal Lua. Luanix?

After I had this little bit of excitement, I remembered that I had configured ELK with the task scheduler and system timer. I figured it might be worth a try to see if at least the interrupt driven timer was working. Sure enough:

> print (os.date())
unhandled system call (174) args: 2, 1241512644, 1241512624, 8, 1207961832, 1241512696
unhandled system call (5) args: 276151, 657408, 0, 1, 1241510728, 1241511520
Thu Jan  1 00:07:22 1970
unhandled system call (174) args: 2, 1241512644, 1241512624, 8, 1207961832, 1241512696
> print (os.date())
unhandled system call (174) args: 2, 1241512644, 1241512624, 8, 1207961832, 1241512696
Thu Jan  1 00:07:45 1970
unhandled system call (174) args: 2, 1241512644, 1241512624, 8, 1207961832, 1241512696
> 

Time keeps on ticking. Fun stuff.

ELLCC Now Available for Windows

Last week someone mentioned that he’d like to run the ELLCC cross development tools on an operating system called “Windows”. Doing a little goolging proved that he might not be alone. Windows seems to be fairly popular, rivaling even Linux in popularity on the desktop. I decided to see what it would take to get ELLCC running on Windows. I spent the day Saturday installing MinGW-w64 on my development system and tweaking the ELLCC build rules to use it. The result is that I can now make a binary snapshot of ELLCC that runs on Windows which I have placed on the ftp site at ftp://ellcc.org/pub.

Here is an example of the results:

C:\cygwin\home\rich>ellcc\bin\ecc.exe -target x86_64-linux-eng hello.c

C:\cygwin\home\rich>ellcc\bin\ecc-size a.out
   text    data     bss     dec     hex filename
  15595     248    1608   17451    442b a.out

C:\cygwin\home\rich>

If I copy a.out to my Linux box, I get:

[~] dev% ellcc/bin/ecc-size a.out
   text    data     bss     dec     hex filename
  15595     248    1608   17451    442b a.out 
[~] dev% ./a.out 
hello world
[~] dev%

You can also use the work-in-progress ELK bare metal environment:

C:\cygwin\home\rich>ellcc\bin\ecc.exe -target arm-elk-engeabi hello.c -g

C:\cygwin\home\rich>ellcc\bin\ecc-size.exe a.out
   text    data     bss     dec     hex filename
  76923    2744   23096  102763   1916b a.out

C:\cygwin\home\rich>

If you have QEMU installed on your Windows box, you can run the bare metal executable as described in a previous blog post.

This makes it theoretically possible for me to do further development of ELLCC on Windows. I don’t think I will though.

Introducing ELK: A Bare Metal Environment for ELLCC

ELLCC (pronounced “elk”) is a C/C++ programming environment based on the clang/LLVM compiler. The goal of ELLCC is to provide a complete tool chain and run-time environment for developing C/C++ programs for embedded systems. ELLCC supports several Linux targets today, specifically ARM, Mips, PowerPC, and x86 systems. ELK (which might mean “embedded little kernel” or “embedded Linux kernel”) is a work-in-progress that allows the ELLCC tool chain to target bare metal environments where an OS is not available or needed.

What will differentiate ELK from other bare metal environments is that the goal is to provide a fairly complete Linux-like environment without the need for actually running Linux. ELK is in the design and development stage right now, but the basic functionality has been implemented based on work done using ELLCC for bare metal development and making the ELLCC tool chain easily configurable.

The thing that makes ELK fairly unique is that it uses the C and C++ libraries compiled for Linux. It does this by handling Linux system calls using the normal system call mechanism for each target. ELK has extensible system call handling capability so that new system call emulators can be added as needed. ELK is being developed by setting up the infrastructure for program start up in the bare metal environment and handling system calls and context switching in a small assembly language file. A fairly functional example for ARM can be seen here. System call support can be added easily. Here is a simple exit() system call example:

/* Handle the exit system call.
 */     
#include <bits/syscall.h>       // For syscall numbers.
#include <stdio.h>
#include <kernel.h>

// Make the simple console a loadable feature.
FEATURE(exit, exit)

static int sys_exit(int status)
{
    printf("The program has exited.\n");
    for( ;; )
      continue;
}       
        
CONSTRUCTOR()   
{       
    // Set up a simple exit system call.
    __set_syscall(SYS_exit, sys_exit);
}       

Here’s an example of a simple ARM program running on QEMU in system (bare metal) mode:

[~] dev% cat hello.cpp
#include <iostream>

int main()
{
  std::cout << "hello world" << std::endl;
}
[~] dev% ~/ellcc/bin/ecc++ -target arm-elk-engeabi hello.cpp -g
[~] dev% ~/ellcc/bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel a.out
audio: Could not init `oss' audio driver
unhandled system call (256) args: 1, 268472376, 1610613215, 1610613023, 1207961673, 1241513736
unhandled system call (45) args: 0, 0, 0, 1241512408, 590752, 0
unhandled system call (45) args: 4096, 4111, 1241512408, 1241512336, 592800, 1254352
unhandled system call (192) args: 0, 8192, 3, 34, -1, 0
unhandled system call (45) args: 4096, 4111, 34, 1241512336, 592800, -1
unhandled system call (192) args: 0, 8192, 3, 34, -1, 0
hello world
unhandled system call (248) args: 0, 1207962904, 1309668, 0, 1207991180, 1241513736
The program has exited.

Notice that there are several unhandled system calls (I did say that ELK is a work-in-progress) but enough has been implemented that “hello world” comes out.
The C version is a little quieter:

[~] dev% ~/ellcc/bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel a.out
audio: Could not init `oss' audio driver
unhandled system call (256) args: 1, 268472376, 1610613215, 1610613023, 1207961673, 1241513736
hello world
unhandled system call (248) args: 0, 0, 1207983756, 0, 1207985256, 1241513736
The program has exited.

The current state of ELK is that only the ARM version is at all functional. It supports multiple threads, memory allocation using malloc, semaphores, and timers. ELK started out as a couple of prototypes to test the feasibility of using Linux libraries on bare metal and to enhance the ability to configure the tool chain for multiple environments. I like the way that the development is going so far and hope to have a complete ARM implementation soon and support for the other targets soon after that. You can browse the complete source code for ELK.

ELLCC Cross Compilation Configuration

In my previous post I described an initial prototype of a scheme to configure ELLCC (my build of the clang compiler and other stuff) to make it easier to use in a cross compilation development environment. I have implemented a functional prototype of the ideas I described in that post and the latest binary snapshots of ELLCC contain the functionality that I’m describing here.

First a little background on what I’m trying to accomplish. Out of the box, ELLCC supports several processors with a C/C++ compiler based on clang/LLVM (ecc), a full set of run-time support libraries (libc++, musl libc, compiler-rt and others), and a set of the GNU binutils and GDB built to support all of the targets. The VERSIONS page has a list of all the current packages contained in ELLCC. ELLCC is built to support several processor families: ARM, Microblaze, Mips, PowerPC, and X86 (32 and 64 bit). ELLCC is intended to support these processors on both Linux and standalone OS-less targets with more environments to follow.

The problem is that each of these processor and OS (or OS-less) environments have a different set of include files and run-time libraries needed to do a successful build. In addition, all of the processor families have several similar but different processor variations that might require object files created by the compiler and those placed in the run-time libraries to be created specifically for the particular processor variant.

ecc has a bazillion command line options to control include file paths, processor types, ABI settings, and all sorts of other things. It is sometimes difficult to determine the exact set of options needed to target a particular environment. I have to admit that I’m sometimes a command line kind of guy and I like to build a simple “hello world” for a particular environment. Having to remember and type in all the correct options is not an ideal situation.

Usually, the main option you need for cross compiling is the -target option, which usually looks something like this:

-target arm-ellcc-linux

The target option takes a target triple, which specifies the processor, the vendor, and the target OS. If I try to build a simple test program for an ARM processor, I get this:

[~] dev% ~/ellcc/bin/ecc -target arm-ellcc-linux test.c
/home/rich/ellcc/bin/ecc-ld: error: /tmp/test-0df798.o uses VFP register arguments, a.out does not
/home/rich/ellcc/bin/ecc-ld: failed to merge target specific data of file /tmp/test-0df798.o
ecc: error: linker command failed with exit code 1 (use -v to see invocation)
[~] dev% 

Obviously I don’t have everything that I need on the command line so I liike around and finally come up with the right combination:

[~] dev% ~/ellcc/bin/ecc -target arm-ellcc-linux test.c -march=armv7 -mfpu=vfp -mfloat-abi=softfp 
[~] dev% 

In addition, if I want to build for another ARM variant I get nice messages like this:

[~] dev% ~/ellcc/bin/ecc -target arm-ellcc-linux test.c -mcpu=cortex-m4         /home/rich/ellcc/bin/ecc-ld: error: /tmp/test-e8da00.o uses VFP register arguments, a.out does not
/home/rich/ellcc/bin/ecc-ld: error: /tmp/test-e8da00.o: Conflicting architecture profiles M/A
/home/rich/ellcc/bin/ecc-ld: failed to merge target specific data of file /tmp/test-e8da00.o
/home/rich/ellcc/bin/ecc-ld: error: /home/rich/ellcc/bin/../libecc/lib/arm/linux/libc.a(syscall.o): Conflicting CPU architectures 13/0
ecc: error: linker command failed with exit code 1 (use -v to see invocation)
[~] dev% 

I decided to add a configuration capability to ecc to allow command line options and other environment specific stuff to be specified once in a common place with the ability to easily override the defaults if necessary.

LLVM has a mechanism to read and write YAML format files easily. It is called YAMLIO. The approach I’ve taken is to use YAML format configuration information to configure ELLCC to handle all the different variations need to build for different target environments.

Now for some gory implementation details. I modified ecc to handle all target triples with the ellcc vendor with YAML formatted configuration information. The default configurations are defined in the Tools.cpp file and look something like this:

namespace {
const char ellcc_linux[] =
  "global:\n"
  "  static_default: true\n"
  "compiler:\n"
  "  cxx_include_dirs:\n"
  "    - $R/include/c++\n"
  "assembler:\n"
  "  output:\n"
  "    - -o $O\n"
  "linker:\n"
  "  exe: $E/ecc-ld\n"
  "  output:\n"
  "    - -o $O\n"
  "  start:\n"
  "    - -e _start\n"
  "  opt_static:\n"
  "    - -Bstatic\n"
  "  opt_rdynamic:\n"
  "    - -export-dynamic\n"
  "  opt_dynamic:\n"
  "    - -Bdynamic\n"
  "  opt_shared:\n"
  "    - -shared\n"
  "  opt_notshared:\n"
  "    - -dynamic-linker /usr/libexec/ld.so\n"
  "  opt_pthread:\n"
  "    - -pthread\n"
  "  cxx_libraries:\n"
  "    - -lc++\n"
  "    - -lm\n"
  "  profile_libraries:\n"
  "    - -lprofile_rt\n"
  "  c_libraries:\n"
  "    - -lc\n"
  "    - -lcompiler_rt\n"
  "";

const char arm_ellcc_linux[] =
  "based_on: ellcc-linux\n"
  "compiler:\n"
  "  options:\n"
  "    - -target arm-ellcc-linux\n"
  "  c_include_dirs:\n"
  "    - $R/include/arm/linux\n"
  "    - $R/include/arm\n"
  "    - $R/include/linux\n"
  "    - $R/include\n"
  "assembler:\n"
  "  exe: $E/arm-elf-as\n"
  "linker:\n"
  "  options:\n"
  "    - -m armelf_linux_eabi\n"
  "    - --build-id\n"
  "    - --hash-style=gnu\n"
  "    - --eh-frame-hdr\n"
  "  static_crt1: $R/lib/arm/linux/crt1.o\n"
  "  dynamic_crt1: $R/lib/arm/linux/Scrt1.o\n"
  "  crtbegin: $R/lib/arm/linux/crtbegin.o\n"
  "  crtend: $R/lib/arm/linux/crtend.o\n"
  "  library_paths:\n"
  "    - -L $R/lib/arm/linux\n"
  "  c_libraries:\n"
  "    - -(\n"                  // This is needed for the ARM.
  "    - -lc\n"
  "    - -lcompiler_rt\n"
  "    - -)\n"
  "";
... (more definitions here)

These definitions are registered at ecc start up time with code like this:

using namespace clang::compilationinfo;
class ELLCC {
public:
  ELLCC() {
    CompilationInfo::DefineInfo("ellcc-linux", ellcc_linux);
    CompilationInfo::DefineInfo("arm-ellcc-linux", arm_ellcc_linux);
    CompilationInfo::DefineInfo("armeb-ellcc-linux", armeb_ellcc_linux);
    CompilationInfo::DefineInfo("i386-ellcc-linux", i386_ellcc_linux);
    CompilationInfo::DefineInfo("microblaze-ellcc-linux",
                                microblaze_ellcc_linux);
    CompilationInfo::DefineInfo("mips-ellcc-linux", mips_ellcc_linux);
    CompilationInfo::DefineInfo("mipsel-ellcc-linux", mipsel_ellcc_linux);
    CompilationInfo::DefineInfo("ppc-ellcc-linux", ppc_ellcc_linux);
    CompilationInfo::DefineInfo("ppc64-ellcc-linux", ppc64_ellcc_linux);
    CompilationInfo::DefineInfo("x86_64-ellcc-linux", x86_64_ellcc_linux);
  }
} ELLCC;

Now, if I use a command line option like “-target arm-ellcc-linux”, the appropriate compliation information is found and used. Notice that there is a hierarchy in these configurations. “arm-ellcc-linux” is based on “ellcc-linux”, which means it inherits that configuration information.

I addition to the predefined configurations, I’ve added a mechanism to read configurations from a file. If the target option is specified with e.g. “-target armv7-linux” and a file exists in the current directory or a predefined config directory (ellcc/libecc/config) named “armv7-linux” then that file is read to define the compilation information. A typical configuration file looks like this:

[~/ellcc/libecc/config] dev% cat armv7-linux 
based_on: arm-ellcc-linux
compiler:
  options:
    - -target arm-ellcc-linux
    - -march=armv7
    - -mfpu=vfp
    - -mfloat-abi=softfp

Note that it is based on the predefined arm-ellcc-linux configuration and just adds a few command line options.

[~] dev% ~/ellcc/bin/ecc -target armv7-linux test.c
[~] dev% 
Voila!

I also added a little debugging capability. If I set a field called "dump" to true like this:
dump: true
based_on: arm-ellcc-linux
compiler:
  options:
    - -target arm-ellcc-linux
    - -march=armv7
    - -mfpu=vfp
    - -mfloat-abi=softfp

I get the complete compilation configuration:

 
[~] dev% ~/ellcc/bin/ecc -target armv7-linux test.c
# based_on `arm-ellcc-linux'
# based_on `ellcc-linux'
---
dump:            true
based_on:        ''
global:          
  static_default:  true
compiler:        
  options:         
    - -target arm-ellcc-linux
    - '-march=armv7'
    - '-mfpu=vfp'
    - '-mfloat-abi=softfp'
  c_include_dirs:  
    - '$R/include/arm/linux'
    - '$R/include/arm'
    - '$R/include/linux'
    - '$R/include'
  cxx_include_dirs: 
    - '$R/include/c++'
assembler:                                                                      
  exe:             '$E/arm-elf-as'                                              
  output:                                                                       
    - '-o $O'                                                                   
linker:                                                                         
  exe:             '$E/ecc-ld'                                                  
  output:                                                                       
    - '-o $O'                                                                   
  start:                                                                        
    - -e _start                                                                 
  opt_static:      
    - -Bstatic
  opt_rdynamic:    
    - -export-dynamic
  opt_dynamic:     
    - -Bdynamic
  opt_shared:      
    - -shared
  opt_notshared:   
    - -dynamic-linker /usr/libexec/ld.so
  opt_pthread:     
    - -pthread
  options:         
    - -m armelf_linux_eabi
    - --build-id
    - '--hash-style=gnu'
    - --eh-frame-hdr
  static_crt1:     '$R/lib/arm/linux/crt1.o'
  dynamic_crt1:    '$R/lib/arm/linux/Scrt1.o'
  crtbegin:        '$R/lib/arm/linux/crtbegin.o'
  crtend:          '$R/lib/arm/linux/crtend.o'
  library_paths:   
    - '-L $R/lib/arm/linux'
  cxx_libraries:   
    - '-lc++'
    - -lm
  profile_libraries: 
    - -lprofile_rt
  c_libraries:     
    - '-('
    - -lc
    - -lcompiler_rt
    - '-)'
...
[~] dev% 

An Initial Prototype of a clang Cross Compile Config File

UPDATE: The latest update to the configuration info is described here.

Today I decided to play around with some ideas for making clang a little more friendly for cross compiling. I added the ability to make the -target option look for and read in a configuration file if one is available. I build and use clang for cross compiling stuff. Up until now I’ve been using makefile magic to differentiate between different processor families and architectures. The original idea came from this mailing list post.

The thought behind my prototype is this: If a -target option is specified, or the clang executable name has a prefix (e.g. cortex-m4-ecc, ecc is my name for clang), then attempt to read a config file from the resource directory with the name of the -target argument or the executable prefix.

The config file would then be used to set things up to compile for a particular target.
Here is an example config file:

compiler:
  options:
    - -target arm-ellcc-linux-eabi5
    - -mcpu=cortex-m4
assembler:
  exe: $E/arm-elf-as
  options:
linker:
  exe: $E/ecc-ld
  options:
    - -m armelf_linux_eabi
    - --build-id
    - --hash-style=gnu
    - --eh-frame-hdr

  static_crt1: $R/lib/arm/linux/crt1.o
  dynamic_crt1: $R/lib/arm/linux/Scrt1.o
  crtbegin: $R/lib/arm/linux/crtbegin.o
  crtend: $R/lib/arm/linux/crtend.o
  library_path: $R/lib/arm/linux

I implemented the prototype this weekend, I like the way it feels. I created a CompilationInfo class that is passed to the Driver if a config file is used. I use the class in Tools.cpp if it is available to configure the assembler and linker invocations. Cool stuff.
This week I’ll use it to build libraries and programs for a variety of ARM variants. I’d also like to think about specifying #include file paths in the config file this week.

A few more details on the implementation so far. I use the -target argument, or the prefix on the compiler name, to try to open the config file. If a file of that name isn’t found, the -target argument passes through unscathed and works as it does now. I use YAMLIO to read the configuration into a structure that currently looks like this:

  typedef std::vector StrSequence;
  struct Compiler {
    StrSequence options;
  };

  struct Assembler {
    std::string exe;
    StrSequence options;
  };

  struct Linker {
    std::string exe;
    StrSequence options;
    std::string static_crt1;
    std::string dynamic_crt1;
    std::string crtbegin;
    std::string crtend;
    std::string library_path;
  };

  struct CompilationInfo {
    Compiler compiler;
    Assembler assembler;
    Linker linker;
  };

After some good discussion on the LLVM mailing list, I realized that with a simple registration process, statically initialized info structures could be registered by preexisting drivers. This would obviously eliminate the need to read the config file for every compilation.

Does this sound like a reasonable approach?