Author Archives: rich

ELLCC Bare Metal ARM Update

An update to the ELLCC bare metal experiment last reported here


  • Targets the vexpress-a9 Cortex-A9 based evaluation card.
  • SP804 timer supported.
  • PL011 Serial port supported for the system console.
  • Fully interrupt driven operation for the timers and serial port.
  • Vectored interrupts supported using the ARM Generic Interrupt Controller.
  • The scheduler now supports multiple thread priorities, with the number of priorities from 1 to N specified at compile time.
  • Round robin scheduling supported.
  • Tick-less kernel scheduling: no timer interrupts occur unless specifically required for time-slicing.

An example of some tests and commands:

../../bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel kernel.bin
audio: Could not init `oss' audio driver
kernel started. Type "help" for a list of commands.
kernel % thread1
thread started foo
kernel % thread2
thread2 started
kernel % thread3
unhandled system call (175) args: 1, 1216308, 0, 8, 8, 0
thread3 started
kernel % thread4
thread4 started
kernel % thread5
thread5 started
kernel % thread2 still running
unrecognized command: is
kernel % ts
     TID  STATE        PRI NAME       
 0x28d58: RUNNING        1 kernel     
 0x2cd98: IDLE           3 idle0      
 0x2f550: MSGWAIT        1 thread1    
 0x305b0: TIMEOUT        1 thread2    
 0x31670: READY          1 clone1     
 0x326e0: SEMWAIT        1 thread4    
 0x33740: SEMTMO         1 thread5    
kernel % thread5 running
thread2 still running
thread5 running
thread2 still running
thread5 running
thread2 still running
thread5 running
thread2 still running
thread5 running
thread2 still runningindepenent
thread5 running
thread2 still running
kernel % date
Thu Jan  1 00:00:02 1970
kernel % help date
                date: show/set the system clock.
                      The argument used to set the time, if given,
                      is in the form [MMDDhhmm[[CC]YY][.ss]].
kernel % date 050408272014
kernel % date
Sun May  4 08:27:48 2014
kernel % 

The ARM specific source code is available here and the processor independent code is here.

If you want to try this out at home, everything you need except QEMU is packaged as a binary download from choose the tarball appropriate for your host system, untar it, go into the ellcc/baremetal/arm directory and type “make run”.

Make sure you have QEMU installed on your system, as I am not currently able to cross make it for all the hosts.

Bare Metal ARM with musl Gets a Little Friendlier

UPDATE: More info on this little project is here.

So, this thing is getting a little out of hand. “This thing”, in case you haven’t been here before, is my experiment with using a standard Linux C library on a bare metal ARM board. The way that I’m making it work is by writing simple Linux system call handlers to do what the normal Linux system call would do but in a much simplified way. In my last post I described a little about how my test kernel is built and run and how to connect to it with GDB for debugging.

This time I have added some interrupt handlers and implemented a few more system calls. My first attempt at interrupt handling is for the SP804 dual timer to handle the POSIX monotonic and realtime system clocks. After one of my last posts, someone asked what ARM I was targeting. It turned out to be a great question for which I had no clue how to answer. There are so many ways to say ARM: arm7, armv7, cortex, … Diving in to this project at first I had no idea about the nuances. It turns out that that what I’m trying to target at first is the Cortex-a9, specifically as emulated by QEMU on the vexpress-a9 board.

Ah! you say. Then why the heck are you using the SP804 timer for that? You should be using the 64 bit timers in the private memory region! Indeed I should, and that will be an exercise for another day. The a15 is even cooler, with its virtual timer, but I digress.

In this update, I’ve also added some simple command line processing:

[~/ellcc/baremetal/arm] dev% make run
../../bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel kernel.bin
audio: Could not init `oss' audio driver
kernel started. Type "help" for a list of commands.
kernel % help
                date: show/set the realtime timer.
                time: time the specified command with arguments.
               sleep: sleep for a time period.
Test Commands:
             syscall: test the syscall interface with an unhandled system call.
               yield: yield the current time slice.
             thread1: start the thread1 test case.
               send1: send a message to the thread1 test thread.
             cancel1: cancel the thread1 test thread.
             thread2: start the thread2 test case.
kernel % time date
Thu Jan  1 00:23:50 1970
elapsed time: 0.007252000 sec
kernel %

Kind of cool, right? Well, just to show that some rudimentary threading is going on:

kernel % thread1
unhandled system call (175) args: 1, 1194164, 0, 8, 8, 0
thread started foo
thread self = 0x00029EB0
kernel % send1
thread running 3
kernel % 

Even better, here’s a test case where the thread periodically sleeps:

kernel % thread2
thread2 started
kernel % thread2 still running
thread2 still running
thread2 still running
thread2 still running

The code for thread2 looks like this:

static void *thread2(void *arg)
    printf ("thread2 started\n");
    for ( ;; ) {
        // Go to sleep.
        printf ("thread2 still running\n");

    return NULL;

Unfortunately it will be running forever. I haven’t implemented pthread_cancel() or pthread_kill() support yet. But the command prompt is still active:

thread2 still running
thread2 still running
thread2 still running
timethread2 still running
 sleep 1
elapsed time: 1.000731000 sec
kernel % 

(I had typed in “time sleep 1″)

As usual, the code is available here. Please don’t look at irq.c. The interrupt controller code is a total hack for now.

UPDATE: If you want to try this out at home, everything you need except QEMU is packaged as a binary download from choose the tarball appropriate for your host system, untar it, go into the ellcc/baremetal/arm directory and type “make run”.

Make sure you have QEMU installed on your system, as I am not currently able to cross make it for all the hosts.

Even More Bare Metal ARM

I’ve spend much of the weekend (it is a holiday, right?) playing around with my bare metal prototype. In the last post in my Bare Metal ARM series, I did a little context switching. Actually it implemented simple co-routines that could give up execution to each other. In this installment, I’ve added a rudimentary ready list and implemented a simple message passing scheme allowing threads to communicate with each other and block if no messages are available.

The full source of my little prototype is here. A little explanation of how I’m developing this might be in order. init.S is the processor initialization and exception handling code, written in assembly because most of it has to be. init.S calls into the C standard library by calling __libc_start_main() after setting up a few stack pointers and zeroing out the uninitialized data (.bss) area. __libc_start_main() ends up calling main() after initializing the library.

The Makefile can be used to build and run the code.

[~/ellcc/baremetal/arm] dev% make
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp  -c init.S
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c main.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c simple_console.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c simple_memman.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c scheduler.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -nostartfiles -T kernel.ld \
    ../../libecc/lib/arm/linux/crtbegin.o \
    init.o main.o simple_console.o simple_memman.o scheduler.o \
    ../../libecc/lib/arm/linux/crtend.o \
    -o kernel.elf -Wl,--build-id=none
../../bin/ecc-objcopy -O binary kernel.elf kernel.bin
[~/ellcc/baremetal/arm] dev% make run
../../bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel kernel.bin
audio: Could not init `oss' audio driver
kernel: hello world
unhandled system call (0) args: 1, 2, 3, 4, 5, 6
__syscall(0) = -1, No error information
hello from context 42 code = 3
hello from context 42 code = 6809
prompt: hi there
got: hi there

To exit out of QEMU use the key sequence Control-A x.

By the way, the send and receiving of messages (the “hello from context” messages) is done in main.c:

Queue queue = {};
static intptr_t context(intptr_t arg1, intptr_t arg2)
    for ( ;; ) {
      Message *msg = get_message(&queue);
      printf("hello from context %" PRIdPTR " code = %d\n", arg1, msg->code);
    return 0;
int main(int argc, char **argv)
    new_thread(context, 4096, 42, 0);
    Message msg = { { NULL, sizeof(msg) }, 3 };
    send_message(&queue, &msg);
    msg.code = 6809;
    send_message(&queue, &msg);

I do debugging by using two windows, one to start QEMU in debug mode and the other to run GDB. The first window:

[~/ellcc/baremetal/arm] dev% make debug
../../bin/qemu-system-arm -s -S -M vexpress-a9 -m 128M -nographic -kernel kernel.bin
audio: Could not init `oss' audio driver

The other window looks like:

[~/ellcc/baremetal/arm] dev% ~/ellcc/bin/ecc-gdb kernel.elf
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kernel.elf...done.
(gdb) target remote :1234
Remote debugging using :1234
0x60000000 in ?? ()
(gdb) break main
Breakpoint 1 at 0x10408: file main.c, line 40.
(gdb) c

Breakpoint 1, main (argc=1, argv=0x100e8 ) at main.c:40
40          printf("%s: hello world\n", argv[0]);

One aspect of this code that might be confusing is that there is no apparent explicit initialization code in main() for the serial port, memory allocator, or scheduler. Since this is implemented with a full C library, I take advantage of the fact that static constructors work and are extensions to C in both clang and GCC. An example is at the bottom of the scheduler.c file:

/* Initialize the scheduler.
static void init(void)
    __attribute__((__constructor__, __used__));
static void init(void)
    // Set up the main and idle threads.
    idle_thread.saved_sp = (Context *)&idle_stack[IDLE_STACK];
    __new_context(&idle_thread.saved_sp, idle, Mode_SYS, NULL,
                  0, 0);
    // The main thread is what's running right now. = &idle_thread;
    ready = &main_thread;
    // Set up a simple set_tid_address system call.
    __set_syscall(SYS_set_tid_address, sys_set_tid_address);

Why did I do it this way? Because I don’t have to change any source code if I want to swap out simple_console.c for a hypothetical interrupt_console.c, for example. I just have to change the SRCS macro in the Makefile.

The next step is to implement interrupt handlers, the timer, and preemptive scheduling. By the way, the full POSIX pthread_create() is still a ways off, but it feels like it is getting closer all the time.

More Bare Metal ARM With Linux C Libraries

As a follow up to my post An ARM Bare Metal Hello World Using Musl I thought I’d give an update. A quick background. I’m trying to build a bare metal (no OS) framework that can use a stock Linux standard C library. To do this I’m building a small assembly infrastructure that traps the Linux system calls used by the library and deals with them. In my previous post, I got “hello world”. In this one, I’ve come a bit farther. I can read stdin, malloc() memory and do simple context switching. You can see the code I’m working on here. In this example, printf(), fgets() and malloc() is usable enough to be used for a simple context switching example.

Most of the example main program (THREAD is not defined yet: don’t get too excited):

long __syscall_ret(unsigned long r);
long __syscall(long, ...);

#define CONTEXT
#if defined(CONTEXT)
static void *main_sa;
static void *context1_sa;
static void *context2_sa;

static int context(intptr_t arg1, intptr_t arg2)
    void **context_sa = (void **)arg2;
    for ( ;; ) {
      printf("hello from context %" PRIdPTR "\n", arg1);
      __switch(context_sa, main_sa);
    return 0;

int main(int argc, char **argv)
    printf("%s: hello world\n", argv[0]);

    int i = __syscall_ret(__syscall(0, 1, 2, 3, 4, 5, 6));
    printf("__syscall(0) = %d, %s\n", i, strerror(errno));

#if defined(THREAD)
    int s;
    pthread_attr_t attr;
    s = pthread_attr_init(&attr);
    pthread_t id;
    if (s != 0)
        printf("pthread_attr_init: %s\n", strerror(errno));
    s = pthread_create(&id, &attr, &thread, NULL);
    if (s != 0)
        printf("pthread_create: %s\n", strerror(errno));
#if defined(CONTEXT)
    char *p = malloc(4096);
    context1_sa = p + 4096;
    __new_context(&context1_sa, context, Mode_SYS, NULL, 42, (intptr_t)&context1_sa);
    p = malloc(4096);
    context2_sa = p + 4096;
    __new_context(&context2_sa, context, Mode_SYS, NULL, 6809, (intptr_t)&context2_sa);
    // Let's do some context switching.
    __switch(&main_sa, context1_sa);
    __switch(&main_sa, context2_sa);
    __switch(&main_sa, context1_sa);
    __switch(&main_sa, context2_sa);
    __switch(&main_sa, context2_sa);

    for ( ;; ) {
        char buffer[100];
        fputs("prompt: ", stdout);
        fgets(buffer, sizeof(buffer), stdin);
        printf("got: %s", buffer);

The output:

~/ellcc/baremetal/arm] dev% qemu-system-arm -cpu any -M versatilepb -m 128M -nographic -kernel kernel.bin
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
kernel: hello world
unhandled system call (0) args: 1, 2, 3, 4, 5, 6
__syscall(0) = -1, Function not implemented
hello from context 42
hello from context 6809
hello from context 42
hello from context 6809
hello from context 6809
prompt: hello world
got: hello world

It’s almost starting to look like Posix at the bare metal level. Fun stuff.

An ARM Bare Metal Hello World Using Musl

I’ve been trying to figure out how to use the excellent musl C standard library in a bare metal environment for quite a while. I didn’t want to damage the musl source code too much but couldn’t figure out an easy way to get at least a subset of musl’s functionality available in a bare metal environment.

Over the weekend I can up with a cute little solution that may be viable for future development. It’s based on a simple concept: Don’t change musl at all, but implement system call handling at the machine level to handle the system calls that musl needs.

I added a baremetal directory to the top-level ELLCC directory as a proof of concept. The arm sub-directory is where the initial work was done. There are just a few files in the directory:

  • Makefile – To build the example
  • init.S – To handle system initialization and exception handling
  • main.c – A simple C source file showing the handling of two system calls
  • kernel.ld – A simple linker command file to put it in the right place in the ARM’s memory.

It turns out that musl’s printf does two system calls to print “hello world\n”. The first is an ioctl() to determine how to buffer. I ignored that call since all the unimplemented system calls return error.
The interesting call that musl did make was to writev(), for which I made a simple replacement in startup.c.

After typing “make”, here is the result of running it:

[~/ellcc/baremetal/arm] dev% qemu-system-arm -cpu any -M versatilepb -m 128M -nographic -kernel kernel.bin
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
unhandled system call 54
hello world
QEMU: Terminated
[~/ellcc/baremetal/arm] dev%

The 54 is the unhandled ioctl(). I’m sure that output will be very handy as I implement more of musl’s functionality for my bare metal environment.

Using C++11 Lambdas with ELLCC

A recent post on C++11 lambdas and a forum question led me to this topic. Lambdas are a new C++ feature, described very well in the first link. The forum post asked how to build a simple test program using lambdas with ellcc:

#include <functional>
#include <iostream>
#include <string>
int main(int argc, char **argv){
    std::string test;
    test += "Hello World!";
    std::cout << test << std::endl;
    auto lambda = [&](const std::string &in) -> void {
        std::cout << "Text was: " << in << std::endl;
    lambda("extra text.");
    return 0;

The simple solution was:

[~] dev% ellcc/bin/ecc++ -target x86_64-ellcc-linux --std=c++11 test.cpp
[~] dev% ./a.out
Hello World!
Text was: extra text.
[~] dev%

The important thing here is that you need the -target option to set the appropriate target environment and you need to specify --std=c++11 to enable C++11 language features.

Interesting Linux Kernel Bug Uncovered by musl

ELLCC uses the very cool musl standard C library as a replacement for the normal Linux standard library. In the latest version of ELLCC, ELLCC was not able to compile itself on an x86_64 Fedora 20 Linux system. I was stumped for a while trying to track down the problem. It was weird: self hosting worked on a 32 bit Linux system (Fedora 19), but failed on a 64 bit system? Furthermore, self hosting only failed with ELLCC compiled with itself and linked with musl, but not with ELLCC compiled with itself and linked with glibc.

Fortunately, Rich Felker of musl fame shed some light on the issue. Here’s an edited IRC log:

04:34:00 PM - rdp: OK. malloc fails on my x86_64 linux after about 65527 4K allocations with musl malloc(). glibc malloc() doesn't, probably because it reverts to mmap() if brk fails. Yet I don't see any resource limits set. The gloibc brk() also failes after about 64K allocations.
04:37:52 PM - dalias: rdp, oh, we've seen this before
04:37:57 PM - dalias: it's a kernel bug with some optional kernel feature
04:38:21 PM - dalias: it keeps the kernel from merging adjacent vma's, so you end up with 64k pages each as their own tiny vma
04:38:36 PM - rdp: Excellent.
04:38:38 PM - dalias: it would happen if we used mmap too
04:38:50 PM - dalias: the reason it doesn't affect glibc is that they allocate huge amounts at a time
04:38:58 PM - rdp: Ah.
04:38:59 PM - dalias: and thereby waste memory if the program doesn't actually need much
04:39:17 PM - dalias: i'll try to find the option
04:39:21 PM - rdp: Any work around?
04:39:25 PM - rdp: OK. Thanks.
04:40:20 PM - dalias: CONFIG_MEM_SOFT_DIRTY
04:40:23 PM - dalias: turn it off
04:40:28 PM - dalias: there might be a way to do it at runtime
04:40:42 PM - dalias: or you could increase the limit on # of vma's
04:40:50 PM - dalias: but basically this option wastes MASSIVE amounts of ram
04:40:57 PM - dalias: by refusing to merge vma's
04:41:47 PM - dalias: it's a hack to make process checkpointing (save and restore running processes) more efficient
04:42:01 PM - dalias: by better tracking what has changed
04:42:50 PM - dalias: i don't see a way to turn it off
04:42:55 PM - dalias: check /proc/$pid/maps tho
04:43:08 PM - dalias: you should see a separate line for each page (i.e. 64k lines)
04:43:16 PM - dalias: if this is the issue that's affecting you
04:43:39 PM - rdp: I do.
04:43:51 PM - dalias: ok then this is the issue
04:43:57 PM - dalias: you can just up the limit if you want
04:44:04 PM - dalias: /proc/sys/vm/max_map_count
04:44:09 PM - dalias: but again this is expensive
04:44:16 PM - dalias: you want to disable CONFIG_MEM_SOFT_DIRTY
04:44:21 PM - dalias: and we really need to report this bug to the kernel folks                                                                             
04:44:25 PM - dalias: i don't think they're aware of it                       
04:45:11 PM - rdp: dalias: Thanks.                                            
04:46:47 PM - rdp: dalias: is it x86_64 specific? Not on i386?                
04:49:13 PM - dalias: rdp, i think it may be                                  
04:50:26 PM - dalias:
04:50:28 PM - feepbot: Analyzing cause of performance regression with different kernel version - Stack Overflow
04:51:38 PM - dalias: the accepted answer tracked down the cause of the soft_dirty bug and seems to cover how to fix it
04:53:02 PM - rdp: gotta love stackoverflow
05:47:19 PM - dalias: rdp, haha with regard to that SO answer:
05:47:26 PM - dalias: Finally fixed in Linux 3.13.3 and Linux 3.12.11, released 2014-02-13. – osgx 21 hours ago
05:57:32 PM - rdp: dalias: :-)
07:00:00 PM - dalias: rdp, i think it would be worth adding the issue you had to the faq on the wiki
07:00:51 PM - dalias: with a link to the stack overflow question/answer and information that it's fixed in 3.13.3, and that you can work around it by turning off CONFIG_MEM_SOFT_DIRTY (good fix) or increasing max_map_count (expensive fix)

For now, I got around the problem by using Rich’s expensive fix option (as superuser):

echo 128000 > /proc/sys/vm/max_map_count

Why didn’t ELLCC linked with glibc fail? Somebody considered it a bug at one point, but the glibc maintainers disagreed, I guess.
Rich pointed out that my guess about why the glibc malloc() doesn’t fail is probably wrong. But it is still a kernel bug nevertheless.

Building NetBSD Userland with ELLCC.

I’ve started a new project to build the core NetBSD userland programs for Linux using ELLCC. I have two motivations for doing this:

  • To have a large set of programs to compile for testing ELLCC. I’ve been looking for something for a while, and my stackoverflow question didn’t come up with any responses.
  • I’d like to eventually have a full set of userland commands for Linux that are not GPL licensed.

So far, the results have been promising. I have a nice subset of the NetBSD userland compiled so far:

ls x86_64-linux/bin x86_64-linux/usr/bin
[*      cp*    domainname*  expr*      ksh*  mkdir*  pwd*    sh*     tar*
cat*    cpio*  echo*        hostname*  ln*   mv*     rm*     sleep*  test*
chmod*  date*  ed*          kill*      ls*   pax*    rmdir*  sync*

apply*     calendar*  crunchgen*  expand*  getconf*  lastcomm*  make*
asa*       checknr*   crunchide*  fgen*    getopt*   leave*     od*
at*        cmp*       csplit*     find*    groups*   locate*    tar@
atq*       col*       ctags*      flock*   head*     lock*      uncompress*
atrm*      colcrt*    cut*        fmt*     hexdump*  logname*   whoami*
banner*    colrm*     deroff*     fold*    id*       look*
basename*  column*    dirname*    fpr*     indent*   m4*
batch*     comm*      du*         from*    join*     mail*
biff*      compress*  env*        fsplit*  jot*      Mail*
cal*       cpio@      error*      ftp*     lam*      mailx*
[~/ellcc-bsd] dev%

The files in that listing represent the low hanging fruit: They are the files that required little or no editing to compile with ELLCC.

If you want to try to build the NetBSD userland using ELLCC, first install ELLCC itself. Then you can grab the current ellcc-bsd project files:

[~] dev% svn co ellcc-bsd

This will create the ellcc-bsd directory right next to the already existing ellcc directory.

The next step is to build the bmake program (the Berkeley make program, used by NetBSD):

[~] dev% cd ellcc-bsd/
[~/ellcc-bsd] dev% ./build

This will put the host bmake executable in the ~/ellcc/bin directory.

Now all you have to do is

[~/ellcc-bsd] dev% ~/ellcc/bin/bmake
[~/ellcc-bsd] dev% ~/ellcc/bin/bmake install

This will create the populated directories list above.

For a little more fun try building ARM executables:

[~/ellcc-bsd] dev% ~/ellcc/bin/bmake TARGET=arm
[~/ellcc-bsd] dev% ~/ellcc/bin/bmake TARGET=arm install
[~/ellcc-bsd] dev% ls arm-linux/bin/ arm-linux/usr/bin/
[*      cp*    domainname*  expr*      ksh*  mkdir*  pwd*    sh*     tar*
cat*    cpio*  echo*        hostname*  ln*   mv*     rm*     sleep*  test*
chmod*  date*  ed*          kill*      ls*   pax*    rmdir*  sync*

apply*      csplit*   hexdump*   mkstr*         sdiff*       uniq*
asa*        ctags*    id*        mktemp*        sed*         units*
at*         cut*      indent*    mkubootimage*  seq*         unvis*
atq*        deroff*   join*      msgc*          shlock*      users*
atrm*       dirname*  jot*       msgs*          shuffle*     uudecode*
banner*     du*       lam*       nbperf*        soelim*      uuencode*
basename*   env*      lastcomm*  newgrp*        sort*        uuidgen*
batch*      error*    leave*     newsyslog*     split*       vis*
biff*       expand*   locate*    nice*          stat*        wall*
cal*        fgen*     lock*      nl*            su*          wc*
calendar*   find*     logname*   nohup*         tabs*        what*
checknr*    flock*    look*      od*            tar@         whereis*
cmp*        fmt*      m4*        paste*         tee*         which*
col*        fold*     mail*      patch*         time*        who*
colcrt*     fpr*      Mail*      pathchk*       touch*       whoami*
colrm*      from*     mailx*     pr*            tr*          whois*
column*     fsplit*   make*      printenv*      tty*         window*
comm*       ftp*      man*       printf*        ul*          write*
compress*   getconf*  menuc*     progress*      uname*       xargs*
cpio@       getopt*   mesg*      readlink*      uncompress*  xstr*
crunchgen*  groups*   mkdep*     renice*        unexpand*    yes*
crunchide*  head*     mkfifo*    rs*            unifdef*
[~/ellcc-bsd] dev% file arm-linux/bin/sh
arm-linux/bin/sh: ELF 32-bit LSB executable, ARM, version 1, statically linked, BuildID[sha1]=2a1b11acf02098cd6b77965f97d3e4031126951e, stripped
[~/ellcc-bsd] dev%

In a future post I’ll describe how to use chroot to play in the new tiny userland environment.

Cross Building tcsh with ELLCC

Now that ELLCC can build itself, it’s time to do some testing of other packages. I thought tcsh (a widely used command shell for *nix) would be a good choice, because it would exercise a bunch of system calls and standard library functions on all the targets.

I created a tcsh directory under the ELLCC test/src directory and got the latest version of tcsh sources.

I made a modified build script and Makefile to do the overall configure and build because I wanted an easy way to create executables for all the targets. The build script creates a directory for each target and runs the tcsh configure program in the directory:

# ELLCC build script.

# Get the staging directory.
prefix=`cd ../../..; pwd`

# Figure out the compilers to use.
. $prefix/build-setup $*

echo Configured to $WHY.
echo C compiler: $cc $CFLAGS
echo C++ compiler: $cxx $CXXFLAGS
echo In: build$builddir

if [ "x$arg1" != "x" ] ; then
    # Build for a single target.

# Configure for all active targets in the target list.
for t in $targets; do
  t=`basename $t -elf`
  if [ -e $prefix/libecc/mkscripts/targets/$t/ ] ; then
    echo Configuring for $t-$os
    mkdir -p build-$t-$os
    make DIR=build-$t-$os CC=$cc CXX=$cxx AR=$ar TARGET=$t OS=$os \
        target=$t haslibs=$haslibs \
        bindir=$bindir prefix=$prefix build=$build \
        configure || exit 1

    make -C build-$t-$os || exit 1

The build script uses a simple make file to get the proper build parameters for each target:

-include $(prefix)/libecc/mkscripts/targets/$(TARGET)/

ifneq ($(TARGET),$(build))

ifneq ($(CC),gcc)
  ifeq ($(haslibs),yes)

        cd $(DIR) ; \
        ../src/configure \
        CC=$(CC) CFLAGS="$(CFLAGS)" \
        --bindir=$(bindir) --prefix=$(prefix) \
        $(HOST) $(BUILD) $(TARGETS)

        rm -fr build-*

Now building is a simple as:

[~/ellcc/test/src/tcsh] dev% ./build

This will do all the configures and makes. The file command can tell what each binary is:

[~/ellcc/test/src/tcsh] dev% file */tcsh
build-armeb-linux/tcsh:      ELF 32-bit MSB executable, ARM, version 1, statically linked, BuildID[sha1]=0x078219cbae606a2a0f587d2ef1ec08cb2f74507d, not stripped
build-arm-linux/tcsh:        ELF 32-bit LSB executable, ARM, version 1, statically linked, BuildID[sha1]=0x1e9b7097aeabe9d87bab6a6c58299d5e9ad8420b, not stripped
build-i386-linux/tcsh:       ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, BuildID[sha1]=0xe22f2010af0036439a4291a813fe52281fc0a854, not stripped
build-microblaze-linux/tcsh: ELF 32-bit MSB executable, version 1 (SYSV), statically linked, not stripped
build-mipsel-linux/tcsh:     ELF 32-bit LSB executable, MIPS, MIPS-I version 1, statically linked, BuildID[sha1]=0x38cea1a3ab4fbc8499d49c9db3d81fb61856694d, not stripped
build-mips-linux/tcsh:       ELF 32-bit MSB executable, MIPS, MIPS-I version 1, statically linked, BuildID[sha1]=0xfb15ef6c9556283827968236f14b182f23ecc1e0, not stripped
build-ppc-linux/tcsh:        ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), statically linked, BuildID[sha1]=0x728ba20e60011cb9e4daff2961c285fa689e23aa, not stripped
build-x86_64-linux/tcsh:     ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=0xbd05c401b4275cb101304547a070d55dd296227c, not stripped
[~/ellcc/test/src/tcsh] dev%

The size command will give a comparison of the executable size of each target:

[~/ellcc/test/src/tcsh] dev% size */tcsh
   text    data     bss     dec     hex filename
 852268   11940   84232  948440   e78d8 build-armeb-linux/tcsh
 850044   11940   84232  946216   e7028 build-arm-linux/tcsh
 836222   11932   83960  932114   e3912 build-i386-linux/tcsh
1211936   11944   84104 1307984  13f550 build-microblaze-linux/tcsh
 963356   11984   84088 1059428  102a64 build-mipsel-linux/tcsh
 963536   11984   84088 1059608  102b18 build-mips-linux/tcsh
 892092   13364   84352  989808   f1a70 build-ppc-linux/tcsh
 878438   15800   89840  984078   f040e build-x86_64-linux/tcsh
[~/ellcc/test/src/tcsh] dev%

It is interesting that the microblaze text section is to much larger than the rest.

I’ll try to run one of the executables.

[~/ellcc/test/src/tcsh] dev% ~/ellcc/bin/qemu-ppc build-ppc-linux/tcsh
[~/ellcc/test/src/tcsh] dev% echo $version
tcsh 6.18.01 (Astron) 2012-02-14 (ppc-apple-linux) options wide,nls,dl,al,kan,rh,color,filec
[~/ellcc/test/src/tcsh] dev%

Sure enough, it is an ppc-apple-linux build (!?!). Fun.