Author Archives: rich

More Bare Metal ARM With Linux C Libraries

As a follow up to my post An ARM Bare Metal Hello World Using Musl I thought I’d give an update. A quick background. I’m trying to build a bare metal (no OS) framework that can use a stock Linux standard C library. To do this I’m building a small assembly infrastructure that traps the Linux system calls used by the library and deals with them. In my previous post, I got “hello world”. In this one, I’ve come a bit farther. I can read stdin, malloc() memory and do simple context switching. You can see the code I’m working on here. In this example, printf(), fgets() and malloc() is usable enough to be used for a simple context switching example.

Most of the example main program (THREAD is not defined yet: don’t get too excited):

long __syscall_ret(unsigned long r);
long __syscall(long, ...);

#define CONTEXT
#if defined(CONTEXT)
static void *main_sa;
static void *context1_sa;
static void *context2_sa;

static int context(intptr_t arg1, intptr_t arg2)
{
    void **context_sa = (void **)arg2;
    for ( ;; ) {
      printf("hello from context %" PRIdPTR "\n", arg1);
      __switch(context_sa, main_sa);
    }
    return 0;
}
#endif

int main(int argc, char **argv)
{
    printf("%s: hello world\n", argv[0]);

    int i = __syscall_ret(__syscall(0, 1, 2, 3, 4, 5, 6));
    printf("__syscall(0) = %d, %s\n", i, strerror(errno));

#if defined(THREAD)
    int s;
    pthread_attr_t attr;
    s = pthread_attr_init(&attr);
    pthread_t id;
    if (s != 0)
        printf("pthread_attr_init: %s\n", strerror(errno));
    s = pthread_create(&id, &attr, &thread, NULL);
    if (s != 0)
        printf("pthread_create: %s\n", strerror(errno));
#endif
#if defined(CONTEXT)
    char *p = malloc(4096);
    context1_sa = p + 4096;
    __new_context(&context1_sa, context, Mode_SYS, NULL, 42, (intptr_t)&context1_sa);
    p = malloc(4096);
    context2_sa = p + 4096;
    __new_context(&context2_sa, context, Mode_SYS, NULL, 6809, (intptr_t)&context2_sa);
    // Let's do some context switching.
    __switch(&main_sa, context1_sa);
    __switch(&main_sa, context2_sa);
    __switch(&main_sa, context1_sa);
    __switch(&main_sa, context2_sa);
    __switch(&main_sa, context2_sa);

#endif
    for ( ;; ) {
        char buffer[100];
        fputs("prompt: ", stdout);
        fflush(stdout);
        fgets(buffer, sizeof(buffer), stdin);
        printf("got: %s", buffer);
    }
}

The output:

~/ellcc/baremetal/arm] dev% qemu-system-arm -cpu any -M versatilepb -m 128M -nographic -kernel kernel.bin
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
kernel: hello world
unhandled system call (0) args: 1, 2, 3, 4, 5, 6
__syscall(0) = -1, Function not implemented
hello from context 42
hello from context 6809
hello from context 42
hello from context 6809
hello from context 6809
prompt: hello world
got: hello world

It’s almost starting to look like Posix at the bare metal level. Fun stuff.

An ARM Bare Metal Hello World Using Musl

I’ve been trying to figure out how to use the excellent musl C standard library in a bare metal environment for quite a while. I didn’t want to damage the musl source code too much but couldn’t figure out an easy way to get at least a subset of musl’s functionality available in a bare metal environment.

Over the weekend I can up with a cute little solution that may be viable for future development. It’s based on a simple concept: Don’t change musl at all, but implement system call handling at the machine level to handle the system calls that musl needs.

I added a baremetal directory to the top-level ELLCC directory as a proof of concept. The arm sub-directory is where the initial work was done. There are just a few files in the directory:

  • Makefile – To build the example
  • init.S – To handle system initialization and exception handling
  • main.c – A simple C source file showing the handling of two system calls
  • kernel.ld – A simple linker command file to put it in the right place in the ARM’s memory.

It turns out that musl’s printf does two system calls to print “hello world\n”. The first is an ioctl() to determine how to buffer. I ignored that call since all the unimplemented system calls return error.
The interesting call that musl did make was to writev(), for which I made a simple replacement in startup.c.

After typing “make”, here is the result of running it:

[~/ellcc/baremetal/arm] dev% qemu-system-arm -cpu any -M versatilepb -m 128M -nographic -kernel kernel.bin
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
unhandled system call 54
hello world
QEMU: Terminated
[~/ellcc/baremetal/arm] dev%

The 54 is the unhandled ioctl(). I’m sure that output will be very handy as I implement more of musl’s functionality for my bare metal environment.

Using C++11 Lambdas with ELLCC

A recent post on C++11 lambdas and a forum question led me to this topic. Lambdas are a new C++ feature, described very well in the first link. The forum post asked how to build a simple test program using lambdas with ellcc:

#include <functional>
#include <iostream>
#include <string>
int main(int argc, char **argv){
    std::string test;
    test += "Hello World!";
    std::cout << test << std::endl;
    auto lambda = [&](const std::string &in) -> void {
        std::cout << "Text was: " << in << std::endl;
        return;
    };
    lambda("extra text.");
    return 0;
}

The simple solution was:

[~] dev% ellcc/bin/ecc++ -target x86_64-ellcc-linux --std=c++11 test.cpp
[~] dev% ./a.out
Hello World!
Text was: extra text.
[~] dev%

The important thing here is that you need the -target option to set the appropriate target environment and you need to specify --std=c++11 to enable C++11 language features.

Interesting Linux Kernel Bug Uncovered by musl

ELLCC uses the very cool musl standard C library as a replacement for the normal Linux standard library. In the latest version of ELLCC, ELLCC was not able to compile itself on an x86_64 Fedora 20 Linux system. I was stumped for a while trying to track down the problem. It was weird: self hosting worked on a 32 bit Linux system (Fedora 19), but failed on a 64 bit system? Furthermore, self hosting only failed with ELLCC compiled with itself and linked with musl, but not with ELLCC compiled with itself and linked with glibc.

Fortunately, Rich Felker of musl fame shed some light on the issue. Here’s an edited IRC log:

04:34:00 PM - rdp: OK. malloc fails on my x86_64 linux after about 65527 4K allocations with musl malloc(). glibc malloc() doesn't, probably because it reverts to mmap() if brk fails. Yet I don't see any resource limits set. The gloibc brk() also failes after about 64K allocations.
04:37:52 PM - dalias: rdp, oh, we've seen this before
04:37:57 PM - dalias: it's a kernel bug with some optional kernel feature
04:38:21 PM - dalias: it keeps the kernel from merging adjacent vma's, so you end up with 64k pages each as their own tiny vma
04:38:36 PM - rdp: Excellent.
04:38:38 PM - dalias: it would happen if we used mmap too
04:38:50 PM - dalias: the reason it doesn't affect glibc is that they allocate huge amounts at a time
04:38:58 PM - rdp: Ah.
04:38:59 PM - dalias: and thereby waste memory if the program doesn't actually need much
04:39:17 PM - dalias: i'll try to find the option
04:39:21 PM - rdp: Any work around?
04:39:25 PM - rdp: OK. Thanks.
04:40:20 PM - dalias: CONFIG_MEM_SOFT_DIRTY
04:40:23 PM - dalias: turn it off
04:40:28 PM - dalias: there might be a way to do it at runtime
04:40:42 PM - dalias: or you could increase the limit on # of vma's
04:40:50 PM - dalias: but basically this option wastes MASSIVE amounts of ram
04:40:57 PM - dalias: by refusing to merge vma's
04:41:47 PM - dalias: it's a hack to make process checkpointing (save and restore running processes) more efficient
04:42:01 PM - dalias: by better tracking what has changed
04:42:50 PM - dalias: i don't see a way to turn it off
04:42:55 PM - dalias: check /proc/$pid/maps tho
04:43:08 PM - dalias: you should see a separate line for each page (i.e. 64k lines)
04:43:16 PM - dalias: if this is the issue that's affecting you
04:43:39 PM - rdp: I do.
04:43:51 PM - dalias: ok then this is the issue
04:43:57 PM - dalias: you can just up the limit if you want
04:44:04 PM - dalias: /proc/sys/vm/max_map_count
04:44:09 PM - dalias: but again this is expensive
04:44:16 PM - dalias: you want to disable CONFIG_MEM_SOFT_DIRTY
04:44:21 PM - dalias: and we really need to report this bug to the kernel folks                                                                             
04:44:25 PM - dalias: i don't think they're aware of it                       
...                                                                           
04:45:11 PM - rdp: dalias: Thanks.                                            
...                                                                           
04:46:47 PM - rdp: dalias: is it x86_64 specific? Not on i386?                
...                                                                           
04:49:13 PM - dalias: rdp, i think it may be                                  
04:50:26 PM - dalias: http://stackoverflow.com/questions/20997809/analyzing-cause-of-performance-regression-with-different-kernel-version
04:50:28 PM - feepbot: Analyzing cause of performance regression with different kernel version - Stack Overflow
04:51:38 PM - dalias: the accepted answer tracked down the cause of the soft_dirty bug and seems to cover how to fix it
...
04:53:02 PM - rdp: gotta love stackoverflow
...
05:47:19 PM - dalias: rdp, haha with regard to that SO answer:
05:47:26 PM - dalias: Finally fixed in Linux 3.13.3 and Linux 3.12.11, released 2014-02-13. – osgx 21 hours ago
05:57:32 PM - rdp: dalias: :-)
...
07:00:00 PM - dalias: rdp, i think it would be worth adding the issue you had to the faq on the wiki
07:00:51 PM - dalias: with a link to the stack overflow question/answer and information that it's fixed in 3.13.3, and that you can work around it by turning off CONFIG_MEM_SOFT_DIRTY (good fix) or increasing max_map_count (expensive fix)

For now, I got around the problem by using Rich’s expensive fix option (as superuser):

echo 128000 > /proc/sys/vm/max_map_count

Why didn’t ELLCC linked with glibc fail? Somebody considered it a bug at one point, but the glibc maintainers disagreed, I guess.
[Update]
Rich pointed out that my guess about why the glibc malloc() doesn’t fail is probably wrong. But it is still a kernel bug nevertheless.

Building NetBSD Userland with ELLCC.

I’ve started a new project to build the core NetBSD userland programs for Linux using ELLCC. I have two motivations for doing this:

  • To have a large set of programs to compile for testing ELLCC. I’ve been looking for something for a while, and my stackoverflow question didn’t come up with any responses.
  • I’d like to eventually have a full set of userland commands for Linux that are not GPL licensed.

So far, the results have been promising. I have a nice subset of the NetBSD userland compiled so far:

ls x86_64-linux/bin x86_64-linux/usr/bin
x86_64-linux/bin:
[*      cp*    domainname*  expr*      ksh*  mkdir*  pwd*    sh*     tar*
cat*    cpio*  echo*        hostname*  ln*   mv*     rm*     sleep*  test*
chmod*  date*  ed*          kill*      ls*   pax*    rmdir*  sync*

x86_64-linux/usr/bin:
apply*     calendar*  crunchgen*  expand*  getconf*  lastcomm*  make*
asa*       checknr*   crunchide*  fgen*    getopt*   leave*     od*
at*        cmp*       csplit*     find*    groups*   locate*    tar@
atq*       col*       ctags*      flock*   head*     lock*      uncompress*
atrm*      colcrt*    cut*        fmt*     hexdump*  logname*   whoami*
banner*    colrm*     deroff*     fold*    id*       look*
basename*  column*    dirname*    fpr*     indent*   m4*
batch*     comm*      du*         from*    join*     mail*
biff*      compress*  env*        fsplit*  jot*      Mail*
cal*       cpio@      error*      ftp*     lam*      mailx*
[~/ellcc-bsd] dev%

The files in that listing represent the low hanging fruit: They are the files that required little or no editing to compile with ELLCC.

If you want to try to build the NetBSD userland using ELLCC, first install ELLCC itself. Then you can grab the current ellcc-bsd project files:

[~] dev% svn co http://ellcc.org/svn/ellcc-bsd/trunk ellcc-bsd

This will create the ellcc-bsd directory right next to the already existing ellcc directory.

The next step is to build the bmake program (the Berkeley make program, used by NetBSD):

[~] dev% cd ellcc-bsd/
[~/ellcc-bsd] dev% ./build

This will put the host bmake executable in the ~/ellcc/bin directory.

Now all you have to do is

[~/ellcc-bsd] dev% ~/ellcc/bin/bmake
...
[~/ellcc-bsd] dev% ~/ellcc/bin/bmake install

This will create the populated directories list above.

For a little more fun try building ARM executables:

[~/ellcc-bsd] dev% ~/ellcc/bin/bmake TARGET=arm
...
[~/ellcc-bsd] dev% ~/ellcc/bin/bmake TARGET=arm install
...
[~/ellcc-bsd] dev% ls arm-linux/bin/ arm-linux/usr/bin/
arm-linux/bin/:
[*      cp*    domainname*  expr*      ksh*  mkdir*  pwd*    sh*     tar*
cat*    cpio*  echo*        hostname*  ln*   mv*     rm*     sleep*  test*
chmod*  date*  ed*          kill*      ls*   pax*    rmdir*  sync*

arm-linux/usr/bin/:
apply*      csplit*   hexdump*   mkstr*         sdiff*       uniq*
asa*        ctags*    id*        mktemp*        sed*         units*
at*         cut*      indent*    mkubootimage*  seq*         unvis*
atq*        deroff*   join*      msgc*          shlock*      users*
atrm*       dirname*  jot*       msgs*          shuffle*     uudecode*
banner*     du*       lam*       nbperf*        soelim*      uuencode*
basename*   env*      lastcomm*  newgrp*        sort*        uuidgen*
batch*      error*    leave*     newsyslog*     split*       vis*
biff*       expand*   locate*    nice*          stat*        wall*
cal*        fgen*     lock*      nl*            su*          wc*
calendar*   find*     logname*   nohup*         tabs*        what*
checknr*    flock*    look*      od*            tar@         whereis*
cmp*        fmt*      m4*        paste*         tee*         which*
col*        fold*     mail*      patch*         time*        who*
colcrt*     fpr*      Mail*      pathchk*       touch*       whoami*
colrm*      from*     mailx*     pr*            tr*          whois*
column*     fsplit*   make*      printenv*      tty*         window*
comm*       ftp*      man*       printf*        ul*          write*
compress*   getconf*  menuc*     progress*      uname*       xargs*
cpio@       getopt*   mesg*      readlink*      uncompress*  xstr*
crunchgen*  groups*   mkdep*     renice*        unexpand*    yes*
crunchide*  head*     mkfifo*    rs*            unifdef*
[~/ellcc-bsd] dev% file arm-linux/bin/sh
arm-linux/bin/sh: ELF 32-bit LSB executable, ARM, version 1, statically linked, BuildID[sha1]=2a1b11acf02098cd6b77965f97d3e4031126951e, stripped
[~/ellcc-bsd] dev%

In a future post I’ll describe how to use chroot to play in the new tiny userland environment.

Cross Building tcsh with ELLCC

Now that ELLCC can build itself, it’s time to do some testing of other packages. I thought tcsh (a widely used command shell for *nix) would be a good choice, because it would exercise a bunch of system calls and standard library functions on all the targets.

I created a tcsh directory under the ELLCC test/src directory and got the latest version of tcsh sources.

I made a modified build script and Makefile to do the overall configure and build because I wanted an easy way to create executables for all the targets. The build script creates a directory for each target and runs the tcsh configure program in the directory:

#!/bin/sh
# ELLCC build script.

# Get the staging directory.
prefix=`cd ../../..; pwd`

# Figure out the compilers to use.
. $prefix/build-setup $*

echo Configured to $WHY.
echo C compiler: $cc $CFLAGS
echo C++ compiler: $cxx $CXXFLAGS
echo In: build$builddir

if [ "x$arg1" != "x" ] ; then
    # Build for a single target.
    targets=$arg1
fi

# Configure for all active targets in the target list.
for t in $targets; do
  t=`basename $t -elf`
  if [ -e $prefix/libecc/mkscripts/targets/$t/setup.mk ] ; then
    echo Configuring for $t-$os
    mkdir -p build-$t-$os
    make DIR=build-$t-$os CC=$cc CXX=$cxx AR=$ar TARGET=$t OS=$os \
        target=$t haslibs=$haslibs \
        bindir=$bindir prefix=$prefix build=$build \
        configure || exit 1

    make -C build-$t-$os || exit 1
  fi
done

The build script uses a simple make file to get the proper build parameters for each target:

-include $(prefix)/libecc/mkscripts/targets/$(TARGET)/setup.mk

ifneq ($(TARGET),$(build))
  HOST=--host=$(TARGET)-$(OS)
  BUILD=--build=$(build)-$(OS)
else
  HOST=
  BUILD=
endif

ifneq ($(CC),gcc)
  ifeq ($(haslibs),yes)
    CFLAGS=$(CFLAGS.$(TARGET))
    CXXFLAGS=$(CXXFLAGS.$(TARGET))
  endif
endif

configure:
        cd $(DIR) ; \
        ../src/configure \
        CC=$(CC) CFLAGS="$(CFLAGS)" \
        CXX=$(CXX) CXXFLAGS="$(CXXFLAGS)" \
        --bindir=$(bindir) --prefix=$(prefix) \
        $(HOST) $(BUILD) $(TARGETS)

clean:
        rm -fr build-*

Now building is a simple as:

[~/ellcc/test/src/tcsh] dev% ./build

This will do all the configures and makes. The file command can tell what each binary is:

[~/ellcc/test/src/tcsh] dev% file */tcsh
build-armeb-linux/tcsh:      ELF 32-bit MSB executable, ARM, version 1, statically linked, BuildID[sha1]=0x078219cbae606a2a0f587d2ef1ec08cb2f74507d, not stripped
build-arm-linux/tcsh:        ELF 32-bit LSB executable, ARM, version 1, statically linked, BuildID[sha1]=0x1e9b7097aeabe9d87bab6a6c58299d5e9ad8420b, not stripped
build-i386-linux/tcsh:       ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, BuildID[sha1]=0xe22f2010af0036439a4291a813fe52281fc0a854, not stripped
build-microblaze-linux/tcsh: ELF 32-bit MSB executable, version 1 (SYSV), statically linked, not stripped
build-mipsel-linux/tcsh:     ELF 32-bit LSB executable, MIPS, MIPS-I version 1, statically linked, BuildID[sha1]=0x38cea1a3ab4fbc8499d49c9db3d81fb61856694d, not stripped
build-mips-linux/tcsh:       ELF 32-bit MSB executable, MIPS, MIPS-I version 1, statically linked, BuildID[sha1]=0xfb15ef6c9556283827968236f14b182f23ecc1e0, not stripped
build-ppc-linux/tcsh:        ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), statically linked, BuildID[sha1]=0x728ba20e60011cb9e4daff2961c285fa689e23aa, not stripped
build-x86_64-linux/tcsh:     ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=0xbd05c401b4275cb101304547a070d55dd296227c, not stripped
[~/ellcc/test/src/tcsh] dev%

The size command will give a comparison of the executable size of each target:

[~/ellcc/test/src/tcsh] dev% size */tcsh
   text    data     bss     dec     hex filename
 852268   11940   84232  948440   e78d8 build-armeb-linux/tcsh
 850044   11940   84232  946216   e7028 build-arm-linux/tcsh
 836222   11932   83960  932114   e3912 build-i386-linux/tcsh
1211936   11944   84104 1307984  13f550 build-microblaze-linux/tcsh
 963356   11984   84088 1059428  102a64 build-mipsel-linux/tcsh
 963536   11984   84088 1059608  102b18 build-mips-linux/tcsh
 892092   13364   84352  989808   f1a70 build-ppc-linux/tcsh
 878438   15800   89840  984078   f040e build-x86_64-linux/tcsh
[~/ellcc/test/src/tcsh] dev%

It is interesting that the microblaze text section is to much larger than the rest.

I’ll try to run one of the executables.

[~/ellcc/test/src/tcsh] dev% ~/ellcc/bin/qemu-ppc build-ppc-linux/tcsh
[~/ellcc/test/src/tcsh] dev% echo $version
tcsh 6.18.01 (Astron) 2012-02-14 (ppc-apple-linux) options wide,nls,dl,al,kan,rh,color,filec
[~/ellcc/test/src/tcsh] dev%

Sure enough, it is an ppc-apple-linux build (!?!). Fun.

Building Lua for ARM with ELLCC

Today someone mentioned Lua the scripting language and how it was a nice embed-able language well suited to small embedded systems. I decided to see how well ELLCC could do building Lua for an ARM Linux target. You can build the ELLCC compiler package by following the instructions here.

I downloaded the latest Lua tarball from their download page and was off to the races. I extracted Lua into my ~/ellcc directory:

[~] dev% cd ~/ellcc
[~/ellcc] dev% tar xvfpz Downloads/lua-5.2.2.tar.gz

I made a few small changes to their configuration to use ecc for the ARM:

[~] dev% diff -r -c lua-5.2.2 ellcc/lua-5.2.2/
diff -r -c lua-5.2.2/src/luaconf.h ellcc/lua-5.2.2/src/luaconf.h
*** lua-5.2.2/src/luaconf.h     2013-03-16 16:10:18.000000000 -0500
--- ellcc/lua-5.2.2/src/luaconf.h       2013-11-08 09:59:56.000000000 -0600
***************
*** 43,49 ****
  #if defined(LUA_USE_LINUX)
  #define LUA_USE_POSIX
  #define LUA_USE_DLOPEN                /* needs an extra library: -ldl */
! #define LUA_USE_READLINE      /* needs some extra libraries */
  #define LUA_USE_STRTODHEX     /* assume 'strtod' handles hex formats */
  #define LUA_USE_AFORMAT               /* assume 'printf' handles 'aA' specifiers */
  #define LUA_USE_LONGLONG      /* assume support for long long */
--- 43,49 ----
  #if defined(LUA_USE_LINUX)
  #define LUA_USE_POSIX
  #define LUA_USE_DLOPEN                /* needs an extra library: -ldl */
! // #define LUA_USE_READLINE   /* needs some extra libraries */
  #define LUA_USE_STRTODHEX     /* assume 'strtod' handles hex formats */
  #define LUA_USE_AFORMAT               /* assume 'printf' handles 'aA' specifiers */
  #define LUA_USE_LONGLONG      /* assume support for long long */
diff -r -c lua-5.2.2/src/Makefile ellcc/lua-5.2.2/src/Makefile
*** lua-5.2.2/src/Makefile      2012-12-27 04:51:43.000000000 -0600
--- ellcc/lua-5.2.2/src/Makefile        2013-11-08 21:34:36.494043682 -0600
***************
*** 6,14 ****
  # Your platform. See PLATS for possible values.
  PLAT= none
  
! CC= gcc
  CFLAGS= -O2 -Wall -DLUA_COMPAT_ALL $(SYSCFLAGS) $(MYCFLAGS)
  LDFLAGS= $(SYSLDFLAGS) $(MYLDFLAGS)
  LIBS= -lm $(SYSLIBS) $(MYLIBS)
  
  AR= ar rcu
--- 6,16 ----
  # Your platform. See PLATS for possible values.
  PLAT= none
  
! CC= /home/rich/ellcc/bin/ecc
  CFLAGS= -O2 -Wall -DLUA_COMPAT_ALL $(SYSCFLAGS) $(MYCFLAGS)
+ CFLAGS+= -target arm-ellcc-linux-eabi -mcpu=armv6z -mfpu=vfp -mfloat-abi=softfp
  LDFLAGS= $(SYSLDFLAGS) $(MYLDFLAGS)
+ LDFLAGS+= -target arm-ellcc-linux-eabi
  LIBS= -lm $(SYSLIBS) $(MYLIBS)
  
  AR= ar rcu
***************
*** 103,109 ****
  generic: $(ALL)
  
  linux:
!       $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E -ldl -lreadline"
  
  macosx:
        $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline"
--- 105,111 ----
  generic: $(ALL)
  
  linux:
!       $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E -ldl"
  
  macosx:
        $(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_MACOSX" SYSLIBS="-lreadline"
[~] dev% 

Then I just typed:

[~/ellcc] dev% cd lua-5.2.2/
[~/ellcc/lua-5.2.2] dev% make linux

I now have my ARM lua exectiable:

[~/ellcc/lua-5.2.2] dev% file src/lua
src/lua: ELF 32-bit LSB executable, ARM, version 1, statically linked, BuildID[sha1]=0x635737dc31e9d493f06b8f9fa5d2e7e3c1fe93ee, not stripped
[~/ellcc/lua-5.2.2] dev% 

Which I can run with QEMU (I use an x86_64 Linux box and don’t have ARM hardware handy):

[~/ellcc/lua-5.2.2] dev% ~/ellcc/bin/qemu-arm src/lua
Lua 5.2.2  Copyright (C) 1994-2013 Lua.org, PUC-Rio
> 

I then went to Lua’s live demo page and cut and pasted an example program:

> -- hello.lua
-- the first program in every language

io.write("Hello world, from ",_VERSION,"!\n")> > > 
Hello world, from Lua 5.2!
> 

Nice!

Using ELLCC to Build Itself, Including Canadian Cross Builds

The ELLCC tool chain can now completely build itself. Currently the build has been tested only on Linux systems. The steps I’ll use here are:

  1. Build ELLCC using the system compiler (bootstrap build)
  2. Build it again with itself (sanity build)
  3. Cross build it for another target machine (show off build)

ELLCC supports several target processors:

  • arm – little endian ARM
  • armeb – big endian ARM
  • i366 – x86 in 32 bit mode
  • microblaze – the Xilinx soft-core processor for FPGAs
  • mips – big endian Mips
  • mipsel – little endian Mips
  • ppc – the Power PC in 32 bit mode
  • x86_64 – x86 in 64 bit mode

The compiler we’ll be building can target all of them.

Go into a directory and get the latest ELLCC top of tree (TOT):

[~] dev% svn checkout http://ellcc.org/svn/ellcc/trunk ellcc

I checkout into a directory called ellcc, but the name isn’t important.
In this example. I checked out the files in my home directory. Now we can do the initial bootstrap build:

[~] dev% cd ellcc
[~/ellcc] dev% ./build

The build takes quite a while because it is building ecc, the clang/LLVM based C/C++ compiler, GNU binutils for the assemblers, linker, and utilities, GDB for debugging, and QEMU for all the target processors for testing purposes. It also builds a compete set of run-time libraries:

  • libc++ for the C++ standard library
  • libc++ABI for C++ run-time suport
  • libunwind for C++ exception handling
  • musl for the C standard library
  • compiler-rt for low-level support routines
  • ncurses for terminal support
  • zlib for compression/decompression

All of these libraries are built for all the supported target processors.

When the bootstrap build is finished, you’ll get a message like this:

Please run the build script again to bootstrap ecc.
This may be done a few times:
1. ecc is built with itself (compiled with gcc) and libecc.
2. ecc is built with itself (compiled with itself) and libecc.

Run build again. This will build the ELLCC tools with themselves to complete the bootstrap.

Just run the same build command again:

[~/ellcc] dev% ./build

This time you won’t get the message above, since ELLCC has been completely bootstrapped.

Now for the fun part, a Canadian cross build.
In this step, we’ll use our newly built compiler to build a compiler that will run on a different target system. In this case, the ELLCC build rules don’t bother with compiling QEMU, since it is only used for ELLCC development. In addition, we can skip building the libraries, since they have already been built. I’ll do a build for an ARM target:

[~/ellcc] dev%de>[~/ellcc] dev% ./build arm

When this build completes, we’ll have a new directory populated with the ARM executables, bin-arm-linux:

[~/ellcc] dev% ls bin-arm-linux/
arm-elf-as*     ecc-gdb*      i386-elf-as*       llvm-extract*     macho-dump*
bugpoint*       ecc-gprof*    llc*               llvm-link*        microblaze-elf-as*
clang-check*    ecc-ld*       lli*               llvm-mc*          mips-elf-as*
clang-format*   ecc-ld.bfd*   lli-child-target*  llvm-mcmarkup*    not*
clang-tblgen*   ecc-nm*       llvm-ar*           llvm-nm*          opt*
ecc*            ecc-objcopy*  llvm-as*           llvm-objdump*     ppc64-elf-as*
ecc++@          ecc-objdump*  llvm-bcanalyzer*   llvm-ranlib@      ppc-elf-as*
ecc-addr2line*  ecc-ranlib*   llvm-config*       llvm-readobj*     sparc-elf-as*
ecc-ar*         ecc-readelf*  llvm-config-host*  llvm-rtdyld*      x86_64-elf-as*
ecc-as*         ecc-size*     llvm-cov*          llvm-size*
ecc-c++filt*    ecc-strings*  llvm-diff*         llvm-stress*
ecc-elfedit*    ecc-strip*    llvm-dis*          llvm-symbolizer*
ecc-embedspu*   FileCheck*    llvm-dwarfdump*    llvm-tblgen*

[~/ellcc] dev% ~/ellcc[~/ellcc] dev% file bin-arm-linux/ecc
bin-arm-linux/ecc: ELF 32-bit LSB executable, ARM, version 1, statically linked, BuildID[sha1]=0x9ff616a316ab010b46062f7fc1dff554ee7a6db8, not stripped
/bin/qemu-arm bin-arm-linux/ecc -v
clang version 3.4 (trunk)
Target: arm-unknown-linux-gnu
Thread model: posix
Selected GCC installation: 
[~/ellcc] dev

Very cool!