Building clang/LLVM with ELLCC Pre-compiled Binaries

ELLCC has periodic pre-compiled binary releases that can be download and installed on various Linux hosts. A coworker wanted to take advantage of that and use ELLCC to bootstrap clang/LLVM instead of GCC. The steps described here can be used to build clang/LLVM on systems where either no GCC exists, or where the version of GCC that’s installed isn’t new enough to handle clang/LLVM C++11 code base.

An advantage of using the pre-compiled ELLCC binaries is that they are statically linked and can be run on just about any new-ish Linux box (post about 2.16).

Here’s how I build clang/LLVM using ELLCC. First I created a directory to work in:

sh-4.3$ mkdir clang
sh-4.3$ cd clang/

Then I downloaded an ELLCC binary tarball from the download page and un-tared it:

sh-4.3$ tar xvfp ~/Downloads/ellcc-x86_64-linux-eng-0.1.21.tgz

This created the ellcc directory and the subdirectories containing the ELLCC release. Then I got the latest LLVM and clang sources:

sh-4.3$ svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
sh-4.3$ cd llvm/tools/
sh-4.3$ svn co http://llvm.org/svn/llvm-project/cfe/trunk clang
sh-4.3$ cd ../..

My coworker ran into problems because currently ELLCC only supports static linking and parts of clang/LLVM are unconditionally built as shared libraries. Also, there are a few small incompatibilities resulting from the fact that ELLCC uses musl as its standard C library. A small set of patches are needed to make the stock clang/LLVM sources compile with ELLCC. Thus is the patch file: http://ellcc.org/clang-ecc.patch. You can download and apply it like this:

sh-4.3$ cd llvm
sh-4.3$ wget http://ellcc.org/clang-ecc.patch
sh-4.3$ patch -p0 < clang-ecc.patch
sh-4.3$ cd ..

Now we're ready to build:

sh-4.3$ mkdir llvm-build
sh-4.3$ cd llvm-build/
sh-4.3$ CC=/home/rich/ellcc/bin/ecc CXX=/home/rich/ellcc/bin/ecc++ CPP="/home/rich/ellcc/bin/ecc -E" ../llvm-ecc-src/configure --enable-optimized --enable-shared=no
sh-4.3$ make -j 6

Your configure line will be a little different if you're running tcsh: You'll need to set CC, etc. as environment variables with setenv. Also, I use "make -j 6" because I happen to have a 6 core system. You'll want to adjust for your system.

After the build, the bin directory looks like this:

sh-4.3$ ls Release+Asserts/bin/
arcmt-test    lli-child-target  llvm-link            llvm-stress
bugpoint      llvm-ar           llvm-lit             llvm-symbolizer
c-arcmt-test  llvm-as           llvm-lto             llvm-tblgen
c-index-test  llvm-bcanalyzer   llvm-mc              not
clang         llvm-config       llvm-mcmarkup        obj2yaml
clang++       llvm-cov          llvm-nm              opt
clang-check   llvm-c-test       llvm-objdump         sancov
clang-format  llvm-cxxdump      llvm-pdbdump         scan-build
clang-tblgen  llvm-diff         llvm-PerfectShuffle  scan-view
count         llvm-dis          llvm-profdata        verify-uselistorder
diagtool      llvm-dsymutil     llvm-ranlib          yaml2obj
FileCheck     llvm-dwarfdump    llvm-readobj         yaml-bench
fpcmp         llvm-dwp          llvm-rtdyld
llc           llvm-extract      llvm-size
lli           llvm-lib          llvm-split

The clang created has been statically linked:

sh-4.3$ file Release+Asserts/bin/clang
Release+Asserts/bin/clang: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=2465acda3f4acf5c88e0491f984f4f4048b47015, not stripped

If you use that clang to rebuild, the result will use the normal system shared libraries.

When everything is finished building you can do a "make install" to install clang and the LLVM tools in the default location which will be under /usr/local unless you supply a --prefix option to the configure command. You now have a compiled version of clang that uses your system header files and libraries. Unlike ecc, it also can create and use shared libraries and in fact uses them by default.

Cross Building Linux With ELLCC: Part 1

Update: Cross Building Linux With ELLCC: Part 2 – Raspberry Pi B+
ELLCC is a clang/LLVM based cross compilation tool chain. It is available pre-compiled for several different host systems and unlike GCC can generate programs for several targets with one set of tools. This week I thought it would be fun to try to compiler the Linux kernel using ELLCC. In this post I’ll build the patched Linux kernel from the LLVMLinux project git repository.

You need to be using ELLCC version 0.1.18 or ELLCC svn-5503 or later to build Linux. Download one of the ELLCC binary releases or follow the instructions for building ELLCC from source.

Create a directory next to the ELLCC directory. For example if the ELLCC directory is ~/ellcc:

[~] dev% mkdir ~/llvmlinux
[~] dev% cd ~/llvmlinux

We’re going to follow the instructions on the LLVMLinux site to get the pre-patched Linux kernel sources:

[~/llvmlinux] dev% git clone git://git.linuxfoundation.org/llvmlinux/kernel.git

There is a copy of the little ‘m’ makefile modified for ELLCC in the ELLCC distribution. Make a symbolic link to it:

[~/llvmlinux/kernel] dev% ln -s ../../ellcc/libecc/llvmlinux/makefile .

This little “m” makefile is set up to build Linux for the ARM. You can modify it to select another target.

Configure the kernel:

[~/llvmlinux/kernel] dev% make menuconfig

For now, under System Type, choose Dummy Virtual Machine.
Disable the following devices (LLVM bugs submitted):

  • Device Drivers->Serial ATA and Parallel ATA drivers (libata)->OPTI FireStar PATA support (Very Experimental) – LLVM bug 25330
  • Device Drivers->Network device support->Ethernet driver support->Exar Xframe 10Gb Ethernet Adapter – LLVM bug 25332
  • Device Drivers->Sound card support->Advanced Linux Sound Architecture->PCI sound devices->Aztech AZF3328 / PCI168 – LLVM bug 25330
  • Kernel hacking->Build targets in Documentation/ tree

Exit and save the configuration.

Build Linux:

[~/llvmlinux] dev% cd kernel/
[~/llvmlinux/kernel] dev% make
make[1]: Entering directory '/home/rich/llvmlinux/kernel'
  CHK     include/config/kernel.release
  CHK     include/generated/uapi/linux/version.h
  CHK     include/generated/utsrelease.h
make[2]: 'include/generated/mach-types.h' is up to date.
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     include/generated/uapi/linux/version.h
  Kernel: arch/arm/boot/Image is ready
  LD      arch/arm/boot/compressed/vmlinux
following symbols must have non local/private scope:
$d
$d
$d
arch/arm/boot/compressed/Makefile:190: recipe for target 'arch/arm/boot/compressed/vmlinux' failed
make[3]: *** [arch/arm/boot/compressed/vmlinux] Error 1
arch/arm/boot/Makefile:52: recipe for target 'arch/arm/boot/compressed/vmlinux' failed
make[2]: *** [arch/arm/boot/compressed/vmlinux] Error 2
arch/arm/Makefile:310: recipe for target 'zImage' failed
make[1]: *** [zImage] Error 2
make[1]: Leaving directory '/home/rich/llvmlinux/kernel'
makefile:71: recipe for target 'all' failed
make: *** [all] Error 2
[~/llvmlinux/kernel] dev%

So the kernel and a bunch of modules get built but subsequent processing fails. In the next part of this post I’ll try to track down what the problem is.

A Little Example of ELLCC Under Windows

As I mentioned previously, it is easy to use ELLCC to compile C and C++ programs under Windows. Now that the latest releases of ELLCC come with the MinGW-w64 header files and libraries, all you have to do is download and install a Windows binary release and you have a full C/C++ development environment including the clang/LLVM based ecc compiler, binutils, and GDB. The tool chain can target Windows as well as various Linux systems as a cross compiler. You can also run ELLCC under Linux to target Windows. In fact, the Windows release has been compiled under Linux.

Here is a simple example of compiling and debugging a simple “hello world” program using the Windows cmd shell:

                                                                            
Microsoft Windows [Version 6.3.9600]                                             
(c) 2013 Microsoft Corporation. All rights reserved.                             
                                                                                 
C:\Users\rich>cd \                                                               
                                                                                 
C:\>cd ellcc                                                                     
                                                                                 
C:\ellcc>bin/ecc examples\hello\main.c -g                                        
'bin' is not recognized as an internal or external command,                      
operable program or batch file.                                                  
                                                                                 
C:\ellcc>bin\ecc examples\hello\main.c -g                                        
                                                                                 
C:\ellcc>bin/ecc-gdb a.exe                                                       
'bin' is not recognized as an internal or external command,                      
operable program or batch file.                                                  
                                                                                 
C:\ellcc>bin\ecc-gdb a.exe                                                       
GNU gdb (GDB) 7.7                                                                
Copyright (C) 2014 Free Software Foundation, Inc.                                
License GPLv3+: GNU GPL version 3 or later     
This is free software: you are free to change and redistribute it.               
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"       
and "show warranty" for details.                                                 
This GDB was configured as "i386-mingw32".                                       
Type "show configuration" for configuration details.                             
For bug reporting instructions, please see:                                      
.                                         
Find the GDB manual and other documentation resources online at:                 
.                                
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.exe...done.
(gdb) break main
Breakpoint 1 at 0x401571: file examples\hello\main.c, line 5.
(gdb) run
Starting program: C:\ellcc\a.exe
[New Thread 2432.0xc38]
warning: Can not parse XML library list; XML support was disabled at compile tim
e

Breakpoint 1, main () at examples\hello\main.c:5
5           printf("hello world\n");
(gdb) list
1       #include <stdio.h>
2
3       int main()
4       {
5           printf("hello world\n");
6       }
(gdb) c
Continuing.
hello world
[Inferior 1 (process 2432) exited normally]
(gdb)

Since I’m a *nix guy I’m more comfortable with a shell, as you can see by my ‘/’ vs. ‘\’ fumbling above. Here is the same thing done using the MSYS command shell:


rich@VirtualWin8-64 MSYS ~
$ cd /c/ellcc

rich@VirtualWin8-64 MSYS /c/ellcc
$ bin/ecc examples/hello/main.c -g

rich@VirtualWin8-64 MSYS /c/ellcc
$ bin/ecc-gdb a.exe
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a.exe...done.
(gdb) break main
Breakpoint 1 at 0x401571: file examples/hello\main.c, line 5.
(gdb) run
Starting program: C:\ellcc\a.exe
[New Thread 1936.0x538]
warning: Can not parse XML library list; XML support was disabled at compile time

Breakpoint 1, main () at examples/hello\main.c:5
5           printf("hello world\n");
(gdb) c
Continuing.
hello world
[Inferior 1 (process 1936) exited normally]
(gdb)

Cool stuff.

ELLCC now has 64 bit ARM AArch64 Support

A new binary release of the ELLCC cross compilation tool chain is available.
ELLCC is a pre-packaged set of tools designed to support cross compilation
for a variety of target processors.

ELLCC includes:

  • ecc (a clang/LLVM based C/C++ compiler)
  • binutils
  • GDB

The ELLCC run-time libraries are supplied pre-built for all targets and
are all covered with BSD-like licenses:

  • libc++ The C++ standard library
  • musl The POSIX compliant C library
  • compiler-rt Low level compiler support
  • expat XML parsing
  • libedit Command history
  • ncurses Terminal handling
  • zlib Compression

ELLCC is entirely self hosting.

From the ChangeLog:

version 0.1.11:

* Added aarch64 support.
* Update to musl 1.1.7.
* Update to LLVM r232721.

Binary tarballs are available via FTP or HTTP from the download page.

The tool chain can build programs for ARM, Mips, PowerPC, and x86 Linux and stand-alone targets and x86 Windows systems.

Pre-built binaries are available for ARM, Mips, PowerPC, and x86 Linux host systems as well as x86 Windows and Mac OS X hosts.

All host systems can build programs for all the supported targets.

Available targets are listed here

Building mbed TLS with ELLCC

When I saw the recent announcement that PolarSSL is now owned by ARM and called mbed TLS, I thought it would be interesting to see how easily it could be built using ELLCC.

I downloaded the latest release (mbedtls-1.3.10-gpl.tgz) from the web site on to my x86_64 Linux box. A quick look at the README.txt file showed that mbed TLS can be built with make, cmake, or Microsoft Visual Studio. I opted for make.

It turns out that using ELLCC to cross build this library is very easy. Here is the command line for ARM:

[~/mbedtls] dev% tar xvfpz ~/Downloads/mbedtls-1.3.10-gpl.tgz
[~/mbedtls] dev% cd mbedtls-1.3.10/
[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target arm-linuxlld-engeabihf"

(I happen to have ELLCC installed in my home directory at ~/ellcc.) This single command built the library as well as a test suite. I have binfmt installed on my Linux box, which a cool way of telling my x86_64 Linux kernel how to run foreign executables (in this case ARM binaries). So I decided to try running the test suite.

[~/mbedtls/mbedtls-1.3.10] dev% make check
Running checks (Success if all tests PASSED)
 - test_suite_aes.ecb
   PASSED (77 / 77 tests (0 skipped))

 - test_suite_aes.cbc

[many tests deleted]

 - test_suite_version
   PASSED (5 / 5 tests (0 skipped))

[~/mbedtls/mbedtls-1.3.10] dev% 

All the tests passed. Pretty easy, huh? To prove I’ve really built for ARM, I used the file command:

[~/mbedtls/mbedtls-1.3.10] dev% file tests/test_suite_aes.cbc
tests/test_suite_aes.cbc: ELF 32-bit LSB executable, ARM, version 1 (SYSV), statically linked, not stripped
[~/mbedtls/mbedtls-1.3.10] dev%

So I thought, “This is cool, how about using ELLCC’s MinGW64 support to try a build for Windows?”. The answer was only slightly more complicated.

First I tried:

[~/mbedtls/mbedtls-1.3.10] dev% make clean
[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target x86_64-w64-mingw32"

This didn’t work out so well. I got:

...
  CC    x509write_crt.c
  CC    x509write_csr.c
  CC    xtea.c
  AR    libmbedtls.a
ar: creating libmbedtls.a
  RL    libmbedtls.a
  LN    libpolarssl.a -> libmbedtls.a
  CC    aes/aescrypt2.c
../library/libmbedtls.a: error adding symbols: Archive has no index; run ranlib to add one
ecc: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [aes/aescrypt2] Error 1
make: *** [all] Error 2
[~/mbedtls/mbedtls-1.3.10] dev%

Apparently my system ar command doesn’t handles Windows PE object files. No problem. I’ll just tell make to use ELLCC’s ecc-ar, which has been built with PE format support:

[~/mbedtls/mbedtls-1.3.10] dev% make clean
[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target x86_64-w64-mingw32" AR=~/ellcc/bin/ecc-ar

This time, building the test suite failed with errors like:

...
../library/libmbedtls.a(net.o):(.text+0x4ce): undefined reference to `__imp_WSAGetLastError'
../library/libmbedtls.a(net.o):(.text+0x503): undefined reference to `__imp_shutdown'
../library/libmbedtls.a(net.o):(.text+0x513): undefined reference to `__imp_closesocket'
ecc: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [pkey/dh_client] Error 1
make: *** [all] Error 2
[~/mbedtls/mbedtls-1.3.10] dev

I seem to be missing a library needed for linking. I needed to find which library was missing, so I used the trusty nm command to find out:

[~/ellcc/libecc/mingw/x86_64-w64-mingw32/sys-root/mingw/lib] dev% ~/ellcc/bin/ecc-nm -A *.a | grep __imp_closesocket 
/home/rich/ellcc/bin/ecc-nm: libruntimeobject.a: File format not recognized
libws2_32.a:dscdbs00149.o:0000000000000000 I __imp_closesocket
libwsock32.a:divxs00037.o:0000000000000000 I __imp_closesocket
[~/ellcc/libecc/mingw/x86_64-w64-mingw32/sys-root/mingw/lib] dev%

Looks like I have two choices. libws2_32.a sounds newer (grin), so I’ll try that. The tests/Makefile has the line:

LDFLAGS += -L../library -lmbedtls $(SYS_LDFLAGS)

All I have to do is add a define for SYS_LDFLAGS:

[~/mbedtls/mbedtls-1.3.10] dev% make CC="~/ellcc/bin/ecc -target x86_64-w64-mingw32" AR=~/ellcc/bin/ecc-ar SYS_LDFLAGS=-lws2_32
...
  Generate      test_suite_rsa.c
  CC            test_suite_rsa.c
  Generate      test_suite_shax.c
  CC            test_suite_shax.c
  Generate      test_suite_x509parse.c
  CC            test_suite_x509parse.c
  Generate      test_suite_x509write.c
  CC            test_suite_x509write.c
  Generate      test_suite_xtea.c
  CC            test_suite_xtea.c
  Generate      test_suite_version.c
  CC            test_suite_version.c
[~/mbedtls/mbedtls-1.3.10] dev% 

Let’s try the file command again.

file tests/test_suite_aes.cbc
tests/test_suite_aes.cbc: PE32+ executable (console) x86-64, for MS Windows
[~/mbedtls/mbedtls-1.3.10] dev%

Perfect. Now the test suite. Since binfmt can handle Windows executables and run them under Wine, things should just work.

[~/mbedtls/mbedtls-1.3.10] dev% make check
...
XTEA Encrypt CBC #4 ............................................... PASS
XTEA Encrypt CBC #5 ............................................... PASS
XTEA Encrypt CBC #6 ............................................... PASS
XTEA Decrypt CBC #1 ............................................... PASS
XTEA Decrypt CBC #2 ............................................... PASS
XTEA Decrypt CBC #3 ............................................... PASS
XTEA Decrypt CBC #4 ............................................... PASS
XTEA Decrypt CBC #5 ............................................... PASS
XTEA Decrypt CBC #6 ............................................... PASS
XTEA Selftest ..................................................... PASS


PASSED (25 / 25 tests (0 skipped))

 - test_suite_version
   Check compiletime library version ................................. PASS
Check runtime library version ..................................... PASS
Check for POLARSSL_VERSION_C ...................................... PASS
Check for POLARSSL_AES_C when already present ..................... PASS
Check for unknown define .......................................... PASS


PASSED (5 / 5 tests (0 skipped))

[~/mbedtls/mbedtls-1.3.10] dev% 

Voila!

Using ELLCC to Build for the Cortex-M3

This weekend I decided to work on getting ELLCC to support the ARM Cortex-M processors. The Cortex-M processors use the Thumb 16 bit instruction set, so adding M support involves adding a configuration file for a member of the Cortex-M family and then building the run-time libraries specifically for the target processor. To start I chose the Cortex-M3. Its configuration file currently looks like this:

[~/ellcc/libecc/config] dev% cat thumb-linux-engeabi
based_on: arm-ellcc-linux
compiler:
options:
- -target arm-ellcc-linux
- -mthumb
- -mcpu=cortex-m3
- -mfpu=none
- -mfloat-abi=softfp
linker:
static_crt1: $R/lib/thumb-linux-engeabi/crt1.o
dynamic_crt1: $R/lib/thumb-linux-engeabi/Scrt1.o
crtbegin: $R/lib/thumb-linux-engeabi/crtbegin.o
crtend: $R/lib/thumb-linux-engeabi/crtend.o
library_paths:
- -L$R/lib/thumb-linux-engeabi

With the config file in place, I attempted to build ELLCC’s standard C library musl for the M. Building a library for a new configuration is pretty easy:

[~/ellcc/libecc] dev% make thumb-linux-engeabi
[~/ellcc/libecc] dev% cd
[~] dev% cd ~/ellcc/libecc
[~/ellcc/libecc] dev% make thumb-linux-engeabi

Well, it isn’t quite that simple. The musl library has several assembly files that are written for the specific target processor. If the assembly language of the desired processor doesn’t match the sources, the sources have to be modified to support the new target. Luckily Thumb instructions are mostly a subset of 32 bit ARM instructions so the changes needed to build musl for the M were fairly minimal.

The next step to support a given processor is to compiler the compiler-rt library. This library contains support functions that handle operations that the processor doesn’t support directly and compiler generated built in functions. In the case of the M processors, this includes things like floating point functions since the M3 doesn’t have a floating point unit. Building compiler-rt was probably the most challenging part of this little project. I had never fully ported the compiler-rt build rules to use the new config files properly so rather than cobbling together something that could handle building for the M, I rewrote the build rules to properly support config files.

The last step is to build the C++ standard libraries, assuming you want C++ support. I ran into one little glitch here. The M doesn’t support 64 bit atomic operations so I had to modify those operations for the M.

Now that the libraries are built, a little experimentation is in order. I decided to try out a simple “hello world”. Here’s the source:

[~/ellcc/examples/hello] dev% cat main.c
#include <stdio.h>

int main()
{
printf("hello world\n");
}

The make files in the example directories can build for multiple targets as long as config files have been set up, so I built for the M:

[~/ellcc/examples/hello] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% ~/ellcc/bin/qemu-arm hello
hello world
[~/ellcc/examples/hello] dev%

Nice! I was curious how the Thumb instruction set affected code size so I did a side by side comparison:

[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
19540 156 1412 21108 5274 hello
[~/ellcc/examples/hello] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hello'
rm -f *.o hello hello.bin hello.log elkconfig.ld
Compiling main.c
Linking hello
make[1]: Leaving directory `/home/rich/ellcc/examples/hello'
[~/ellcc/examples/hello] dev% size hello
text data bss dec hex filename
27432 156 1412 29000 7148 hello

ELLCC creates static binaries by default, so those sizes include all the standard library bits that the program needs, i.e. printf() and software floating point support functions. It is kind of cool that the M is about 15% smaller than the equivalent ARM code for this example.

I decided to try C++. Here’s the code:

[~/ellcc/examples/hellocpp] dev% cat main.cpp
#include <iostream>

int main()
{
std::cout << "hello world" << std::endl; }

Here are the results:

[~/ellcc/examples/hellocpp] dev% make thumb-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% ~/ellcc/bin/qemu-arm hellocpp
hello world

And a size comparison:

[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
867540 500 10824 878864 d6910 hellocpp
[~/ellcc/examples/hellocpp] dev% make arm-linux-engeabi
make[1]: Entering directory `/home/rich/ellcc/examples/hellocpp'
rm -f *.o hellocpp hellocpp.bin hellocpp.log elkconfig.ld
Compiling main.cpp
Linking hellocpp
make[1]: Leaving directory `/home/rich/ellcc/examples/hellocpp'
[~/ellcc/examples/hellocpp] dev% size hellocpp
text data bss dec hex filename
1412708 500 10824 1424032 15baa0 hellocpp

Cool stuff.

Finally a size comparison of ELLCC's ecc clang/LLVM based compiler compiled for Thumb and ARM:

[~/ellcc] dev% size bin-thumb-linux-engeabi/ecc
text data bss dec hex filename
37889520 33760 63000 37986280 2439fe8 bin-thumb-linux-engeabi/ecc
[~/ellcc] dev% size bin-arm-linux-engeabi/ecc
text data bss dec hex filename
49504456 33760 63000 49601216 2f4dac0 bin-arm-linux-engeabi/ecc

Looks like the Thumb code is about 25% smaller.

Precompiled binaries supporting the Thumb and all the other ELLCC targets are available in the 0.1.10 release available on the Download page.

Using Config Files to use ELLCC+MinGW to Cross Compile for Windows

ELLCC is based on clang/LLVM of course. Previously I mentioned that I was experimenting with YAML based config files for the compiler driver. When I made that post and subsequently, several people had objections to the approach because it potentially required reading a file each time the compiler is involked and that it would make the already complicated driver code even more complicated. It came up again today in the LLVM mailing list and it got me to thinking that maybe another real world example might help.

I used MinGW to cross compile ELLCC for Windows hosts. I started to wonder what an ELLCC config file might look like for using clang to cross compile for Windows instead. I came up with this as a first cut example:

# based_on `i386-ellcc-linux'
# based_on `ellcc-linux'
---
based_on:        ''
global:          
  static_default:  true
compiler:        
  options:         
    - -target i386-ellcc-win32
    # - -no-integrated-as
  c_include_dirs:  
    - /usr/lib64/gcc/i686-w64-mingw32/4.8.4/include
    - /usr/lib64/gcc/i686-w64-mingw32/4.8.4/include-fixed
    - /usr/i686-w64-mingw32/sys-root/mingw/include
  cxx_include_dirs: 
    - '$R/include/c++'
assembler:       
  exe: '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/bin/as'
  output:          
    - '-o $O'
linker:          
  exe: '/usr/libexec/gcc/i686-w64-mingw32/4.8.4/collect2'
  output:          
    - '-o $O'
  start:           
    #- -e _start
  opt_static:      
    - -Bstatic
  opt_rdynamic:    
    - -export-dynamic
  opt_dynamic:     
    - -Bdynamic
  opt_shared:      
    - -shared
  opt_notshared:   
    - -dynamic-linker /usr/libexec/ld.so
  opt_pthread:     
    - -pthread
  options:         
    - -m i386pe
  static_crt1: '/usr/i686-w64-mingw32/sys-root/mingw/lib/../lib/crt2.o'
  dynamic_crt1: '$R/lib/i386-linux-eng/Scrt1.o'
  crtbegin:    '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/crtbegin.o'
  crtend:      '/usr/lib64/gcc/i686-w64-mingw32/4.8.4/crtend.o'
  library_paths:   
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4'
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/lib/../lib'
    - '-L/usr/i686-w64-mingw32/sys-root/mingw/lib/../lib'
    - '-L/usr/lib64/gcc/i686-w64-mingw32/4.8.4/../../../../i686-w64-mingw32/lib'
    - '-L/usr/i686-w64-mingw32/sys-root/mingw/lib'
  cxx_libraries:   
    - '-lc++'
    - -lm
  profile_libraries: 
    - -lprofile_rt
  c_libraries:     
    - '-lmingw32'
    - '-lgcc'
    - '-lgcc_eh'
    - '-lmoldname'
    - '-lmingwex'
    - '-lmsvcrt'
    - '-ladvapi32'
    - '-lshell32'
    - '-luser32'
    - '-lkernel32'
    - '-lmingw32'
    - '-lgcc'
    - '-lgcc_eh'
    - '-lmoldname'
    - '-lmingwex'
    - '-lmsvcrt'
...

I called it i386-w64-mingw32, and, Hey Presto! a Windows cross compiler on my Linux box.

[~] dev% cat main.c
#include <stdio.h>

int main()
{
    printf("hello world\n");
}
[~] dev% ~/ellcc/bin/ecc -target i386-w64-mingw32 main.c
[~] dev% ./a.exe 
fixme:winediag:start_process Wine-Compholio is a Wine testing version containing experimental patches.
fixme:winediag:start_process Please don't report bugs at winehq.org and use our issue tracker instead:
fixme:winediag:start_process https://github.com/compholio/wine-compholio/issues
hello world
[~] dev% file a.exe
a.exe: PE32 executable (console) Intel 80386, for MS Windows
[~] dev%

Adding Networking Support to ELK: Part 3

Again I’m posting this as a post-in-progress. I was hoping to finish this before real life intrudes tomorrow. Things are going well but tomorrow looks like a real stretch goal.

In Part 1 of this post I wrote about some design goals for ELK networking. In Part 2, I described some of the implementation details that evolved as I integrated the LwIP TCP/IP stack into ELK. In this post, I’ll describe adding an Ethernet driver to ELK’s network implementation.

As in Part 2, I’m writing this as development is progressing, so be please patient if I tend to go off on tangents.

LwIP comes with a skeleton example Ethernet driver in src/netif/ethernetif.c. My ARM development environment is currently QEMU emulating the Versatile Express A9 board, which emulates the LAN 91C111 Ethernet controller. Eventually, I’d like to get ELK networking running on the Raspberry Pi, but since it uses a USB based Ethernet controller (the smsc9512), that will require a bit more work.

OK, enough with Google. Time to get to work. I’ll start or by making a copy of ethernetif.c which I have called lan91c111.c. Now I’ll start filling in the details.

I’ve made some basic changes to the driver and now I’ll add it to the list of ELK source files in sources.mk. I know the first build is going to spit out a bunch of errors, and sure enough it does. I’ll add a few “#if RICH” lines to mark places I have to add functionality. I got through the “#if RICH” part when I realized that I’m missing the whole infrastructure for socket ioctl() calls. I realized that when I started about how to initialize Ethernet devices. So I’m side tracking to add ioctl() support to the network interfaces.

As an aside, it turns out I had to make another change to the LwIP sources. One of the ioctl() calls supported on sockets is named IP_HDRINCL. It turns out that LwIP uses that name at the lower levels to represent a packet that already contains an address. I changed all instances of IP_HDRINCL to IP_HDRINCLUDED in LwIP to get around the naming conflict.

There are two interfaces that LwIP supports to configure Ethernet devices, basically because LwIP has two modes of operation as I mentioned in Part 1. The core of LwIP is meant to be used by a single (or the only) thread. That is the first mode, and is usually reserved for smaller systems where all network activity is done by a single thread. The second mode has a network thread which has a message based interface to all the other threads in the system. ELK uses the second mode, so we have to use the netifapi functions, defined in the lwip/netifapi.h file. I had to to enable this API in the ELK specific lwipopts.h file.

The ioctl() calls for sockets are documented in the netdevice(7) man page. We’ll see how well the LwIP API maps to Linux’s idea of network configuration.

There are a set of ioctl() calls defined that are used to configure network devices as described in netdevice(7). I’ve implemented a fairly large subset of socket ioctl() calls to prepare for actually adding the Ethernet driver. Here is an example of some of them:

/* ELK running as a VM enabled OS.
 */
#define _GNU_SOURCE
#include <sys/ioctl.h>
#include <net/if.h>
#include <net/if_arp.h>
#include <arpa/inet.h>
#include <errno.h>
#include <string.h>

#include <sys/cdefs.h>
#include <stdio.h>

#include "command.h"

int main(int argc, char **argv)
{
  setprogname("elk");
  printf("%s started. Type \"help\" for a list of commands.\n", getprogname());

  int sfd = socket(AF_INET, SOCK_STREAM /*|SOCK_NONBLOCK */, 0);
  if (sfd < 0) {
    printf("socket(AF_INET) failed: %s\n", strerror(errno));
  }

  int s;
  struct ifreq ifreq;
  struct in_addr in_addr;

  // Set the device name for subsequent calls.
  strcpy(ifreq.ifr_name, "ln01");

  // Set the interface IP{ address.
  inet_aton("192.124.43.4", &in_addr);
  ifreq.ifr_addr.sa_family = AF_INET;
  memcpy(ifreq.ifr_addr.sa_data, &in_addr, sizeof(in_addr));
  s = ioctl(sfd, SIOCSIFADDR, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFADDR) failed: %s\n", strerror(errno));
  }

  // Set ine interface netmask.
  inet_aton("255.255.255.0", &in_addr);
  ifreq.ifr_netmask.sa_family = AF_INET;
  memcpy(ifreq.ifr_netmask.sa_data, &in_addr, sizeof(in_addr));
  s = ioctl(sfd, SIOCSIFNETMASK, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFNETMASK) failed: %s\n", strerror(errno));
  }

  // Set the interface MAC address.
  ifreq.ifr_hwaddr.sa_family = ARPHRD_ETHER;
  ifreq.ifr_hwaddr.sa_data[0] = 0x01;
  ifreq.ifr_hwaddr.sa_data[1] = 0x02;
  ifreq.ifr_hwaddr.sa_data[2] = 0x03;
  ifreq.ifr_hwaddr.sa_data[3] = 0x04;
  ifreq.ifr_hwaddr.sa_data[4] = 0x05;
  ifreq.ifr_hwaddr.sa_data[5] = 0x06;
  s = ioctl(sfd, SIOCSIFHWADDR, &ifreq);
  if (s < 0) {
    printf("ioctl(SIOCSIFHWADDR) failed: %s\n", strerror(errno));
  }
  printf("Try the command 'inetif'\n");

  // Enter the kernel command processor.
  do_commands(getprogname());
}

Where is what the compiled example produces:

[~/ellcc/examples/elk] dev% make run
Compiling main.c
Linking elk
Running elk
enter 'control-A x' to exit QEMU
audio: Could not init `oss' audio driver
elk started. Type "help" for a list of commands.
Try the command 'inetif'
elk % inetif
0 lo: active flags=137 (0x01)  mtu 0
        inet 127.0.0.1  netmask 255.0.0.0  broadcast 127.255.255.255
1 ln01: active flags=66 (0x32)  mtu 1500
        inet 192.124.43.4  netmask 255.255.255.0  broadcast 192.124.43.255
        ether 01:02:03:04:05:06
elk % 

ln01 is my stubbed driver. It looks like the ioctl() calls have worked. lwip_network.c is the source file implementing ELK's LwIP socket interface. Now it's time to get the driver to do something.

As I started to flesh out the driver I realized that the vexpress-a9 board that QEMU is emulating uses the LAN9118 Ethernet controller, not the SMC91C111 that I thought. That got me thinking. I really don't want to reinvent the wheel each time for all the different Ethernet controllers out there and it is always better to find something that exists and adapt it rather than to write it from scratch. With that in mind, I decided what it would take to use Ethernet drivers borrowed from NetBSD for ELK. As I see it, the first step is to get a NetBSD driver to compile in the ELK environment with as few source changes as possible. Step two is to make some glue code to make the NetBSD driver fit into the LwIP environment.

Yikes! I just spent a harrowing half a day trying to isolate the NetBSD driver for the LAN9118 so that it could "drop in" as a LwIP driver. I went down the path of stubbing out all the include files that were directly and indirectly included and compiling the source file, over and over, adding definitions to the include files as needed. It turns out that the driver has lots of dependencies on the NetBSD kernel, which is not surprising. I've decided to try a different route, but I did stumble on this interesting post about rump kernels, which are basically doing what ELK is doing, but with NetBSD as the basis.

I like the direction this new approach is taking. I'm using an idea from the LwIP wiki to create a driver framework. The driver skeleton has been fleshed out in ethernetif_driver.c and low level support for the LAN9118 is taking shape in lan9118.c. I hope I find some time over the next few days to fill in the reast of the details.

Adding Networking Support to ELK: Part 2

In a previous post I described how I started to add LwIP networking support to ELK, along with some of the design decisions that guide ELK development. In this post, I’ll describe how socket related system calls are added to ELK and how they interface with the LwIP network stack and the rest of the ELK modules.

A minimal source file that allows LwIP to be brought in to ELK at link time looks like this:

#include "config.h"
#include "kernel.h"
#include "lwip/tcpip.h"

// Make LwIP networking a select-able feature.
FEATURE_CLASS(lwip_network, network)

// Define socket related system calls.
ELK_CONSTRUCTOR()
{
}

// Start up LwIP.
C_CONSTRUCTOR()
{
  tcpip_init(0, 0);
}

Almost all symbols in an ELK select-able module are static. The only external linkage symbols in this module are defined by the FEATURE_CLASS() macro. In this case, the symbols __elk_lwip_network and __elk_feature_network are defined. The first symbol is used to pull this module in at link time. The second symbol is used to cause an error if another (currently non-existent) network module is linked in at the same time.

ELK used two phases of constructor functions to preform system initialization. ELK_CONSTRUCTOR() functions are called at system start up before the C library is initialized. These functions are typically used to initialize system call definitions. C_CONSTRUCTOR() functions are normal constructors called after the C library has been initialized but before main() is called. No system calls have been defined in this example, but the LwIP initialization function is called in the C_CONSTRUCTOR() phase.

There are several system calls that are specific to sockets and networking. I’ll create stub functions for all of them now. Even though I’m concentrating on getting ELK running on an ARM target right now, I always build ELK for all targets. My first attempt to create a stub handler for accept4() failed to compile for the i386:

#include <sys/socket.h>

#include "config.h"
#include "kernel.h"
#include "syscalls.h"
#include "lwip/tcpip.h"
#include "crt1.h"

// Make LwIP networking a select-able feature.
FEATURE_CLASS(lwip_network, network)

static int sys_accept4(int sockfd, struct sockaddr *addr, socklen_t *addrlen)
{
  return -ENOSYS;
}

// Define socket related system calls.
ELK_CONSTRUCTOR()
{
  SYSCALL(accept4);
}

// Start up LwIP.
C_CONSTRUCTOR()
{
  tcpip_init(0, 0);
}

It turns out that the i386 socket calls (and perhaps other targets) all go through one system call called SYS_socketcall. When I modified my source file like this:

#include <sys/socket.h>

#include "config.h"
#include "kernel.h"
#include "syscalls.h"
#include "lwip/tcpip.h"
#include "crt1.h"

// Make LwIP networking a select-able feature.
FEATURE_CLASS(lwip_network, network)

static int sys_accept4(int sockfd, struct sockaddr *addr, socklen_t *addrlen)
{
  return -ENOSYS;
}

#ifdef SYS_socketcall
static int sys_socketcall(int call, unsigned long *args)
{
  long arg[6];

  // Get the call arguments.
  copyin(arg, args, sizeof(arg));

  switch (call) {
  case __SC_accept4:
    return sys_accept4(args[0], (struct sockaddr *)arg[1], (socklen_t *)arg[2]);

  default:
    return -ENOSYS;
  }
}
#endif

// Define socket related system calls.
ELK_CONSTRUCTOR()
{
#ifndef SYS_socketcall
  SYSCALL(accept4);
#else
  SYSCALL(socketcall);
#endif
}

// Start up LwIP.
C_CONSTRUCTOR()
{
  tcpip_init(0, 0);
}

I’ll follow a similar pattern for the rest of the system calls. The result, with all the socket functions stubbed in, is here. Note that another little wrinkle is that at least some of the Linux ports, in this case the x86_64 port, don’t have the recv() and send() system calls. They use recvfrom() and sendto() instead.

Now I’ll start adding meat to the empty system call framework. I’ll start with the socket() system call, since it is the only way to get a socket and several design decisions will need to be made about how to interface with the rest of ELK. The normal LwIP socket descriptors index into a static array of structures containing the state of any open sockets. This means that all threads that use sockets share a socket namespace and that that socket namespace is different from the normal ELK file descriptor namespace. That is the first thing that has to change. To do that, I have to integrate LwIP sockets into the ELK VFS (Virtual File System) module. That is, a LwIP socket, when created, should result in the creation of a socket vnode. Subsequently, all operations on the socket should go through the vnode. The beauty of this approach is that socket descriptors and file descriptors will share the same namespace and that other operations that should be legal on sockets, like read() and write() (and select() when it gets implemented), will just work on all types of file descriptors as long as the low level support exists in the vnode implementation.

As I was delving into the implementation of the socket system call handling code, I realized that LwIP only implements the AF_INET and AF_INET6 domains. Since I’d like to support other domains, especially AF_UNIX, I decided to split the LwIP interface code out of the socket system call handling code. If I can set up the interfaces correctly, this should also allow other networking stacks to be dropped in in place of LwIP. So now I have network.c with the generic system call handling code and lwip_network.c with the LwIP interface glue.

Now’s the time to add error handling code to the system call handlers. This is really a stalling tactic while I’m thinking about how to integrate sockets into the existing virtual file system framework, but I suspect I’ll get some insights as I flesh out the generic code. After some thought and feverish coding, I came up with something that feels reasonable. I implemented the getsockopt() and setsockopt() system calls, which started giving me a better idea of what I need for socket integration. I’ve started to implement unix_network.c, which will implement the AF_UNIX (AF_LOCAL) domain. It looks like most of the code for the socket interface will be in network.c, with callbacks to the individual domain handlers where the semantics differ between domains. One of the interesting parts of the evolving design is that support for the various domain handlers can be specified at link time and eventually will be available as loadable modules.

Now I have much of the socket infrastructure done. I can create a socket file and open it, and use the socket() and socketpair() system calls. setsocketopt() and getsocketopt() are implemented and can be used to manipulate information in a generic socket structure. Although socket files exist in the normal virtual file system namespace, I had to add support for vnodes that don’t live in the normal space for non-file sockets. All socket operations go through a vnode however and socket file descriptors and regular file descriptors are indistinguishable. My first little test program looks like this:

[~/ellcc/examples/socket] dev% cat main.c
/* Simple socket tests.                                                                             
 */                                                                                                 
#include <sys/socket.h>                                                                             
#include <sys/stat.h>                                                                               
#include <fcntl.h>                                                                                  
#include <unistd.h>                                                                                 
#include <stdio.h>                                                                                  
#include <stdlib.h>                                                                                 
#include <errno.h>                                                                                  
#include <string.h>                                                                                 
                                                                                                    
int main(int argc, char **argv)                                                                     
{                                                                                                   
  int sv[2];                                                                                        
  int s = socketpair(AF_UNIX, SOCK_STREAM, 0, sv);                                                  
  if (s < 0) {                                                                                      
    printf("socketpair() failed: %s\n", strerror(errno));                                           
    exit(1);                                                                                        
  }

  s = write(sv[0], "hello world\n", sizeof( "hello world\n"));
  if (s < 0) {
    printf("write() failed: %s\n", strerror(errno));
  }
  char buffer[100];
  s = read(sv[1], buffer, 1);
  if (s < 0) {
    printf("read() failed: %s\n", strerror(errno));
  }

  s = mknod("/socket", S_IFSOCK|S_IRWXU, 0);
  if (s < 0) {
    printf("mknod() failed: %s\n", strerror(errno));
  }

  int fd = open("/socket", O_RDWR);
  if (fd < 0) {
    printf("open() failed: %s\n", strerror(errno));
  }
  s = read(fd, buffer, 1);
  if (s < 0) {
    printf("read() failed: %s\n", strerror(errno));
  }
}

Here is the result of running the program:

[~/ellcc/examples/socket] dev% make run
Preprocessing elkconfig.cfg
Compiling main.c
Linking socket
Running socket
enter 'control-A x' to exit QEMU
audio: Could not init `oss' audio driver
write() failed: Protocol not supported
read() failed: Protocol not supported
read() failed: Protocol not supported

Not bad for a day's work. The errors on the read() and write() calls are expected because I haven't yet implement the read and write buffers for AF_UNIX sockets, but the fact that the error returned is EPROTONOSUPPORT shows that the socket infrastructure is working as it's supposed to.

I've taken a little break to think about how I'd like to implement socket buffers. I think I'd like a design that

  • Allocates buffers a page (usually 4K) at a time.
  • Is coded to be shared between the different socket domains.
  • Expand and contract as needed.
  • Is as simple as possible.

I'm thinking that one approach would be to use two arrays of page pointers, each empty initially. The empty arrays would reside in the socket structure and be limited in size by a kernel compile time constant, maybe 64 entries each. This would allow a maximum 262,144 bytes for each of the buffers given 4K pages. The size would be controlled by the send and receive buffer size socket options, so sockets could be configured to use less memory in a memory constrained system.

It's been a busy day. I've implemented the socketpair() and bind() for AF_UNIX sockets. The buffering scheme seems to be working well, but I have to thing a bit more about how and when the buffer size can be reduced. I'm currently implementing listen() so I can move on to connect() and accept(). The nice thing about concentrating on AF_UNIX sockets first is that it is giving me a good feel for what I'll need to implement AF_INET using the LwIP stack.

It turns out that listen() in interesting. It is supposed to set up a backlog queue of pending connections, but what should that look like? I can see what it might look like for AF_UNIX sockets, but it is unclear what it should look like for remote connections. I guess my plan of implementing AF_UNIX first needs a slight diversion: I'm going to switch back to LwIP and see what a connection queue looks like there to try to come up with a common solution.

It turns out that interfacing LwIP with the current interface is pretty easy. I've had to make one change to the LwIP sources so far. I had to make the type of the socket member of the netconn structure definable at compile time. I needed this because sockets need to be represented by their socket structure pointer, not by a simple integer socket descriptor since socket file descriptors share the same descriptor namespace in different processes. I changed the declaration of the socket member in lwip/api.h to

#if LWIP_SOCKET
  LWIP_SOCKET_TYPE(socket);
#endif /* LWIP_SOCKET */

and added this definition in lwip/opt.h:

#ifndef LWIP_SOCKET_TYPE
#define LWIP_SOCKET_TYPE(name)          int name
#endif

In my lwipopts.h file I overrode the definition with

#define LWIP_SOCKET_TYPE(name) struct { int name; void *priv; }

This anonymous structure replaces the previous definition of "int socket;" with both an int and a pointer. I use the pointer to keep the higher level socket pointer for lwip_interface.c.

I've finished the first phase of LwIP integration. A few simple tests work, like in examples/socket/main.c. Next comes more extensive testing. I'm pretty happy with the way both AF_UNIX and AF_INET handling is implemented. It looks like it will be easy to drop in different stacks for these protocols or for other protocols as necessary.

Part 3 of this little saga will be about adding an Ethernet driver, so I can talk to something beyond 127.0.0.1.

Adding Networking Support to ELK: Part 1

The holidays are a great time. A little time away from my day job and I can concentrate a little on bigger ELLCC sub-projects. This holiday, I decided to concentrate on adding networking to ELK, ELLCC’s bare metal run time environment, previously mentioned here and here.

ELLCC is, for the most part, made up of other open source projects. ELLCC (the cross compilation tool chain) leverages the clang/LLVM compiler and for cross compiling C and C++ source code. I decided early on that the ELLCC run time support libraries would all have permissive BSD or BSD-like licensing so I use libc++ and libc++ABI for C++ library support, musl for standard C library support, and compiler-rt for low level processor support.

For those of you unfamiliar with ELK (which is probably all of you), I’ll give a brief synopsis of ELK’s design. The major design goals of ELK are

  • Use the standard run time libraries, compiled for Linux, unchanged in a bare metal environment.
  • Allow fine grained configuration of ELK at link time to support target environments with widely different memory and processor resources.
  • Have BSD or BSD-like licensing so that it can be used with no encumbrances for commercial and non-commercial projects.

The implications of the first goal are interesting. I wanted a fully configured ELK environment to support the POSIX environment that user space programs enjoy in kernel space. In addition, all interactions between the application and the bare metal environment would be through system calls, whether or not an MMU is present or used on the system. I can feel embedded programmers shuddering at the last statement: “What!?! A system call just to write a string to a serial port? What a waste!”. I completely understand, being an old embedded guy myself. But it turns out that there are a couple of good reasons to use the system call approach. The first is that system calls are the place where a context switch will often occur in a multi-threaded environment. Other than in the most bare metal of environments, ELK supports multi-threading and can take advantage of the context saved at the time of the system call to help implement threading. The second reason for using system calls is that modern Linux user space programs try to do system calls as infrequently as possible. For example, POSIX mutexes and semaphores are built on Linux futexes. A futex is a wonderful synchronization mechanism. The only time a system call is needed when taking a mutex, for example, is when the mutex is contested. Finally, using system calls allows ELK to be implemented by implemented system call functionality and you only need to include the system calls that your program needs. I gave a simple example of a system call definition in this post.

At the very lowest level, ELK consists of start-up code that initializes the processor and provides hooks for context switching and system call handling. Here is an example of the ARM start-up code. Above that, ELK consists of several modules, each of which provide system calls related to their functionality. The system call status page gives a snapshot of the system calls currently implemented by ELK, along with the name of the module that the system call has (or will be) implemented in. When I started working on adding networking to ELK, the set of modules supported were

  • thread – thread related system calls and data structures.
  • memman – memory management: brk(), mmap(), etc.
  • time – time related calls: clock_gettime(), nanosleep(), etc.
  • vfs – Virtual File System: File systems, device nodes, etc.
  • vm – Virtual Memory.

Some of the modules have multiple variations that can be selected at link time. For example, memman has a very simple for (supporting malloc() only), and a full blown mmap() supporting version, while a version of vm exists for both MMU and non-MMU systems. Much of the functionality of these modules were derived from the cool but seemingly abandoned Prex project.

Looking back at what I’ve written here so far, I’m guessing more that one of any potential readers are thinking “I thought this post was about adding networking support”. Well, I guess it is about time to get to the point.

I had several options for ELK networking. I could write a network stack myself and spent years debugging it or, like may other components of ELLCC, I could look around for a suitably licensed existing open source alternative. Unlike Prex, I wanted the networking code to show signs of being actively maintained and I wanted to be able to import it and updated with as little change to the source code as possible. I didn’t want to get into the business of doing ongoing maintenance of a one of a kind network stack. I finally settled on LwIP, which I had heard of over the years, but never actually used. LwIP has the right kind of license, and even though the last release was in 2012, it is being actively maintained as evidenced by this recent CERT advisory. In addition LwIP was originally designed for small, resource limited systems and is highly configurable.

LwIP consists of the core functionality, which is a single threaded network stack designed to provide a low level API providing callback functions for network events. In addition, LwIP provides two higher level APIs. The netconn API provides a multi-threaded interface by making the core functionality a thread and communicating with it via messages. Above that, LwIP also provides a Berkeley socket interface API. For ELK, I decided to use the core and netconn functionality and provide my own socket interface API that integrate fully into the existing ELK thread and vfs modules so that file descriptors and vnode interfaces would be consistent.

The first step was to get LwIP to compile within the ELK build framework. That was easy: I got the latest GIT clone and imported it into the ELK source tree, I added the core and netconn sources to ELK’s build rules and provided a couple of configuration headers and glue source files to tie it all together. Fortunately, LwIP has been ported to Linux, and ELK provides a Linux like environment, so even the glue files already existed.

I was very curious how much adding networking would add to the size of an ELK program, so I built an ELK example (http://ellcc.org/viewvc/svn/ellcc/trunk/examples/elk/) both with and without networking linked in. A full blown configuration, without networking, and with this main.c:

/* ELK running as a VM enabled OS.
 */
#include 
#include 

#include "command.h"


int main(int argc, char **argv)
{
#if 0
  // LwIP testing.
  void tcpip_init(void *, void *);
  tcpip_init(0, 0);
#endif
  setprogname("elk");
  printf("%s started. Type \"help\" for a list of commands.\n", getprogname());
  // Enter the kernel command processor.
  do_commands(getprogname());
}

Had a size like this:

[~/ellcc/examples/elk] dev% size elk
   text    data     bss     dec     hex filename
 162696    3364   64240  230300   3839c elk
[~/ellcc/examples/elk] dev%

When I enabled the LwIP initialization, I got

[~/ellcc/examples/elk] dev% size elk
   text    data     bss     dec     hex filename
 367390    4024   68428  439842   6b622 elk
[~/ellcc/examples/elk] dev%

Not bad, considering two things. First, I have just about all the LwIP bells and whistles turned on, and secondly I am compiling with no optimization. Total program size is about 100K smaller with -O3.

The other cool things is that at least at this stage, LwIP is starting with no complaint. I can run the example and see that the networking thread has been started:

[~/ellcc/examples/elk] dev% make run
Running elk
enter 'control-A x' to exit QEMU
audio: Could not init `oss' audio driver
elk started. Type "help" for a list of commands.
elk % ps
Total pages: 8192 (33554432 bytes), Free pages: 8131 (33304576 bytes)
   PID    TID       TADR STATE        PRI NAME       
     0      0 0x800683a0 RUNNING        1 kernel
     0      2 0x8006d150 IDLE           3 [idle0]
     0      3 0x80099000 SLEEPING       1 [kernel]
elk % 

That third thread (TID 3) is the network thread in all its glory. Now to make it do something.

In the next installment of this blog, I’ll describe how the ELK network module handles socket related system calls and interacts with the other ELK modules and the LwIP netconn API.