Tag Archives: LWIP

Adding Networking Support to ELK: Part 1

The holidays are a great time. A little time away from my day job and I can concentrate a little on bigger ELLCC sub-projects. This holiday, I decided to concentrate on adding networking to ELK, ELLCC’s bare metal run time environment, previously mentioned here and here.

ELLCC is, for the most part, made up of other open source projects. ELLCC (the cross compilation tool chain) leverages the clang/LLVM compiler and for cross compiling C and C++ source code. I decided early on that the ELLCC run time support libraries would all have permissive BSD or BSD-like licensing so I use libc++ and libc++ABI for C++ library support, musl for standard C library support, and compiler-rt for low level processor support.

For those of you unfamiliar with ELK (which is probably all of you), I’ll give a brief synopsis of ELK’s design. The major design goals of ELK are

  • Use the standard run time libraries, compiled for Linux, unchanged in a bare metal environment.
  • Allow fine grained configuration of ELK at link time to support target environments with widely different memory and processor resources.
  • Have BSD or BSD-like licensing so that it can be used with no encumbrances for commercial and non-commercial projects.

The implications of the first goal are interesting. I wanted a fully configured ELK environment to support the POSIX environment that user space programs enjoy in kernel space. In addition, all interactions between the application and the bare metal environment would be through system calls, whether or not an MMU is present or used on the system. I can feel embedded programmers shuddering at the last statement: “What!?! A system call just to write a string to a serial port? What a waste!”. I completely understand, being an old embedded guy myself. But it turns out that there are a couple of good reasons to use the system call approach. The first is that system calls are the place where a context switch will often occur in a multi-threaded environment. Other than in the most bare metal of environments, ELK supports multi-threading and can take advantage of the context saved at the time of the system call to help implement threading. The second reason for using system calls is that modern Linux user space programs try to do system calls as infrequently as possible. For example, POSIX mutexes and semaphores are built on Linux futexes. A futex is a wonderful synchronization mechanism. The only time a system call is needed when taking a mutex, for example, is when the mutex is contested. Finally, using system calls allows ELK to be implemented by implemented system call functionality and you only need to include the system calls that your program needs. I gave a simple example of a system call definition in this post.

At the very lowest level, ELK consists of start-up code that initializes the processor and provides hooks for context switching and system call handling. Here is an example of the ARM start-up code. Above that, ELK consists of several modules, each of which provide system calls related to their functionality. The system call status page gives a snapshot of the system calls currently implemented by ELK, along with the name of the module that the system call has (or will be) implemented in. When I started working on adding networking to ELK, the set of modules supported were

  • thread – thread related system calls and data structures.
  • memman – memory management: brk(), mmap(), etc.
  • time – time related calls: clock_gettime(), nanosleep(), etc.
  • vfs – Virtual File System: File systems, device nodes, etc.
  • vm – Virtual Memory.

Some of the modules have multiple variations that can be selected at link time. For example, memman has a very simple for (supporting malloc() only), and a full blown mmap() supporting version, while a version of vm exists for both MMU and non-MMU systems. Much of the functionality of these modules were derived from the cool but seemingly abandoned Prex project.

Looking back at what I’ve written here so far, I’m guessing more that one of any potential readers are thinking “I thought this post was about adding networking support”. Well, I guess it is about time to get to the point.

I had several options for ELK networking. I could write a network stack myself and spent years debugging it or, like may other components of ELLCC, I could look around for a suitably licensed existing open source alternative. Unlike Prex, I wanted the networking code to show signs of being actively maintained and I wanted to be able to import it and updated with as little change to the source code as possible. I didn’t want to get into the business of doing ongoing maintenance of a one of a kind network stack. I finally settled on LwIP, which I had heard of over the years, but never actually used. LwIP has the right kind of license, and even though the last release was in 2012, it is being actively maintained as evidenced by this recent CERT advisory. In addition LwIP was originally designed for small, resource limited systems and is highly configurable.

LwIP consists of the core functionality, which is a single threaded network stack designed to provide a low level API providing callback functions for network events. In addition, LwIP provides two higher level APIs. The netconn API provides a multi-threaded interface by making the core functionality a thread and communicating with it via messages. Above that, LwIP also provides a Berkeley socket interface API. For ELK, I decided to use the core and netconn functionality and provide my own socket interface API that integrate fully into the existing ELK thread and vfs modules so that file descriptors and vnode interfaces would be consistent.

The first step was to get LwIP to compile within the ELK build framework. That was easy: I got the latest GIT clone and imported it into the ELK source tree, I added the core and netconn sources to ELK’s build rules and provided a couple of configuration headers and glue source files to tie it all together. Fortunately, LwIP has been ported to Linux, and ELK provides a Linux like environment, so even the glue files already existed.

I was very curious how much adding networking would add to the size of an ELK program, so I built an ELK example (http://ellcc.org/viewvc/svn/ellcc/trunk/examples/elk/) both with and without networking linked in. A full blown configuration, without networking, and with this main.c:

/* ELK running as a VM enabled OS.

#include "command.h"

int main(int argc, char **argv)
#if 0
  // LwIP testing.
  void tcpip_init(void *, void *);
  tcpip_init(0, 0);
  printf("%s started. Type \"help\" for a list of commands.\n", getprogname());
  // Enter the kernel command processor.

Had a size like this:

[~/ellcc/examples/elk] dev% size elk
   text    data     bss     dec     hex filename
 162696    3364   64240  230300   3839c elk
[~/ellcc/examples/elk] dev%

When I enabled the LwIP initialization, I got

[~/ellcc/examples/elk] dev% size elk
   text    data     bss     dec     hex filename
 367390    4024   68428  439842   6b622 elk
[~/ellcc/examples/elk] dev%

Not bad, considering two things. First, I have just about all the LwIP bells and whistles turned on, and secondly I am compiling with no optimization. Total program size is about 100K smaller with -O3.

The other cool things is that at least at this stage, LwIP is starting with no complaint. I can run the example and see that the networking thread has been started:

[~/ellcc/examples/elk] dev% make run
Running elk
enter 'control-A x' to exit QEMU
audio: Could not init `oss' audio driver
elk started. Type "help" for a list of commands.
elk % ps
Total pages: 8192 (33554432 bytes), Free pages: 8131 (33304576 bytes)
   PID    TID       TADR STATE        PRI NAME       
     0      0 0x800683a0 RUNNING        1 kernel
     0      2 0x8006d150 IDLE           3 [idle0]
     0      3 0x80099000 SLEEPING       1 [kernel]
elk % 

That third thread (TID 3) is the network thread in all its glory. Now to make it do something.

In the next installment of this blog, I’ll describe how the ELK network module handles socket related system calls and interacts with the other ELK modules and the LwIP netconn API.