Monthly Archives: April 2014

Bare Metal ARM with musl Gets a Little Friendlier

UPDATE: More info on this little project is here.

So, this thing is getting a little out of hand. “This thing”, in case you haven’t been here before, is my experiment with using a standard Linux C library on a bare metal ARM board. The way that I’m making it work is by writing simple Linux system call handlers to do what the normal Linux system call would do but in a much simplified way. In my last post I described a little about how my test kernel is built and run and how to connect to it with GDB for debugging.

This time I have added some interrupt handlers and implemented a few more system calls. My first attempt at interrupt handling is for the SP804 dual timer to handle the POSIX monotonic and realtime system clocks. After one of my last posts, someone asked what ARM I was targeting. It turned out to be a great question for which I had no clue how to answer. There are so many ways to say ARM: arm7, armv7, cortex, … Diving in to this project at first I had no idea about the nuances. It turns out that that what I’m trying to target at first is the Cortex-a9, specifically as emulated by QEMU on the vexpress-a9 board.

Ah! you say. Then why the heck are you using the SP804 timer for that? You should be using the 64 bit timers in the private memory region! Indeed I should, and that will be an exercise for another day. The a15 is even cooler, with its virtual timer, but I digress.

In this update, I’ve also added some simple command line processing:

[~/ellcc/baremetal/arm] dev% make run
../../bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel kernel.bin
audio: Could not init `oss' audio driver
kernel started. Type "help" for a list of commands.
kernel % help
                date: show/set the realtime timer.
                time: time the specified command with arguments.
               sleep: sleep for a time period.
Test Commands:
             syscall: test the syscall interface with an unhandled system call.
               yield: yield the current time slice.
             thread1: start the thread1 test case.
               send1: send a message to the thread1 test thread.
             cancel1: cancel the thread1 test thread.
             thread2: start the thread2 test case.
kernel % time date
Thu Jan  1 00:23:50 1970
elapsed time: 0.007252000 sec
kernel %

Kind of cool, right? Well, just to show that some rudimentary threading is going on:

kernel % thread1
unhandled system call (175) args: 1, 1194164, 0, 8, 8, 0
thread started foo
thread self = 0x00029EB0
kernel % send1
thread running 3
kernel % 

Even better, here’s a test case where the thread periodically sleeps:

kernel % thread2
thread2 started
kernel % thread2 still running
thread2 still running
thread2 still running
thread2 still running

The code for thread2 looks like this:

static void *thread2(void *arg)
{
    printf ("thread2 started\n");
    for ( ;; ) {
        // Go to sleep.
        sleep(10);
        printf ("thread2 still running\n");
    }

    return NULL;
}

Unfortunately it will be running forever. I haven’t implemented pthread_cancel() or pthread_kill() support yet. But the command prompt is still active:

thread2 still running
thread2 still running
thread2 still running
timethread2 still running
 sleep 1
elapsed time: 1.000731000 sec
kernel % 

(I had typed in “time sleep 1”)

As usual, the code is available here. Please don’t look at irq.c. The interrupt controller code is a total hack for now.

UPDATE: If you want to try this out at home, everything you need except QEMU is packaged as a binary download from ftp://ellcc.org/pub/ choose the tarball appropriate for your host system, untar it, go into the ellcc/baremetal/arm directory and type “make run”.

Make sure you have QEMU installed on your system, as I am not currently able to cross make it for all the hosts.

Even More Bare Metal ARM

I’ve spend much of the weekend (it is a holiday, right?) playing around with my bare metal prototype. In the last post in my Bare Metal ARM series, I did a little context switching. Actually it implemented simple co-routines that could give up execution to each other. In this installment, I’ve added a rudimentary ready list and implemented a simple message passing scheme allowing threads to communicate with each other and block if no messages are available.

The full source of my little prototype is here. A little explanation of how I’m developing this might be in order. init.S is the processor initialization and exception handling code, written in assembly because most of it has to be. init.S calls into the C standard library by calling __libc_start_main() after setting up a few stack pointers and zeroing out the uninitialized data (.bss) area. __libc_start_main() ends up calling main() after initializing the library.

The Makefile can be used to build and run the code.

[~/ellcc/baremetal/arm] dev% make
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp  -c init.S
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c main.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c simple_console.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c simple_memman.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -march=armv7 -mfpu=vfp -mfloat-abi=softfp -g -MD -MP -Werror -Wall -Wno-unused-function -c scheduler.c
../../bin/ecc -target arm-ellcc-linux-eabi5 -nostartfiles -T kernel.ld \
    ../../libecc/lib/arm/linux/crtbegin.o \
    init.o main.o simple_console.o simple_memman.o scheduler.o \
    ../../libecc/lib/arm/linux/crtend.o \
    -o kernel.elf -Wl,--build-id=none
../../bin/ecc-objcopy -O binary kernel.elf kernel.bin
[~/ellcc/baremetal/arm] dev% make run
../../bin/qemu-system-arm -M vexpress-a9 -m 128M -nographic -kernel kernel.bin
audio: Could not init `oss' audio driver
kernel: hello world
unhandled system call (0) args: 1, 2, 3, 4, 5, 6
__syscall(0) = -1, No error information
hello from context 42 code = 3
hello from context 42 code = 6809
prompt: hi there
got: hi there
prompt: 

To exit out of QEMU use the key sequence Control-A x.

By the way, the send and receiving of messages (the “hello from context” messages) is done in main.c:

...
Queue queue = {};
static intptr_t context(intptr_t arg1, intptr_t arg2)
{
    for ( ;; ) {
      Message *msg = get_message(&queue);
      printf("hello from context %" PRIdPTR " code = %d\n", arg1, msg->code);
    }
    return 0;
}
...
int main(int argc, char **argv)
{
...
    new_thread(context, 4096, 42, 0);
    Message msg = { { NULL, sizeof(msg) }, 3 };
    send_message(&queue, &msg);
    msg.code = 6809;
    send_message(&queue, &msg);
...

I do debugging by using two windows, one to start QEMU in debug mode and the other to run GDB. The first window:

[~/ellcc/baremetal/arm] dev% make debug
../../bin/qemu-system-arm -s -S -M vexpress-a9 -m 128M -nographic -kernel kernel.bin
audio: Could not init `oss' audio driver

The other window looks like:

[~/ellcc/baremetal/arm] dev% ~/ellcc/bin/ecc-gdb kernel.elf
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kernel.elf...done.
(gdb) target remote :1234
Remote debugging using :1234
0x60000000 in ?? ()
(gdb) break main
Breakpoint 1 at 0x10408: file main.c, line 40.
(gdb) c
Continuing.

Breakpoint 1, main (argc=1, argv=0x100e8 ) at main.c:40
40          printf("%s: hello world\n", argv[0]);
(gdb) 

One aspect of this code that might be confusing is that there is no apparent explicit initialization code in main() for the serial port, memory allocator, or scheduler. Since this is implemented with a full C library, I take advantage of the fact that static constructors work and are extensions to C in both clang and GCC. An example is at the bottom of the scheduler.c file:

/* Initialize the scheduler.
 */
static void init(void)
    __attribute__((__constructor__, __used__));
    
static void init(void)
{   
    // Set up the main and idle threads.
    idle_thread.saved_sp = (Context *)&idle_stack[IDLE_STACK];
    __new_context(&idle_thread.saved_sp, idle, Mode_SYS, NULL,
                  0, 0);
    
    // The main thread is what's running right now.
    main_thread.next = &idle_thread;
    ready = &main_thread;
        
    // Set up a simple set_tid_address system call.
    __set_syscall(SYS_set_tid_address, sys_set_tid_address);
}

Why did I do it this way? Because I don’t have to change any source code if I want to swap out simple_console.c for a hypothetical interrupt_console.c, for example. I just have to change the SRCS macro in the Makefile.

The next step is to implement interrupt handlers, the timer, and preemptive scheduling. By the way, the full POSIX pthread_create() is still a ways off, but it feels like it is getting closer all the time.

More Bare Metal ARM With Linux C Libraries

As a follow up to my post An ARM Bare Metal Hello World Using Musl I thought I’d give an update. A quick background. I’m trying to build a bare metal (no OS) framework that can use a stock Linux standard C library. To do this I’m building a small assembly infrastructure that traps the Linux system calls used by the library and deals with them. In my previous post, I got “hello world”. In this one, I’ve come a bit farther. I can read stdin, malloc() memory and do simple context switching. You can see the code I’m working on here. In this example, printf(), fgets() and malloc() is usable enough to be used for a simple context switching example.

Most of the example main program (THREAD is not defined yet: don’t get too excited):

long __syscall_ret(unsigned long r);
long __syscall(long, ...);

#define CONTEXT
#if defined(CONTEXT)
static void *main_sa;
static void *context1_sa;
static void *context2_sa;

static int context(intptr_t arg1, intptr_t arg2)
{
    void **context_sa = (void **)arg2;
    for ( ;; ) {
      printf("hello from context %" PRIdPTR "\n", arg1);
      __switch(context_sa, main_sa);
    }
    return 0;
}
#endif

int main(int argc, char **argv)
{
    printf("%s: hello world\n", argv[0]);

    int i = __syscall_ret(__syscall(0, 1, 2, 3, 4, 5, 6));
    printf("__syscall(0) = %d, %s\n", i, strerror(errno));

#if defined(THREAD)
    int s;
    pthread_attr_t attr;
    s = pthread_attr_init(&attr);
    pthread_t id;
    if (s != 0)
        printf("pthread_attr_init: %s\n", strerror(errno));
    s = pthread_create(&id, &attr, &thread, NULL);
    if (s != 0)
        printf("pthread_create: %s\n", strerror(errno));
#endif
#if defined(CONTEXT)
    char *p = malloc(4096);
    context1_sa = p + 4096;
    __new_context(&context1_sa, context, Mode_SYS, NULL, 42, (intptr_t)&context1_sa);
    p = malloc(4096);
    context2_sa = p + 4096;
    __new_context(&context2_sa, context, Mode_SYS, NULL, 6809, (intptr_t)&context2_sa);
    // Let's do some context switching.
    __switch(&main_sa, context1_sa);
    __switch(&main_sa, context2_sa);
    __switch(&main_sa, context1_sa);
    __switch(&main_sa, context2_sa);
    __switch(&main_sa, context2_sa);

#endif
    for ( ;; ) {
        char buffer[100];
        fputs("prompt: ", stdout);
        fflush(stdout);
        fgets(buffer, sizeof(buffer), stdin);
        printf("got: %s", buffer);
    }
}

The output:

~/ellcc/baremetal/arm] dev% qemu-system-arm -cpu any -M versatilepb -m 128M -nographic -kernel kernel.bin
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
kernel: hello world
unhandled system call (0) args: 1, 2, 3, 4, 5, 6
__syscall(0) = -1, Function not implemented
hello from context 42
hello from context 6809
hello from context 42
hello from context 6809
hello from context 6809
prompt: hello world
got: hello world

It’s almost starting to look like Posix at the bare metal level. Fun stuff.

An ARM Bare Metal Hello World Using Musl

I’ve been trying to figure out how to use the excellent musl C standard library in a bare metal environment for quite a while. I didn’t want to damage the musl source code too much but couldn’t figure out an easy way to get at least a subset of musl’s functionality available in a bare metal environment.

Over the weekend I can up with a cute little solution that may be viable for future development. It’s based on a simple concept: Don’t change musl at all, but implement system call handling at the machine level to handle the system calls that musl needs.

I added a baremetal directory to the top-level ELLCC directory as a proof of concept. The arm sub-directory is where the initial work was done. There are just a few files in the directory:

  • Makefile – To build the example
  • init.S – To handle system initialization and exception handling
  • main.c – A simple C source file showing the handling of two system calls
  • kernel.ld – A simple linker command file to put it in the right place in the ARM’s memory.

It turns out that musl’s printf does two system calls to print “hello world\n”. The first is an ioctl() to determine how to buffer. I ignored that call since all the unimplemented system calls return error.
The interesting call that musl did make was to writev(), for which I made a simple replacement in startup.c.

After typing “make”, here is the result of running it:

[~/ellcc/baremetal/arm] dev% qemu-system-arm -cpu any -M versatilepb -m 128M -nographic -kernel kernel.bin
pulseaudio: set_sink_input_volume() failed
pulseaudio: Reason: Invalid argument
pulseaudio: set_sink_input_mute() failed
pulseaudio: Reason: Invalid argument
unhandled system call 54
hello world
QEMU: Terminated
[~/ellcc/baremetal/arm] dev%

The 54 is the unhandled ioctl(). I’m sure that output will be very handy as I implement more of musl’s functionality for my bare metal environment.