USB seems to be damaged in some way as it reports nothing but rubbish to the kernel and something causes bad noises to come from the beeper. Some forum threads pointed out that might be caused by a bad power supply. However, after changing the AC adapter it doesn’t run any better in my case. Since I wasn’t able to flash the original firmware correctly (it was just too obvious that I manipulated that box) I decided to open it up and take a look at its PCB.

Unfortunately I can’t see any physical damages on that board so I must assume there’s some hidden defect somewhere. I doubt I can fix it, so I must consider that box to be broken after just 4 months. After all I read I must advice everyone NOT to buy a NSLU2 – except you are out for some soldering on the board. The hardware fails much too often from what I could find, be it for overheating, manufacturing or just design errors (like the fancy “feature” that you can power the box solely by an external USB hub). Sadly, if I cannot find the error or a workaround for it, I may not continue on this project since I am not willing to buy another box for full price that’s broken by design.

Either I buy one very very cheap at eBay or I will just let it die. Maybe someday I might be able to fix it – could be my studies might be of some help at some later point.

Anyway; this project was no complete loss of time. Much of the knowledge I gained can be helpful in getting Gentoo onto the Pandora when it finally arrives (which unfortunately may take another 3-4 months depending on LCD production since I assume I’m somewhere in the last third of the preorder queue).

For months I have been trying to figure out why my cross-compiled kernel and rootfs won’t boot on the NSLU2 aka Slug. Since I didn’t want to solder pins for the serial I/O onto the board I tried a quite unique approach: Since the Slug has 4 LEDs that can be easily controlled through GPIO, I could hook up at some function and send all output through the LEDs. Sure this would be slow (~1 character per second, maybe 1.5) but that would be enough to at least read kernel panic messages while retaining full warranty on the hardware.

Finding out how LEDs were controlled took me about 1,5 hours, hooking up and testing first things another hour. I hooked up before the call of init (disk 2 on), right after init (disk 1 on) and on kernel panics (both disk leds on). So what happened: It got init and then crashed with a panic after a few seconds. I figured out I could hook up at uart_console_write() or panic() and then read any output by blinking a byte in 4 steps (one byte on each disk LED, signal indication by setting power to green and “clock” indication through power amber). Well it started blinking for hours and hours and hours… But all I could decode was just infinite rubbish, no matter what I tried. Even a comparison i

Nothing worked - until now....

For some reason I chose msleep() over mdelay(). That was the critical fault.

msleep() seems to suspend the currently running task, so it is non-blocking regarding the whole system.

mdelay() blocks the system (or at least the active CPU) if running single-threaded.

So why was that small change critical to my code? I don’t know exactly. But I know that panic() disables scheduling before any further action. So what happens if some code fragment used by panic() tries to relay on scheduling? Something seems to get corrupted very seriously, maybe some kind of heap or stack overflow happens. Maybe some process/scheduler data gets screwed up. I don’t know. But that seemingly tiny difference of blocking vs. non-blocking functions (what function does what isn’t always that clear if you’re new to Linux kernel programming) really makes a very big difference.

I finally recorded my kernel panic on dv tape and will decode it tomorrow using a simple tool I wrote. If it is finished I will make it available from this website.

Here’s a small excerpt from the ~20 minutes long message transmitting “BUG: sched[...]” (I did not decode more than that yet):

Edit: I decoded all 20 minutes. What was readable (the decoder was a quick and dirty solution since yet) led me to “BUG: scheduling while atomic:” in kernel source and was simply caused by a remaining msleep in my LED function. I got rid of it and now got a clean message “Attempted to kill init!”. Now, that’s where the debugging begins…

If that device is completely new to you, the article on Wikipedia (en/de) may provide a good starting point for more information on what is possible. If you get interested in it, nslu2-linux.org provides a great resource to answer almost all your questions.

My goal is to get Asterisk, DHCP, DNS, OpenVPN and maybe a small webserver to run on it. However I haven’t reached that yet. (Click the link below to read more.)I initially booted the original system to verify everything is working. Since I don’t have any USB HDD and intended to attach a cheap 8GB USB flash drive I (surprisingly) couldn’t do anything with Linksys’ firmware: It detects the stick but wouldn’t use it without a real hard drive attached to USB port 1 to install onto. With just 8 minutes uptime I decided to flash it right away.

I wanted to install Gentoo but installation on an embedded system is not that straight at all. Since there are some ready-to-use distros for the slug I flashed SlugOS/BE onto it. As you may guess from the suffix, it’s a Big Endian version of that (Debian-based) distribution. The CPU could switch to Little Endian as well but since the unmodified RedBoot from the NSLU2 expects the system to be in BE mode it’s easier to set it up that way. It booted up with no problems and I could log in and start installing an outdated 2005.0 release of Gentoo using uclibc instead of glibc – unfortunately this is the latest (experimental) release of Gentoo for armeb (ARM Big Endian) using softfloat (to emulate the missing floating point unit) and uclibc (for saving memory).

I got quite far updating the base system but shortly before I could run a world update I experienced a severe blocker issue: uclibc is slowly updated and reported to be bugged in its latest revision, so Gentoo only supports the previous version. Unfortunately that version is reported to be unable to work with kernels/kernel headers >2.6.17 and caused a lot of bugs lately due to its age.

Now I had two options:

1. Forget about Gentoo and use one of the Debian based distros for the slug. However, SlugOS has been released before the critical kernel bugfix from February 2008, so it must be considered vulnerable to attacks if exposed to the Internet. Maybe I could update that system but I didn’t really want to be stuck with Debian.

2. Read into cross-compiling a complete system for embedded systems, manually applying kernel patches and transferring everything to the slug for trial-and-error testing until it finally boots and gives me a remote shell. Eventually this is the way I chose a week ago and that’s where the fun starts.

I just want to give a short introduction with this post. During the next weeks/months however, I will blog more and more articles about the issues I have with getting the system to run. So far I finally understood cross-compiling on Gentoo, got a basic system that could theoretically be chrooted into but since I compiled with a more recent version of glibc than SlugOS has been linked against, it won’t let me in yet. Today I prepared a “TinyGentoo” to (possibly) boot into, but it’s not ready yet. Next I will need to slim the boot system and try to build a kernel and transfer everything onto the slug for a first try to boot it blindly (without soldering the serial port connector onto the board).

Unfortunately I have two exams next monday so it’s unlikely I will get any further this week. I will keep this blog category updated.