The first Mesh Potato

Yesterday (Monday June 1) the courier arrived with the very first Mesh Potato (MP) prototype, which had been hand assembled by the good people at Atcom. This is always an exciting and risky time in the life of any hardware project, as you don’t really know if it’s going to work. In fact, there is a hell of a lot that could go wrong. Like one solder ball under the Atheros BGA chip not soldered, or a nasty PCB or schematic design error that means another re-spin of the PCB and 1 month delay. Or smoke might come out of the Atheros chip when you first apply power. I have seen all of these on past projects…..

Fortunately Alen at Atcom had done some initial tests, like checking for the 3V3 and 1V8 power rails, and making sure the 40 MHz clock was present. So we had some encouraging early signs of life.

My initial goal was to get the boot loader to run – this would prove that most of the AR2317 circuit is OK (CPU, RAM, flash etc). I chose the boot loader image (ap61.rom) that we have been using on the DIR-300 router as this router uses the same Atheros AR2317 chip. So I connected the JTAG cable. The CPU was detected (excellent!) and the jtagspi software started flashing the boot loader. This is a slow process (2.5 hours) using my home-brew JTAG cable, so I headed out for a nice Chinese lunch and a walk with my wife! Alright, so we also went to the bakery and bought a custard filled cake. You need energy to hack.

Two hours later I returned and tentatively connected the RS232 cable and power up the MP. This is what I saw:

No board config data found!+flash_hwr_init: Unsupported flash device - id=22FLASH: driver init failed: Driver does not support deviceSorry, FLASH config exceeds available space in FIS directoryCouldn't find valid MAC address for enet0. Using default!Invalid PHY ID1 for enet0 port0.  Expected 0x0243, read 0xffff/ar2317-prj/LSDK5.0.2.46/src/redboot_cobra/ecos/packages/devs/eth/mips/ar531x/ccEthernet eth0: MAC address 00:03:7f:e0:02:bfIP: 0.0.0.0/255.255.255.0, Gateway: 0.0.0.0Default server: 0.0.0.0RedBoot(tm) bootstrap and debug environment [ROMRAM]Non-certified release, version UNKNOWN - built 00:32:22, Aug  7 2007Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc.Board: ap61RAM: 0x80000000-0x80800000, [0x80040760-0x807f1000] availableFLASH: 0x00000000 - 0x00000001, 0 blocks of 0x00000000 bytes each.RedBoot>

This is actually pretty good – the boot loader is running – with a few complaints about the 8M flash and Ethernet PHY. However a running boot loader means much of the hardware must be OK. Nice.

I have some RedBoot hacking to do (to accommodate the 8M flash and Ethernet PHY) so the next step was to get the Ethernet up. That way I can tftp new boot loader images rather than put up with 2 hour JTAG flash sessions. Actually in hind sight I should have bought a commercial JTAG cable that can flash in 5 seconds. However the Ethernet was dead (no link lights, ping didn’t work), and I didn’t like the look of those complaints about the PHY chip.

So I perused the DP83848I PHY chip data sheet. I decided to probe a few pins on the PHY chip with my oscilloscope to see if it looked alive. One pin I tested was the CLK_OUT pin 25, which should have a 25 MHz signal on it. However it was silent. Hmmmm, not good.

So I contacted Alen via IM and he confirmed that on other designs using that chip p25 definetely had a 25 MHz signal on it. So I checked the second MP01 I had been sent. No CLK_OUT signal either.

When both my MP01′s didn’t have the CLK_OUT signal I suspected it was a wiring/design error. Well, it was just a guess really. Telling these bug hunt stories in review it looks like the outcome was pre-ordained. However at the time you are never really sure. You usually go down many blind alleys, which results in frustration and lack of sleep shutting brain cells down!

Anyway clocks should usually come straight up without any software initialization. So I just checked the reset/interrupt lines (in case the chip was being held in the reset state), then I checked the various power supply lines to the chip. A-HA! One power supply line (pin 22 AVDD33) was not connected! Via IM Alen then double checked his other designs and confirmed that yes, it was a bug.

When I added a wire to supply 3V3 to pin 22 CLK_OUT came up and the network link light started working. Yayyyyyyy!

There is also a golden rule in hardware (and indeed software) development. Anything you change will have bugs. That PHY was changed from the reference design so it’s a natural source of bugs. A bug “emitter” if you like.

Then it was time for some network tests. Ping, and even tftp worked. Very cool. The default state of the PHY chip seems to be just enough to get basic network connectivity. This means I can download new RedBoot images quickly as I modify for thew new flash and PHY.

It was getting late by this stage so I decided to call it a night. So I emailed Elektra about where we were at and went to watch Top Gear which my three year old. However our story does not end there. Elektra was working hard while I was resting and sleeping. When I woke up this morning she sent me several emails detailing how to set up the RedBoot development environment. This is great – saves me a mornings work. Amazing how working 12 hours apart (Berlin and Adelaide, Australia) can be helpful!

Right. Time for some a fresh cup of coffee and some RedBoot hacking.

Originally, we had imagined having a management interface for the Village Telco very similar to the Meraki and/or Open-Mesh network management “dashboards” which are  graphically-intuitive and easy for anyone to understand at a glance what is going on in the network.  However, Antoine introduced a new twist which opens up interesting possibilities.  He has designed the Afrimesh software so that it can run on any OpenWRT-based device.  It is written entirely in JavaScript. Actually Afrimesh will run on just about anything but, for me, the interesting new possibility is have a dashboard interface that runs on the mesh devices themselves.  I didn’t initially appreciate initially the potential that this opens up.

You can see at the left here what the management interface currently looks like.  As things currently stand, Afrimesh is designed to:

• monitor the health of the network and display connectivity information for each mesh node and connections between the nodes in a visually intuitive manner;
• provide an interface for the configuration of a B.A.T.M.A.N.-based mesh network; and,
• be an extensible GUI framework for interfacing with a mesh network.

From self organising data networks to self-organising phone networks

So here is where my minor epiphany came.  What is great about mesh networks is that you just plug them in and they work.  They self-organise in terms of upstream connectivitity, redundant routing, etc.  They are a very, very clever way of quickly establishing a pervasive, reliable IP-based network.

With the Mesh Potato, we are trying to leverage the power of mesh networks to set up a phone network.  Each Mesh Potato access point will have an RJ11 port, into which an everyday phone can be plugged.  However, to use the Mesh Potato, some configuration of the phone network detail is still required.  People need to have phone numbers to dial and a phone number themselves that people can call.  More than that, people have to have a way of finding out other people’s phone numbers.

I began to think of the Mesh Potato in different environments to that which we originally conceived.  For instance, would Mesh Potatoes work in an crisis environment where there was a need to set up a instant phone network?  Could they be designed to self-configure out of the box as a phone mesh?  Conversations with David Rowe and Antoine have encouraged me to think so.

Imagine you power up your Mesh Potato and an interactive voice response system says:

This is quite a reasonable scenario.  We could set up the Mesh Potatoes to do something like the above and the impact would be that you could unpack a box of 24 Mesh Potatoes, power them up (battery or solar), and within minutes have a working phone network.  This would work with or without upstream connectivity.  It would mean you could immediately start delivering value with a Village Telco even before you have a server, billing, and upstream connectivity set up.

The Afrimesh Potato

But it gets better.  With the Afrimesh software, you could conceivably combine data network information with phone network information.  Consider an enhanced version of the Afrimesh interface which not only provided network information but also voice call information.  The image at the right is my mock-up but it conveys the sense of how Afrimesh could show you who was on the phone network, what their extension is, whether they’re “available”, etc.

I think that having a plug-and-play Mesh Potato that will allow people to start talking with no initial technical configuration would substantially increase the take-up rate of the Mesh Potato.  I also think the Afrimesh interface offers opportunities for interesting voice/data/presence/twitter/jabber cross-over innovation on the MPs.  Love to hear thoughts from others on this.

P.S. For those waiting patiently to hold a Mesh Potato in your hand, David and Elektra are in the final stage of PCB design with Atcom.  I don’t have a date but we should have prototypes very shortly.