On Friday I set about testing Wifi on the two Mesh Potato prototypes. The N-type antenna connectors attach to the MP PCB via a pigtail with a tiny Hi-rose style connector. This looks a bit fragile so I decided to mount the two MPs in some weatherproof boxes so I had a firm mounting for the antenna pig tail. I chose one metal and one plastic, to see if the case material makes any difference to the Wifi performance. They wont be completely weatherproof (I’ll just run the phone power, and Ethernet cables through holes), but enough to take outside for some tests and withstand some handling.

I mounted both the PCB and the antenna connector in the lid, so the whole assembly can be removed without putting any strain on that tiny PCB antenna connector. Only problem is I can’t see the LEDs – will need to make a little window on the bottom of the box.

Mesh Potatoes mounted on box lids

Having secured the antenna connector I brought Wifi up. I thought I would take the small, sensible step of trying to associate with my home Wifi network first. It was all looking good until my MP seemed to slow down then stop responding entirely. Then my laptop (also connected via Wifi) started to slow down and lost Internet connectivity. Oops, routing loop between Ethernet and Wifi on the Mesh Potato! Didn’t know you could bring down a whole network that way.

As Wifi was coming up automatically on boot it was hard to stop, so I ended up re-flashing the router and going straight to Batman. This came up straight away and I had two MPs and a Nanostation happily meshing with each other.

The blinking of the two Wifi lights as batman sends out messages every second is mesmerising

Voice Over B.A.T.M.A.N.

After a bit of messing around with Asterisk conf files I made a MP-MP phone call over Batman and the Mesh Network. The audio in one direction was fine but a little distorted in the other direction. After a break of a few days I took a look at this bug. The problem ended up being the Asterisk channel driver, it was accidentally sending both RTP audio and ring tone to the FXS driver during a call, i.e. ring tone wasn’t being switched off when the call was answered. This was overloading the FXS driver FIFO and causing distorted audio. Curiously, it worked OK until I tried a call over Wifi.

Anyway I have fixed this bug now and tested a few different codec configurations over the mesh including ulaw, gsm with 20ms frames, and GSM with 80ms frames. We can also make calls between a MP and a SIP phone connected to the Ethernet port of the MP. The load av is around 0.5 during a call (using the GSM codec).

Here is a block diagram of the test setup. The Laptop is used for monitoring. It’s cool that I can ssh to the remote MP over the mesh network and mess with Asterisk conf files, run top etc.

MP tests for first calls over the mesh

The audio sounds great. We can’t hear any echo, thanks to the Oslec echo canceller which seems to run fine on this platform. To stress the system a bit I tried ping flooding from my laptop. This affected the audio quality but the call was still very intelligible. Like driving under a bridge with a cell phone. This is interesting – perhaps a mesh potato network will have a “soft” degradation of quality as network traffic increases. Call quality will drop a little, but the network may not fall over.

The reliability of the Mesh Potato hardware has been remarkably good so far. I left the prototypes running for about 5 days, and when I came back the mesh was still running, and picking up a phone gave me dial tone. Can’t complain about that on hardware where the solder has barely cooled!

Almost exactly 1 year since the Mesh Potato project was kicked off at the first Village Telco workshop in June 2008.

Alen from Atcom has been closely following our steps, which has encouraged Elektra and I to write a Mesh Potato How To on the Village Telco Wiki.

So I carefully retraced my steps and tried to re-flash the MP from JTAG. Still dead. Huh? There followed several days of futzing about, looking at the jtagspi code, trying this, trying that, and getting more and more confused and not a little upset! I just didn’t make sense – if ap61.rom worked the first time why not now?

Mesh Potatoes Maternity Ward

Then I noticed something funny in the ap61.rom boot log from the DIR-300:

a) RedBoot(tm) bootstrap and debug environment [ROMRAM]
production release, version "2.1.3" - built 18:43:19, Sep 20 2007

compared to the first, miraculous boot of the MP01 on Monday:

b) RedBoot(tm) bootstrap and debug environment [ROMRAM]
Non-certified release, version UNKNOWN - built 00:32:22, Aug 7 2007

Different dates!

So this idea popped into my head:

1/ What if Alen (from Atcom) had flashed a boot loader before sending me the MP01?

2/ From my records on Monday, and examining the jtagspi code, I know that my write address for my initial MP01 flash on Monday was wrong: 0x1fc00000

The jtagspi software discards the top 8 bits, leaving 0xc00000. Now on a DIR-300 with 4M the top 2 bits get discarded leaving 0×00000, or the first three blocks of flash. As you would expect.

However when I used the same 0x1fc00000 address on a MP01 with 8M flash, I you get 400000, or half way through the 8M flash. So on Monday I think I flashed ap61.rom half way through the 8M flash.

When I booted, it ran code from the first 3 sectors, which was whatever was in the SPI flash before I flashed it!

So I checked with Alen, and yes he had programmed the SPI flash chips before soldering! So I had been following a blind alley all week – the idea that ap61.rom boots OK on the MP01!

4/ I continued to chat with Alen via IM. Guess what? The MP01s have 8M flash chips fitted – as that is what the reference design had! That also explains why ap61.rom won’t boot – it was compiled for 16M! The mp01.bin Alen used was set up for a 2M flash/8M ram router. You can even see it in the boot log RAM reports:

MP01:

Board: ap61
RAM: 0x80000000-0x80800000, [0x80040760-0x807f1000] available
FLASH: 0x00000000 - 0x00000001, 0 blocks of 0x00000000 bytes each.
RedBoot>

DIR-300:

Board: DLINK DIR-300
RAM: 0x80000000-0x81000000, [0x80040580-0x80fe1000] available
FLASH: 0xbfc00000 - 0xbfff0000, 64 blocks of 0x00010000 bytes each.


So with my 3 yrs old in tow I ran down to my soldering tools guy and bought some “chip quick”, a special low-melting-point solder than can be used to remove surface mount chips with just an ordinary soldering iron. Here is a 30 second video of how Chip Quick works. It’s actually good fun to use, and the low melting point is very kind to the PCB. It also saved me a couple of days finding some one with proper SM rework tools (it’s Friday afternoon before a long weekend here). No way I wanted to wait until Tuesday to fix this bug!

Now I am pretty geeky, but even I don’t have 16Mbyte SDRAM chips just laying about. So it was time for some open-heart surgery. I removed a 16M flash chip from the “donor” DIR-300, and soldered it onto MP001-001 (the first prototype). I powered up and no power LED – there was a short circuit between 3V3 and GND! Bloody Hell – some days you just can’t win! I looked for the elusive short for one hour then gave up on MP001-001 for now. Instead I turned my attentions to the second prototype, MP01-002. I carefully removed the same precious 16M chip from MP01-001 and soldered it onto MP01-002. No shorts this time. I applied 12V and the boot loader comes alive. YAaaaaayyyy!!

As an encore I tried to get Linux to boot….however I had displeased the Gods of embedded systems this week and it was not to be – the Linux image we use for the DIR-300 is stopping part way. For some reason available memory is reported as 32Mbyte rather than the correct 16MByte. However I am going to work on that problem later – time for a few days break!

Hardware for First Mesh Potato Phone Call

Yesterday we made the first phone call using the Mesh Potato (MP) Architecture – a very important moment in the life of any telephony project! The Prototype Mesh Potato hardware isn’t available yet, so we connected a DIR-300 router to a FXS Interface PCB. The DIR-300 router is based on the same chip set as the MP (Atheros AR2317) so it’s as close a a real MP as we can get.

This gives us a development environment almost identical to the MP which allows us to work on the firmware prior to having the actual MP hardware. This allow for faster development compared to waiting for the prototype MP hardware. When the prototype MP hardware arrives we will have tested firmware to use on the untested prototype MP hardware – greatly reducing the scope for bugs.

The firmware consists of several device drivers (8250mp.ko, mp.ko) and an Asterisk Channel Driver (chan_mp). To test we connected a SIP phone to Asterisk running on the DIR-300. Asterisk then routes the call to chan_mp, and via the device drivers to the FXS interface hardware. The audio samples actually flow through the AR2317 RS-232 UART, before being converted to TDM bus samples by a CPLD and Atmega microcontroller. Control and signaling is handled by a SPI port constructed from the AR2317 GPIO lines.

After a bit of tweaking the full duplex audio quality sounded just fine, CPU load was less than 1%. This was really just a basic Channel Driver designed to test the most important feature (full duplex audio) first. The next step is to add a bunch of other functionality like dial tone, echo cancellation, ringing and on/off hook detection.

We are very happy with this step – it’s a big chunk of firmware written and electronics tested.

Originally, we had imagined having a management interface for the Village Telco very similar to the Meraki and/or Open-Mesh network management “dashboards” which are  graphically-intuitive and easy for anyone to understand at a glance what is going on in the network.  However, Antoine introduced a new twist which opens up interesting possibilities.  He has designed the Afrimesh software so that it can run on any OpenWRT-based device.  It is written entirely in JavaScript. Actually Afrimesh will run on just about anything but, for me, the interesting new possibility is have a dashboard interface that runs on the mesh devices themselves.  I didn’t initially appreciate initially the potential that this opens up.

You can see at the left here what the management interface currently looks like.  As things currently stand, Afrimesh is designed to:

• monitor the health of the network and display connectivity information for each mesh node and connections between the nodes in a visually intuitive manner;
• provide an interface for the configuration of a B.A.T.M.A.N.-based mesh network; and,
• be an extensible GUI framework for interfacing with a mesh network.

From self organising data networks to self-organising phone networks

So here is where my minor epiphany came.  What is great about mesh networks is that you just plug them in and they work.  They self-organise in terms of upstream connectivitity, redundant routing, etc.  They are a very, very clever way of quickly establishing a pervasive, reliable IP-based network.

With the Mesh Potato, we are trying to leverage the power of mesh networks to set up a phone network.  Each Mesh Potato access point will have an RJ11 port, into which an everyday phone can be plugged.  However, to use the Mesh Potato, some configuration of the phone network detail is still required.  People need to have phone numbers to dial and a phone number themselves that people can call.  More than that, people have to have a way of finding out other people’s phone numbers.

I began to think of the Mesh Potato in different environments to that which we originally conceived.  For instance, would Mesh Potatoes work in an crisis environment where there was a need to set up a instant phone network?  Could they be designed to self-configure out of the box as a phone mesh?  Conversations with David Rowe and Antoine have encouraged me to think so.

Imagine you power up your Mesh Potato and an interactive voice response system says:

This is quite a reasonable scenario.  We could set up the Mesh Potatoes to do something like the above and the impact would be that you could unpack a box of 24 Mesh Potatoes, power them up (battery or solar), and within minutes have a working phone network.  This would work with or without upstream connectivity.  It would mean you could immediately start delivering value with a Village Telco even before you have a server, billing, and upstream connectivity set up.

The Afrimesh Potato

But it gets better.  With the Afrimesh software, you could conceivably combine data network information with phone network information.  Consider an enhanced version of the Afrimesh interface which not only provided network information but also voice call information.  The image at the right is my mock-up but it conveys the sense of how Afrimesh could show you who was on the phone network, what their extension is, whether they’re “available”, etc.

I think that having a plug-and-play Mesh Potato that will allow people to start talking with no initial technical configuration would substantially increase the take-up rate of the Mesh Potato.  I also think the Afrimesh interface offers opportunities for interesting voice/data/presence/twitter/jabber cross-over innovation on the MPs.  Love to hear thoughts from others on this.

P.S. For those waiting patiently to hold a Mesh Potato in your hand, David and Elektra are in the final stage of PCB design with Atcom.  I don’t have a date but we should have prototypes very shortly.

Elektra is working on hardening the various ports of the Mesh Potato against environmental and accidental damage. For example making sure the power port is robust to reverse DC connection, and protecting the Antenna and Ethernet ports from static. She is also taking particular care to ensure that the power supply is as efficient as possible; saving a few watts really matters in solar powered applications.

I am working on the FXS interface hardware. The Atheros SoC chips have a lot of functionality, but unfortunately don’t have a Time Division Multiplexed (TDM) bus peripheral which is required to connect to the FXS chipset. The current plan is to make a RS232 serial to TDM adaptor, using some programmable logic. Speech samples will flow through the Atheros SoC RS232 serial port to and from the TDM bus.

The software side of the FXS interface involves modifying the serial driver (8250.c in the kernel), to store speech samples in buffers that are then uploaded to user mode for Asterisk. It is important that the interrupt service routine is really fast, so I am bypassing much of the standard serial code and writing a custom ISR.

Over the last few weeks I have written the modified driver and tested it using RS232 signals from a PC. This week I have been designing and testing the hardware side of the interface. This involves writing a bunch of Verilog code, then testing it using a PC based simulator. I am using the (open source) Icarus and GTKwave programs to simulate and visualise the waveforms. It’s much easier to test and debug all this complex logic in simulation form before firing up any hardware.

Over the last few days I have been testing all this logic using a prototype jig:

Two IP04s are used. IP04s happen to have a Xilinx CPLD chip on board that is suitable for running the FXS Interface logic. So on the LH IP04 I reprogrammed the CPLD with the FXS Interface logic. The right hand IP04 in the picture above is used as “life support” for a FXS module. The FXS module runs as per normal on the IP04 but we tap off the various TDM bus signals we need to drive the CPLD on the IP04 on the left. So to test I can speak into a phone connected to the IP04 and the CPLD does it’s thing with the signals, converting them to and from RS232 serial.

So far I have tested both the receive and transmit side of the FXS Interface by connecting the serial signals to a PC. I managed to sample my own voice by capturing the RS232 bytes to a file and playing it back on my PC’s sound card. So far so good.

Today I have started connecting the FXS Interface to a Nanostation 2. My first tests crashed the Nanonstation! I guess due to something I changed on the serial driver. So time for some debugging……

Links

Verilog Tutorial
FXS Interface Verilog source and test benches
README documenting FXS Interface design
FXS Interface timing diagrams in gschem format