The Mesh Potato is a mix of many different technologies: embedded Linux, telephony hardware, VOIP and Wifi. The Radio Frequency (RF) side of the Mesh Potato is a loose end we need to tie up. Unfortunately, data on the RF side is nearly impossible to obtain. To get the full support package from Atheros costs USD$100k. That is beyond our budget and sits uncomfortably with the open source nature of the Village Telco project.

Each AR2317 chip is a little different and consequently each chips requires individual RF calibration on the production line. This calibration ensures each AR2317 product has uniform transmit power and meets certain performance criteria. The calibration data is stored at the end of the SPI flash chip and loaded by the WiFi driver at boot time. Calibration is performed on the production line using a bunch of expensive RF test equipment connected via GPIB to a host computer running special software.

The calibration procedure and most of the data relating to the AR2317 RF section is part of the Atheros “secret sauce”.

On our prototype Mesh Potatoes we were lucky enough to have the WiFi fire up first time. We just copied the calibration data from another AR2317 based product as a starting point. The Wifi worked, but without calibration Wifi performance is likely to be poor. As we had no RF test equipment our visibility was limited – it was hard to even measure the RF performance.

Elektra managed to hook a V1.0 MP01 up to a borrowed spectrum analyser, thanks to kind guys at the CSIR in South Africa. A spectrum analyser graphs the power at each frequency. Here is what the spectrum of a calibrated DIR-300 looks like:

DIR-300 1Mbit Channel 11 spectrum

And here is the output from our uncalibrated V1.0 Mesh Potato:

MP01 1Mbit Channel 11 spectrum

So after the 2nd Village Telco workshop we decided to tackle Calibration. Fortunately, our friends at Atcom found some friends in Shenzhen who have the production line equipment for AR2317 calibration. Here in Adelaide I bought some basic test equipment – an old Tek 492 spectrum analyser from e-bay and a frequency counter. The Tek 492 is an analogue spectrum analyser (with some digital storage ability) from the early 1980′s. In their day they cost $30,000 but are available 2nd hand for around $2,000. Microwave hacking on a budget! The following photo is courtesy of Ben (see below):

In early October we made our first attempt at calibration on one the V1.1 Betas. The automatic test equipment in Shenzhen “failed” us. There were several bugs including low output power (10dB down) and a large frequency offset (60ppm instead of 20ppm). Another concern was that several Mesh Potatoes tested gave different results. Great. Just what we need when we are trying to get 100 Betas out the door for eager developers.

Thus began a two week frenzy of RF debugging, with Mesh Potatoes being couriered back and forth between Shenzhen and Adelaide and much soldering of tiny 0402 size parts under the microscope. These parts appear the size of a small crumb to the naked eye (about 1mm by 0.4mm), but surprisingly you can hand solder them under a microscope with a little patience.

None of the core team are experienced in Wifi or 2.4GHz RF, and we are working without 95% of the data and test equipment we need for proper RF development. So we had a few challenges.

After about two weeks of hard work we made some progress. I managed to bring 3 out of 4 betas up to our target power of 17dBm (the 4th I blew up accidentally). We discovered some areas where the PCB layout could be improved to get even more power (perhaps 20dBm) out of the MP. The system clock was a few kHz off 40.0 MHz which when multiplied up to 2.4 GHz caused the 60 ppm frequency offset. This was actually the largest problem – it caused packet errors at the low data rates which messed up long range performance. Once this was fixed the packet error rate performance started to look as good as the reference DIR-300 unit we were testing against.

The variable power output of different MPs had a simple reason – some of the tiny 0402 parts were loaded in the wrong place. This is very easy to do as the parts have no writing on them. I only spotted this when I noticed two parts with the same value looked slightly different in colour. Usually all parts from the same reel look identical.

With our new-found experience, Atcom decided to make another revision of the PCB (V1.2) to tighten up the RF side. That should be ready for testing in November. If calibration checks out on V1.2 we will then kick off a Beta run.

Given our lack of RF experience, lack of AR2317 data, lack of support from the chip vendor, and very basic test equipment I feel pretty happy with our progress in RF performance over those two hard weeks. The %$%^ Asterisk channel driver for the Mesh Potato took me three weeks to get stable and I am meant to know something about Asterisk driver development!

As well at the core team of Elektra, myself, the Atcom team (Edwin, Alen, Mr. Lee, Peter), their Shenzhen friends and of course Steve, we had some help from Jeff (our RF consultant) and two other people who were especially kind:

I found Ben Johnson while looking for information on the Tek 492. Ben had posted a page on how he repaired his Tek 492. I emailed Ben to ask if he thought it was suitable for Wifi. Ben was kind enough to actually test his Tek 492 on some Wifi signals and email me the results! This gave me the confidence to bid for a used 492 on ebay here in Australia.

I met Dieter through an article I published on Low cost Electric Cars in Renew magazine. He just happened to email me on a day I was messing with the Tek 492 and I found out he was a microwave engineer working in Melbourne. Throughout the RF debugging process Dieter emailed me virtually every day and gave much needed advice and moral support. He even sent over some semi-rigid RF cable with an N-connector to help with the testing.

The Internet is amazing place. I am constantly bowled over by the kindness of people – especially when you are working on open projects.

Links

[1] Measuring Wifi Transmit Power – An in-depth look on how I used the Tek 492 to measure Wifi transmit power.

Stability

One simple stability test is uptime. I have left both my MP prototypes running for up to 5 days at a time with good results – Linux and Wifi stayed up, and we still had dial tone from the FXS port. This simple test says a lot: e.g no memory leaks, CPU instability, drastic over temperature problems.

Another test I like to do to all my telephony devices is hammer them with phone calls for 24 hours. Here is the test set-up:

Stability Test

An x86 Asterisk box acts as the call generator. It starts a new call to the MP. When the MP receives the call it starts the FXS port ringing. Another Asterisk box (an IP04 with an FXO port) answers the call, plays a prompt for a few seconds, then hangs up. The entire sequence is then repeated – make thousands of calls over a 24 hour period. The SIP call is placed over Mesh Wifi to make sure all parts of the MP system are being exercised.

This test gives the MP quite a hard time as much of the call processing load is involved in set up and tear down of calls, and unless you are my teenage daughter you can’t really make 3500 calls in 24 hours.

When I started the tests it threw up problems nearly immediately – Asterisk was seg faulting with no feedback as to why it was crashing. This led to a week of fun and games where the problem was eventually traced to thread problems in the Asterisk channel driver I had written (chan_mp). This channel driver has been a painful experience for me (even though I have written a few before), and has taken perhaps 3 weeks of part time work to write and debug. Lots of fun with threads and null pointers. Even now I still don’t quite understand all the issues around Asterisk channel drivers. I would appreciate a code review of the driver if there are any Asterisk channel driver experts out there.

Anyway I finally tracked down the source of the problem and modified the channel driver. It passed the 24 hour stability tests and made 3500 calls.

Mesh Load Test

This is a repeat of the load test from Phase 1 of the Mesh Potato Project, but this time on the actual MP hardware with the FXS driver running:

Mesh Load Test

A big difference between the Village Telco and Cell Phone network is no base stations (Cell Phone Towers). Instead, the calls are routed via the client nodes. It’s a peer-peer network rather than a client-server architecture.

The idea of this test is to make sure that a given mesh potato node can relay 15 phone calls for other people while simultaneously making a phone call of it’s own. This scenario places significant CPU load on the router due to the number of Wifi packets that must be processed at the same time as DSP intensive code like echo cancellation and speech compression.

The set up details of this test are on the wiki and have been blogged about earlier. Speech quality was fine, and loadav about 0.71, similar to the Phase 1 results. Occasionally I could hear some packet loss but overall the call was fine. A good result considering all that processing (Linux, Wifi, mesh, Asterisk, echo cancelling, GSM speech compression, FXS driver) on that little router chip!

One interesting fact is that even with a total of 16 calls the bit rate is only around 500 kbit/s, so the bandwidth of the mesh is quite lightly loaded. However speech packets are rather short, so raising the number of phone calls would likely run into CPU load (rather than Wifi bandwidth) issues due to the per-packet processing load.

Wifi Range Test

Wifi is mainly the domain of Elektra and Jeff on this project but I was keen to get some rough idea of how well the radio was working. I had made a few calls around the house OK but noticed that the received signal level was quite low. A bit of thought led us to a static protection diode we had placed across the antenna circuit. Although a good idea for the RX side it was conducting when transmit signal was present. After removing this diode the TX level jumped right up to roughly comparable levels to a DIR-300 router that uses the same Atheros AR2317 SoC.

However using radiated signal levels it is hard to get consistent results, they bounce around all over the place. Jeff suggested connecting a combination of DIR-300 and Mesh Potatoes in A-B tests via cables and calibrated attenuators to obtain consistent measurements.

Over a few days I made several attempts at range tests. However where I live is quite flat so it’s difficult to get a good Line-Of-Sight (LOS). I did manage to make phone calls over about 250m which is pretty much what we need for a Village Telco Network.

Elektra and I feel the radio performance could probably be improved with some adjustment of the RF circuit components and possibly PCB layout. Elektra has access to a RF test lab later this week so we will get some more concrete information of Wifi performance.

One area we really need more information on is radio calibration data and the Hardware Design Guide document for the AR2317, however we have been unable to obtain this important information from Atheros.

More blogging to come from Elektra on Wifi radio and power supply performance.

On Friday I set about testing Wifi on the two Mesh Potato prototypes. The N-type antenna connectors attach to the MP PCB via a pigtail with a tiny Hi-rose style connector. This looks a bit fragile so I decided to mount the two MPs in some weatherproof boxes so I had a firm mounting for the antenna pig tail. I chose one metal and one plastic, to see if the case material makes any difference to the Wifi performance. They wont be completely weatherproof (I’ll just run the phone power, and Ethernet cables through holes), but enough to take outside for some tests and withstand some handling.

I mounted both the PCB and the antenna connector in the lid, so the whole assembly can be removed without putting any strain on that tiny PCB antenna connector. Only problem is I can’t see the LEDs – will need to make a little window on the bottom of the box.

Mesh Potatoes mounted on box lids

Having secured the antenna connector I brought Wifi up. I thought I would take the small, sensible step of trying to associate with my home Wifi network first. It was all looking good until my MP seemed to slow down then stop responding entirely. Then my laptop (also connected via Wifi) started to slow down and lost Internet connectivity. Oops, routing loop between Ethernet and Wifi on the Mesh Potato! Didn’t know you could bring down a whole network that way.

As Wifi was coming up automatically on boot it was hard to stop, so I ended up re-flashing the router and going straight to Batman. This came up straight away and I had two MPs and a Nanostation happily meshing with each other.

The blinking of the two Wifi lights as batman sends out messages every second is mesmerising

Voice Over B.A.T.M.A.N.

After a bit of messing around with Asterisk conf files I made a MP-MP phone call over Batman and the Mesh Network. The audio in one direction was fine but a little distorted in the other direction. After a break of a few days I took a look at this bug. The problem ended up being the Asterisk channel driver, it was accidentally sending both RTP audio and ring tone to the FXS driver during a call, i.e. ring tone wasn’t being switched off when the call was answered. This was overloading the FXS driver FIFO and causing distorted audio. Curiously, it worked OK until I tried a call over Wifi.

Anyway I have fixed this bug now and tested a few different codec configurations over the mesh including ulaw, gsm with 20ms frames, and GSM with 80ms frames. We can also make calls between a MP and a SIP phone connected to the Ethernet port of the MP. The load av is around 0.5 during a call (using the GSM codec).

Here is a block diagram of the test setup. The Laptop is used for monitoring. It’s cool that I can ssh to the remote MP over the mesh network and mess with Asterisk conf files, run top etc.

MP tests for first calls over the mesh

The audio sounds great. We can’t hear any echo, thanks to the Oslec echo canceller which seems to run fine on this platform. To stress the system a bit I tried ping flooding from my laptop. This affected the audio quality but the call was still very intelligible. Like driving under a bridge with a cell phone. This is interesting – perhaps a mesh potato network will have a “soft” degradation of quality as network traffic increases. Call quality will drop a little, but the network may not fall over.

The reliability of the Mesh Potato hardware has been remarkably good so far. I left the prototypes running for about 5 days, and when I came back the mesh was still running, and picking up a phone gave me dial tone. Can’t complain about that on hardware where the solder has barely cooled!

Almost exactly 1 year since the Mesh Potato project was kicked off at the first Village Telco workshop in June 2008.

Alen from Atcom has been closely following our steps, which has encouraged Elektra and I to write a Mesh Potato How To on the Village Telco Wiki.

The first Mesh Potato

Yesterday (Monday June 1) the courier arrived with the very first Mesh Potato (MP) prototype, which had been hand assembled by the good people at Atcom. This is always an exciting and risky time in the life of any hardware project, as you don’t really know if it’s going to work. In fact, there is a hell of a lot that could go wrong. Like one solder ball under the Atheros BGA chip not soldered, or a nasty PCB or schematic design error that means another re-spin of the PCB and 1 month delay. Or smoke might come out of the Atheros chip when you first apply power. I have seen all of these on past projects…..

Fortunately Alen at Atcom had done some initial tests, like checking for the 3V3 and 1V8 power rails, and making sure the 40 MHz clock was present. So we had some encouraging early signs of life.

My initial goal was to get the boot loader to run – this would prove that most of the AR2317 circuit is OK (CPU, RAM, flash etc). I chose the boot loader image (ap61.rom) that we have been using on the DIR-300 router as this router uses the same Atheros AR2317 chip. So I connected the JTAG cable. The CPU was detected (excellent!) and the jtagspi software started flashing the boot loader. This is a slow process (2.5 hours) using my home-brew JTAG cable, so I headed out for a nice Chinese lunch and a walk with my wife! Alright, so we also went to the bakery and bought a custard filled cake. You need energy to hack.

Two hours later I returned and tentatively connected the RS232 cable and power up the MP. This is what I saw:

No board config data found!+flash_hwr_init: Unsupported flash device - id=22FLASH: driver init failed: Driver does not support deviceSorry, FLASH config exceeds available space in FIS directoryCouldn't find valid MAC address for enet0. Using default!Invalid PHY ID1 for enet0 port0.  Expected 0x0243, read 0xffff/ar2317-prj/LSDK5.0.2.46/src/redboot_cobra/ecos/packages/devs/eth/mips/ar531x/ccEthernet eth0: MAC address 00:03:7f:e0:02:bfIP: 0.0.0.0/255.255.255.0, Gateway: 0.0.0.0Default server: 0.0.0.0RedBoot(tm) bootstrap and debug environment [ROMRAM]Non-certified release, version UNKNOWN - built 00:32:22, Aug  7 2007Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc.Board: ap61RAM: 0x80000000-0x80800000, [0x80040760-0x807f1000] availableFLASH: 0x00000000 - 0x00000001, 0 blocks of 0x00000000 bytes each.RedBoot>

This is actually pretty good – the boot loader is running – with a few complaints about the 8M flash and Ethernet PHY. However a running boot loader means much of the hardware must be OK. Nice.

I have some RedBoot hacking to do (to accommodate the 8M flash and Ethernet PHY) so the next step was to get the Ethernet up. That way I can tftp new boot loader images rather than put up with 2 hour JTAG flash sessions. Actually in hind sight I should have bought a commercial JTAG cable that can flash in 5 seconds. However the Ethernet was dead (no link lights, ping didn’t work), and I didn’t like the look of those complaints about the PHY chip.

So I perused the DP83848I PHY chip data sheet. I decided to probe a few pins on the PHY chip with my oscilloscope to see if it looked alive. One pin I tested was the CLK_OUT pin 25, which should have a 25 MHz signal on it. However it was silent. Hmmmm, not good.

So I contacted Alen via IM and he confirmed that on other designs using that chip p25 definetely had a 25 MHz signal on it. So I checked the second MP01 I had been sent. No CLK_OUT signal either.

When both my MP01′s didn’t have the CLK_OUT signal I suspected it was a wiring/design error. Well, it was just a guess really. Telling these bug hunt stories in review it looks like the outcome was pre-ordained. However at the time you are never really sure. You usually go down many blind alleys, which results in frustration and lack of sleep shutting brain cells down!

Anyway clocks should usually come straight up without any software initialization. So I just checked the reset/interrupt lines (in case the chip was being held in the reset state), then I checked the various power supply lines to the chip. A-HA! One power supply line (pin 22 AVDD33) was not connected! Via IM Alen then double checked his other designs and confirmed that yes, it was a bug.

When I added a wire to supply 3V3 to pin 22 CLK_OUT came up and the network link light started working. Yayyyyyyy!

There is also a golden rule in hardware (and indeed software) development. Anything you change will have bugs. That PHY was changed from the reference design so it’s a natural source of bugs. A bug “emitter” if you like.

Then it was time for some network tests. Ping, and even tftp worked. Very cool. The default state of the PHY chip seems to be just enough to get basic network connectivity. This means I can download new RedBoot images quickly as I modify for thew new flash and PHY.

It was getting late by this stage so I decided to call it a night. So I emailed Elektra about where we were at and went to watch Top Gear which my three year old. However our story does not end there. Elektra was working hard while I was resting and sleeping. When I woke up this morning she sent me several emails detailing how to set up the RedBoot development environment. This is great – saves me a mornings work. Amazing how working 12 hours apart (Berlin and Adelaide, Australia) can be helpful!

Right. Time for some a fresh cup of coffee and some RedBoot hacking.

What FON and Meraki have in common is that they are both consumer-deployed infrastructure and both trade on the benefit of network effects.   The paper presents a useful analysis of both organisations but strains a bit in making an effective comparision between them because the models are so different.  FON is a distribution network of “Foneros” who share their Internet connections via WiFi in exchange for gaining low-cost access to other Fonero access points.  Thus FON access points can be anywhere there is an Internet connection.  Meraki on the other hand depends on nodes being connected to each other via a mesh network protocol.  You can see how it is hard then to make a head-to-head comparison.  I would be more interested in seeing a head-to-head comparison of Meraki and OpenMesh.

I still found the paper very useful, particularly in its analysis of FON.  It points out the critical failing that FON doesn’t achieve anything like ubiquity in distribution and particularly availability of access points and consequently turn out to be not so useful to people who might want to take advantage of roaming access.  The notion for driving around a city looking for a live FON access point can hardly be very appealing.  It would be more interesting if FON were able to link up with a service like IPASS so that you could leverage access where you want it most in cafes and airports (well, those are my priority spots .  However, it would probably be difficult to find a fit with IPASS’s purely commercial model.  Anyway, the take home for me about FON was the importance of ubiquity. The paper rates Meraki a little higher than FON because it achieves “local ubiquity” which in the context of Meraki is probably as good as is needed.

Middleton argues that Meraki’s model, while more successful than FON’s, still suffers from a reliability issue because the infrastructure is dependent on the community.  She highlights an incident with Meraki where a number of people turned off their routers in order to plug in Christmas trees causing the otherwise robust, redundant mesh network to fail.  It highlights the fact that a Meraki user can’t be a simple consumer, they need to be a participant in the community.  This may not be a bad thing. 

http://www.cwirp.ca/files/CWIRP_FON_Meraki_slides.pdf

However, Middleton’s final assessment is that both FON and Meraki are challenged when it comes to going to scale.  She argues that there is a need for cental coordination that is not easily achieved through the ad-hoc nature of community efforts.  This is something of a curious conclusion as the Internet itself is an ad-hoc network that succeeds very well.  It may be that FON and Meraki are doomed precisely because they attempt to have centralised control over their networks.  It may be that a federated model that links heterogenous community WiFi initiatives may be more successful; a situation in which an Île Sans Fil user in Montreal might have privileges on a FON or a Meraki network.

So what does this mean for the Village Telco?  I think the Village Telco is likely to address the key sustainability issues that Middleton raises about these networks.  I like the general criteria (pictured at left) that she comes up with to assess the networks.  I think they will be useful benchmarks to keep in mind as the Village Telco evolves.