Stability

One simple stability test is uptime. I have left both my MP prototypes running for up to 5 days at a time with good results – Linux and Wifi stayed up, and we still had dial tone from the FXS port. This simple test says a lot: e.g no memory leaks, CPU instability, drastic over temperature problems.

Another test I like to do to all my telephony devices is hammer them with phone calls for 24 hours. Here is the test set-up:

Stability Test

An x86 Asterisk box acts as the call generator. It starts a new call to the MP. When the MP receives the call it starts the FXS port ringing. Another Asterisk box (an IP04 with an FXO port) answers the call, plays a prompt for a few seconds, then hangs up. The entire sequence is then repeated – make thousands of calls over a 24 hour period. The SIP call is placed over Mesh Wifi to make sure all parts of the MP system are being exercised.

This test gives the MP quite a hard time as much of the call processing load is involved in set up and tear down of calls, and unless you are my teenage daughter you can’t really make 3500 calls in 24 hours.

When I started the tests it threw up problems nearly immediately – Asterisk was seg faulting with no feedback as to why it was crashing. This led to a week of fun and games where the problem was eventually traced to thread problems in the Asterisk channel driver I had written (chan_mp). This channel driver has been a painful experience for me (even though I have written a few before), and has taken perhaps 3 weeks of part time work to write and debug. Lots of fun with threads and null pointers. Even now I still don’t quite understand all the issues around Asterisk channel drivers. I would appreciate a code review of the driver if there are any Asterisk channel driver experts out there.

Anyway I finally tracked down the source of the problem and modified the channel driver. It passed the 24 hour stability tests and made 3500 calls.

Mesh Load Test

This is a repeat of the load test from Phase 1 of the Mesh Potato Project, but this time on the actual MP hardware with the FXS driver running:

Mesh Load Test

A big difference between the Village Telco and Cell Phone network is no base stations (Cell Phone Towers). Instead, the calls are routed via the client nodes. It’s a peer-peer network rather than a client-server architecture.

The idea of this test is to make sure that a given mesh potato node can relay 15 phone calls for other people while simultaneously making a phone call of it’s own. This scenario places significant CPU load on the router due to the number of Wifi packets that must be processed at the same time as DSP intensive code like echo cancellation and speech compression.

The set up details of this test are on the wiki and have been blogged about earlier. Speech quality was fine, and loadav about 0.71, similar to the Phase 1 results. Occasionally I could hear some packet loss but overall the call was fine. A good result considering all that processing (Linux, Wifi, mesh, Asterisk, echo cancelling, GSM speech compression, FXS driver) on that little router chip!

One interesting fact is that even with a total of 16 calls the bit rate is only around 500 kbit/s, so the bandwidth of the mesh is quite lightly loaded. However speech packets are rather short, so raising the number of phone calls would likely run into CPU load (rather than Wifi bandwidth) issues due to the per-packet processing load.

Wifi Range Test

Wifi is mainly the domain of Elektra and Jeff on this project but I was keen to get some rough idea of how well the radio was working. I had made a few calls around the house OK but noticed that the received signal level was quite low. A bit of thought led us to a static protection diode we had placed across the antenna circuit. Although a good idea for the RX side it was conducting when transmit signal was present. After removing this diode the TX level jumped right up to roughly comparable levels to a DIR-300 router that uses the same Atheros AR2317 SoC.

However using radiated signal levels it is hard to get consistent results, they bounce around all over the place. Jeff suggested connecting a combination of DIR-300 and Mesh Potatoes in A-B tests via cables and calibrated attenuators to obtain consistent measurements.

Over a few days I made several attempts at range tests. However where I live is quite flat so it’s difficult to get a good Line-Of-Sight (LOS). I did manage to make phone calls over about 250m which is pretty much what we need for a Village Telco Network.

Elektra and I feel the radio performance could probably be improved with some adjustment of the RF circuit components and possibly PCB layout. Elektra has access to a RF test lab later this week so we will get some more concrete information of Wifi performance.

One area we really need more information on is radio calibration data and the Hardware Design Guide document for the AR2317, however we have been unable to obtain this important information from Atheros.

More blogging to come from Elektra on Wifi radio and power supply performance.

On Friday I set about testing Wifi on the two Mesh Potato prototypes. The N-type antenna connectors attach to the MP PCB via a pigtail with a tiny Hi-rose style connector. This looks a bit fragile so I decided to mount the two MPs in some weatherproof boxes so I had a firm mounting for the antenna pig tail. I chose one metal and one plastic, to see if the case material makes any difference to the Wifi performance. They wont be completely weatherproof (I’ll just run the phone power, and Ethernet cables through holes), but enough to take outside for some tests and withstand some handling.

I mounted both the PCB and the antenna connector in the lid, so the whole assembly can be removed without putting any strain on that tiny PCB antenna connector. Only problem is I can’t see the LEDs – will need to make a little window on the bottom of the box.

Mesh Potatoes mounted on box lids

Having secured the antenna connector I brought Wifi up. I thought I would take the small, sensible step of trying to associate with my home Wifi network first. It was all looking good until my MP seemed to slow down then stop responding entirely. Then my laptop (also connected via Wifi) started to slow down and lost Internet connectivity. Oops, routing loop between Ethernet and Wifi on the Mesh Potato! Didn’t know you could bring down a whole network that way.

As Wifi was coming up automatically on boot it was hard to stop, so I ended up re-flashing the router and going straight to Batman. This came up straight away and I had two MPs and a Nanostation happily meshing with each other.

The blinking of the two Wifi lights as batman sends out messages every second is mesmerising

Voice Over B.A.T.M.A.N.

After a bit of messing around with Asterisk conf files I made a MP-MP phone call over Batman and the Mesh Network. The audio in one direction was fine but a little distorted in the other direction. After a break of a few days I took a look at this bug. The problem ended up being the Asterisk channel driver, it was accidentally sending both RTP audio and ring tone to the FXS driver during a call, i.e. ring tone wasn’t being switched off when the call was answered. This was overloading the FXS driver FIFO and causing distorted audio. Curiously, it worked OK until I tried a call over Wifi.

Anyway I have fixed this bug now and tested a few different codec configurations over the mesh including ulaw, gsm with 20ms frames, and GSM with 80ms frames. We can also make calls between a MP and a SIP phone connected to the Ethernet port of the MP. The load av is around 0.5 during a call (using the GSM codec).

Here is a block diagram of the test setup. The Laptop is used for monitoring. It’s cool that I can ssh to the remote MP over the mesh network and mess with Asterisk conf files, run top etc.

MP tests for first calls over the mesh

The audio sounds great. We can’t hear any echo, thanks to the Oslec echo canceller which seems to run fine on this platform. To stress the system a bit I tried ping flooding from my laptop. This affected the audio quality but the call was still very intelligible. Like driving under a bridge with a cell phone. This is interesting – perhaps a mesh potato network will have a “soft” degradation of quality as network traffic increases. Call quality will drop a little, but the network may not fall over.

The reliability of the Mesh Potato hardware has been remarkably good so far. I left the prototypes running for about 5 days, and when I came back the mesh was still running, and picking up a phone gave me dial tone. Can’t complain about that on hardware where the solder has barely cooled!

Almost exactly 1 year since the Mesh Potato project was kicked off at the first Village Telco workshop in June 2008.

Alen from Atcom has been closely following our steps, which has encouraged Elektra and I to write a Mesh Potato How To on the Village Telco Wiki.