Stability

One simple stability test is uptime. I have left both my MP prototypes running for up to 5 days at a time with good results – Linux and Wifi stayed up, and we still had dial tone from the FXS port. This simple test says a lot: e.g no memory leaks, CPU instability, drastic over temperature problems.

Another test I like to do to all my telephony devices is hammer them with phone calls for 24 hours. Here is the test set-up:

Stability Test

An x86 Asterisk box acts as the call generator. It starts a new call to the MP. When the MP receives the call it starts the FXS port ringing. Another Asterisk box (an IP04 with an FXO port) answers the call, plays a prompt for a few seconds, then hangs up. The entire sequence is then repeated – make thousands of calls over a 24 hour period. The SIP call is placed over Mesh Wifi to make sure all parts of the MP system are being exercised.

This test gives the MP quite a hard time as much of the call processing load is involved in set up and tear down of calls, and unless you are my teenage daughter you can’t really make 3500 calls in 24 hours.

When I started the tests it threw up problems nearly immediately – Asterisk was seg faulting with no feedback as to why it was crashing. This led to a week of fun and games where the problem was eventually traced to thread problems in the Asterisk channel driver I had written (chan_mp). This channel driver has been a painful experience for me (even though I have written a few before), and has taken perhaps 3 weeks of part time work to write and debug. Lots of fun with threads and null pointers. Even now I still don’t quite understand all the issues around Asterisk channel drivers. I would appreciate a code review of the driver if there are any Asterisk channel driver experts out there.

Anyway I finally tracked down the source of the problem and modified the channel driver. It passed the 24 hour stability tests and made 3500 calls.

Mesh Load Test

This is a repeat of the load test from Phase 1 of the Mesh Potato Project, but this time on the actual MP hardware with the FXS driver running:

Mesh Load Test

A big difference between the Village Telco and Cell Phone network is no base stations (Cell Phone Towers). Instead, the calls are routed via the client nodes. It’s a peer-peer network rather than a client-server architecture.

The idea of this test is to make sure that a given mesh potato node can relay 15 phone calls for other people while simultaneously making a phone call of it’s own. This scenario places significant CPU load on the router due to the number of Wifi packets that must be processed at the same time as DSP intensive code like echo cancellation and speech compression.

The set up details of this test are on the wiki and have been blogged about earlier. Speech quality was fine, and loadav about 0.71, similar to the Phase 1 results. Occasionally I could hear some packet loss but overall the call was fine. A good result considering all that processing (Linux, Wifi, mesh, Asterisk, echo cancelling, GSM speech compression, FXS driver) on that little router chip!

One interesting fact is that even with a total of 16 calls the bit rate is only around 500 kbit/s, so the bandwidth of the mesh is quite lightly loaded. However speech packets are rather short, so raising the number of phone calls would likely run into CPU load (rather than Wifi bandwidth) issues due to the per-packet processing load.

Wifi Range Test

Wifi is mainly the domain of Elektra and Jeff on this project but I was keen to get some rough idea of how well the radio was working. I had made a few calls around the house OK but noticed that the received signal level was quite low. A bit of thought led us to a static protection diode we had placed across the antenna circuit. Although a good idea for the RX side it was conducting when transmit signal was present. After removing this diode the TX level jumped right up to roughly comparable levels to a DIR-300 router that uses the same Atheros AR2317 SoC.

However using radiated signal levels it is hard to get consistent results, they bounce around all over the place. Jeff suggested connecting a combination of DIR-300 and Mesh Potatoes in A-B tests via cables and calibrated attenuators to obtain consistent measurements.

Over a few days I made several attempts at range tests. However where I live is quite flat so it’s difficult to get a good Line-Of-Sight (LOS). I did manage to make phone calls over about 250m which is pretty much what we need for a Village Telco Network.

Elektra and I feel the radio performance could probably be improved with some adjustment of the RF circuit components and possibly PCB layout. Elektra has access to a RF test lab later this week so we will get some more concrete information of Wifi performance.

One area we really need more information on is radio calibration data and the Hardware Design Guide document for the AR2317, however we have been unable to obtain this important information from Atheros.

More blogging to come from Elektra on Wifi radio and power supply performance.

So I carefully retraced my steps and tried to re-flash the MP from JTAG. Still dead. Huh? There followed several days of futzing about, looking at the jtagspi code, trying this, trying that, and getting more and more confused and not a little upset! I just didn’t make sense – if ap61.rom worked the first time why not now?

Mesh Potatoes Maternity Ward

Then I noticed something funny in the ap61.rom boot log from the DIR-300:

a) RedBoot(tm) bootstrap and debug environment [ROMRAM]
production release, version "2.1.3" - built 18:43:19, Sep 20 2007

compared to the first, miraculous boot of the MP01 on Monday:

b) RedBoot(tm) bootstrap and debug environment [ROMRAM]
Non-certified release, version UNKNOWN - built 00:32:22, Aug 7 2007

Different dates!

So this idea popped into my head:

1/ What if Alen (from Atcom) had flashed a boot loader before sending me the MP01?

2/ From my records on Monday, and examining the jtagspi code, I know that my write address for my initial MP01 flash on Monday was wrong: 0x1fc00000

The jtagspi software discards the top 8 bits, leaving 0xc00000. Now on a DIR-300 with 4M the top 2 bits get discarded leaving 0×00000, or the first three blocks of flash. As you would expect.

However when I used the same 0x1fc00000 address on a MP01 with 8M flash, I you get 400000, or half way through the 8M flash. So on Monday I think I flashed ap61.rom half way through the 8M flash.

When I booted, it ran code from the first 3 sectors, which was whatever was in the SPI flash before I flashed it!

So I checked with Alen, and yes he had programmed the SPI flash chips before soldering! So I had been following a blind alley all week – the idea that ap61.rom boots OK on the MP01!

4/ I continued to chat with Alen via IM. Guess what? The MP01s have 8M flash chips fitted – as that is what the reference design had! That also explains why ap61.rom won’t boot – it was compiled for 16M! The mp01.bin Alen used was set up for a 2M flash/8M ram router. You can even see it in the boot log RAM reports:

MP01:

Board: ap61
RAM: 0x80000000-0x80800000, [0x80040760-0x807f1000] available
FLASH: 0x00000000 - 0x00000001, 0 blocks of 0x00000000 bytes each.
RedBoot>

DIR-300:

Board: DLINK DIR-300
RAM: 0x80000000-0x81000000, [0x80040580-0x80fe1000] available
FLASH: 0xbfc00000 - 0xbfff0000, 64 blocks of 0x00010000 bytes each.


So with my 3 yrs old in tow I ran down to my soldering tools guy and bought some “chip quick”, a special low-melting-point solder than can be used to remove surface mount chips with just an ordinary soldering iron. Here is a 30 second video of how Chip Quick works. It’s actually good fun to use, and the low melting point is very kind to the PCB. It also saved me a couple of days finding some one with proper SM rework tools (it’s Friday afternoon before a long weekend here). No way I wanted to wait until Tuesday to fix this bug!

Now I am pretty geeky, but even I don’t have 16Mbyte SDRAM chips just laying about. So it was time for some open-heart surgery. I removed a 16M flash chip from the “donor” DIR-300, and soldered it onto MP001-001 (the first prototype). I powered up and no power LED – there was a short circuit between 3V3 and GND! Bloody Hell – some days you just can’t win! I looked for the elusive short for one hour then gave up on MP001-001 for now. Instead I turned my attentions to the second prototype, MP01-002. I carefully removed the same precious 16M chip from MP01-001 and soldered it onto MP01-002. No shorts this time. I applied 12V and the boot loader comes alive. YAaaaaayyyy!!

As an encore I tried to get Linux to boot….however I had displeased the Gods of embedded systems this week and it was not to be – the Linux image we use for the DIR-300 is stopping part way. For some reason available memory is reported as 32Mbyte rather than the correct 16MByte. However I am going to work on that problem later – time for a few days break!

The first Mesh Potato

Yesterday (Monday June 1) the courier arrived with the very first Mesh Potato (MP) prototype, which had been hand assembled by the good people at Atcom. This is always an exciting and risky time in the life of any hardware project, as you don’t really know if it’s going to work. In fact, there is a hell of a lot that could go wrong. Like one solder ball under the Atheros BGA chip not soldered, or a nasty PCB or schematic design error that means another re-spin of the PCB and 1 month delay. Or smoke might come out of the Atheros chip when you first apply power. I have seen all of these on past projects…..

Fortunately Alen at Atcom had done some initial tests, like checking for the 3V3 and 1V8 power rails, and making sure the 40 MHz clock was present. So we had some encouraging early signs of life.

My initial goal was to get the boot loader to run – this would prove that most of the AR2317 circuit is OK (CPU, RAM, flash etc). I chose the boot loader image (ap61.rom) that we have been using on the DIR-300 router as this router uses the same Atheros AR2317 chip. So I connected the JTAG cable. The CPU was detected (excellent!) and the jtagspi software started flashing the boot loader. This is a slow process (2.5 hours) using my home-brew JTAG cable, so I headed out for a nice Chinese lunch and a walk with my wife! Alright, so we also went to the bakery and bought a custard filled cake. You need energy to hack.

Two hours later I returned and tentatively connected the RS232 cable and power up the MP. This is what I saw:

No board config data found!+flash_hwr_init: Unsupported flash device - id=22FLASH: driver init failed: Driver does not support deviceSorry, FLASH config exceeds available space in FIS directoryCouldn't find valid MAC address for enet0. Using default!Invalid PHY ID1 for enet0 port0.  Expected 0x0243, read 0xffff/ar2317-prj/LSDK5.0.2.46/src/redboot_cobra/ecos/packages/devs/eth/mips/ar531x/ccEthernet eth0: MAC address 00:03:7f:e0:02:bfIP: 0.0.0.0/255.255.255.0, Gateway: 0.0.0.0Default server: 0.0.0.0RedBoot(tm) bootstrap and debug environment [ROMRAM]Non-certified release, version UNKNOWN - built 00:32:22, Aug  7 2007Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc.Board: ap61RAM: 0x80000000-0x80800000, [0x80040760-0x807f1000] availableFLASH: 0x00000000 - 0x00000001, 0 blocks of 0x00000000 bytes each.RedBoot>

This is actually pretty good – the boot loader is running – with a few complaints about the 8M flash and Ethernet PHY. However a running boot loader means much of the hardware must be OK. Nice.

I have some RedBoot hacking to do (to accommodate the 8M flash and Ethernet PHY) so the next step was to get the Ethernet up. That way I can tftp new boot loader images rather than put up with 2 hour JTAG flash sessions. Actually in hind sight I should have bought a commercial JTAG cable that can flash in 5 seconds. However the Ethernet was dead (no link lights, ping didn’t work), and I didn’t like the look of those complaints about the PHY chip.

So I perused the DP83848I PHY chip data sheet. I decided to probe a few pins on the PHY chip with my oscilloscope to see if it looked alive. One pin I tested was the CLK_OUT pin 25, which should have a 25 MHz signal on it. However it was silent. Hmmmm, not good.

So I contacted Alen via IM and he confirmed that on other designs using that chip p25 definetely had a 25 MHz signal on it. So I checked the second MP01 I had been sent. No CLK_OUT signal either.

When both my MP01′s didn’t have the CLK_OUT signal I suspected it was a wiring/design error. Well, it was just a guess really. Telling these bug hunt stories in review it looks like the outcome was pre-ordained. However at the time you are never really sure. You usually go down many blind alleys, which results in frustration and lack of sleep shutting brain cells down!

Anyway clocks should usually come straight up without any software initialization. So I just checked the reset/interrupt lines (in case the chip was being held in the reset state), then I checked the various power supply lines to the chip. A-HA! One power supply line (pin 22 AVDD33) was not connected! Via IM Alen then double checked his other designs and confirmed that yes, it was a bug.

When I added a wire to supply 3V3 to pin 22 CLK_OUT came up and the network link light started working. Yayyyyyyy!

There is also a golden rule in hardware (and indeed software) development. Anything you change will have bugs. That PHY was changed from the reference design so it’s a natural source of bugs. A bug “emitter” if you like.

Then it was time for some network tests. Ping, and even tftp worked. Very cool. The default state of the PHY chip seems to be just enough to get basic network connectivity. This means I can download new RedBoot images quickly as I modify for thew new flash and PHY.

It was getting late by this stage so I decided to call it a night. So I emailed Elektra about where we were at and went to watch Top Gear which my three year old. However our story does not end there. Elektra was working hard while I was resting and sleeping. When I woke up this morning she sent me several emails detailing how to set up the RedBoot development environment. This is great – saves me a mornings work. Amazing how working 12 hours apart (Berlin and Adelaide, Australia) can be helpful!

Right. Time for some a fresh cup of coffee and some RedBoot hacking.

As part of the proof-of-concept work around the Mesh Potato, David and Elektra have more or less settled on the Atheros AR2317 chip as a likely candidate for the Mesh Potato.  This is partly because it is a very affordable chip yet still has all the features needed for the Mesh Potato but also because it is a chip that Atcom have indicate they have ready access to.

In order to confirm that the AR2317 would perform adequately, Elektra purchased a D-Link DIR-300 which is based on the AR2317.  Elektra points out that the DIR-300:

is less then half the price of a Linksys WRT54GL and about a third the price of a Ubiquiti NS-2. It features 4MB flash and 16MB RAM, 4 port switch, 1 WAN port, a 180MHz Mips (Big Endian) CPU, Redboot Open-Source bootloader, a switched mode onboard DC/DC converter, one R-SMA antenna socket, on-board serial port and JTAG port. The device is much smaller than the Linksys WRT54GL, so outdoor boxes can be much cheaper and easier to mount than outdoor boxes for the Linksys.

As an added bonus, Elektra has posted instructions on how to set up OpenWRT a DIR-300 on the Village Telco wiki.