The Mesh Potato is a mix of many different technologies: embedded Linux, telephony hardware, VOIP and Wifi. The Radio Frequency (RF) side of the Mesh Potato is a loose end we need to tie up. Unfortunately, data on the RF side is nearly impossible to obtain. To get the full support package from Atheros costs USD$100k. That is beyond our budget and sits uncomfortably with the open source nature of the Village Telco project.

Each AR2317 chip is a little different and consequently each chips requires individual RF calibration on the production line. This calibration ensures each AR2317 product has uniform transmit power and meets certain performance criteria. The calibration data is stored at the end of the SPI flash chip and loaded by the WiFi driver at boot time. Calibration is performed on the production line using a bunch of expensive RF test equipment connected via GPIB to a host computer running special software.

The calibration procedure and most of the data relating to the AR2317 RF section is part of the Atheros “secret sauce”.

On our prototype Mesh Potatoes we were lucky enough to have the WiFi fire up first time. We just copied the calibration data from another AR2317 based product as a starting point. The Wifi worked, but without calibration Wifi performance is likely to be poor. As we had no RF test equipment our visibility was limited – it was hard to even measure the RF performance.

Elektra managed to hook a V1.0 MP01 up to a borrowed spectrum analyser, thanks to kind guys at the CSIR in South Africa. A spectrum analyser graphs the power at each frequency. Here is what the spectrum of a calibrated DIR-300 looks like:

DIR-300 1Mbit Channel 11 spectrum

And here is the output from our uncalibrated V1.0 Mesh Potato:

MP01 1Mbit Channel 11 spectrum

So after the 2nd Village Telco workshop we decided to tackle Calibration. Fortunately, our friends at Atcom found some friends in Shenzhen who have the production line equipment for AR2317 calibration. Here in Adelaide I bought some basic test equipment – an old Tek 492 spectrum analyser from e-bay and a frequency counter. The Tek 492 is an analogue spectrum analyser (with some digital storage ability) from the early 1980′s. In their day they cost $30,000 but are available 2nd hand for around $2,000. Microwave hacking on a budget! The following photo is courtesy of Ben (see below):

In early October we made our first attempt at calibration on one the V1.1 Betas. The automatic test equipment in Shenzhen “failed” us. There were several bugs including low output power (10dB down) and a large frequency offset (60ppm instead of 20ppm). Another concern was that several Mesh Potatoes tested gave different results. Great. Just what we need when we are trying to get 100 Betas out the door for eager developers.

Thus began a two week frenzy of RF debugging, with Mesh Potatoes being couriered back and forth between Shenzhen and Adelaide and much soldering of tiny 0402 size parts under the microscope. These parts appear the size of a small crumb to the naked eye (about 1mm by 0.4mm), but surprisingly you can hand solder them under a microscope with a little patience.

None of the core team are experienced in Wifi or 2.4GHz RF, and we are working without 95% of the data and test equipment we need for proper RF development. So we had a few challenges.

After about two weeks of hard work we made some progress. I managed to bring 3 out of 4 betas up to our target power of 17dBm (the 4th I blew up accidentally). We discovered some areas where the PCB layout could be improved to get even more power (perhaps 20dBm) out of the MP. The system clock was a few kHz off 40.0 MHz which when multiplied up to 2.4 GHz caused the 60 ppm frequency offset. This was actually the largest problem – it caused packet errors at the low data rates which messed up long range performance. Once this was fixed the packet error rate performance started to look as good as the reference DIR-300 unit we were testing against.

The variable power output of different MPs had a simple reason – some of the tiny 0402 parts were loaded in the wrong place. This is very easy to do as the parts have no writing on them. I only spotted this when I noticed two parts with the same value looked slightly different in colour. Usually all parts from the same reel look identical.

With our new-found experience, Atcom decided to make another revision of the PCB (V1.2) to tighten up the RF side. That should be ready for testing in November. If calibration checks out on V1.2 we will then kick off a Beta run.

Given our lack of RF experience, lack of AR2317 data, lack of support from the chip vendor, and very basic test equipment I feel pretty happy with our progress in RF performance over those two hard weeks. The %$%^ Asterisk channel driver for the Mesh Potato took me three weeks to get stable and I am meant to know something about Asterisk driver development!

As well at the core team of Elektra, myself, the Atcom team (Edwin, Alen, Mr. Lee, Peter), their Shenzhen friends and of course Steve, we had some help from Jeff (our RF consultant) and two other people who were especially kind:

I found Ben Johnson while looking for information on the Tek 492. Ben had posted a page on how he repaired his Tek 492. I emailed Ben to ask if he thought it was suitable for Wifi. Ben was kind enough to actually test his Tek 492 on some Wifi signals and email me the results! This gave me the confidence to bid for a used 492 on ebay here in Australia.

I met Dieter through an article I published on Low cost Electric Cars in Renew magazine. He just happened to email me on a day I was messing with the Tek 492 and I found out he was a microwave engineer working in Melbourne. Throughout the RF debugging process Dieter emailed me virtually every day and gave much needed advice and moral support. He even sent over some semi-rigid RF cable with an N-connector to help with the testing.

The Internet is amazing place. I am constantly bowled over by the kindness of people – especially when you are working on open projects.

Links

[1] Measuring Wifi Transmit Power – An in-depth look on how I used the Tek 492 to measure Wifi transmit power.

So I carefully retraced my steps and tried to re-flash the MP from JTAG. Still dead. Huh? There followed several days of futzing about, looking at the jtagspi code, trying this, trying that, and getting more and more confused and not a little upset! I just didn’t make sense – if ap61.rom worked the first time why not now?

Mesh Potatoes Maternity Ward

Then I noticed something funny in the ap61.rom boot log from the DIR-300:

a) RedBoot(tm) bootstrap and debug environment [ROMRAM]
production release, version "2.1.3" - built 18:43:19, Sep 20 2007

compared to the first, miraculous boot of the MP01 on Monday:

b) RedBoot(tm) bootstrap and debug environment [ROMRAM]
Non-certified release, version UNKNOWN - built 00:32:22, Aug 7 2007

Different dates!

So this idea popped into my head:

1/ What if Alen (from Atcom) had flashed a boot loader before sending me the MP01?

2/ From my records on Monday, and examining the jtagspi code, I know that my write address for my initial MP01 flash on Monday was wrong: 0x1fc00000

The jtagspi software discards the top 8 bits, leaving 0xc00000. Now on a DIR-300 with 4M the top 2 bits get discarded leaving 0×00000, or the first three blocks of flash. As you would expect.

However when I used the same 0x1fc00000 address on a MP01 with 8M flash, I you get 400000, or half way through the 8M flash. So on Monday I think I flashed ap61.rom half way through the 8M flash.

When I booted, it ran code from the first 3 sectors, which was whatever was in the SPI flash before I flashed it!

So I checked with Alen, and yes he had programmed the SPI flash chips before soldering! So I had been following a blind alley all week – the idea that ap61.rom boots OK on the MP01!

4/ I continued to chat with Alen via IM. Guess what? The MP01s have 8M flash chips fitted – as that is what the reference design had! That also explains why ap61.rom won’t boot – it was compiled for 16M! The mp01.bin Alen used was set up for a 2M flash/8M ram router. You can even see it in the boot log RAM reports:

MP01:

Board: ap61
RAM: 0x80000000-0x80800000, [0x80040760-0x807f1000] available
FLASH: 0x00000000 - 0x00000001, 0 blocks of 0x00000000 bytes each.
RedBoot>

DIR-300:

Board: DLINK DIR-300
RAM: 0x80000000-0x81000000, [0x80040580-0x80fe1000] available
FLASH: 0xbfc00000 - 0xbfff0000, 64 blocks of 0x00010000 bytes each.


So with my 3 yrs old in tow I ran down to my soldering tools guy and bought some “chip quick”, a special low-melting-point solder than can be used to remove surface mount chips with just an ordinary soldering iron. Here is a 30 second video of how Chip Quick works. It’s actually good fun to use, and the low melting point is very kind to the PCB. It also saved me a couple of days finding some one with proper SM rework tools (it’s Friday afternoon before a long weekend here). No way I wanted to wait until Tuesday to fix this bug!

Now I am pretty geeky, but even I don’t have 16Mbyte SDRAM chips just laying about. So it was time for some open-heart surgery. I removed a 16M flash chip from the “donor” DIR-300, and soldered it onto MP001-001 (the first prototype). I powered up and no power LED – there was a short circuit between 3V3 and GND! Bloody Hell – some days you just can’t win! I looked for the elusive short for one hour then gave up on MP001-001 for now. Instead I turned my attentions to the second prototype, MP01-002. I carefully removed the same precious 16M chip from MP01-001 and soldered it onto MP01-002. No shorts this time. I applied 12V and the boot loader comes alive. YAaaaaayyyy!!

As an encore I tried to get Linux to boot….however I had displeased the Gods of embedded systems this week and it was not to be – the Linux image we use for the DIR-300 is stopping part way. For some reason available memory is reported as 32Mbyte rather than the correct 16MByte. However I am going to work on that problem later – time for a few days break!

The first Mesh Potato

Yesterday (Monday June 1) the courier arrived with the very first Mesh Potato (MP) prototype, which had been hand assembled by the good people at Atcom. This is always an exciting and risky time in the life of any hardware project, as you don’t really know if it’s going to work. In fact, there is a hell of a lot that could go wrong. Like one solder ball under the Atheros BGA chip not soldered, or a nasty PCB or schematic design error that means another re-spin of the PCB and 1 month delay. Or smoke might come out of the Atheros chip when you first apply power. I have seen all of these on past projects…..

Fortunately Alen at Atcom had done some initial tests, like checking for the 3V3 and 1V8 power rails, and making sure the 40 MHz clock was present. So we had some encouraging early signs of life.

My initial goal was to get the boot loader to run – this would prove that most of the AR2317 circuit is OK (CPU, RAM, flash etc). I chose the boot loader image (ap61.rom) that we have been using on the DIR-300 router as this router uses the same Atheros AR2317 chip. So I connected the JTAG cable. The CPU was detected (excellent!) and the jtagspi software started flashing the boot loader. This is a slow process (2.5 hours) using my home-brew JTAG cable, so I headed out for a nice Chinese lunch and a walk with my wife! Alright, so we also went to the bakery and bought a custard filled cake. You need energy to hack.

Two hours later I returned and tentatively connected the RS232 cable and power up the MP. This is what I saw:

No board config data found!+flash_hwr_init: Unsupported flash device - id=22FLASH: driver init failed: Driver does not support deviceSorry, FLASH config exceeds available space in FIS directoryCouldn't find valid MAC address for enet0. Using default!Invalid PHY ID1 for enet0 port0.  Expected 0x0243, read 0xffff/ar2317-prj/LSDK5.0.2.46/src/redboot_cobra/ecos/packages/devs/eth/mips/ar531x/ccEthernet eth0: MAC address 00:03:7f:e0:02:bfIP: 0.0.0.0/255.255.255.0, Gateway: 0.0.0.0Default server: 0.0.0.0RedBoot(tm) bootstrap and debug environment [ROMRAM]Non-certified release, version UNKNOWN - built 00:32:22, Aug  7 2007Copyright (C) 2000, 2001, 2002, 2003, 2004 Red Hat, Inc.Board: ap61RAM: 0x80000000-0x80800000, [0x80040760-0x807f1000] availableFLASH: 0x00000000 - 0x00000001, 0 blocks of 0x00000000 bytes each.RedBoot>

This is actually pretty good – the boot loader is running – with a few complaints about the 8M flash and Ethernet PHY. However a running boot loader means much of the hardware must be OK. Nice.

I have some RedBoot hacking to do (to accommodate the 8M flash and Ethernet PHY) so the next step was to get the Ethernet up. That way I can tftp new boot loader images rather than put up with 2 hour JTAG flash sessions. Actually in hind sight I should have bought a commercial JTAG cable that can flash in 5 seconds. However the Ethernet was dead (no link lights, ping didn’t work), and I didn’t like the look of those complaints about the PHY chip.

So I perused the DP83848I PHY chip data sheet. I decided to probe a few pins on the PHY chip with my oscilloscope to see if it looked alive. One pin I tested was the CLK_OUT pin 25, which should have a 25 MHz signal on it. However it was silent. Hmmmm, not good.

So I contacted Alen via IM and he confirmed that on other designs using that chip p25 definetely had a 25 MHz signal on it. So I checked the second MP01 I had been sent. No CLK_OUT signal either.

When both my MP01′s didn’t have the CLK_OUT signal I suspected it was a wiring/design error. Well, it was just a guess really. Telling these bug hunt stories in review it looks like the outcome was pre-ordained. However at the time you are never really sure. You usually go down many blind alleys, which results in frustration and lack of sleep shutting brain cells down!

Anyway clocks should usually come straight up without any software initialization. So I just checked the reset/interrupt lines (in case the chip was being held in the reset state), then I checked the various power supply lines to the chip. A-HA! One power supply line (pin 22 AVDD33) was not connected! Via IM Alen then double checked his other designs and confirmed that yes, it was a bug.

When I added a wire to supply 3V3 to pin 22 CLK_OUT came up and the network link light started working. Yayyyyyyy!

There is also a golden rule in hardware (and indeed software) development. Anything you change will have bugs. That PHY was changed from the reference design so it’s a natural source of bugs. A bug “emitter” if you like.

Then it was time for some network tests. Ping, and even tftp worked. Very cool. The default state of the PHY chip seems to be just enough to get basic network connectivity. This means I can download new RedBoot images quickly as I modify for thew new flash and PHY.

It was getting late by this stage so I decided to call it a night. So I emailed Elektra about where we were at and went to watch Top Gear which my three year old. However our story does not end there. Elektra was working hard while I was resting and sleeping. When I woke up this morning she sent me several emails detailing how to set up the RedBoot development environment. This is great – saves me a mornings work. Amazing how working 12 hours apart (Berlin and Adelaide, Australia) can be helpful!

Right. Time for some a fresh cup of coffee and some RedBoot hacking.

As part of the proof-of-concept work around the Mesh Potato, David and Elektra have more or less settled on the Atheros AR2317 chip as a likely candidate for the Mesh Potato.  This is partly because it is a very affordable chip yet still has all the features needed for the Mesh Potato but also because it is a chip that Atcom have indicate they have ready access to.

In order to confirm that the AR2317 would perform adequately, Elektra purchased a D-Link DIR-300 which is based on the AR2317.  Elektra points out that the DIR-300:

is less then half the price of a Linksys WRT54GL and about a third the price of a Ubiquiti NS-2. It features 4MB flash and 16MB RAM, 4 port switch, 1 WAN port, a 180MHz Mips (Big Endian) CPU, Redboot Open-Source bootloader, a switched mode onboard DC/DC converter, one R-SMA antenna socket, on-board serial port and JTAG port. The device is much smaller than the Linksys WRT54GL, so outdoor boxes can be much cheaper and easier to mount than outdoor boxes for the Linksys.

As an added bonus, Elektra has posted instructions on how to set up OpenWRT a DIR-300 on the Village Telco wiki.