Mesh-Potato smoke testing with 230 Volt AC from Elektra Berlin on Vimeo.

The list of tests included a reverse DC voltage test from a unfused 36 Volt source (consisting of three powerful 12 V lead acid batteries in series), excessive DC voltage tests and finally a really scary test involving 230 Volt AC (330 Volt peak) from mains. Please don’t try this at home. Even if the Mesh-Potato survives, there is a 50% chance that you might have mains potential on ground of all components connected to the MP. Also don’t try this with your alpha or beta series Mesh-Potatos, because the previous over-voltage protection circuit is not up to that challenge.

Technically inexperienced people can easily make mistakes, so our idea was to design the Mesh-Potato as robust as possible. In 2005 I was helping to set up a large scale WLAN network in the Sylhet area in Bangladesh. The network consisted of high towers (up to 100 feet tall) and strong directional antennas, interconnecting towns and a school with wireless long shots (up to 32km). One of the trainees damaged a important wireless relay on a tower by taking the open ends of a 12 Volt cable and plugging it straight into the mains socket. Of course the equipment (a Mesh-Cube from 4G Systems) subsequently looked like a lightning strike had hit it, which was actually what I supposed first. However there had been no thunderstorm in the night before. It took me a while to find the reason. The trainee either hadn’t realized what he had done, or he didn’t want to admit it. He watched me trying to find the problem without saying anything. It is common practice in Bangladesh to plug cables into sockets without plugs. The quality of sockets and plugs is miserable, so loose contacts are the rule, not the exception. Now a important relay was down and it was hard to get a replacement. The problem wasn’t so much the financial loss. Shipping and particularly customs can take weeks in Bangladesh. So during the first Villagetelco workshop I suggested to design the MP as robust as possible.

Last year a bold and creative group of geeks conceived of an Open Hardware / Open Software device that would both simplify and lower the cost of deploying wireless mesh voice networks.  We called it the Mesh Potato which is a mashup of Mesh + POTS + ATA.  Well , it works in Spanish at least.  A year later we had working Mesh Potato prototypes to hack at the workshop. And that is what the workshop focused on, getting the Mesh Potato into production and helping it go to scale.  The rest of this post is a summary of the key elements of the workshop.

Technical Design Trade-offs

One of the first conversations we had at the workshop was to review the technical design decisions to date and discuss the pros and cons of each decision.  From the beginning we have tried to balance a desire to make the Mesh Potato as hard-to-break (or brick for that matter) as possible with a need to keep it affordable.

Antenna Type

Based on her experience in Bangladesh, Elektra proposed the more robust N-type antenna for the Mesh Potato as opposed to the smaller but less rugged Reverse-SMA antenna typically found on low cost wireless APs.  Because the N-type was more expensive there was some discussion as to which was the best option.  This debate was rendered somewhat moot when we realised that we could both save money and increase robustness by going with an internal antenna etched on the motherboard of the Mesh Potato.  That was a big design step for us as it rule out the flexibility of attaching different types of antennae to the device but it seemed like a reasonable trade-off.  Even now though a connector for an external antenna may find its way onto the motherboard somewhere.

FXS Interface

In the design of the Mesh Potato, we have opted for a compromise of speed in getting the Mesh Potato to market and optimal cost-effective design.  David pointed out in the workshop that with another month or so of design work, he would be able to bring the cost of the FXS module by over 50%.  Clearly, this is something to prioritise for the next generation of Mesh Potatoes in terms of producing a very affordable device.

AC Adaptor

A surprise for me in the discussions was recognising what a weak point the AC adaptors are in the context of the whole system.  We have gone the extra mile to harden the ports on the Mesh Potatoes to handle variable power sources, lightning strikes, etc but we haven’t discussed the AC adaptors at all.  These are obviously the first things to go when plugged into an unstable or incorrect power source.  While designing a custom AC adaptor would possibly turn out to be too expensive, it is possible that we will be able to get a higher rated power supply for the Mesh Potato.  David and Edwin are looking into this.

Power Supply Chipset

We had a good discussion about the trade-offs of between a high quality power supply chipset which would give us a 10% savings on the power consumption versus the cost of the chipset which was almost an order of magnitude more expensive than the cheaper option.    One insight for me was that the big win on power for the Mesh Potato is simply that of combining an AP and an ATA in a single device.  That halves the typical power consumption right there.  A 10% savings on top of that is desirable but has to be balanced against overall cost.  There was also a discuss about handling unexpected high voltages such as from an unregulated solar panel and here I have to defer to David and Elektra to chime in as it was beyond my modest capacity to fully grok.  The bottom line is that we are opting for the cheaper power chipset but some additional design is required for this chipset to cope with the high voltage scenario.

PoE and PoTL Injectors

The discussion around PoE (Power of Ethernet) an PoTL (Power over Telephone Line) injectors was pretty wide ranging.  We’d like to create the most flexible scenario possible for the Mesh Potatoes so that they can be powered by standard power cable or PoE using a single ethernet cable or PoTL using a standard telephone line.  We even discussed putting power and ethernet and telephone line through a single cable but the consensus at the end was to be as standard as possible to avoid misconfiguration.  PoTL in particular is a little hazardous because of the comparatively high voltage going over the phone wires.

Rael had a very innovative solution in which the Mesh Potato ports could be snapped on or off the Mesh Potato and extended via PoE.  This would make the Mesh Potato ideal for indoor or outdoor installation but we decided that it was going to add a level of complexity to the design that would slow getting the MPs to market.  Something to look at seriously for the next iteration though.

Case Design

Another significant topic of discussion was the case design.  We all agreed that we wanted the following:

• a water resistant case that can withstand a rain storm;
• UV resistant plastic that will work for years under an African sun;
• white or beige case to keep the device as cool as possible;
• ports should be covered inside the device;
• an LED display to aid configuration; and,
• fittings for both pole and wall mountings as well as possibly a desktop mounting.

In the end it transpired that what we were talking about would look remarkably like an Ubiquiti Nanostation Loco.  Hopefully we may even improve on Ubiquiti’s mounting bracket design.  After seeing some Vodacom payphone units in Khayelitsha we also discussed possibly replacing the LEDs with an LCD display that would open up possibilities for innovation in terms of debugging, configuration, payment status, etc.  We’re still investigating this possibility.

Software Development

I’ll break the software discussion down into the Mesh Potato, Afrimesh on the Mesh Potato, Server Configuration, and Miscellaneous.  Needless to say the conversation was not that structured. 

Mesh Potato Initialisation

We all agreed that to the extent it is possible, we’d like the Mesh Potatoes to work as a telephone network right out of the box.  This applies both to the scenario where they are working in peer mode i.e. not connected to an upstream server and when they are connected to an upstream asterisk server which could be on the local network, at an ISP, or even in the cloud.  Key to making that work is to ensure that each Mesh Potato that comes off the production line has a unique IP address.  Type wireless AP devices have a standard default IP address.  This would not work for us because Mesh Potatoes would then need to be individually configured by the user.  Thus, each MP needs to begin its life with a unique IP address.  Jeff Wishnie proposed a similar strategy to that used by Meraki which is to generate an IP address from a hash of the MAC address of the devices.  We decided that each MP should not only have a unique IP address but also a unique phone number which would be derived from the last 9 digits of the IP address.  We even came up with an unused country code (+288) which might be used to give the Mesh Potatoes a fully addressable unique DID at some point in the future. More about this on the wiki.

We also established that it would be desirable to be to switch any Mesh Potato into a ” voice master” mode so that other Mesh Potatoes on the local mesh would automatically recognise a configured Asterisk server on the network.  Elektra suggested that this could be achieved by having the “voice master” node announce a voice gateway class to the network in a very similar manner to how the mesh network announces upstream gateways via a class.  [Need a link to more info on this]

EasyTAG UI

Finally, because the Mesh Potatoes will have effectively random IP addresses out of the box, which will allow them to work as a network from power-on, the Village Telco entrepreneur is likely to want tools for mass configuration of the network, e.g. putting all the nodes on a single subnet, re-numbering the phone numbers, etc.  The whole mass-reconfiguration thing made me think of my music collection and the elegant simplicity of tools like EasyTag which allow you to batch edit an individual attribute of a single mp3 file or a thousand.  Having an EasyTag-like editor to edit the attributes of the Mesh Potatoes on your network would make network configuration a breeze.

ToDo:  Develop all of the above.

Afrimesh

Over the course of the workshop, Antoine managed to get his very elegant Afrimesh software running on the Mesh Potatoes.  He has posted some instructions on how to install Afrimesh on a fresh OpenWRT install.   Afrimesh on the Mesh Potato will serve three core functions.

1. Default GUI:  Afrimesh will be the  default web GUI for the Mesh Potato.  If you plug your laptop into the Mesh Potato via the ethernet port and point your browser at the MP,  Afrimesh is what you should see.  Any other software should be accessible via Afrimesh.
2. Network performance: Afrimesh provides the user with a dynamic map which shows the active mesh nodes on the network and their relative health.
3. Basic configuration: Afrimesh should allow the user to configure basic things like network name, IP address, etc.  For more sophisticated configuration, Afrimesh provides a link to Luci which can open a Pandora’s box of configuration options for the more intrepid user.

ToDo:

• Develop an Afrimesh package is easily installable on OpenWRT and which is also part of a Mesh Potato “target” firmware for OpenWRT
• Develop the network monitoring aspect of the Afrimesh software to include visual monitoring of network performance over time

Interactive Voice Response (IVR) Configuration

Naturally we don’t wnat to just have one way to configure the Mesh Potatoes, there should be several.  In particular, the Mesh Potatoes should be configurable without a web interface i.e. just by picking up a handset.  David made some initial progress in setting up IVR to be able to configure the IP address of the Mesh Potatoes, even working in a 2001 Space Odyssey clip to catch the attention…. “What are you doing, Dave?”  The goal is ultimately to have a suite of commands that could be issued via the phone keypad to be able to both diagnose and configure the Mesh Potatoes.

Default Asterisk Configuration

In order to make it as easy as possible for people testing out the Mesh Potatoes to set up their own VoIP networks, we need to develop some basic Asterisk configuration instructions, files, snippets and even bootable ISOs for Mesh Potato networks.  Work on that has begun on the VT wiki.

Points of Pain

It seems worthwhile ending off this workshop summary with a review of a sessions we called “points of pain”.  That is, what do we need to work on on the Mesh Potato and the Village Telco that will have the biggest impact on both the ease and cost of installing an MP network.  Here are what we came with:

• From a cost perspective “ease of installation” as just important as driving down the purchase price of the Mesh Potato.  A Mesh Potato that is difficult or simply complex to install can add as much as USD 50 to the total cost of the device.  A reminder to make the Mesh Potato blindingly easy to install, monitor, and debug.
• Less to do translates into lower error rates.  Pre-configure as much as possible and anticipate  as many scenarios as possible.
• Investment in software affects the hardware cost.  In this case we are talking about things like have David re-design the FXS interface.  In the spirit of hardware is the new software, smarter software/firmware can allow us to use less expensive (but not lower quality) components in the Mesh Potato.
• Scalability is critical.  What does an MP network look like at 10, 100, 100, 100 000 devices?  All software and design choices need to scale.
• Distinguish between technical infrastructure and payment infrastructure (I actually can’t remember exactly what we meant by this… other than the obvious)
• Protection is key.  Ensure that the Mesh Potatoes are as invulnerable as possible to natural (accidents, lightning, surges, brown-outs, etc) and un-natural (misconfigurations, theft, etc) events.

That’s about as much as I am able to capture from the event.  I hope participants will chip in the comments to flesh out and/or correct memory blips on my part in the post.  It was amazing (again) having such creative and brilliant minds crafting a future for the Mesh Potato and the Village Telco.

-Steve

Originally, we had imagined having a management interface for the Village Telco very similar to the Meraki and/or Open-Mesh network management “dashboards” which are  graphically-intuitive and easy for anyone to understand at a glance what is going on in the network.  However, Antoine introduced a new twist which opens up interesting possibilities.  He has designed the Afrimesh software so that it can run on any OpenWRT-based device.  It is written entirely in JavaScript. Actually Afrimesh will run on just about anything but, for me, the interesting new possibility is have a dashboard interface that runs on the mesh devices themselves.  I didn’t initially appreciate initially the potential that this opens up.

You can see at the left here what the management interface currently looks like.  As things currently stand, Afrimesh is designed to:

• monitor the health of the network and display connectivity information for each mesh node and connections between the nodes in a visually intuitive manner;
• provide an interface for the configuration of a B.A.T.M.A.N.-based mesh network; and,
• be an extensible GUI framework for interfacing with a mesh network.

From self organising data networks to self-organising phone networks

So here is where my minor epiphany came.  What is great about mesh networks is that you just plug them in and they work.  They self-organise in terms of upstream connectivitity, redundant routing, etc.  They are a very, very clever way of quickly establishing a pervasive, reliable IP-based network.

With the Mesh Potato, we are trying to leverage the power of mesh networks to set up a phone network.  Each Mesh Potato access point will have an RJ11 port, into which an everyday phone can be plugged.  However, to use the Mesh Potato, some configuration of the phone network detail is still required.  People need to have phone numbers to dial and a phone number themselves that people can call.  More than that, people have to have a way of finding out other people’s phone numbers.

I began to think of the Mesh Potato in different environments to that which we originally conceived.  For instance, would Mesh Potatoes work in an crisis environment where there was a need to set up a instant phone network?  Could they be designed to self-configure out of the box as a phone mesh?  Conversations with David Rowe and Antoine have encouraged me to think so.

Imagine you power up your Mesh Potato and an interactive voice response system says:

This is quite a reasonable scenario.  We could set up the Mesh Potatoes to do something like the above and the impact would be that you could unpack a box of 24 Mesh Potatoes, power them up (battery or solar), and within minutes have a working phone network.  This would work with or without upstream connectivity.  It would mean you could immediately start delivering value with a Village Telco even before you have a server, billing, and upstream connectivity set up.

The Afrimesh Potato

But it gets better.  With the Afrimesh software, you could conceivably combine data network information with phone network information.  Consider an enhanced version of the Afrimesh interface which not only provided network information but also voice call information.  The image at the right is my mock-up but it conveys the sense of how Afrimesh could show you who was on the phone network, what their extension is, whether they’re “available”, etc.

I think that having a plug-and-play Mesh Potato that will allow people to start talking with no initial technical configuration would substantially increase the take-up rate of the Mesh Potato.  I also think the Afrimesh interface offers opportunities for interesting voice/data/presence/twitter/jabber cross-over innovation on the MPs.  Love to hear thoughts from others on this.

P.S. For those waiting patiently to hold a Mesh Potato in your hand, David and Elektra are in the final stage of PCB design with Atcom.  I don’t have a date but we should have prototypes very shortly.