Factors Affecting Village Telco Performance

This post discusses the factors that affect the quality of Village Telco phone calls, in particular under marginal conditions like heavy mesh load and self-interference. This information was gathered from VOIP over Mesh papers (e.g. [1]) and discussion threads on the Village Telco Google Group in October and November 2009.

Each node in the Village Telco mesh will be a Mesh Potato (MP) or similar device such as a Nanostation 2 with similar processing power and Wifi performance.

In the typical Use Case a Mesh Potato (MP) is mounted on a short mast 1m above a metallic roof of a dwelling in a village or town ship. A secondary use case may see the MP mounted inside the dwelling. Other MPs will be located a few 100m away to form the mesh. The MPs will often have line of site, but are unlikely to have a free Fresnel zone. Signal strengths from adjacent nodes will be relatively high however there will be significant multipath.

In this environment:

  • Multipath effects will have a similar effect to increasing transmitter power and receiver sensitivity. For example moving the antenna a few cm up and down on the mast may increase or decrease the signal strength several dB due to signal re enforcement and annulment in the multipath environment.
  • Antenna directivity could be a useful to remove the effects of directional interference. However by definition a mesh network needs to be able to detect nodes in many directions so at least in the centre of the mesh a roughly omnidirectional pattern is desirable (Note there is strong debate on this point on the mailing list).
  • Surprisingly the speech codec and channel bit rate have a fairly small effect on mesh capacity for VOIP traffic. This is because of large overheads in (i) VOIP packet structure and (ii) the 802.11 MAC protocol. This is explained in the next section.

Packet and 802.11 MAC Overheads: Packet Rate is Key

VOIP packets are very small compared to other traffic on a Wifi network. Consider a 33 byte GSM codec packet compared to a packet of web traffic that may be up to 1500 bytes. To transmit this GSM codec packet using VOIP we must add a RTP header (12 bytes), a UDP header (8 bytes) and an IP header (20 bytes), giving a total IP packet size of 73 bytes. To send one GSM codec payload packet every 20ms (13.2 kbit/s) therefore requires an IP level bit stream of 29.2 kbit/s.

There are futher overheads due to the 802.11 MAC headers and protocol used to reliably transmit data over the Wifi channel. These are optimised for larger packets (1500 bytes). The time required to send 73 bytes at 11 Mbit/s is just 53 uS, however in practice around 800-1000 uS is required before the next packet can be sent. This time is consumed by MAC data, ACKs, physical layer synchronisation and various programmed delays and timers.

The Wifi channel capacity for VOIP is therefore dominated by 802.11 MAC layer overhead. This sets an upper limit on the number of packets/second we can send over the channel, and hence the number of VOIP calls that can be supported.

Packet rate (measured in packets/s) is the key factor for VOIP over Wifi capacity.

The channel bit rate has a relatively small effect as it only affects the small amount of time spent transmitting the payload IP packet. The speech codec bit rate has an even smaller effect as it only affects the size of a small part of the IP packet.

Using an x86 PC with built-in Wifi hardware Elektra has measured the maximum packet rate for 802.11b as 1350 packets/s and 802.11g as 3510 packets/s. This was measured using 100 byte packets with a small amount of ambient Wifi interference. Alex managed over 4000 packets/s on 802.11g. The use of x86 PCs ensures the tests were not CPU limited, i.e. the limits are likely to be the 802.11 MAC protocol.

The inefficiency of the VOIP over Wifi channel can be measured by comparing the throughput compared to the channel bit rate. The throughput at 11 Mbit/s (100 byte packets at 1350 packet/s) is 1.08 Mbit/s. The though put at 54 Mbit/s (100 byte packets at 3510 packet/s) is 2.81 Mbit/s.

For comparison with larger packets (1500 bytes) the Wifi protocols achieve throughputs of around half the channel bit rate, for example 5 Mbit/s over an 11 Mbit/s 802.11b link.

One way to improve throughput is to send multiple codec packets in every Wifi packet. In the literature this is known as source aggregation. We have been experimenting with 4 GSM codec packets per Wifi Packet. This effectively increases the VOIP call capacity by a factor of 4, at the expense of increased delay and possibly robustness under packet loss conditions.

Given the higher potential packet rates of 802.11g Elektra has suggested we configure the nodes to run 802.11g only – in 802.11bg compatibility mode the 802.11g packet rates are limited to 802.11b levels.

Self Interference and a Model for Mesh Capacity

In the use case above we assume good physical layer links. In this case the main interference source will be other nodes on our own mesh. The Wifi protocols arbitrate use of the channel between nodes, but clearly the capacity of the link must decrease if multiple nearby nodes all wish to transmit on the same channel.

The purpose of a mesh network is to relay data between distant nodes, e.g. node A sends data to C via B as A and C cannot send data directly. However only one node can transmit at any one time. Sending a single packet between A & C requires two packet transmissions, consuming 2 packets from the channel capacity.

Therefore the number of VOIP calls the mesh can support depends on the number of hops between nodes. In practice calls will span different numbers of hops. However we can model the number of calls as the (packet capacity)/(average number of hops).

Here is an example:

For a full duplex call using GSM codec we require 50 codec packets/s in each direction or a total of 100 packets/s. We aggregate 4 codec packets in each Wifi packet which reduces the mesh load to 100/4 = 25 packet/s for each call.

In this example we assume we have measured our mesh capacity as 1000 packets/s. Therefore the call capacity is 1000/25 = 40 calls for single hop links. If the average number of hops is 3, we have a call capacity of 1000/(25*3) = 13 calls.

CPU Load and Test Mesh Results

The VOIP over Wifi channel is characterised by many small packets, rather than a smaller number of large packets found in other forms of Internet traffic (e.g. fetching Web pages).

Just like the 802.11 protocols, low speed commodity router CPUs are designed to effectively handle large packets. Many small packets create many interrupts and context switches inside the router CPU which increases the CPU load. Data throughput at high packet rates tends to drop.

In addition to routing Mesh Wifi packets, each Mesh Potato must perform CPU-intensive operations such as echo cancellation, DTMF detection, and speech compression (the speech codec).

Some real world tests were performed using a two hop mesh. Two of the nodes were Nanostation 2s, the third node was a V1.1 Mesh Potato. All of the CPUs are Atheros AR231x variants, of approximately the same CPU performance (180 MHz). The nodes were all in the same room giving a good physical layer links, presumably 802.11g at a high bit rate. The mesh was loaded using iperf in UDP mode. This allowed a variable number of packets/s to be sent over the 2 hop mesh.

It was found that a throughput of around 900 packet/s (at < 1% packet loss, 200 byte packets) was possible over the two hop mesh with the Mesh Potato in the on-hook (inactive) state. Over a single hop around 1900 packets/s was possible with a 1% packet loss. For comparison the 802.11g x86 Wifi tests performed above achieved around 3500 packets/s.

During the two-hop tests the CPU load of the Mesh Potato was measured as 85% (all in the kernel). This suggests that receiving, routing, and re-transmitting mesh Wifi packets consumes a considerable amount of CPU on the Mesh Potato.

These tests suggest that the VOIP capacity of the Village Telco using 802.11g is currently constrained by CPU. This could be improved by optimising the kernel code responsible for receiving, routing, and transmitting the Wifi packets, possible via kernel compilation options.

When we try to send more packets than the mesh network can support we experience packet loss. To test the effect of packet loss we loaded the two-mesh network at 900 packet/s, then placed a VOIP test call across the mesh. The quality of the call could be monitored at variable packet loss rates. At 2% packet loss some click and pops started to become noticeable, however the call was still intelligible at 5-10% packet loss.

This is a fortunate result – call quality degrades gracefully in the presence packet loss rather than all calls on the mesh falling over. Adding a few more calls to a heavily loaded mesh should not cause a drastic drop in quality.

Consider a typical Village Telco mesh containing 150 nodes. We estimate a 10% activity factor, so we shall design the networks such that 15 calls may be operating at any one time. As the number of nodes (150) is much greater than the capacity of the mesh (15 calls) some control mechanism may be necessary to prevent a large number of calls being made at the same time. This could be achieved via scripts running on the Village Telco’s Asterisk server.

Conclusions

In the typical Village Telco Use Case we assume good physical layer links between mesh nodes. We have therefore not considered weak signal performance of performance in the presence of interfering signals from other services. After the initial deployments we should revisit these areas based on practical experience.

The throughput the mesh network can support is about 10% of the channel bit rate due to the effects of short VOIP packets.

For VOIP over mesh the maximum packet rate that the mesh can support is key. This is constrained by the 802.11 MAC algorithms to be around 3500 packets/s for 802.11g and around 1350 packets/s for 802.11b.

In our Use Case self-interference from other nodes on our mesh causes limits call capacity. A model for estimating the number of simultaneous calls possible on a mesh network is given above.

In practice the low end routers we are using are CPU-limited which further limits the packet rate to around 1900 packets/s for a single hop link (900 packets/s for a two-hop link).

VOIP signals are reasonably robust to packet loss when under high load, but it is advisable to limit the total number of active calls to avoid overload of the mesh.

In the next post we will discuss various ways to improve Village Telco performance.

References

[1] There are many papers discussing Voip of Mesh network performance, a good starting point is the list of publications published as a comment by Saritha Kalyanam on my blog. The list includes the following papers:

Performance Optimizations for Deploying VoIP Services in Mesh Networks
http://www.cs.sunysb.edu/~samir/Pubs/jsac-mesh-06.pdf

VoIP over WLAN 802.11b simulations for infrastructure and ad-hoc networks
http://www.it.uc3m.es/~acuevasr/publicaciones/LCS06.pdf

10 things you should know about VoIP over wireless
http://blogs.techrepublic.com.com/10things/?p=205

Simultaneous VoIP Calls Capacity Over an 802.11 Ad Hoc Network
http://ww1.ucmss.com/books/LFS/CSREA2006/ICW5018.pdf

Analysis of UDP, TCP and Voice Performance in IEEE 802.11b Multihop Networks
http://www.ew2007.org/papers/1569014366.pdf

12 Responses to «Factors Affecting Village Telco Performance»


  1. November 29, 2009 at 8:57 pm, Alexander Chemeris said:
    > The quality of the call could be monitored > at variable packet loss rates. At 2% packet > loss some click and pops started to become > noticeable, however the call was still intelligible > at 5-10% packet loss. Hum. IIRC, my impression was that this quality drop was not due to packet loss. I may be wrong, but I recall that you measured call quality on MP's phone port. In this case bad quality on the MP may be caused by local audio drops due to high CPU load. I.e. the real problem was not with packet loss, but with local audio processing. Sorry if I misunderstood your test setup. re: Packet loss rate. 20% of randomly distributed packet loss is very well concealed by PLC. You may check out these samples wito 20% of packet loss, concealed with PLC, developed by us for SIPez: http://ipse.chemeris.ru/plc/ There are two dirs with samples: "MaleFast" and "MaleSlow". Each dir contains original audio and this audio with 20% loss, concealed by different 5 PLC algorithms. Lost fragments are 20ms long. Algorithm 3 is very simple, 4 is slightly more complicated, 2 is even more complicated, 0 and 1 are advanced. You may see here, that even simple algorithm provides enough intelligibility at 20% of uniform random loss. Thus you either (1) have no PLC at all, (2) your distribution of loss is not uniform random loss (e.g. you may loose more then one packet at a time) or (3) your audio issues are caused by different issue.

  2. November 29, 2009 at 9:01 pm, Alexander Chemeris said:
    PS My suggestion is to somehow record loss pattern on receiving site to get better understanding of the issue. PPS One more possible reason of bad audio quality: (4) you loose one packet a time, but your packet is longer then 20ms (which is actually equivalent to case (2)).

  3. December 02, 2009 at 2:49 am, David Rowe said:
    Hi Alex, The packet loss test involved sending a VOIP call over a mesh network that was loaded with UDP traffic such that it had a measurable packet loss (2%, 5% etc). The VOIP call was sent and received by lightly loaded x86 Asterisk boxes, the MP was not being used for voice processing, just passing on packets. Nothing special was done to handle packet loss, I just used some standard Asterisk boxes with the GSM codec. Wow 20% packet loss concealment is very impressive. What sort of codec were you using? Are you concealment algorithms open source? I would like to see more work done in the area of packet loss but it is very time consuming and we have many other tasks to do on the Village Telco project. If we use 4 codec packet aggregation it is likely that 4 consecutive packets will be lost (80ms) so the distribution won't be random. Cheers, David

  4. December 02, 2009 at 3:19 pm, Alexander Chemeris said:
    Hi David, Ok, you've convinced me, that this is a PLC problem :) Examples I've shown are for generic PLC, i.e. it works on any raw-audio stream. So you just decode packet to raw audio and then apply this PLC. We developed it for the cases when codec does not have internal PLC, like G.711. It's a shame Asterisk still does not have PLC. Algorithm 3 (which produces audio with clicks) is actually a dumb frame repetition algorithm - something you can code in an evening by yourself. Algorithm 4 is frame repetition with smoothing just to remove clicks. It requires slightly more work. Seems I can't tell you a lot about algorithms 0-2, as it's a IP of SIPez LLC. I just can tell that it took my co-worker quite some time to implement and polish them to remove all clicks and pops and make it as smooth as possible while introducing zero delay. I would love to see this algorithms in open-source, but that's not in my competence. I've created examples of the same audio with 80ms audio frames and 20% of loss, but I can't put them online right now, because my ISP is having difficulties in receiving money from me, grr.. I'll post the link here when my connection brings up. I still advise you to capture real audio (on sending side) with real packet loss (on receiving side), so we can model it and ply with. PS Why have you chosen to use Asterisk as a SIP UA? It seems it's more of a server, then UA. Writing some client will take some more time, but will likely be lighter the Asterisk and will likely have things like PLC.

  5. December 02, 2009 at 11:40 pm, David Rowe said:
    Hi Alex, Yes I tried to check out your samples yesterday but received a 404 error when I hit the link. BTW for some reason I am not receiving email notifications from this site so my responses depend on me checking this blog. Feel free to continue this discussion on the Google Group if I am slow to respond. The reasons for using Asterisk are described on the Village Telco FAQ". I understand Asterisk does have some PLC algorithms, but we haven't experimented with them yet. I guess we would require some notification of missed frames, and perhaps experimentation with the jitter buffer. Thanks, David

  6. December 03, 2009 at 12:05 am, Alexander Chemeris said:
    Moved to the mailing list.

  7. December 18, 2009 at 11:41 am, Aaron said:
    I've been following villagetelco for some time. I find it very interesting. After reading this article I would like to point out a possible solution for the cpu usage. I set out to create a stronger, faster, and more realiable router at a low cost for the users at Open-Mesh. Some of the technology you guys use seems very similar to what I'm used to. Check out http://www.anaptyxmesh.com You'll find indoor and outdoor models that will easily run your OpenWRT variant. Add to the fact it's designed for dual radio usage, where one card serves the mesh and the other serves the client. I suspect you would of course alter this layout, but maybe it would help you with hardware for the more demanding situation.

  8. November 15, 2011 at 2:37 am, Codec 2 at 1400 bits/s « Rowetel said:
    [...] 1400 bit/s Codec 2 uses 56 bit (7 byte) packets, sent every 40ms. If used for VOIP the RTP+UDP+IP overhead is 40 bytes/packet. So the payload is just 15 % of the total VOIP [...]

  9. November 20, 2011 at 8:30 am, Rick van Rein said:
    Headless Idea #1: Combine RTP packets The overhead of sending RTP packets individually is surprising, and I've been thinking about this problem in the past. In a trunkish setup, with a number of RTP streams flowing between the same endpoints, it would be possible to combine a number of RTP packets into one, thus spreading the header overhead over a number of RTP streams. This would work best if some concentration of traffic takes place; for instance in a PBX per village. Or, since broadcasting is free in a WiFi setting, multiple recipients within reach could be addressed in the same message. RFC 2198 defines a binary format for piling up various RTP pieces, at a tagging overhead of 4 bytes. And voila, Codec2 is once again a bandwidth-slurping beast ;-) I'm pretty sure you'll be able to sacrafice several of these bytes by specialising this approach for your codec, and thereby find the motivation to make Codec 2 even more impressive than it already is. You would end up with nodes collecting a few RTP packets from different sources, taking them apart, splitting them each to their destination, and shipping those out once in a while. This would lead to extra delays, but that may be compensated: it is possible to send smaller packets at more regular intervals since there is less waste -- 40 ms could become 10 ms, with 10-20 ms delay at intermediate nodes. The real disadvantage would be that you would need to do something similar to VPI setup in ATM; when the RTP payload type is used so much, it is probably necessary to change it on each RTP-packing-unpacking hop; a lookup table depending on the sender with the sent payload type could serve to lookup the outgoing hop and payload type. I'll leave it to you to decide if this is inspired or insane ;-)

  10. November 20, 2011 at 8:53 am, Rick van Rein said:
    Headless Idea #2: Step down from UDP to LLC1 As we pile protocol on protocol, we often forget what can be done at the lower layers. The mirror image of UDP on IP is LLC1 on Ethernet. For a mesh, you probably don't need full-blown IP anyway -- and as a radio amateur you'll like it -- LLC1 and LLC2 are look a lot like AX.25. If I was to save on RTP bandwidth, I would start off by using IPv6, not IPv4. That is a good idea for SIP/RTP anyway, and perhaps it is time to stop the stampede of NAT. All rarely transmitted stuff, so SIP and RTCP, would just be shipped as IPv6 traffic, but RTP would be siphoned through another procedure. If the entire country uses a fixed /64 prefix to cover its telephone net, than that prefix would not add any information. And the lower 64 bits can simply be an EUI-64 address, so basically a MAC address. This means that such IPv6 addresses can be reduced to a MAC address. Now on to the UDP port used for RTP. Assuming that a small range of (even) UDP ports is assigned to RTP and statically mapped to the SAP values of an LLC1 link, it would be possible to remove the UDP port as well. It is now up to you to decide if the RTP header adds valuable information, or if one or more RTP packets could be packaged with a payload type annex 8-bit VPI. Another bit of information that seems to be forgotten, is that early ethernets could use 2-byte addresses instead of the common 6-byte, globally unique addresses. In a mesh architecture, it should be quite possible to use this abbreviated form, provided of course that modern WiFi hardware still supports it. Finally, do we need 802.1p/q headers? I don't think so -- the transmission over the air is not going to be sped up and your mesh probably has no switches; and upon arrival, the special transmission format makes these packets easily recognisable and subjectable to internal prioritisation / traffic shaping. So, what do we have now? - 2 or 6 bytes ethernet destination - 2 or 6 bytes ethernet source - 2 bytes length < 1500 - 2 bytes for LLC1 - n RTP dressed-down packets, each - 1 byte payload type - 7 bytes of compressed voice If n=5 and 2-byte ethernet still works, we'd have 13 bytes of "overhead" and 35 bytes of data, the valuable content being 73% of the total. More than enough room to challenge yourself to an even more radical Codec 2... I'll leave it to you to decide if this is inspired or insane ;-)

  11. November 20, 2011 at 10:11 pm, David Rowe said:
    Some great ideas Rick, I will link to them from the Codec 2 post. Cheers, David

  12. November 29, 2011 at 3:42 pm, Rick van Rein said:
    Hi David, I've been thinking a lot about header overhead over the past few weeks, to figure out how to downtune the overhead (also for mobile VoIP). I worked on a method that could compress the combined IPv6/UDP/RTP headers to about 1 byte, which is possible because they are highly predictable. But, as so often, such generally useful work has already been itching with someone else: http://www.rohc.net/ I didn't know about the RoHC line of RFC's before, but there is at least one implementation in open source that you might want to port to OpenWRT for your purposes ;-) although I think it might need expansion with RTP-mode compression. It looks like a normal IPv6/UDP/RTP header combo would reduce to about 1 byte, exactly as what I found to be possible. Codec2 and RoHC could combine to IP-level telephony traffic at 5+1 bytes once per 40 ms... or 1200 b/s...! As for the Ethernet level, it's good to know that there is an ethernet payload type for RoHC, namely 0x22F1. So you don't need to worry about compatibility issues with uncompressed traffic, and you won't even need the two bytes for LLC1. I thought that was worth bothering you once more for ;-) Cheers, -Rick

Leave a Reply