Metaswitch CEO Kevin DeNuccio focuses on new development, purchases, other countries

As widely reported today in press releases,

Metaswitch Networks today signaled its ambition to become a major telecommunications vendor, with the appointment of successful industry executive Kevin DeNuccio as chief executive officer. At the same time, John Lazar has been promoted to chairman.

On the conference call discussing the move, DeNuccio focused on some key points.

1. Product Line Expansion.Metaswitch wants to expand their current product line, primarily using their existing engineering base. Metaswitch has strong software-based telecom and network-protocol support, but you don't see see them doing a lot of hardware design with FPGAs or CAMs with their current development staff. Their DSPs, for example, are products from Texas Instruments. Products like the Acme Packet SBC or Cisco Routers have sophisticated fast path hardware.

So I think we can look for more software-based system-control products, and probably not as many line-rate products like routers, switches. Could Metaswitch make a competent firewall? Absolutely. Could they make a great Call-Quality Enhancer? Yep. Could they make an SBC? They'd actually have a lot of work.

Could they create a some real competition in the Application Server / Registrar world? Certainly. Since the demise of Sylantro and Tekelec's feature servers, and the inability to launch Sonus's feature server in a significant way. Right now, Metaswitch's product is tightly integrated to their TDM gateway. Some customers (ITSPs) don't actually want an SS7-capable gateway, and don't need T1 voice interfaces.

2. Purchase of new companies. DeNuccio mentioned purchasing other companies several times. You haven't seen many publicly-announced mergers out of Metaswitch in the past; they've built their own tools, but they sometimes do integrate with others' products. I wonder if we'll hear a merger announcement soon.


3. Global expansion. Metaswitch is very strong with switches in North America. Just based on the technology, I'd expect the fastest growth in the Caribbean (who share most of the US's telecom technology), and Japan (ditto). I suspect, but don't know, that the British Commonwealth nations might be good targets next, because engineers tend to know about the technologies in use nearby, and many of Metaswitch's developers are in London. I suspect many commonwealth nations use the UK telecom standards and C7 variant.

If iPhone development was complete, the Apple Store wouldn't have Avaya phones

When you go to the Apple store, do you see an IBM cash register? No, you see store employees using iPod Touchs to do everything.

When you go to the Apple store, do you see a PC at the back for use by the manager? No, all the business machines are Macs.

When you go to the Apple store, do you see PBX business phones? Yes -- you see a store employee talking on an Avaya PBX phone. He's telling a customer about the iPhone.

So then: Why isn't Apple eating its own dogfood? Why are they selling a phone, but not using it themselves?

• The PBX business phones are coupled to the location, but iPhones are coupled to individuals. Answer 1: Use a SIP client on the iPhone to connect to the phone system. Employees would carry an own iPhone that can only REGISTER to the local SIP system when it's in the store. Answer 2: Use an AT&T Femtocell, and add some call routing to the Femtocell. When an employees phone is within the store (and within range of the Femtocell), that phone receives business phone calls.
  
  
• The iPhone doesn't have the call control features a store would want; e.g., to put a call back on hold, or to transfer calls to another desk in the store. Answer 1: Provide a custom call control interface for managing calls after their received. This would require calls route through an Application Server (such as BroadWorks or Metaswitch) that has some external call-control interface. Answer 2: Use Centrex features from the AT&T telephone switch.
  
  
  

And then there were two: BroadSoft notices Metaswitch

At the BroadSoft Connections 2009 "Executive User's Forum" in
Scottsdale, AZ, I got my first positive confirmation that BroadSoft
has actually noticed Metaswitch.

I've written before about the two companies. BroadSoft makes quality
software, while MetaSwitch makes both good software and good hardware.
While the two companies are very different, both BroadSoft and
Metaswitch can deliver phone calls and calling features reliably.
Their customers are telephone companies: giants like Verizon or
CenturyLink (formerly Embarq and CenturyTel), or mom-and-pop's like
NGTelecom. These telephone companies want to use VoIP to offer
telephone service.

BroadSoft has long provided features for businesses, like line-hunting
and Automated Attendant. Metaswitch's carrier base is primarily
telephone companies offering residential service, but now provides
many of the same features, but Metaswitch also sells network gateways
used to connect to the old-school traditional telephone equipment.

On one hand, the BroadSoft and Metaswitch are not really competitors.
BroadSoft couldn't clock a T1 or parse an SS7 ISUP message to save its
life -- i.e., BroadSoft did not function as the VoIP-to-oldschool
network gateway. Whereas Metaswitch does SS7, CAS, ISDN Q.931, GR-303,
and many of Metaswitch's carrier customers depend heavily on these
technologies. On the same hand, BroadSoft has long had integrated
Voicemail, automated attendant, call center queueing, incoming and
outgoing calling policy enforcement, music-on-hold, line hunting, and
a customer-facing web interface -- all built into the most basic
BroadWorks system.

On the other hand, BroadSoft and Metaswitch are closer than ever, and
not because BroadWorks handles robbed-bit signaling. Metaswitch has
been gradually enhancing its MetaSphere-brand products. According to
sources within the company, they've added all of the features popular
among BroadSoft's carrier customers.

Several years ago, I heard from some key technical Metaswitch
employees their regard for BroadSoft. But for years, I had never heard
BroadSoft folks mention Metaswitch. But that changed this week, when
one of BroadSoft's long-time technical folks started talking smack
about the "jerks over at MetaSwitch."

There's a hierarchical, asymmetrical college rivalry between NC State
University (NCSU), the University of North Carolina at Chapel Hill
(UNC-CH), and Duke University. While I was at UNC-CH for grad school,
I heard much about Duke. But once I married into an NCSU family, I
heard a lot about UNC-CH. I'm not sure who the Duke people care about.

NCSU thinks more of UNC-CH than UNC-CH thinks of NCSU. And UNC-CH
thinks more of Duke than Duke thinks of NCSU. At UNC-CH, the "big
game" is the Duke-UNC match. At NCSU, the "big game" is the NCSU-UNC
match. To use an operator from Relational Theory: NCSU < UNC-CH < Duke.

In 2005, MetaSwitch looked up to BroadSoft as a rival, but I never
heard evidence of the contrary. The rivalry was asymmetrical. But now
the influence is becoming more balanced: Broadsoft and Metaswitch both
see each other as rivals.

Competition is a good thing and can evoke out good hard work. I am not
biased toward either vendor: if the equipment and software can work,
I'll make it work. With the consumption of Sylantro and the General
Bandwidth (GenBand) M6, BroadSoft has removed nearly(*) all the other
serious competitors. Now there are only two mature VoIP platforms --
MetaSwitch and BroadSoft.

In the old days, nearly every telephone company of any size had either
a Nortel switch or a Lucent switch. (**) And at this point, there seem
to be two major core-telecom equipment vendors: Metaswitch and
BroadSoft. Will they be the Nortel and Lucent of the next era of
telecom?

----------------
(*) I would say "all," but I've heard Sonus is out there. I've heard
rumors of people using their application server platform, but I've
never had the pleasure to see it in person.

(**) OK, a few had Siemens switches, but Nortel and Lucent are
definitely kings.

IT'S THE LATENCY, STUPID. OH, AND PACKET LOSS TOO. The Problems with 3G Mobile Networks.

All the carriers brag about their 3G Networks. I mostly attend to AT&T 3G and Sprint EV-DO; these are the two carriers I use.

Email and file transfer performance is usually disappointing. Transfer speeds as measured by the Ookla "Speedtest" application vary wildly, but the round-trip-time (RTT) latency is consistently quite high.

I recently did a study for a client about the effects of network delay in cross-country file transmission. With fixed networks (such as normal Internet links), cross-country latency is often 40 to 50 ms; much of that time due to the speed of light to get from New York to Seattle, for example. So the round-trip time (RTT) is 80 ms to 100 ms, typically. The overall goal is to transfer a 4 GB file across the US in a matter of minutes. We demonstrated through real lab tests that nontrivial delays under 100 ms do slow TCP transfers, so that FTP will not get the job done. The effects are exaggerated as the network throughput increases. E.g., at 100 Mbps, a link with 16 ms RTT slows to about 52 Mbps (assuming all the TCP buffers/windows are tuned properly.)

Many national labs, such as LANL and CERN, deal with problems of large data transfers. Their scientific instruments output gigabytes of data within seconds, and they have to transport it across the network or around the world. They have the money to buy 100 Mbps links from the US to Switzerland.

But many ordinary folks are on the other end of the spectrum, below 1 Mbps, and experiencing large delays. The vaunted 3G networks fall into this category: even with 3 Mbps or better downstream paths, the large delays can also kill performance.

A reasonable AT&T 3G speed is around 350 Kbps, i.e., 0.35 Mbps. Consider these lab tests with varying levels of delay:

• 350 Kbps downstream, 45 Kbps upstream, 0 ms RTT delay, 10 kB TCP window: 330 Kbps TCP connection throughput
  
  
• 350 Kbps downstream, 45 Kbps upstream, 200 ms RTT delay, 10 kB TCP window: 230 Kbps TCP connection throughput
  
  
• 350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 10 kB TCP window: 135 Kbps TCP connection throughput
  
  
• 350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 64 kB TCP window: 309 Kbps TCP connection throughput
  

The TCP window size is normally configured in the operating system by default; 64 kB (i.e., 65,536 bytes) is a typical value.

In the AT&T 3G and Sprint EV-DO networks, 400 ms RTT to the first-hop router is quite typical. Consider these examples with higher capacity and 400 ms RTT:

• 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 10 kB TCP window: 150 Kbps TCP connection throughput
  
  
• 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 64 kB TCP window: 430 Kbps TCP connection throughput
  
  
• 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 64 kB TCP window, 2% downstream packet loss: 220 Kbps TCP connection throughput
  
  
• 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 527 kB TCP window: 600 Kbps TCP connection throughput
  
  
• 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 782 kB TCP window: 650 Kbps TCP connection throughput
  
  
• 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 3500 kB TCP window: 980 Kbps TCP connection throughput
  

As you can see, as network delay increases, the TCP window must also increase. But even a tiny amount of packet loss -- 2%, fully reasonable in my tests of these networks -- can slow things down as well.

Conclusions

• The TCP Buffer sizes need to be tuned to 3G networks, not just high-performance networks: OS Vendors need to dynamically adjust TCP buffer sizing based on the first-hop round trip time.
  
• Users may need to explore non-TCP mechanisms to fully exploit network transport.
  
• High latency and even moderate packet loss can significantly reduce the usefulness of 3G wireless networks.
  

I performed these lab tests using two Mac OS X 10.5 machines and the Dummynet Delay/bandwidth emulator in the kernel, using 1 MB buffers on each end of the link. iperf was used to generate traffic.

Metered Broadband Won't Work, or Won't Matter: Stop focusing on the heavy-tail customers

In a Telephony Online article, we're told that Verizon (Landline) CTO thinks the end will come for flat-rate broadband. He means that you won't be able to pay a flat rate and get unlimited access to the Internet. Apparently, AT&T and others have come out and said the same thing.

The problem is that CTOs are focusing on the heavy-tail customers; it works like this:
a

• Most of your users are clustered toward the bottom. They don't use very much Internet access.
  
• A few of your users are serious; they use twice or three times as much as most of your users.
  
• A very few of the users never stop downloading. They use thousands or millions of times as much bandwidth as their friends.
  

Heavy-tailed distributions skew familiar values like averages and medians; so if you have users that follow a heavy-tailed distribution as described above, and you look at your "average" customer use, you'll get something much larger than your "typical" customer's use.

The simplest approach to avoid metering broadband use is to just fire the customers who are using way too much. If you drop the top 5% of users, you'll regain tons of network capacity.

If you want to use versioning, then you can charge one price that covers nearly everybody, then a higher price for people who go over their limits. This is the standard US cell phone billing plan; you get N minutes, and if you use more than N minutes, then you get charged per minute. But people don't want to have the meter running.

Google Next Big Service: Plain-text transcriptions of Phone Call, Podcast, and Youtube Videos

Google provides some very nice services; Google Books is useful and technically interesting. They started with printed books, and figured out how to de-warp those images to produce nice flat pages.

Then they used Optical Character Recognition (OCR) to convert the photographs of text into actual computer-manageable text. This brings image data into the domain of structured data, and it makes the text searchable. On top of that, it's effectively 500-to-1 compression of the text data (assuming that a 1 MB photograph of a document compresses to about 2 kB of text).

Google is poised to make a similar transformation by converting audio to text. Witness:

• Google purchased "Grand Central" and called it "Google Voice". Each Google Voice user gets voicemail service, and incoming voicemails are automatically transcribed to text for email.
  
• Many people complained about the transcription quality, but last month, in August 2009, Google Voice announced improved transcription service. Therefore, Google has and is improving the technology to automatically transcribe spoken-word audio.
  
• Google Voice functions as type of telephone service, allowing you to place and receive calls. All of the call audio can route through Google Voice's system. This gives Google access to your call audio when using Google Voice.
  
• Google owns Youtube, which has much useful and educational spoken-word material.
  
• Google Books has demonstrated that Google is interested in converting from analog-domain data (printed books) to structured textual data.
  
• Lots of useful, unique spoken-word information is presented in podcast and video form. A few podcasts are recordings of other written material (e.g., the Economist Magazine's podcast and audio edition), but most is probably never published in written format.
  
• Spoken word audio is not directly searchable by Google. Further, audio recordings are quite large; the number of bytes per word is very large. (The previous sentence in written form requires about 200 bytes of storage, including all the formatting around it. A 64 kbps MP3/Ogg Vorbis audio stream of the same sentence would require around 80,000 bytes of storage. The ratio is 400:1, similar to the ratio for printed books to printed text.)
  

Therefore, we should fully expect Google to implement new services:

• Podcasts can be transcribed. This would unlock lots of useful information that is currently inaccessible without a huge commitment of human time.
  
• Youtube videos can be transcribed. For ordinary, spoken-word presentation material, the benefits would be huge; the spoke-word material could be readily indexed, searched, and read. Google could use the temporal redundancy clues (i.e., whether this frame looks a lot like the previous frame; if there is motion in the video, the frames have little temporal redundancy) in the video to determine whether the video is changing often; if it's just a speaker with some text beside his head, they could even show us what's on the screen as the speaker speaks: At the beginnings of sentences, an individual frame can be extracted and shown along with the text.
  
• Google voice telephone calls can be recorded and transcribed. This would be a great way to record notes on work-related phone calls, and then go back and review notes from old calls. Recordings of phone calls are tedious, but transcriptions of those recordings would be fabulously useful.
  

IANAL Analysis of Patent 5,912,888 Claims against VoIP Gateway developers (MetaSwitch, etc.)

Last week, "Network Gateway Solutions LLC" filed a patent infringement case against many (but not all) VoIP gateway vendors. It's court case 1:09-cv-00667-UNA at the moment. (Justia)

You can download the complete court filings here. (I paid the PACER document access fees for you.)

• Complete ZIP file of documents available as of 2009 September 11 (4.5 MB)
  
• Document 1: Main Document, 38 pages
  
• Document 2: PDF of patent, Exhibit A, 46 pages
  
• Document 3: Civil Cover Sheet
  

US Patent 5,912,888 is in question. (US Patent and Trademark Office version.) The Patent was filed originally by US Robotics. It describes a Network Access Server, such as the US Robotics Total Control system. I worked extensively with the US Robotics TC system at a regional ISP in the 1990s. It was eventually sold to and marketed by 3Com.

The patent is a continuation of a patent that was filed June 9, 1994. It describes an "all-digital" system, where the Network Access Server has a T1 link to the telephone network. Back in 1994 and 1995, it was much more common for smallish ISPs to have analog modems connected via RS232 port to a network access server.

What companies are being sued?

Many (but not all) of the big names in VoIP network gateways.

• Adtran Inc -- sued for TA900
  
• Audiocodes Ltd. -- sued for Mediant 1000 MSBG
  
• Audiocodes Inc.
  
• Avaya Inc. -- G350 media gateway
  
• Cisco Systems Inc. -- AS5350 universal gateway
  
• Genband Inc. -- G6
  
• Juniper Networks,Inc. -- J2320
  
• Alcatel-Lucent -- CellPipe CELL-IAD-8T
  
• Alcatel-Lucent USA Inc. -- CELL-IAD-8T
  
• Media5 Corporation -- Mediatrix 3000
  
• Mediatrix Telecom Inc.-- Mediatrix 3000
  
• Metaswitch Inc.-- PB3100 card
  
• Mitel Networks Corporation -- MNC SX-200
  
• Mitel Networks Inc. -- MNi SX-200
  
• Multi-Tech Systems Inc.-- MultiVOIP MVP2410
  
• Patton Electronics Co. -- SmartNode 4960
  
• Quintum Technologies LLC -- Tenor Gateway DX2008
  
• Siemens AG -- HiPath RG2500
  
• Siemens Corporation -- HiPath RG2500
  
• Sonus Networks Inc. -- Sonus GSX9000
  
• Zhone Technologies Inc. -- IMACS-200
  

How did Digium, Taqua, Tekelec, and Nortel get left out of the list? Certainly they didn't actually buy a license for this patent.

So does this patent apply to modern VoIP gateways?

It's a real stretch to adapt this to VoIP equipment. The actual complaint sues on the basis of Claim 2 in the patent. The standard complaint is,

Defendant X, within United States, manufactures, uses, offers for sale, or sells network gateway systems including, but not limited to, Y, that fall within thte '888 patent. These devices have an all-digital network access system that connects remote computers for generating digital data to a network computer on a local or wide are a network via a digital telephone line. The Y includes modems, a telephone control interface, telephone bus, digital signal processing system, parallel bus, and a network gateway controller. At a minimum, X's Y contains each limitation set forth in at least claim 2 of the '888 patent.

The limitations of Claim 2 require (summarized in my language)

• At least one modem
  
• A telephone control interface for TDM management
  
• A telephone bus
  
• A digital signal processor (DSP) system for demodulating time-spaced bus signals into incoming binary data without conversion to analog form
  
• A parallel bus connected to the DSP
  
• A network gateway controller connected to that parallel bus, that formats the data for transmission on the local area network (like Ethernet.
  

The key problem for Network Gateway Solutions, LLC is that VoIP gateways have no modem. They have a digital network interface connected to the PSTN, usually in the form of DS1/T1 or DS3/T3 CSU/DSUs, but they don't have a "modem" in any conventional sense. And the lack of a modem is key to the distinction: this patent '888 relates to computers dialing in over modems, through the PSTN, to a network access server (NAS).

Claims of the Patent vs Modern VoIP Gateways

But let's look at all the claims, because they all hang together.

Patent Claims 1 and 2, "An all-digital network access server connecting remote computers generating digital data to a network computer on a local or wide area network": The inventor claims a system that connects a computer to an analog modem which connects to the digital PSTN, and eventually gets connected to another modem in the Network Access Server. But in a VoIP network, there isn't a computer on the PSTN side: it's a telephone in the kitchen.

Patent Claim 3 discusses "said incoming binary data", which data was generated by the digital computer connected to the analog modem of Claim 1.

Patent Claim 4 discusses the "packet of data from said network computer" (at the service provider) "destined for said remote computers" (connected to the analog modems) "via said digital telephone line". Again, there are no computers at the other end of the PSTN. Just phones in kitchens.

Patent Claim 5 also relates to the digital computer at the other end.

Patent Claim 6 discusses "an all-digital network access server for receiving and transmitting calls representing digital data", but in VoIP calls, the calls don't represent digital data. They represent calls.

The key to the Network Access Server is a digital computer at the far end of the connection, using an analog modem to connect to the PSTN. But in modern VoIP there is no analog modem. You might stretch to say that SIP phones are a type of digital computer, but there is no analog modem involved.

So if none of the claims relate to telephone calls, what could they possibly have on MetaSwitch, AudioCodes, and others?

So who DOES this patent apply to?

I believe this patent does not properly apply to VoIP network gateways because there is no modem involved in a VoIP gateway, as described in patent '888.

I'm not opposed to patents in principle. This all-digital network access system is a useful and innovative invention from the early 1990s. I do think Network Gateway Solutions LLC got one defendant right: The Cisco AS5300, AS5350, AS5400, etc that can operate with MICA modems do have modem capabilities. It appears that they are covered under this patent.

---

I am not a lawyer, but patents are about technology and precise thinking. Both of those are interesting. The Verizon Patent Win Against Vonage is also interesting to study because it applies quite broadly to VoIP Carriers.

TMC Blogger, Keating, wowed by new Decades-Old technology

Tom Keating of the TMC Net "VoIP and Gadgets Blog", please consider researching the facts before posting. You can't trust every company president who booths at the TMC IT Expo.

In a September 1, 2009 posting, "XCast Labs Cuts VoIP Bandwidth Requirements In Half," Keating oohs and ahs over XCast Lab's use of ordinary SIP technology. The technology innovation is described thus:

XCast Labs on the other hand uses their patented "direct RTP" which is able to tunnel through both user's firewalls and setup a direct peer-to-peer (P2P) RTP session between the two users. Once the call is setup, the two users are able to send the RTP media directly to each other. Since both callers are in California, they are just a few hops/routers away from each other thus dramatically reducing latency and jitter. XCast Labs simply maintains a small signaling connection to determine when the call ends for billing purposes.

Keating was "dumbfounded that no one else had thought of this". He should be dumbfounded if no one else had done this. XCast labs is using the the STANDARD way of doing RTP (prior to the hosted VoIP industry). There's nothing new here.

• Look in RFC 3261 that devices SIPv2: In the basic "SIP Trapezoid", the signaling flows between SIP proxies, but the RTP flows along the shortest, most efficient path between the endpoints.
  
  
• Look at the Acme Packet SBC: it supports Media Release directly. It's simple to configure (mm-in-realm=disabled), and used very commonly.
  
  
• Look at the Ditech PeerPoint C100: it supports direct media between endpoints, including those who are NAT'd with certain types of NAT.
  
  
• Look at the MetaSwitch: You can disable a setting called "restricted media" and get the exact behavior they're describing. So any MetaSwitch customer can do it.
  
  
• Look at BroadWorks: It does direct, shortest-path RTP *by default*. The only way to make BroadWorks control the media is to use advanced call mixing or recording features.
  

And let's not forget Verizon's patent, US Patent 6,104,711. In 2006, Vonage was found to be infringing of that patent. That patent covers cases where the IP address of the called party is returned to the calling party. This is exactly the requirement for XCast's direct media to occur!

So this is prior art -- and actually covered under another patent. Verizon has not apparently tried to enforce this patent, yet, against all the other service provides. In fact, Verizon USES BroadWorks and Acme Packet itself in their new fIOS VoIP architecture.

---

Update 2009 September 15: Tom Keating revised his post with some more details on the claim of new technology:

Ok, some clarification from XConnect. The key point which I guess I may not have made clear in my article is that their direct RTP is able to penetrate firewalls and initiate direct RTP sessions. That's their "secret sauce".

Here's what Vlad Smelyansky from XConnect told me:

Your description of our RTP is wrong and the comments (except reference to the patent) are correct. The guy actually picked on the core mistakes and ignored the rest.

There are gazillions of other cases when people are using direct RTP. Our patent and technology is not about the idea of sending traffic directly between end points because that is part of the RFC. Our patent and technology is that we actually can do it between endpoints behind firewall(s) without any special configuration of firewalls (NAT) or deploying additional technologies (STUN) on those endpoints. ACME, Nextone, and Ditech tried to do this and came to the conclusion that it is impossible without additional protocol or special firewall configuration. But if endpoint devices are on a public IP or the user is willing to do special NAT or use STUN, they all can do Direct RTP for last eight-nine years. Our specialty is that we can do it with almost any endpoint behind a firewall and do it totally plug-n-play. No one else can do that.

He referred to my comment about the patent. I am not a lawyer, but Vonage was found infringing of the claim in the patent where the IP address of the called party is provided to the calling party. AFAIK, Verizon hasn't tried to sue anybody else about that claim in US patent 6104711. Perhaps XCast has some way of setting up direct media between the calling and called parties without returning their IP addresses to each other; telepathy, maybe?

Nevertheless, I only mentioned the patent because it's a dated legal document showing that people were thinking about sending media directly between the two voice-over-packet endpoints.

---

(SIP Trapezoid image from https://wiki.sch.bme.hu/bin/view/Infoszak/IpTetelKidolgozasMedia?CGISESSID=0d198e2a22067629087267719d075294, but obviously drawn by Cisco.)

When 160 bytes is worth $175,000,000 (and when it is not)

From a Reuters Article:

"BANGALORE, Aug 25 (Reuters) - Syniverse Holdings (SVR.N) said it would buy Verisign Inc's (VRSN.O) messaging business for $175 million and expects the combined business to grow about 10 percent annually, sending its shares soaring more than a quarter."

"The wireless voice and data services provider, whose customers include AT&T Inc (T.N), Telefonica (TEF.MC) and China Telecom (0728.HK), hopes to ride on rising demand for smart phones and netbooks."

"Smartphones would account for 29 percent of the entire mobile phone market in 2012, compared with 14 percent in 2009, Samsung Mobile Display predicted in a statement in April."

Why is Syniverse buying Verizon's MMS and SMS business for $175 million? Syniverse obviously thinks these are useful services. And you only buy a thing if it's going to earn you more than it costs, so Syniverse must believe they will make more than $175 million off of the business.

And people really do pay for SMS and MMS. Speaking just for SMS, My mother's Sprint/Nextel plan charges $0.40 for each SMS. She could upgrade to a pricier plan, pre-paying for a few hundred SMS messages. Is this a sustainable price?

Why is Verizon selling their MMS and SMS business? I think this is the fun part: the price of SMS will rapidly approach zero. Why will it be free? People, like my mom, are using an Internet-capable device to send a 160-byte message to another Internet-capable device. That's ONE PACKET to deliver a message, and then an acknowledgment.

To migrate AWAY from MMS/SMS, won't the carriers need to develop competing technology? Not really: AIM, MSN, Yahoo! Instant Messenger, and Google Talk have already developed a perfectly reasonable way of doing it.

So why aren't people using Internet-based Instant Messaging on their phones right now? There are some genuine accidental advantages to SMS:

• User Interface. The MMS/SMS user interface is very simple; it doesn't require starting a special application, or booting a java subsystem. Instant Messenger clients, at the moment, usually do.
  
• Integrated into mobile phone signaling. This lets you use SMS without going through an elaborate "connecting to the Internet" ritual. And it lets you receive messages even if you are "offline" -- whatever that means for a mobile phone that's always connected to its carrier.
  

But these are accidental advantages -- easily extinguished by better software and service design.

There was an era when a mobile phone plan did not include voicemail. Then there was an era when it didn't include text messaging. Now we're in an era where some mobile plans don't include Internet access. But that, too will change. Every phone will have Internet access, just as every plan now has voicemail.

Samsung, LG, and the other mobile phone developers could easily build on this to add an Instant Messenger client into the phone. It could work as seemlessly as SMS/MMS does today. And it relieves the network operator, such as Verizon, from running servers to process all of these messages. Just let AIM, Yahoo, and Google do that for you!

SMS is part of the mobile phone signaling protocol, so you can get your SMS anywhere you can get phone service. Apple has harnessed this by using specially-encoded SMS messages to control its features, and implement "Push" notifications for the iPhone. Other vendors could follow this lead. Or, they could simply ensure that the mobile device is always on the Internet; so that if it can receive mobile phone signaling, it can also receive packets from the Instant Messenger server. IM clients, and servers, already accommodate mobile and weakly-connected clients.

So is MMS and SMS worth $175,000,000? For Verizon, the answer is clearly no -- they'd rather have the cash than the business! And I think it's easy to see why: SMS and MMS are great services that can easily be re-implemented with cheap and free Internet-based technology.

Syniverse sees this as a "strategic decision" that expands their customer base -- but in reality, they're just in love with SS7. It's like the old Strategic Planning saw about the locomotive industry; they saw themselves as a train industry, instead of a transportation industry. Syniverse sees itself as an SS7/C7 company, not a messaging and call control company.

PRAGMATIC SECURITY: "I'm from the Government Cyber Security Agency, and I'm here to help."

National governments around the world are starting to focus heavily on "Cyber Security," the politician's buzzword for computer and network security. I'm gravely concerned.

May 29, 2009: "Obama Announces Cyber Security Czar"
http://thehill.com/leading-the-news/obama-announces-cyber-security-czar-2009-05-29.html

July 9, 2009: "France Creates New Cyber Security Agency"
http://www.pcworld.com/article/168135/france_creates_new_national_it_security_agency.html

In big companies, producers (the employees who make the product, as contrasted with the overhead staff) have an antagonistic relationship with the network security policies and their enforcers. In most cases, corporate security policies tend to militate *against* successful work and productivity. Cisco did a study in 2008 that found that a huge number of employees believe they must circumvent security measures to get their work done. Ironically for this week's news, France had the "worst" record of all. "Of all the countries, France (84 percent) has the most employees who admitted defying policies, whether rarely or routinely." http://www.cisco.com/web/BE/about/press/press08/10282008.html

The fundamental problem is that security policies often build too big of a hedge around the actual requirements for security. Scott Adams jokes about this with Mordac The Preventer of Information Services; Mordac once decided Dilbert's password was too easy, so he replaced it with the entire text of the "Da Vinci Code"(*).

At one big corporation based in Portland, Oregon, they have a security policy that prevents any kind of audio recording. Yet this company builds and maintains voicemail platforms! The policy is silly; if they're worried about corporate spying, bug detectors, not silly policies posted on the receptionist's desk. And so the policy is actively ignored. (I used a publicly-visible voice recorder at one meeting there.)

At another firm I work with, they have firewalls and VPNs, but most of the passwords are the same common, english word. The truth is that they probably don't need VPNs to begin with, and VPNs with trivial passwords just provide an illusion of security.

So now, the US and French government will be getting into the act. They'll find a way to make rules requiring certain security policies to be enforced. The state of Nevada already require some companies operating in its state to comply with the Payment Card Industry (PCI) Security Standards, for example.

http://www.boazgelbord.com/2009/06/nevada-mandates-pci-standard.html

The "safe harbor" provision of Nevada's law means that if you can pass the PCI DSS audit in Nevada, then you're not liable for doing anything else to ensure security of your customer's data.

But if big companies make security policies, what's wrong with the civil government making security policies? Big companies can't put you in jail for violating their policies. Big companies can't declare you to be in violation, and therefore give all your assets to somebody else.

The federal government's involvement is another expansion of power for government. I'm skeptical that it may do much good, since we already have laws against intrusion into somebody else's computer, or theft of data, and because security compliance standards can be faked, and because security compliance audits don't catch even the big stuff.

Even in profit-driven companies, security policies tend to be over-reaching, and prevent good work from getting done. I'm very skeptical that national governments are going to do any better since productivity is not a concern.


(*) The password didn't include the parts Mordac didn't believe.

Want a job in VoIP? Consider BroadSoft Training

BroadSoft Inc's BroadWorks platform is undoubtably one of the leaders in VoIP platforms out there. They've created two certification programs: BroadSoft Certified Platform Administrator (BCPA), and BroadSoft Certified Application Administrator (BCAA).

They're also using Ziiva, aka Prosperty Learning Management Systems (LMS), to host a site http://certification.broadsoft.com where you can take the tests and study for the tests.

One of the frustrating things about the voice-telecom industry is that their documentation is typically kept closed, and only available to customers. But why? Documentation from Cisco, Oracle, and IBM is all publicly available; it's not hurting them in their industries. But you'll see companies like Adtran open their data-telecom documentation, while keeping some of their voice-telecom documentation behind a login and password. (I consider documentation to be "open" if you can get it without being a customer of the company.)

BroadSoft's certification site bucks that trend. The courseware is free and open. If you're trying to break-in to carrier-grade VoIP systems, studying the free BroadWorks documentation on their certification site is a good way to learn.

And if you're actually willing to do the work to read and learn that material, my hat's off to you. This industry needs people who are willing to crack open a PDF and do some studying.

Nortel Networks Bankruptcy and the CS2000 / CS2K: Finding Alternatives

Nortel Networks has gone bankrupt, and is selling bits and pieces to Nokia Siemens. I'm not sure if the CS2K is one of those pieces.

Nortel has a lot of really smart people. It's difficult to change from charging $16M for a telephone switch servicing 100,000 lines in 1997 (which is what they charged BellSouth for the DMS100 in my home town) to the current marketplace.

There are interesting engineering and technical questions that arise when a company does this. People have long complained about Nortel's support for the CS2K. At least one carrier is rumored to have stopped provisioning new subscribers on their Nortel CS2K as soon as the bankruptcy was announced.

If you need a modern alternative to the CS2K, what are the choices? There are plenty, but a few front-runners:

• MetaSwitch is the first to come to mind because CS2K users often have copious TDM and SS7 interconnections. The MetaSwitch platform can definitely do this, Class-5 and Class-4/tandem type features.
  
  
• BroadSoft may be a fine choice if you were using the CS2K primarily for Class-5-like features, and you have some other way of interconnecting SIP to the PSTN.
  
  
• Alcatel-Lucent has the old Telica Plexus 9000 platform (not to be confused with the Plexus 8000). I think the name-of-the-month is the "Alcatel-Lucent Network Gateway". It's a nice box, but somewhat difficult to troubleshoot. (Especially if you don't have the super-secret list of debug commands and logging that you can run on the individual CPU and VSM cards.) Still, if you're looking for solid SIP-to-PSTN interworking, it's a good option to consider.
  
  
• Taqua has a platform worth considering.
  
  
• You might consider Cisco because of the PGW or BTS10200. These come across as pretty weak players, compared to other folks in this list. If you're interconnecting the VoIP world with the PSTN, then you can use AS5400s and have great joy. (Just be sure you add access-lists to control the paths SIP can take. IOS will accept a call from anybody!)
  
  

Three Rules for Effective Naming

1. ONE OF THE RULES that has served me well is this: things should have unique names so I can distinguish them. Filenames are easy examples; I've noticed it's a lot easier if, for output I don't intend to edit, I don't re-use filenames.

Output I don't intend to edit includes things like the output of mysqldump (which dumps the contents of a database) for backup purposes; or the output of wget (as it downloads a web page or site). Or the output of "show tech-support" on a network box.

At the moment, I put a timestamp in the filename; such as "show_tech_support_200906131557".

This raises some questions:

(a) Can't we depend on the filesystem for this? After all, every filesystem has time and date stamps. BUT: Files don't just live on filesystems: they're often sent through email, or attached to ticket systems on web sites.

(b) What timezone should be used? I just use the timezone I'm actually in, but I'm not sure that's "ideal". It *CAN* be confusing when a system (such as a server) is in one timezone and I'm working in another timezone. I probably need a better rule for this case.

2. FILESYSTEM HIERARCHIES ARE GREAT, but unfortunately they're not useful once the file leaves the filesystem (such as when it's sent by email). So if you're file is named "tech-support.txt", you only know the least bit about its content. If you add the timestamp as I suggest above, you get "tech-support_200906131607.txt". It's also smart to add your organization to the name if it's going to be crossing organizational boundaries, such as when you email the output to a vendor. It's also smart to include the name of the system that generated the output.

So you'd end up with "acmecorp_router_1_tech-support_200906131607.txt".

Occasionally, I'll get a complaint that the files named this way are too long. I attribute this to sloth on the part of the complainer, but perhaps other theories may explain the complaints.

3. ANOTHER RULE is keeping a unique tag on datasets introduced to inter-mingled sets. For example, I happened to harvest some MP4 audio files as they were flying by one month, and I added those to iTunes. Later on I decided the quality was too low on those files, so I wanted to delete them all.

Fortunately, I had included an underscore "_" in the song names when I added those files to the iTunes library, so it's trivial to select and delete them all.

Or take a less-trivial case: suppose you're managing SIP VoIP Phones, and you're adding entries to the database for 10,000 of them. The database may already have 5,000 entries. After you've added these entries to the database, you may later need to go back and work with these entries. (Maybe you didn't get it perfect the first time.) In this case, it's convenient to have some marker in the database entries for the SIP phones you added in this way. If the database has a way of marking the source of the entries explicitly, then that's good to use. If not, then you can sometimes add a tag to the name. At the very least, you can keep keep a list of the individual entries added, which you can use later to access the new data set.

Reclaiming Pragmatic Security: Why Standard Computer Security approaches Stink

The standard approach to computer security goes something like this: "Remember, Job #1 of any security professional is to make sure nothing bad happens. You don't have to be elegant, you just need to get the job done." [1] That statement is from Mike Rothman, the author of the "Pragmatic CSO". This is a book to explain to security techies how to function in a business environment.

But this statement is hogwash! I haven't read the book, but Rothman seems to be teaching you how to (a) feel influential because managers are asking you questions, and (b) make project plans and business cases for security equipment purchases. The premise has some truth: computer-system security specialists within companies need to function like grownups. But his statement "make sure nothing bad happens" reveals a misperception of any role in business.

Security always has TWO sides [2]:

(a) Safety: Be sure bad things DON'T happen.

(b) Liveness: Be sure good things DO happen.

But these ideas are not well known outside the formal Computer Science Research community.

There's an old joke: the best firewall (network security device) is the air-gap between a network cable and the port. The joke is funny because so many security people actually function as if their real job is to ensure nothing bad happens.

This absurdity is promulgated by Rothman's statement. Its practice creates the crazy situation in which the security technicians -- Chief Security Officer (CSO), IT technicians, etc. -- work *against* getting Good Things Done, leaving the rest of the business to work in favor of getting Good Things done.

Cisco published a relevant study in 2008:

"One of the most significant findings was the difference in employee and IT perspectives on policy non-compliance. According to IT, employees defy policies for a variety of reasons . . . [E]mployees said the top reason for non-compliance is their belief that policies do not align with the reality of what they need to do their jobs. More than two of five employees (42 percent) made this claim globally. In Germany, even though the majority of employees felt their companies' policies were fair, more than half of them (55 percent) said they would break them to complete their jobs." [3]

Of course the security staff feels like the users don't care: the security staff doesn't really care if the users to get their jobs done. It's as if the security staff feel that it's better not to get the job done at all rather than have it done in an insecure way.


PRINCIPLES FOR BEING PRAGMATIC ABOUT SECURITY

I. The computer security insanity has to stop. Being pragmatic about Computer System Security doesn't mean learning how to make fake business plans to buy equipment.

II. Being pragmatic about computer security must include both enabling good things to happen, and preventing bad things from happening.

III. The MAIN job of anybody in an organization is to do the business of the organization; i.e., to get good things done, to provide the service, make the product, satisfy the customer, help the world.

IV. All ventures involve risk; there's always a risk of attack and a cost to defend against it. The probability and costs of many risks cannot be quantified because you don't know what the worm or attacker will do.

V. Security is NOT about buying equipment or software. (Most serious network attacks go right through firewalls. Anti-Virus software is almost never up-to-date enough to stop new viruses.)

VI. Complex system security is sometimes worse than no security. For example, firewalls with 100-line access lists are almost always wrong.

VII. Security Controls that Prevent activity are fundamentally defense against specific attacks. For every security control (firewall, access list, etc.) you have, you should know what you're defending against.

VIII. Realize that Compliance can be a farce; being "complaint" to some standard may mean you're not defending against the right threats.Auditors are often ex- accountants with checklists and no real understanding of systems. Don't rely on policy/law compliance for anything besides satisfying paperwork requirements.



[1] Mike Rothman's 2006 blog post, "'Pragmatic Security' Coming Into View", posted at: http://securityincite.com/blog/mike-rothman/pragmatic-security-coming-into-view

[2] Fred Schneider, "Defining Liveness", http://portal.acm.org/citation.cfm?id=867712

[3] Cisco Systems, "Global Cisco Study Applies Reality Check to Corporate Security Policies, Draws Connection to Data Leakage Risk",http://newsroom.cisco.com/dlls/2008/prod_102808.html

Securing VoIP Carriers: Start with the CPE Management

It amazes me that VoIP carriers continue to make a very simple mistake: they setup an FTP server (which is good) to provide the configurations for their SIP Phones (such as PolyCom, and Aastra SIP phones) (which is also good), but they use the manufacturer's default "PlcmSpIp" username and password on the servers (which is not ideal, but that's debatable). (Google reports 411 web pages documenting "PlcmSpIp" on the Internet right now. 47 of them are published by "site:polycom.com".)

But worst of all, they allow anyone in the world to FTP in using these well-publicized credentials, and then actually list all of the CPE configuration files in the directory using FTP. An attacker can then download these configuration files and:

• Steal phone service by registering using the credentials documented in the configuration file.
  
• Interrupt a legitimate customer's phone service.
  
• In some cases, he may be able to stealthily intercept telephone calls going to and from the user.
  

There are a number of defenses against this type of exploit. First of all, don't allow the PlcmSpIp (or any other user used by your customers) to actually view the contents of the directory via FTP. This makes it more difficult to download files, because attackers cannot directly view the list of SIP phones in your system. Instead, attackers are forced to guess the MAC addresses of the SIP phones in your system. You can then use approximate defenses, such as blocking users who attempt to download too many non-existent config files.

Protecting IP Phone configurations is one key element in a quality VoIP System Security Analysis.

Choosing BroadSoft and MetaSwitch

I regularly spend time working with BroadSoft BroadWorks systems, and with MetaSwitch systems. I rarely have a client that has both, but I have many clients in both camps. Both are winning -- in separate camps. The BroadSoft service providers and the MetaSwitch service providers really don't seem like the same market set.

There have been a lot of competitors -- consider Sylantro, General Bandwidth M6, Lignup, Alcatel-Lucent's Feature Server 4000, and Asterisk, as examples. Some of these systems work, but some of them are pretty flaky. It takes a lot of work to catch up with the features and maturity of BroadSoft and MetaSwitch.

Since my readers are interested in this subject, I'll lay out some thoughts on the strengths of the two platforms.

BroadSoft's Strengths

BroadSoft makes some really strong, reliable software. Making and supporting mature software is an amazing feat. They should really be proud of their software platform, and the new features they've added.

It seems BroadSoft's customers (service providers) want to sell sophisticated features; they want to replace the office PBX. BroadSoft customers ask questions like, "do you have the features to replace a hotel PBX?" or "can we use BroadWorks to provide roaming features to users homed on another Class-5 switch?"

Many of BroadSoft's customers have in-house programming teams to handle the CDRs. Many of them work with outside SIP peering partners, like Level(3), Global Crossing, and Dash. Some of them have a sophisticated telephone gateway capable of SS7/ISUP and SIP, but some of them just use ISDN PRI gateways, like the AudioCodes or Cisco IOS gateways (AS5400, e.g.). BroadSoft doesn't really care how you connect to the PSTN.

BroadSoft Inc doesn't have a technical conference per se; they have an "Executive User's Forum", called Connections. You don't go to BroadSoft Connections to talk shop about telecom primarily -- you go there to learn about how to market your services and grow your business. You're generally supposed to get technical details directly from your salesman and your sales engineer separately. (They will answer technical questions and talk about some details -- but the purpose of Connections is all about networking -- in the human sense.)

BroadSoft operates a TAC (Technical Assistance Center); they prefer to interact primarily through their ticketing system, ExtraView. You may not get the same TAC engineer twice. If you have a general question, it's sometimes best to work through your sales team. If you have an outage, the TAC works very hard to pull in engineers who can help understand things quickly.

BroadSoft has the "Xchange" site where people can post questions. Most questions will receive a reply from a BroadSoft employee. In my experience, BroadSoft customers don't tend to interact a lot on the Internet. It might be because so many of them consider themselves to be nationwide players, selling service all over the US. And if there's a discussion group full of your competitors, you're not likely to talk about your questions and concerns.

BroadSoft has a really neat, simple to use voicemail platform built right in. I like it a lot; it's integrated well, and it can use your existing email account as a voicemail store.

BroadSoft's system runs on Linux or Solaris servers, so you need Unix system administration skills.

BroadSoft's documentation is split into many hundreds of small files; it's definitely improving over the past few years.

BroadSoft has some very smart business people at the top.

MetaSwitch's Strengths

MetaSwitch, too, makes some really strong software. And they build some really strong hardware, too. They have some really smart people developing and maintaining their software, and providing customer support as well. MetaSwitch has many sophisticated features; they don't sell every PBX-replacement feature that BroadSoft offers, but they sell many of the major features in the new SDP platform.

MetaSwitch's customers are often replacing Lucent 5ESS, Nortel DMS, or other SS7-capable telephone switches. MetaSwitch customers often ask questions like, "can the MetaSwitch support point code proxy?" and "can I split an ISUP trunk group across gateways at two different locations?" and "how many DS3s can I get in a gateway?" MetaSwitch tries very hard to provide PSTN switch and gateway features that telephone companies are accustomed to using.

Because MetaSwitch provides the network gateways, they do a lot with the audio (RTP). For example, MetaSwitch can directly perform echo cancellation for calls that traverse the gateway. (And, in fact, you could send SIP-to-SIP calls through their gateway.)

MetaSwitch has a User Forum; they talk a lot about their product, enhancements, and features. They talk about marketing your services a little bit. Many of the technical people from MetaSwitch are there.

MetaSwitch customers often rely heavily on their Customer Support Engineer, CSE. The CSE is directly assigned to a service provider, and he'll remember the problem you were discussing earlier. In my experience, you can have an informal conversation directly with the CSE. Beyond the CSE, MetaSwitch some extremely sharp CSE managers; the overall support team is incredibly bright and well educated.

MetaSwitch has a very lively customer forum email list, where people talk about their technical problems. It's probably in part because so many of MetaSwitch's customers are regional carriers who only serve one area of their state, and they don't feel threatened by other carriers.

MetaSwitch has a voicemail platform; as I understand it, it's in use by some major carriers who have hundreds-of-thousands of voicemail boxes. They scale it way down for smaller folks.

MetaSwitch runs on Solaris, but you really don't need to know that. They expose separate interfaces for your normal maintenance.

MetaSwitch's documentation is bundled into about 10 files, and is reasonably good. (Note: this is high praise. Most technical documentation is a waste of everyone's time to write and to read.)

MetaSwitch has some very smart Engineering people at the top of their business.

How do you choose?

Do you need an SS7-capable gateway?

Do you need voicemail for every subscriber?

Do you have Unix system administrators?

Do you expect to integrate the platform with your provisioning system?

Do you need to have multiple people modifying translations/routing at the same time?

Both of these are remarkably capable systems. They do have their strengths, and their target markets. Of course, BroadSoft and MetaSwitch would each like to have all of the customers, but since BroadSoft doesn't sell media gateways and SS7-capable equipment, and MetaSwitch doesn't try to provide every PBX feature known to man, they're really not direct competitors in my mind.

VoIP-NSP Mailing list

There's no general discussion for people using VoIP in a Network
Service Provider forum, until now.

http://voip-nsp.e-c-group.com/

VoIP-NSP: for people using VoIP in a Network Service Provider
environment. These should include people using BroadSoft BroadWorks,
Acme Packet, MetaSwitch, Sylantro, VocalData, General Bandwidth M6,
Alcatel-Lucent, Asterisk, and anybody else interested.

VoIP Network Service Providers have stringent network requirements,
and unusual requirements.

VoIP Security is a Different Animal

A lot of Network Security Expert grew up protecting conventional
enterprise applications. Clients are PCs. Servers are web servers, or
run MS Exchange. Maybe they get involved in web proxies. Perhaps they
have to create a special rule for Microsoft SQL Server.

VoIP Security is entirely different. The ports, connections, services,
and behavior are completely different than conventional PC services.

Does your Security Consultant understand SIP or MGCP? (Certain ports
have to be opened to allow access -- but they shouldn't allow access
for the whole Internet.)

Do they understand how RTP ports are allocated? (There's no standard
defined set of RTP ports.)

Do they understand how SIP phones acquire their configurations? (Do
this wrong, and you may be exposing all of your customers to fraud and
interception of calls.)

Do they understand how Session Border Controllers work, and what
traffic they should allow through?

Do they understand how SIP authentication works? (Get this wrong, and
you might be giving away free phone service.)

MetaSwitch "Innovators" and BroadSoft's "Marketplace"

Last week, MetaSwitch (part of Data Connection Limited, DCL) launched the "Innovators" site. It's similar in spirit to BroadSoft Inc's "Marketplace". They're both intended to spur "Client Development".

Both are major VoIP software companies. MetaSwitch also handles hardware. Both keep their ordinary documentation locked in customer-only web sites. But both have released just a tiny bit of their documentation through these sites -- documents about "client" application development.

What's a "Client" for BroadSoft and MetaSwitch? What's it good for?

Both MetaSwitch and BroadSoft make call control systems. They're "VoIP Softswitches". They replace "traditional" telephone switches, like the Lucent 5ESS or Nortel DMS. A telephone company would buy BroadSoft's BroadWorks, or buy a MetaSwitch system, then connect customers to it. As a customer, you'd pick up your phone and make a phone call, and BroadWorks or the MetaSwitch does all the work necessary to connect your call to the person you called. They also handle things like Voicemail, and fancy features like find-me-follow-me routing.

They both have software interfaces to write "Client" applications. Carbon-12 was a company that wrote a BroadWorks client "toolbar"; it let you dial calls, answer calls, transfer calls, etc. from a toolbar installed in MS Internet Explorer. BroadSoft bought them and now it's called the "BroadSoft Assistant". MetaSwitch has also developed a similar client for their system.

Both BroadSoft Inc and MetaSwitch/DCL apparently believe that there's a market for other clients. And to help make their case, my company did develop a Mac client for BroadWorks, called Attache.. We did it mostly because we had plenty of time in the slowdown of 2008, and we're a desktop-Mac OS X shop. We wanted a client for ourselves. Attache is a lot like the BroadSoft Assistant (toolbar), but it runs on Apple Mac OS X.

BroadSoft's Marketplace / Developer Sites

Marketplace has been going strong for a while. It has several applications posted on it, including Attache. There's a sister site, developer.broadsoft.com that has documentation. They have some of BroadSoft's documentation -- but not all:

• OCI-P -- The Provisioning Interface
  
• OCI-C aka CAP -- The Call-Control Interface (used by BroadSoft Assistant, ADP, Attache, etc.)
  
• Xtended Interfaces, aka XSI -- The new HTTP-based "REST-ful" interfaces for call control. Unfortunately, the XSI requires a new server that most BroadWorks service providers don't have (yet), but should.
  

In addition to the Developer and Marketplace sites, BroadSoft runs its customer portal, Xchange. That site has the actual documentation and software patches used by Customers. All three of those sites (Developer, Marketplace, and Xchange) have discussion forums and question-and-answer capabilities, but BroadSoft customers know that the fourth site, ExtraView, is where you actually ask real support questions to get official answers from BroadSoft Inc.

BroadSoft is eager to see new applications built against its platform. They've announced a contest with cash prizes to encourage developers to build applications.

MetaSwitch Innovators

The new MetaSwitch Innovators site started just last week. It also has limited documentation:

• CommPortal Customization Information
  
• Call Control API Information
  

Like BroadSoft, MetaSwitch has a separate site, the MetaSwitch customer portal, where the real documentation lives.

VoIP System Agnosticism

In addition to BroadSoft and MetaSwitch, Asterisk is the elephant in the room. Sure, it's not "carrier grade", but carriers are using it -- the same way Carriers and Enterprises were using Linux in 1995 before it was "enterprise ready".

Many open-source database-driven applications are built to run on different databases. For example, Request Tracker, RT, from BestPractical runs on MySQL and Oracle. By similar logic, any software developer with Computer-Scientific dignity will shriek with joy to build new layer of abstraction over a clever new service. By building their tools to span VoIP platforms, a developer can market his tool to a wider market. MetaSwitch, BroadSoft, and Asterisk all have some external call-control logic. So if there's a good tool that someone can develop for one platform, they should be able to port it to others.

Where's all this going?

Only So Many Click-to-Dial Systems Needed

Most of the client applications are effectively click-to-dial tools. For example, there's a nifty way to use Salesforce.com and a BroadSoft BroadWorks account to place calls to people in your contact list there. But you can also do neat things with incoming calls: For example, ADP Inc, the Payroll Services and Car-Dealer Services company, has integrated both BroadWorks and Cisco Call Manager into their Dealer Management System. When a car-buying prospect calls a car dealer using that software and the right phone service, the salesman receiving the call can automatically jump to the information about that person calling.

The world only needs so many desktop phone control tools like Attache or BroadSoft Assistant. The real question is: what other interesting things can people do, in addition to desktop clients?

Already Other Ways To Control Calls

VoIP lets you connect your computer to the PSTN; to make and receive telephone calls. We already have SIP for doing this. For "serious applications", like Call Centers or Call Recording, you typically see external application servers that function as SIP Back To Back User Agents (B2BUAs).

What do the Client APIs add on top of SIP?

The clients add Third Party Call Control (3PCC); i.e., in addition to the two ends of the conversation, the client can manipulate the call -- placing the call initially, putting it on hold, transferring, etc.

And the 3PCC is without the use of SIP. I.e., I don't have to implement a full SIP stack, and function as a B2BUA, to control the calls. The Call Control APIs try to reduce the accidental complexity of controlling calls.

Fundamental Human Limitations on Client Applications

In what circumstances is it appropriate for a system to manipulate the call? Both sides of the conversation have to be OK with the call manipulation; either by being unaware of it (as happens when a call is dialed), or by causing it (e.g., by clicking the "Transfer to Bob" button), or by understanding that it will happen automatically (such as happens with a call center).

FAIL: Isolated Development Engineers

I work with numerous telecommunications equipment providers who
isolate their Development Engineers from customers -- and sometimes
even from their internal staff. The Wall of Isolation becomes evident
when you need to really understand how something works precisely.

All I want is to talk to somebody who can read the source code and
tell me what it's intended to do:

A major VoIP software developer with engineers in Montreal seems to
have a barrier between the US-based support staff and the Canadian
developers. Their professional services folks are pretty smart, but it
can be really quite difficult to get somebody to tell you how the
system is *supposed* to work in certain circumstances. And it's nearly
impossible to talk to somebody who can even view the source code. FAIL.

A VoIP software/equipment company based in Boston has a similar
problem; when trying to track down specific functionality, we've been
told by an employee that they dare not bother the Engineering people
who really understand it all. FAIL.

Another VoIP equipment company, based in Texas has a similar problem.
We've gotten the impression that the people who know the details are
really just too busy to describe precisely how it works. FAIL.

But it doesn't have to be this way:

A significant VoIP equipment company with offices in the Alameda seems
to have a different approach. It *is* possible, at times, to get help
from the developers. And there's an appreciation -- even at "lower
levels" of support -- for understanding *precisely* how the system
works. The support managers at this company are Software or Computer
Engineers, for the most part.

A major networking equipment company based in California sometimes
doesn't fail: their support engineers have access to the source code,
and occasionally they can really tell you precisely how it's supposed
to work.

The War over DNS and VoIP

Many people don't like using DNS to route VoIP traffic. I suspect other distributed applications have similar questions. They've had too many problems with DNS being reliable, or too slow for real-time call processing. In addition, many VoIP carriers use private IP addresses for their servers! This breaks Internet Engineering principles, and severely complicates name-to-address lookup services.

So instead of using DNS, they usually go with one of these options:

(a) Don't use DNS at all; instead, just do everything with IP addresses. The upside is that there aren't and DNS or directory lookups. The downsides are that they lose a level of indirection, which makes the system less flexible for reconfiguration. They also lose the mnemonic that makes the system manageable; traffic is now going to "4.2.2.2" instead of "ns1.level3.net"

(b) Use /etc/hosts-type name lookups; i.e., they have an OS-level or application-level lookup service that maps names to IP addresses. The problem here is most name-to-IP-address lookups only support the equivalent of a single DNS A record. They can't do sophisticated records, like MX records, or SRV records.

Today, I learned that some folks are using a third option in a leading VoIP Carrier Softswitch system with a Quebecoise accent:

(c) "namedefs", a fake DNS zone file integrated into the application itself. This "namedefs" file vaguely resembles the DNS zonefile, but doesn't include an SOA (Start of Authority) section. It supports more complex lookups, like SRV, but doesn't seem to support MX, and has an extremely restricted syntax. The application consults this file before accessing the

The "namedefs" is a new idea. It gives the application the power of certain fancier lookups, but it can contradict DNS or /etc/hosts lookups. It's only used by the VoIP Softswitch application, so it's possible that the application will use these lookups, while another component will use DNS or /etc/hosts to do related lookups.

Why duplicate the functionality of DNS? Network Politics, probably. The DNS servers are usually operated by the ISP / Data group, and the VoIP softswitch is operated by the Switch Engineering and Translations groups. Fred Brooks predicted this in his book, "The Mythical Man Month: Essays on Software Engineering", first published in 1975. The organization of Software Systems will match that of the Business Organization that created them.

Modern VoIP Networks are large software systems and fit this model neatly. The sad thing is that DNS already provides a mechanism for this organizational boundary: DNS zones and delegation.

I haven't had time to analyze this new approach in depth. But my hunch is that it's not a good step. Rather than bypassing or avoiding DNS, Engineers should have the courage to make DNS work properly --

1. Setup dedicated, redundant DNS servers for important applications.

2. Use DNS zone delegation to give complete control over important applications.

3. Avoid breaking Internet Engineering rules: VoIP servers should have public IP addresses, and be integrated into the global DNS hierarchy, even if they're protected by firewalls.

4. Learn how to manage zone files safely.

Black-Box Testing Fring for SIP and SkypeOut

"Fring" is a Instant Messaging and VoIP client for the iPod and iPhone Touch. It's a free download, and I decided to try it.

Fring Setup

When you start Fring, you have to register for a Fring User ID -- before you even enter your IM, SIP, or Skype contact info. That suggested to me that it was doing this stuff through a central server.

I setup my MSN, AOL, and Yahoo IM contacts and sent a few IMs. It sets your IM status to "Mobile with Fring".

Configuring VoIP

Then I setup my Skype account. I bought some SkypeOut credit. But I was never able to get it to place a call outbound. I deleted my Skype login on Fring, then tried to set it up again -- but it wouldn't let me login.

I also configured a SIP account with a VoIP carrier, Vwave.net, an NGTelecom Company. NGTelecom is a sister company; they're a very modern SIP VoIP provider: they have an Acme Packet SBC and DNS SRV records. I entered my SIP username, SIP authentication password, and the Vwave domain, "vwave.net".

The Fring system did a proper DNS SRV lookup, then registered through the NGTelecom Acme Packet SBC. I noticed that it registered from 63.215.199.71, an IP address assigned to CWIE, LLC of Tempe, AZ.

The CWIE IP Address did not stay registered after I left the Fring application on my iPod Touch. In fact, when I restarted Fring, it wasn't always able to re-register; the progress spinner would keep on spinning.

Call Quality Problems

While Fring was registered, I placed several (maybe 6) calls. Two (2) of them carried intelligible speech. In a couple of cases, I heard weird feedback, on my iPod Touch, before the call was even connected. But I don't know of any similar examples from The Cisco "Duck Quack" page of Voice Quality Symptoms. In some of the call cases, I had one way audio. (Vwave's Acme Packet SBC generally solves all NAT problems for moderately-well-behaved SIP clients, but symmetric RTP is required if the RTP flows through a NAT device or stateful firewall.)


Comments and Conclusions

I was hoping for a SIP client that actually runs on the iPod Touch. Fring apparently sends the call across the Internet, so that means your SIP proxy has to be reachable from the Internet.

Selling more than you have for sale

Occasionally I get to observe a client who has made a mistake; sometimes they come to us to help clean things up.

One such mistake is selling something that you don't have to sell. There are two forms of this mistake: the "Maximum Scale Error", and the "Unknown Cost Error".

Maximum Scale Error. Suppose you have a service, like Shared Call Appearance (SCA), also called Shared Line Appearance (SLA). This feature allows one phone to appear to have extensions for many other lines just like an old Key system. So you press one button to pick up "line one", and another button to pick up "line two", and you have the same buttons on several difference VoIP phones in your office. The mistake is assuming that you can reasonably expand that to any number of lines. For example, if you have a 10-button phone, you might assume you could share 10 lines across all those phones. 

This is a simple naiveté: while some of the technology scales arbitrarily large, there are practical limitations on current implementations. In this case, a SIP message can carry arbitrarily large numbers of shared lines. But SIP over UDP can only handle a few, and UDP is what is often in use.

To avoid this error, you have to actually prove what the limits are. Are two SCA instances supported? Are five SCA instances supported? Are ten? In the case of SCA, there are actually two dimensions: (a) number of SCA lines, and (b) number of phones involved in the SCA line. The number of SCA lines relates to the maximum size of the messaging sent to any one SIP phone. But if you have only one SCA instance shared among 100 phones, then there's another scaling problem: the signaling cost of notifying all 100 phones may overwhelm the link connecting those 100 phones to the Application Server.

How do you prove the capability? The simplest form of proof is analysis. In this example, that's estimating signaling requirements network capacity/bandwidth. But these are also complex systems, and it can be difficult to know that  you're accounting for every component in your analysis. Thus, you should also prove by testing the scale you wish to sell in advance.

Unknown Cost Error. Let's extend the example of Shared Call Appearance, and you've proven that sharing a line across six (6) SIP phones works just fine for customers connected via T1. Let's start selling! But then your Application Server (AS) / Call Agent (CA) or Session Border Controller (SBC) starts to have overload problems. What happened?

It turns out that SCA incurs a high cost on the system because all of the simultaneous messaging that must occur: every phone in the group is sent a SIP INVITE, and also SIP NOTIFY messages about the status of the other phones in the group. In a study with the popular VoIP platform BroadSoft's BroadWorks, placing single telephone call using SCA with six phones can create a burst 2.4 Mbps in SIP messaging. Using SCA in this call, a single telephone call can consume roughly six times the server/SBC capacity of a single "ordinary" phone call. Suddenly, your server capacity has dropped by 87% (since each call uses 6x capacity of one call, you have only 1/6 server capacity remaining.)

The naiveté here is not knowing how much the feature would cost, but selling it anyway. To continue the example, if you sell SCA heavily and thus "fill up" your system sooner than expected, will you have the revenue to upgrade or augment your system?

Care and Feeding of Pet Ideas

Thorbjoern Mann's book, "Time Management for Architects and Designers" has a lot of good advice that applies to people doing network, system, or software design! No, I don't design physical structures or objects, but it's still design. There's always more than one way to satisfy a goal. And making Design Decisions that harmonize to create a good system or network overall is important.



He makes a great point about Pet Ideas. A "pet idea" is a way of solving a problem, or designing the system, and you have an emotional commitment to it. The danger is that you won't examine other options, because you'll be defending your pet.

For example, suppose you're designing a system to manage IP Phones. These devices -- like the Polycom SoundPoint IP SIP phones, Cisco, or Aastra phones -- all download their software and configuration from server, usually using FTP. These phones are Customer Premise Equipment, CPE, so we call it a "CPE Management" server, or "CPEM" server.

Suppose you need to separate one group of phones from another; for example, some phones belong to a legacy platform, but some belong to a newer platform. Normally, all the phone configurations and software are grouped together in one place, but you need to separate them. The first idea that comes to mind is to build a new CPEM server, then have all the new-platform phones talk to the new server.

This can easily become a pet idea; you might feel the need to defend it, and to point out deficiencies in alternatives. The danger is that another idea -- say, establishing a new FTP login and new file directory for the new phones -- might be much easier. But if you're committed to the first idea, you might not really consider any other ideas.

I get automatically attached to the first idea I have. But this really isn't healthy, because I spend too much time considering the advantages of that idea. I really need to consider both advantages and limitations for this idea -- and others too.

Dr. Mann recommends that you come up with at least two ideas, so that you have something to consider.

On the CPEM server, I've already mentioned a second idea. Here's a third: instead of a different server, or a different login, you a different file transfer protocol. Simply retain the use of FTP on the old phones, and HTTP on the new phones. This means you can keep the same login credentials, but use a different service on the server, and therefore a new directory.

What is "Enterprise Quality Video"?

Psytechnics has a troubleshooting product that sounds neat. Expensive, but pretty neat.

Their marketing folks have sent me this email, announcing they'd teach me how to ensure "Enterprise-Quality" Video and Voice.

What is "Enterprise Quality"? To be honest, most "enterprises" I know do a lot of business over cell phones. Is cell phone quality what we're aiming for?

And even if we accept that "enterprise quality audio" means some standard of goodness related to phone calls, what is "enterprise quality video"? Most of the video teleconferencing out there is Skype -- Grandma Shirley talking to Grandson Oren. At 320x200, and 15 fps.

If they had said, "broadcast quality video", I would have a better idea about what they meant.

Writing Detailed MOPs, and the Distinction between Planning and Doing

Service Providers often want a detailed Method of Procedure (MOP) for any change in their network. Some service providers, like Level(3),
have Engineering people plan the procedure, while Operations people actually do the procedure.

This does encourage careful planning. But sometimes things go wrong;
most MOPs have a back-out procedure, so that any changes can be reversed.

But wouldn't it be better if the change could be made in a planned way, but minor problems be resolved when doing the change? When a
surgeon cuts you ope, he has a plan. But if he gets inside and discovers complications, he'll often try to actually fix the problem.
"There were complications, so the surgery took longer."

Let's step back from big organizations that have all these people and
procedures. In smaller, looser, MOP-free organizations, smart people
do the changes. They often login -- and figure out precisely what
needs to be done while they're logged in. If they run into
complications, then they fix the problems then and there. There's
something attractive about just getting the problem solved.

But then the smart people get burned: sometimes mistakes happen and
create outages. Or, the change appears to work fine, but then the
problems erupt later because of the change. So one force at work is
that the stop-talking-about-it-and-fix-it approach sometimes leads to
outages. The technicians implement change control; for example, they
only make their changes at night.

But another force is this: what if the technicians break things, but
the managers have to take the calls from angry users? The managers
start to implement change control. Another form of change control is
that they want to approve every change, because they don't want to
suffer for the outage caused by the technicians. It seems this kind of
control sometimes prevents problems from being fixed in the
maintenance window.

Change Control systems seem to have these dimensions:

-- When (what times and dates) changes are made

-- Who plans the changes to be made

-- Who approves the changes that are made

-- How the changes are recorded or logged

-- How the changes are tested to ensure that they worked properly.

-- Degree of latitude the changers have to work around complications

On Organizing People and Work at a VoIP Service Provider

VoIP service providers these days face the technical challenges of huge flexibility, and no single integrated solution with interop-tested partner devices. You can't just buy a "switch", plug in some TDM/SONET transport and turn up "smart remotes" made by the switch manufacturer.

Even integrated VoIP systems like MetaSwitch leave a lot of design space:

-- What signaling protocols?

-- Which of the many packet networks will be used (Ethernet-only, DSL, Mix of IP transports)?

-- How will high-performance network capacity be ensured? (prioritization? reserved bandwidth? Classified by DSCP? Port number?)

-- How will unauthorized access be prevented? (SBC? Firewall? No default route?)

-- What CPE will be used? And which features will be used? Which signaling variants will be used for, say, putting calls on hold?

-- How will you detect faults? Dry alarm contacts? End-to-end testing? SNMP traps? SNMP polling? Threshold/Statistical-norm monitoring?

My point is that there's a lot to it. Once you outgrow trivially small, you need to split up the work.

I've been brainstorming about ways to organize people to design and operate a VoIP carrier. If you need more than one person, then there's a question: how do you divide work among the people?

This reminds me of Fred Brooks's essays on organizing the development of large software systems. Voice network design operation is somewhat more constrained than are large software systems. But it's still important to have overall coherence.

I wonder how carriers would think about this if VoIP were delivered in a few 84-inch-tall cabinets with opaque slide-in cards? I.e., what if it looked like a 5ESS or DMS?

But VoIP isn't built that way. A cabinet at a VoIP carrier might have equipment from half a dozen independent vendors. And the carrier has to deal with them separately.

Below are some models I've considered. I evaluate the models with these factors in mind:

(a) Does any ONE person within the technical team claim responsibility for correct operation of the whole product/service? This is critical.

(b) Is there normally enough work to keep a person busy? ...intellectually stim-yoo-lated?

(c) Can the model accomodate staff turnover, vacation, around-the-clock emergency work?

(d) Is authority to design the system and fix problems vested in one group, or spread out all over the place?

Though competent engineers should be able to run the whole platform, these tasks are fairly easy to factor out of the engineering group:

(a) Watch the NOC screen and determine when something seems wrong

(b) Take tier-1 customer calls/complaints

(c) On-site installation

One thing I don't consider here is busywork-avoidance: how do you ensure engineers spend their time doing important things? Because people are really good at thinking of minor enhancements and seeking input. For example, if I'm allowed to spend three hours writing a blog article, it might end up really long and thorough, but it might only be 20% better than the one-hour article. But imagine if I'm given all day to write it! It might be a full 30% better than the one-hour article. But it WILL be better.

If this tendency to comprehensiveness is allowed to run unrestrained, engineers will never get a product off the ground. There's always a new way to look at the problem and build the system.

Alternately, we might have a reasonable understanding of the problem, but one more group conference call might make things slightly better.

I call this work that doesn't lead to significantly better results as "busywork". And I don't really address it directly in the organizational structures described here.

------------

The Integrated Model: everybody in the team can do everything. There's a senior-most engineer overall, but he can delegate any task to any member of the group. Some members have more experience on a specific topic. The compensation structure encourages teaching each other, but encourages doing the work yourself even more.

Advantages:

-- Every engineer understands the whole system, and can work on any problem

-- Often intellectually satisfying to engineers

-- Ensures whole team stays up-to-date and no member becomes obsolete

-- Senior-most engineer has overall responsibility for a working design

-- Work scales as overall workload increases

Limitations:

-- Requires fearless, hard-working staff

-- Strong leadership required to avoid over-specialization

-- Long training process

-- Engineers might feel like jacks of all trades, but masters of none.

-- Certain tasks have to be arbitrarily assigned. E.g., who will monitor for critical software updates? Otherwise these tasks might go un-done.

-------

Layered model: Responsibility is broken into layers mimicking the network. Somebody is responsible for layers 1-3 (cables, switches, transport, routing), while somebody else is responsible for layer 4 (operating systems), somebody else does the application layer.

Advantages:

-- The top-layer application engineer has responsibility for the overall product.

-- Exploits deep expertise in specializations. E.g. The network guy understands networking regardless of who manufactured the device; the application guy knows RTP codecs and how the whole real-time audio process works.

-- Encourages cross-platform experience. E.g. SIP guy knows SIP across all platforms

-- Once the layers are divided up amongst the staff, the responsibilities are well defined because they match the software/system responsibilities.

Limitations:

-- The top-layer application engineer may not have knowledge/resources to troubleshoot all problems, since he's specialized in only one area.

-- Some layers won't have enough to do, while other layers will be overworked. So this model may only make sense if you have a large group so the slowest job can keep busy.

-- ...but, it's not clear how to divide work as you scale up. E.g., what if there's too much work for one the application-layer guy?

---------

Device model: each physical object is "owned" by somebody, who's responsible for everything on that.

Advantages:

-- Neatly fits asset ownership within an organization (maybe)

-- Engineers can have deep experience in their devices. E.g., Cisco cert here, Nortel cert there

-- Easy to point fingers...I mean, divide responsibility...corresponding to physical interfaces on the devices.

Limitations:

-- Wastes expertise. e.g., if one guy owns a cisco router and another owns a nortel router, you've got two people with half a job with largely overlapping experience, probably bored with their job

-- Nobody really understands the whole product

-- Easy to point fingers...I mean, divide responsibility...corresponding to physical interfaces on the devices. "The packet is leaving my box...it must be on your end."

---------

The SME model: the system is broken into specific specialties by subject matter, and individuals are assigned to be experts in those areas. SM's might include: SBC, App Server, Net Server, Billing, Provisioning, Client Call Control, PSTN access, SIP, MGCP, fault detection, etc.

Advantages:

-- Promotes deep expertise in the subject matter

-- Matches an academic model of divide research topics

Limitations:

-- Very weak on design -- no engineer has overall design authority or comprehension

-- SMEs may know their area in the abstract, but have trouble with the overall application

-- Being an SME may not be enough to keep a person busy

----

The Knuth model (SME-graph): Like the SME model, but each specialty has at least two experts, and each expert has at least two specialties. The purpose is to ensure all the specialties are connected, so it's best to require dissimilar expertise.

Advantages:

-- Overall system is owned by the whole team, so decisions aren't made in isolation

Disadvantages:

-- No single technical mind responsible for overall product operation and uniform design. This may result in an overly-complicated, fragile system.

---------

Network Region Model: The network is divided into segments, and engineers are assigned regions. Then they do everything within their region. E.g., one engineer might handle the customer access network, another handles the outside of the SBC (i.e. the core network part connected to the Internet), and another handles the inside of the core.

Advantages:

-- Engineers work across functions within their network region, e.g., physical layer, switching, routing, application, performance, etc.

Limitations:

-- Nobody technical owns the whole product