Want a job in VoIP? Consider BroadSoft Training

BroadSoft Inc's BroadWorks platform is undoubtably one of the leaders in VoIP platforms out there. They've created two certification programs: BroadSoft Certified Platform Administrator (BCPA), and BroadSoft Certified Application Administrator (BCAA).

They're also using Ziiva, aka Prosperty Learning Management Systems (LMS), to host a site http://certification.broadsoft.com where you can take the tests and study for the tests.

One of the frustrating things about the voice-telecom industry is that their documentation is typically kept closed, and only available to customers. But why? Documentation from Cisco, Oracle, and IBM is all publicly available; it's not hurting them in their industries. But you'll see companies like Adtran open their data-telecom documentation, while keeping some of their voice-telecom documentation behind a login and password. (I consider documentation to be "open" if you can get it without being a customer of the company.)

BroadSoft's certification site bucks that trend. The courseware is free and open. If you're trying to break-in to carrier-grade VoIP systems, studying the free BroadWorks documentation on their certification site is a good way to learn.

And if you're actually willing to do the work to read and learn that material, my hat's off to you. This industry needs people who are willing to crack open a PDF and do some studying.

Nortel Networks Bankruptcy and the CS2000 / CS2K: Finding Alternatives

Nortel Networks has gone bankrupt, and is selling bits and pieces to Nokia Siemens. I'm not sure if the CS2K is one of those pieces.

Nortel has a lot of really smart people. It's difficult to change from charging $16M for a telephone switch servicing 100,000 lines in 1997 (which is what they charged BellSouth for the DMS100 in my home town) to the current marketplace.

There are interesting engineering and technical questions that arise when a company does this. People have long complained about Nortel's support for the CS2K. At least one carrier is rumored to have stopped provisioning new subscribers on their Nortel CS2K as soon as the bankruptcy was announced.

If you need a modern alternative to the CS2K, what are the choices? There are plenty, but a few front-runners:

• MetaSwitch is the first to come to mind because CS2K users often have copious TDM and SS7 interconnections. The MetaSwitch platform can definitely do this, Class-5 and Class-4/tandem type features.
  
  
• BroadSoft may be a fine choice if you were using the CS2K primarily for Class-5-like features, and you have some other way of interconnecting SIP to the PSTN.
  
  
• Alcatel-Lucent has the old Telica Plexus 9000 platform (not to be confused with the Plexus 8000). I think the name-of-the-month is the "Alcatel-Lucent Network Gateway". It's a nice box, but somewhat difficult to troubleshoot. (Especially if you don't have the super-secret list of debug commands and logging that you can run on the individual CPU and VSM cards.) Still, if you're looking for solid SIP-to-PSTN interworking, it's a good option to consider.
  
  
• Taqua has a platform worth considering.
  
  
• You might consider Cisco because of the PGW or BTS10200. These come across as pretty weak players, compared to other folks in this list. If you're interconnecting the VoIP world with the PSTN, then you can use AS5400s and have great joy. (Just be sure you add access-lists to control the paths SIP can take. IOS will accept a call from anybody!)
  
  

Three Rules for Effective Naming

1. ONE OF THE RULES that has served me well is this: things should have unique names so I can distinguish them. Filenames are easy examples; I've noticed it's a lot easier if, for output I don't intend to edit, I don't re-use filenames.

Output I don't intend to edit includes things like the output of mysqldump (which dumps the contents of a database) for backup purposes; or the output of wget (as it downloads a web page or site). Or the output of "show tech-support" on a network box.

At the moment, I put a timestamp in the filename; such as "show_tech_support_200906131557".

This raises some questions:

(a) Can't we depend on the filesystem for this? After all, every filesystem has time and date stamps. BUT: Files don't just live on filesystems: they're often sent through email, or attached to ticket systems on web sites.

(b) What timezone should be used? I just use the timezone I'm actually in, but I'm not sure that's "ideal". It *CAN* be confusing when a system (such as a server) is in one timezone and I'm working in another timezone. I probably need a better rule for this case.

2. FILESYSTEM HIERARCHIES ARE GREAT, but unfortunately they're not useful once the file leaves the filesystem (such as when it's sent by email). So if you're file is named "tech-support.txt", you only know the least bit about its content. If you add the timestamp as I suggest above, you get "tech-support_200906131607.txt". It's also smart to add your organization to the name if it's going to be crossing organizational boundaries, such as when you email the output to a vendor. It's also smart to include the name of the system that generated the output.

So you'd end up with "acmecorp_router_1_tech-support_200906131607.txt".

Occasionally, I'll get a complaint that the files named this way are too long. I attribute this to sloth on the part of the complainer, but perhaps other theories may explain the complaints.

3. ANOTHER RULE is keeping a unique tag on datasets introduced to inter-mingled sets. For example, I happened to harvest some MP4 audio files as they were flying by one month, and I added those to iTunes. Later on I decided the quality was too low on those files, so I wanted to delete them all.

Fortunately, I had included an underscore "_" in the song names when I added those files to the iTunes library, so it's trivial to select and delete them all.

Or take a less-trivial case: suppose you're managing SIP VoIP Phones, and you're adding entries to the database for 10,000 of them. The database may already have 5,000 entries. After you've added these entries to the database, you may later need to go back and work with these entries. (Maybe you didn't get it perfect the first time.) In this case, it's convenient to have some marker in the database entries for the SIP phones you added in this way. If the database has a way of marking the source of the entries explicitly, then that's good to use. If not, then you can sometimes add a tag to the name. At the very least, you can keep keep a list of the individual entries added, which you can use later to access the new data set.

Reclaiming Pragmatic Security: Why Standard Computer Security approaches Stink

The standard approach to computer security goes something like this: "Remember, Job #1 of any security professional is to make sure nothing bad happens. You don't have to be elegant, you just need to get the job done." [1] That statement is from Mike Rothman, the author of the "Pragmatic CSO". This is a book to explain to security techies how to function in a business environment.

But this statement is hogwash! I haven't read the book, but Rothman seems to be teaching you how to (a) feel influential because managers are asking you questions, and (b) make project plans and business cases for security equipment purchases. The premise has some truth: computer-system security specialists within companies need to function like grownups. But his statement "make sure nothing bad happens" reveals a misperception of any role in business.

Security always has TWO sides [2]:

(a) Safety: Be sure bad things DON'T happen.

(b) Liveness: Be sure good things DO happen.

But these ideas are not well known outside the formal Computer Science Research community.

There's an old joke: the best firewall (network security device) is the air-gap between a network cable and the port. The joke is funny because so many security people actually function as if their real job is to ensure nothing bad happens.

This absurdity is promulgated by Rothman's statement. Its practice creates the crazy situation in which the security technicians -- Chief Security Officer (CSO), IT technicians, etc. -- work *against* getting Good Things Done, leaving the rest of the business to work in favor of getting Good Things done.

Cisco published a relevant study in 2008:

"One of the most significant findings was the difference in employee and IT perspectives on policy non-compliance. According to IT, employees defy policies for a variety of reasons . . . [E]mployees said the top reason for non-compliance is their belief that policies do not align with the reality of what they need to do their jobs. More than two of five employees (42 percent) made this claim globally. In Germany, even though the majority of employees felt their companies' policies were fair, more than half of them (55 percent) said they would break them to complete their jobs." [3]

Of course the security staff feels like the users don't care: the security staff doesn't really care if the users to get their jobs done. It's as if the security staff feel that it's better not to get the job done at all rather than have it done in an insecure way.


PRINCIPLES FOR BEING PRAGMATIC ABOUT SECURITY

I. The computer security insanity has to stop. Being pragmatic about Computer System Security doesn't mean learning how to make fake business plans to buy equipment.

II. Being pragmatic about computer security must include both enabling good things to happen, and preventing bad things from happening.

III. The MAIN job of anybody in an organization is to do the business of the organization; i.e., to get good things done, to provide the service, make the product, satisfy the customer, help the world.

IV. All ventures involve risk; there's always a risk of attack and a cost to defend against it. The probability and costs of many risks cannot be quantified because you don't know what the worm or attacker will do.

V. Security is NOT about buying equipment or software. (Most serious network attacks go right through firewalls. Anti-Virus software is almost never up-to-date enough to stop new viruses.)

VI. Complex system security is sometimes worse than no security. For example, firewalls with 100-line access lists are almost always wrong.

VII. Security Controls that Prevent activity are fundamentally defense against specific attacks. For every security control (firewall, access list, etc.) you have, you should know what you're defending against.

VIII. Realize that Compliance can be a farce; being "complaint" to some standard may mean you're not defending against the right threats.Auditors are often ex- accountants with checklists and no real understanding of systems. Don't rely on policy/law compliance for anything besides satisfying paperwork requirements.



[1] Mike Rothman's 2006 blog post, "'Pragmatic Security' Coming Into View", posted at: http://securityincite.com/blog/mike-rothman/pragmatic-security-coming-into-view

[2] Fred Schneider, "Defining Liveness", http://portal.acm.org/citation.cfm?id=867712

[3] Cisco Systems, "Global Cisco Study Applies Reality Check to Corporate Security Policies, Draws Connection to Data Leakage Risk",http://newsroom.cisco.com/dlls/2008/prod_102808.html

Securing VoIP Carriers: Start with the CPE Management

It amazes me that VoIP carriers continue to make a very simple mistake: they setup an FTP server (which is good) to provide the configurations for their SIP Phones (such as PolyCom, and Aastra SIP phones) (which is also good), but they use the manufacturer's default "PlcmSpIp" username and password on the servers (which is not ideal, but that's debatable). (Google reports 411 web pages documenting "PlcmSpIp" on the Internet right now. 47 of them are published by "site:polycom.com".)

But worst of all, they allow anyone in the world to FTP in using these well-publicized credentials, and then actually list all of the CPE configuration files in the directory using FTP. An attacker can then download these configuration files and:

• Steal phone service by registering using the credentials documented in the configuration file.
  
• Interrupt a legitimate customer's phone service.
  
• In some cases, he may be able to stealthily intercept telephone calls going to and from the user.
  

There are a number of defenses against this type of exploit. First of all, don't allow the PlcmSpIp (or any other user used by your customers) to actually view the contents of the directory via FTP. This makes it more difficult to download files, because attackers cannot directly view the list of SIP phones in your system. Instead, attackers are forced to guess the MAC addresses of the SIP phones in your system. You can then use approximate defenses, such as blocking users who attempt to download too many non-existent config files.

Protecting IP Phone configurations is one key element in a quality VoIP System Security Analysis.

Choosing BroadSoft and MetaSwitch

I regularly spend time working with BroadSoft BroadWorks systems, and with MetaSwitch systems. I rarely have a client that has both, but I have many clients in both camps. Both are winning -- in separate camps. The BroadSoft service providers and the MetaSwitch service providers really don't seem like the same market set.

There have been a lot of competitors -- consider Sylantro, General Bandwidth M6, Lignup, Alcatel-Lucent's Feature Server 4000, and Asterisk, as examples. Some of these systems work, but some of them are pretty flaky. It takes a lot of work to catch up with the features and maturity of BroadSoft and MetaSwitch.

Since my readers are interested in this subject, I'll lay out some thoughts on the strengths of the two platforms.

BroadSoft's Strengths

BroadSoft makes some really strong, reliable software. Making and supporting mature software is an amazing feat. They should really be proud of their software platform, and the new features they've added.

It seems BroadSoft's customers (service providers) want to sell sophisticated features; they want to replace the office PBX. BroadSoft customers ask questions like, "do you have the features to replace a hotel PBX?" or "can we use BroadWorks to provide roaming features to users homed on another Class-5 switch?"

Many of BroadSoft's customers have in-house programming teams to handle the CDRs. Many of them work with outside SIP peering partners, like Level(3), Global Crossing, and Dash. Some of them have a sophisticated telephone gateway capable of SS7/ISUP and SIP, but some of them just use ISDN PRI gateways, like the AudioCodes or Cisco IOS gateways (AS5400, e.g.). BroadSoft doesn't really care how you connect to the PSTN.

BroadSoft Inc doesn't have a technical conference per se; they have an "Executive User's Forum", called Connections. You don't go to BroadSoft Connections to talk shop about telecom primarily -- you go there to learn about how to market your services and grow your business. You're generally supposed to get technical details directly from your salesman and your sales engineer separately. (They will answer technical questions and talk about some details -- but the purpose of Connections is all about networking -- in the human sense.)

BroadSoft operates a TAC (Technical Assistance Center); they prefer to interact primarily through their ticketing system, ExtraView. You may not get the same TAC engineer twice. If you have a general question, it's sometimes best to work through your sales team. If you have an outage, the TAC works very hard to pull in engineers who can help understand things quickly.

BroadSoft has the "Xchange" site where people can post questions. Most questions will receive a reply from a BroadSoft employee. In my experience, BroadSoft customers don't tend to interact a lot on the Internet. It might be because so many of them consider themselves to be nationwide players, selling service all over the US. And if there's a discussion group full of your competitors, you're not likely to talk about your questions and concerns.

BroadSoft has a really neat, simple to use voicemail platform built right in. I like it a lot; it's integrated well, and it can use your existing email account as a voicemail store.

BroadSoft's system runs on Linux or Solaris servers, so you need Unix system administration skills.

BroadSoft's documentation is split into many hundreds of small files; it's definitely improving over the past few years.

BroadSoft has some very smart business people at the top.

MetaSwitch's Strengths

MetaSwitch, too, makes some really strong software. And they build some really strong hardware, too. They have some really smart people developing and maintaining their software, and providing customer support as well. MetaSwitch has many sophisticated features; they don't sell every PBX-replacement feature that BroadSoft offers, but they sell many of the major features in the new SDP platform.

MetaSwitch's customers are often replacing Lucent 5ESS, Nortel DMS, or other SS7-capable telephone switches. MetaSwitch customers often ask questions like, "can the MetaSwitch support point code proxy?" and "can I split an ISUP trunk group across gateways at two different locations?" and "how many DS3s can I get in a gateway?" MetaSwitch tries very hard to provide PSTN switch and gateway features that telephone companies are accustomed to using.

Because MetaSwitch provides the network gateways, they do a lot with the audio (RTP). For example, MetaSwitch can directly perform echo cancellation for calls that traverse the gateway. (And, in fact, you could send SIP-to-SIP calls through their gateway.)

MetaSwitch has a User Forum; they talk a lot about their product, enhancements, and features. They talk about marketing your services a little bit. Many of the technical people from MetaSwitch are there.

MetaSwitch customers often rely heavily on their Customer Support Engineer, CSE. The CSE is directly assigned to a service provider, and he'll remember the problem you were discussing earlier. In my experience, you can have an informal conversation directly with the CSE. Beyond the CSE, MetaSwitch some extremely sharp CSE managers; the overall support team is incredibly bright and well educated.

MetaSwitch has a very lively customer forum email list, where people talk about their technical problems. It's probably in part because so many of MetaSwitch's customers are regional carriers who only serve one area of their state, and they don't feel threatened by other carriers.

MetaSwitch has a voicemail platform; as I understand it, it's in use by some major carriers who have hundreds-of-thousands of voicemail boxes. They scale it way down for smaller folks.

MetaSwitch runs on Solaris, but you really don't need to know that. They expose separate interfaces for your normal maintenance.

MetaSwitch's documentation is bundled into about 10 files, and is reasonably good. (Note: this is high praise. Most technical documentation is a waste of everyone's time to write and to read.)

MetaSwitch has some very smart Engineering people at the top of their business.

How do you choose?

Do you need an SS7-capable gateway?

Do you need voicemail for every subscriber?

Do you have Unix system administrators?

Do you expect to integrate the platform with your provisioning system?

Do you need to have multiple people modifying translations/routing at the same time?

Both of these are remarkably capable systems. They do have their strengths, and their target markets. Of course, BroadSoft and MetaSwitch would each like to have all of the customers, but since BroadSoft doesn't sell media gateways and SS7-capable equipment, and MetaSwitch doesn't try to provide every PBX feature known to man, they're really not direct competitors in my mind.

VoIP-NSP Mailing list

There's no general discussion for people using VoIP in a Network
Service Provider forum, until now.

http://voip-nsp.e-c-group.com/

VoIP-NSP: for people using VoIP in a Network Service Provider
environment. These should include people using BroadSoft BroadWorks,
Acme Packet, MetaSwitch, Sylantro, VocalData, General Bandwidth M6,
Alcatel-Lucent, Asterisk, and anybody else interested.

VoIP Network Service Providers have stringent network requirements,
and unusual requirements.

VoIP Security is a Different Animal

A lot of Network Security Expert grew up protecting conventional
enterprise applications. Clients are PCs. Servers are web servers, or
run MS Exchange. Maybe they get involved in web proxies. Perhaps they
have to create a special rule for Microsoft SQL Server.

VoIP Security is entirely different. The ports, connections, services,
and behavior are completely different than conventional PC services.

Does your Security Consultant understand SIP or MGCP? (Certain ports
have to be opened to allow access -- but they shouldn't allow access
for the whole Internet.)

Do they understand how RTP ports are allocated? (There's no standard
defined set of RTP ports.)

Do they understand how SIP phones acquire their configurations? (Do
this wrong, and you may be exposing all of your customers to fraud and
interception of calls.)

Do they understand how Session Border Controllers work, and what
traffic they should allow through?

Do they understand how SIP authentication works? (Get this wrong, and
you might be giving away free phone service.)

MetaSwitch "Innovators" and BroadSoft's "Marketplace"

Last week, MetaSwitch (part of Data Connection Limited, DCL) launched the "Innovators" site. It's similar in spirit to BroadSoft Inc's "Marketplace". They're both intended to spur "Client Development".

Both are major VoIP software companies. MetaSwitch also handles hardware. Both keep their ordinary documentation locked in customer-only web sites. But both have released just a tiny bit of their documentation through these sites -- documents about "client" application development.

What's a "Client" for BroadSoft and MetaSwitch? What's it good for?

Both MetaSwitch and BroadSoft make call control systems. They're "VoIP Softswitches". They replace "traditional" telephone switches, like the Lucent 5ESS or Nortel DMS. A telephone company would buy BroadSoft's BroadWorks, or buy a MetaSwitch system, then connect customers to it. As a customer, you'd pick up your phone and make a phone call, and BroadWorks or the MetaSwitch does all the work necessary to connect your call to the person you called. They also handle things like Voicemail, and fancy features like find-me-follow-me routing.

They both have software interfaces to write "Client" applications. Carbon-12 was a company that wrote a BroadWorks client "toolbar"; it let you dial calls, answer calls, transfer calls, etc. from a toolbar installed in MS Internet Explorer. BroadSoft bought them and now it's called the "BroadSoft Assistant". MetaSwitch has also developed a similar client for their system.

Both BroadSoft Inc and MetaSwitch/DCL apparently believe that there's a market for other clients. And to help make their case, my company did develop a Mac client for BroadWorks, called Attache.. We did it mostly because we had plenty of time in the slowdown of 2008, and we're a desktop-Mac OS X shop. We wanted a client for ourselves. Attache is a lot like the BroadSoft Assistant (toolbar), but it runs on Apple Mac OS X.

BroadSoft's Marketplace / Developer Sites

Marketplace has been going strong for a while. It has several applications posted on it, including Attache. There's a sister site, developer.broadsoft.com that has documentation. They have some of BroadSoft's documentation -- but not all:

• OCI-P -- The Provisioning Interface
  
• OCI-C aka CAP -- The Call-Control Interface (used by BroadSoft Assistant, ADP, Attache, etc.)
  
• Xtended Interfaces, aka XSI -- The new HTTP-based "REST-ful" interfaces for call control. Unfortunately, the XSI requires a new server that most BroadWorks service providers don't have (yet), but should.
  

In addition to the Developer and Marketplace sites, BroadSoft runs its customer portal, Xchange. That site has the actual documentation and software patches used by Customers. All three of those sites (Developer, Marketplace, and Xchange) have discussion forums and question-and-answer capabilities, but BroadSoft customers know that the fourth site, ExtraView, is where you actually ask real support questions to get official answers from BroadSoft Inc.

BroadSoft is eager to see new applications built against its platform. They've announced a contest with cash prizes to encourage developers to build applications.

MetaSwitch Innovators

The new MetaSwitch Innovators site started just last week. It also has limited documentation:

• CommPortal Customization Information
  
• Call Control API Information
  

Like BroadSoft, MetaSwitch has a separate site, the MetaSwitch customer portal, where the real documentation lives.

VoIP System Agnosticism

In addition to BroadSoft and MetaSwitch, Asterisk is the elephant in the room. Sure, it's not "carrier grade", but carriers are using it -- the same way Carriers and Enterprises were using Linux in 1995 before it was "enterprise ready".

Many open-source database-driven applications are built to run on different databases. For example, Request Tracker, RT, from BestPractical runs on MySQL and Oracle. By similar logic, any software developer with Computer-Scientific dignity will shriek with joy to build new layer of abstraction over a clever new service. By building their tools to span VoIP platforms, a developer can market his tool to a wider market. MetaSwitch, BroadSoft, and Asterisk all have some external call-control logic. So if there's a good tool that someone can develop for one platform, they should be able to port it to others.

Where's all this going?

Only So Many Click-to-Dial Systems Needed

Most of the client applications are effectively click-to-dial tools. For example, there's a nifty way to use Salesforce.com and a BroadSoft BroadWorks account to place calls to people in your contact list there. But you can also do neat things with incoming calls: For example, ADP Inc, the Payroll Services and Car-Dealer Services company, has integrated both BroadWorks and Cisco Call Manager into their Dealer Management System. When a car-buying prospect calls a car dealer using that software and the right phone service, the salesman receiving the call can automatically jump to the information about that person calling.

The world only needs so many desktop phone control tools like Attache or BroadSoft Assistant. The real question is: what other interesting things can people do, in addition to desktop clients?

Already Other Ways To Control Calls

VoIP lets you connect your computer to the PSTN; to make and receive telephone calls. We already have SIP for doing this. For "serious applications", like Call Centers or Call Recording, you typically see external application servers that function as SIP Back To Back User Agents (B2BUAs).

What do the Client APIs add on top of SIP?

The clients add Third Party Call Control (3PCC); i.e., in addition to the two ends of the conversation, the client can manipulate the call -- placing the call initially, putting it on hold, transferring, etc.

And the 3PCC is without the use of SIP. I.e., I don't have to implement a full SIP stack, and function as a B2BUA, to control the calls. The Call Control APIs try to reduce the accidental complexity of controlling calls.

Fundamental Human Limitations on Client Applications

In what circumstances is it appropriate for a system to manipulate the call? Both sides of the conversation have to be OK with the call manipulation; either by being unaware of it (as happens when a call is dialed), or by causing it (e.g., by clicking the "Transfer to Bob" button), or by understanding that it will happen automatically (such as happens with a call center).

FAIL: Isolated Development Engineers

I work with numerous telecommunications equipment providers who
isolate their Development Engineers from customers -- and sometimes
even from their internal staff. The Wall of Isolation becomes evident
when you need to really understand how something works precisely.

All I want is to talk to somebody who can read the source code and
tell me what it's intended to do:

A major VoIP software developer with engineers in Montreal seems to
have a barrier between the US-based support staff and the Canadian
developers. Their professional services folks are pretty smart, but it
can be really quite difficult to get somebody to tell you how the
system is *supposed* to work in certain circumstances. And it's nearly
impossible to talk to somebody who can even view the source code. FAIL.

A VoIP software/equipment company based in Boston has a similar
problem; when trying to track down specific functionality, we've been
told by an employee that they dare not bother the Engineering people
who really understand it all. FAIL.

Another VoIP equipment company, based in Texas has a similar problem.
We've gotten the impression that the people who know the details are
really just too busy to describe precisely how it works. FAIL.

But it doesn't have to be this way:

A significant VoIP equipment company with offices in the Alameda seems
to have a different approach. It *is* possible, at times, to get help
from the developers. And there's an appreciation -- even at "lower
levels" of support -- for understanding *precisely* how the system
works. The support managers at this company are Software or Computer
Engineers, for the most part.

A major networking equipment company based in California sometimes
doesn't fail: their support engineers have access to the source code,
and occasionally they can really tell you precisely how it's supposed
to work.

The War over DNS and VoIP

Many people don't like using DNS to route VoIP traffic. I suspect other distributed applications have similar questions. They've had too many problems with DNS being reliable, or too slow for real-time call processing. In addition, many VoIP carriers use private IP addresses for their servers! This breaks Internet Engineering principles, and severely complicates name-to-address lookup services.

So instead of using DNS, they usually go with one of these options:

(a) Don't use DNS at all; instead, just do everything with IP addresses. The upside is that there aren't and DNS or directory lookups. The downsides are that they lose a level of indirection, which makes the system less flexible for reconfiguration. They also lose the mnemonic that makes the system manageable; traffic is now going to "4.2.2.2" instead of "ns1.level3.net"

(b) Use /etc/hosts-type name lookups; i.e., they have an OS-level or application-level lookup service that maps names to IP addresses. The problem here is most name-to-IP-address lookups only support the equivalent of a single DNS A record. They can't do sophisticated records, like MX records, or SRV records.

Today, I learned that some folks are using a third option in a leading VoIP Carrier Softswitch system with a Quebecoise accent:

(c) "namedefs", a fake DNS zone file integrated into the application itself. This "namedefs" file vaguely resembles the DNS zonefile, but doesn't include an SOA (Start of Authority) section. It supports more complex lookups, like SRV, but doesn't seem to support MX, and has an extremely restricted syntax. The application consults this file before accessing the

The "namedefs" is a new idea. It gives the application the power of certain fancier lookups, but it can contradict DNS or /etc/hosts lookups. It's only used by the VoIP Softswitch application, so it's possible that the application will use these lookups, while another component will use DNS or /etc/hosts to do related lookups.

Why duplicate the functionality of DNS? Network Politics, probably. The DNS servers are usually operated by the ISP / Data group, and the VoIP softswitch is operated by the Switch Engineering and Translations groups. Fred Brooks predicted this in his book, "The Mythical Man Month: Essays on Software Engineering", first published in 1975. The organization of Software Systems will match that of the Business Organization that created them.

Modern VoIP Networks are large software systems and fit this model neatly. The sad thing is that DNS already provides a mechanism for this organizational boundary: DNS zones and delegation.

I haven't had time to analyze this new approach in depth. But my hunch is that it's not a good step. Rather than bypassing or avoiding DNS, Engineers should have the courage to make DNS work properly --

1. Setup dedicated, redundant DNS servers for important applications.

2. Use DNS zone delegation to give complete control over important applications.

3. Avoid breaking Internet Engineering rules: VoIP servers should have public IP addresses, and be integrated into the global DNS hierarchy, even if they're protected by firewalls.

4. Learn how to manage zone files safely.

Black-Box Testing Fring for SIP and SkypeOut

"Fring" is a Instant Messaging and VoIP client for the iPod and iPhone Touch. It's a free download, and I decided to try it.

Fring Setup

When you start Fring, you have to register for a Fring User ID -- before you even enter your IM, SIP, or Skype contact info. That suggested to me that it was doing this stuff through a central server.

I setup my MSN, AOL, and Yahoo IM contacts and sent a few IMs. It sets your IM status to "Mobile with Fring".

Configuring VoIP

Then I setup my Skype account. I bought some SkypeOut credit. But I was never able to get it to place a call outbound. I deleted my Skype login on Fring, then tried to set it up again -- but it wouldn't let me login.

I also configured a SIP account with a VoIP carrier, Vwave.net, an NGTelecom Company. NGTelecom is a sister company; they're a very modern SIP VoIP provider: they have an Acme Packet SBC and DNS SRV records. I entered my SIP username, SIP authentication password, and the Vwave domain, "vwave.net".

The Fring system did a proper DNS SRV lookup, then registered through the NGTelecom Acme Packet SBC. I noticed that it registered from 63.215.199.71, an IP address assigned to CWIE, LLC of Tempe, AZ.

The CWIE IP Address did not stay registered after I left the Fring application on my iPod Touch. In fact, when I restarted Fring, it wasn't always able to re-register; the progress spinner would keep on spinning.

Call Quality Problems

While Fring was registered, I placed several (maybe 6) calls. Two (2) of them carried intelligible speech. In a couple of cases, I heard weird feedback, on my iPod Touch, before the call was even connected. But I don't know of any similar examples from The Cisco "Duck Quack" page of Voice Quality Symptoms. In some of the call cases, I had one way audio. (Vwave's Acme Packet SBC generally solves all NAT problems for moderately-well-behaved SIP clients, but symmetric RTP is required if the RTP flows through a NAT device or stateful firewall.)


Comments and Conclusions

I was hoping for a SIP client that actually runs on the iPod Touch. Fring apparently sends the call across the Internet, so that means your SIP proxy has to be reachable from the Internet.

Selling more than you have for sale

Occasionally I get to observe a client who has made a mistake; sometimes they come to us to help clean things up.

One such mistake is selling something that you don't have to sell. There are two forms of this mistake: the "Maximum Scale Error", and the "Unknown Cost Error".

Maximum Scale Error. Suppose you have a service, like Shared Call Appearance (SCA), also called Shared Line Appearance (SLA). This feature allows one phone to appear to have extensions for many other lines just like an old Key system. So you press one button to pick up "line one", and another button to pick up "line two", and you have the same buttons on several difference VoIP phones in your office. The mistake is assuming that you can reasonably expand that to any number of lines. For example, if you have a 10-button phone, you might assume you could share 10 lines across all those phones. 

This is a simple naiveté: while some of the technology scales arbitrarily large, there are practical limitations on current implementations. In this case, a SIP message can carry arbitrarily large numbers of shared lines. But SIP over UDP can only handle a few, and UDP is what is often in use.

To avoid this error, you have to actually prove what the limits are. Are two SCA instances supported? Are five SCA instances supported? Are ten? In the case of SCA, there are actually two dimensions: (a) number of SCA lines, and (b) number of phones involved in the SCA line. The number of SCA lines relates to the maximum size of the messaging sent to any one SIP phone. But if you have only one SCA instance shared among 100 phones, then there's another scaling problem: the signaling cost of notifying all 100 phones may overwhelm the link connecting those 100 phones to the Application Server.

How do you prove the capability? The simplest form of proof is analysis. In this example, that's estimating signaling requirements network capacity/bandwidth. But these are also complex systems, and it can be difficult to know that  you're accounting for every component in your analysis. Thus, you should also prove by testing the scale you wish to sell in advance.

Unknown Cost Error. Let's extend the example of Shared Call Appearance, and you've proven that sharing a line across six (6) SIP phones works just fine for customers connected via T1. Let's start selling! But then your Application Server (AS) / Call Agent (CA) or Session Border Controller (SBC) starts to have overload problems. What happened?

It turns out that SCA incurs a high cost on the system because all of the simultaneous messaging that must occur: every phone in the group is sent a SIP INVITE, and also SIP NOTIFY messages about the status of the other phones in the group. In a study with the popular VoIP platform BroadSoft's BroadWorks, placing single telephone call using SCA with six phones can create a burst 2.4 Mbps in SIP messaging. Using SCA in this call, a single telephone call can consume roughly six times the server/SBC capacity of a single "ordinary" phone call. Suddenly, your server capacity has dropped by 87% (since each call uses 6x capacity of one call, you have only 1/6 server capacity remaining.)

The naiveté here is not knowing how much the feature would cost, but selling it anyway. To continue the example, if you sell SCA heavily and thus "fill up" your system sooner than expected, will you have the revenue to upgrade or augment your system?

Care and Feeding of Pet Ideas

Thorbjoern Mann's book, "Time Management for Architects and Designers" has a lot of good advice that applies to people doing network, system, or software design! No, I don't design physical structures or objects, but it's still design. There's always more than one way to satisfy a goal. And making Design Decisions that harmonize to create a good system or network overall is important.



He makes a great point about Pet Ideas. A "pet idea" is a way of solving a problem, or designing the system, and you have an emotional commitment to it. The danger is that you won't examine other options, because you'll be defending your pet.

For example, suppose you're designing a system to manage IP Phones. These devices -- like the Polycom SoundPoint IP SIP phones, Cisco, or Aastra phones -- all download their software and configuration from server, usually using FTP. These phones are Customer Premise Equipment, CPE, so we call it a "CPE Management" server, or "CPEM" server.

Suppose you need to separate one group of phones from another; for example, some phones belong to a legacy platform, but some belong to a newer platform. Normally, all the phone configurations and software are grouped together in one place, but you need to separate them. The first idea that comes to mind is to build a new CPEM server, then have all the new-platform phones talk to the new server.

This can easily become a pet idea; you might feel the need to defend it, and to point out deficiencies in alternatives. The danger is that another idea -- say, establishing a new FTP login and new file directory for the new phones -- might be much easier. But if you're committed to the first idea, you might not really consider any other ideas.

I get automatically attached to the first idea I have. But this really isn't healthy, because I spend too much time considering the advantages of that idea. I really need to consider both advantages and limitations for this idea -- and others too.

Dr. Mann recommends that you come up with at least two ideas, so that you have something to consider.

On the CPEM server, I've already mentioned a second idea. Here's a third: instead of a different server, or a different login, you a different file transfer protocol. Simply retain the use of FTP on the old phones, and HTTP on the new phones. This means you can keep the same login credentials, but use a different service on the server, and therefore a new directory.

What is "Enterprise Quality Video"?

Psytechnics has a troubleshooting product that sounds neat. Expensive, but pretty neat.

Their marketing folks have sent me this email, announcing they'd teach me how to ensure "Enterprise-Quality" Video and Voice.

What is "Enterprise Quality"? To be honest, most "enterprises" I know do a lot of business over cell phones. Is cell phone quality what we're aiming for?

And even if we accept that "enterprise quality audio" means some standard of goodness related to phone calls, what is "enterprise quality video"? Most of the video teleconferencing out there is Skype -- Grandma Shirley talking to Grandson Oren. At 320x200, and 15 fps.

If they had said, "broadcast quality video", I would have a better idea about what they meant.

Writing Detailed MOPs, and the Distinction between Planning and Doing

Service Providers often want a detailed Method of Procedure (MOP) for any change in their network. Some service providers, like Level(3),
have Engineering people plan the procedure, while Operations people actually do the procedure.

This does encourage careful planning. But sometimes things go wrong;
most MOPs have a back-out procedure, so that any changes can be reversed.

But wouldn't it be better if the change could be made in a planned way, but minor problems be resolved when doing the change? When a
surgeon cuts you ope, he has a plan. But if he gets inside and discovers complications, he'll often try to actually fix the problem.
"There were complications, so the surgery took longer."

Let's step back from big organizations that have all these people and
procedures. In smaller, looser, MOP-free organizations, smart people
do the changes. They often login -- and figure out precisely what
needs to be done while they're logged in. If they run into
complications, then they fix the problems then and there. There's
something attractive about just getting the problem solved.

But then the smart people get burned: sometimes mistakes happen and
create outages. Or, the change appears to work fine, but then the
problems erupt later because of the change. So one force at work is
that the stop-talking-about-it-and-fix-it approach sometimes leads to
outages. The technicians implement change control; for example, they
only make their changes at night.

But another force is this: what if the technicians break things, but
the managers have to take the calls from angry users? The managers
start to implement change control. Another form of change control is
that they want to approve every change, because they don't want to
suffer for the outage caused by the technicians. It seems this kind of
control sometimes prevents problems from being fixed in the
maintenance window.

Change Control systems seem to have these dimensions:

-- When (what times and dates) changes are made

-- Who plans the changes to be made

-- Who approves the changes that are made

-- How the changes are recorded or logged

-- How the changes are tested to ensure that they worked properly.

-- Degree of latitude the changers have to work around complications

On Organizing People and Work at a VoIP Service Provider

VoIP service providers these days face the technical challenges of huge flexibility, and no single integrated solution with interop-tested partner devices. You can't just buy a "switch", plug in some TDM/SONET transport and turn up "smart remotes" made by the switch manufacturer.

Even integrated VoIP systems like MetaSwitch leave a lot of design space:

-- What signaling protocols?

-- Which of the many packet networks will be used (Ethernet-only, DSL, Mix of IP transports)?

-- How will high-performance network capacity be ensured? (prioritization? reserved bandwidth? Classified by DSCP? Port number?)

-- How will unauthorized access be prevented? (SBC? Firewall? No default route?)

-- What CPE will be used? And which features will be used? Which signaling variants will be used for, say, putting calls on hold?

-- How will you detect faults? Dry alarm contacts? End-to-end testing? SNMP traps? SNMP polling? Threshold/Statistical-norm monitoring?

My point is that there's a lot to it. Once you outgrow trivially small, you need to split up the work.

I've been brainstorming about ways to organize people to design and operate a VoIP carrier. If you need more than one person, then there's a question: how do you divide work among the people?

This reminds me of Fred Brooks's essays on organizing the development of large software systems. Voice network design operation is somewhat more constrained than are large software systems. But it's still important to have overall coherence.

I wonder how carriers would think about this if VoIP were delivered in a few 84-inch-tall cabinets with opaque slide-in cards? I.e., what if it looked like a 5ESS or DMS?

But VoIP isn't built that way. A cabinet at a VoIP carrier might have equipment from half a dozen independent vendors. And the carrier has to deal with them separately.

Below are some models I've considered. I evaluate the models with these factors in mind:

(a) Does any ONE person within the technical team claim responsibility for correct operation of the whole product/service? This is critical.

(b) Is there normally enough work to keep a person busy? ...intellectually stim-yoo-lated?

(c) Can the model accomodate staff turnover, vacation, around-the-clock emergency work?

(d) Is authority to design the system and fix problems vested in one group, or spread out all over the place?

Though competent engineers should be able to run the whole platform, these tasks are fairly easy to factor out of the engineering group:

(a) Watch the NOC screen and determine when something seems wrong

(b) Take tier-1 customer calls/complaints

(c) On-site installation

One thing I don't consider here is busywork-avoidance: how do you ensure engineers spend their time doing important things? Because people are really good at thinking of minor enhancements and seeking input. For example, if I'm allowed to spend three hours writing a blog article, it might end up really long and thorough, but it might only be 20% better than the one-hour article. But imagine if I'm given all day to write it! It might be a full 30% better than the one-hour article. But it WILL be better.

If this tendency to comprehensiveness is allowed to run unrestrained, engineers will never get a product off the ground. There's always a new way to look at the problem and build the system.

Alternately, we might have a reasonable understanding of the problem, but one more group conference call might make things slightly better.

I call this work that doesn't lead to significantly better results as "busywork". And I don't really address it directly in the organizational structures described here.

------------

The Integrated Model: everybody in the team can do everything. There's a senior-most engineer overall, but he can delegate any task to any member of the group. Some members have more experience on a specific topic. The compensation structure encourages teaching each other, but encourages doing the work yourself even more.

Advantages:

-- Every engineer understands the whole system, and can work on any problem

-- Often intellectually satisfying to engineers

-- Ensures whole team stays up-to-date and no member becomes obsolete

-- Senior-most engineer has overall responsibility for a working design

-- Work scales as overall workload increases

Limitations:

-- Requires fearless, hard-working staff

-- Strong leadership required to avoid over-specialization

-- Long training process

-- Engineers might feel like jacks of all trades, but masters of none.

-- Certain tasks have to be arbitrarily assigned. E.g., who will monitor for critical software updates? Otherwise these tasks might go un-done.

-------

Layered model: Responsibility is broken into layers mimicking the network. Somebody is responsible for layers 1-3 (cables, switches, transport, routing), while somebody else is responsible for layer 4 (operating systems), somebody else does the application layer.

Advantages:

-- The top-layer application engineer has responsibility for the overall product.

-- Exploits deep expertise in specializations. E.g. The network guy understands networking regardless of who manufactured the device; the application guy knows RTP codecs and how the whole real-time audio process works.

-- Encourages cross-platform experience. E.g. SIP guy knows SIP across all platforms

-- Once the layers are divided up amongst the staff, the responsibilities are well defined because they match the software/system responsibilities.

Limitations:

-- The top-layer application engineer may not have knowledge/resources to troubleshoot all problems, since he's specialized in only one area.

-- Some layers won't have enough to do, while other layers will be overworked. So this model may only make sense if you have a large group so the slowest job can keep busy.

-- ...but, it's not clear how to divide work as you scale up. E.g., what if there's too much work for one the application-layer guy?

---------

Device model: each physical object is "owned" by somebody, who's responsible for everything on that.

Advantages:

-- Neatly fits asset ownership within an organization (maybe)

-- Engineers can have deep experience in their devices. E.g., Cisco cert here, Nortel cert there

-- Easy to point fingers...I mean, divide responsibility...corresponding to physical interfaces on the devices.

Limitations:

-- Wastes expertise. e.g., if one guy owns a cisco router and another owns a nortel router, you've got two people with half a job with largely overlapping experience, probably bored with their job

-- Nobody really understands the whole product

-- Easy to point fingers...I mean, divide responsibility...corresponding to physical interfaces on the devices. "The packet is leaving my box...it must be on your end."

---------

The SME model: the system is broken into specific specialties by subject matter, and individuals are assigned to be experts in those areas. SM's might include: SBC, App Server, Net Server, Billing, Provisioning, Client Call Control, PSTN access, SIP, MGCP, fault detection, etc.

Advantages:

-- Promotes deep expertise in the subject matter

-- Matches an academic model of divide research topics

Limitations:

-- Very weak on design -- no engineer has overall design authority or comprehension

-- SMEs may know their area in the abstract, but have trouble with the overall application

-- Being an SME may not be enough to keep a person busy

----

The Knuth model (SME-graph): Like the SME model, but each specialty has at least two experts, and each expert has at least two specialties. The purpose is to ensure all the specialties are connected, so it's best to require dissimilar expertise.

Advantages:

-- Overall system is owned by the whole team, so decisions aren't made in isolation

Disadvantages:

-- No single technical mind responsible for overall product operation and uniform design. This may result in an overly-complicated, fragile system.

---------

Network Region Model: The network is divided into segments, and engineers are assigned regions. Then they do everything within their region. E.g., one engineer might handle the customer access network, another handles the outside of the SBC (i.e. the core network part connected to the Internet), and another handles the inside of the core.

Advantages:

-- Engineers work across functions within their network region, e.g., physical layer, switching, routing, application, performance, etc.

Limitations:

-- Nobody technical owns the whole product

The "Just Do It" approach to network design

The natural way networks are designed is "just do it". We just do
whatever seems obvious; very little thought is given to design.
Instead, neophytes think it's "just" a matter of configuration.

This approach leads to incomprehensibly complex designs. Cables going
in every which way, poorly-planned fault tolerance, VLAN
inconsistencies (e.g., VLAN 200 is one broadcast domain on this
switch, but a different broadcast domain on another switch), and
incredible difficulties when troubleshooting.

Sales folks and work avoidance

Sales folks are a funny breed. It's a salesman's job to work to
convince you to buy his product. But he's only willing to do so much
work!

The amount of work he's willing to do is vaguely correlated to the
amount of money he might make from you. For example, when I was at
BellSouth, if I called Empirix to get information on their product,
they wanted to get on a plane and come explain it to my. They heard
"BellSouth" and saw big dollar signs. The frustrating thing was that I
wanted answers today, not some future date after they fly to visit me.

Big companies (like large telephone carriers) get lots of attention
from the vendors. The vendors do tons of work -- often for free -- at
the hope that the big-company customer will buy lots.

But I work a lot with smaller companies too, and it's strange to see
the vendors balk at my request. Recently, a DSLAM vendor was
explaining his product to a smallish midwestern CLEC telephone company
I work with. There was one feature they had in particular -- Option
0x82 support for bridged-mode DSL -- and this telco would need to
depend on how that feature works. The salesman and his sales engineer
explained the features and benefits for us. But when we asked him to
give us some examples -- packet captures, or configurations -- they
complained. They did do some web searching for us, but never actually
provided us with hard information.

Fortunately, the CLEC is large enough that the vendor is providing
some evaluation hardware. This way, we can do the work and actually
prove out the feature.

But it's funny to get the complaint. They didn't understand that we're
really just asking for very detailed specifications of their product.
They admitted that they'd have to test it in a lab to give our answer,
but then tried to redirect us to the technical support organization.
Is the support organization going to do a better job of running the lab?

Wireshark Memory Bloat on VoIP capture files

Wireshark is a really neat tool for analyzing phone calls. But when
you load a 100 MB capture file of VoIP calls, you need much more than
100 MB of RAM. But how much more?

Here's a data point from which you can make a line: a 326.15 MB PCAP
file contained lots of SIP, and a little RTP. This wasn't a raw
capture file; I had thrown away a lot of the RTP and RTCP.

The file compressed to 121.15 MB using gzip. Then when Wireshark
opened it, and then generated a "VoIP Calls" analysis, the resident
memory size was 610.52 MB. That means Wireshark needs about 1.87-times
as much RAM as there are bytes in the input file, in this case.

So if I can dedicate 1 GB RAM to opening a capture file, I can
probably handle a PCAP file that's about 530 MB uncompressed.

BroadSoft Connections 2008: the obvious ideas are all here

From the flight from Phoenix, AZ to Raleigh, NC: The BroadSoft Connections 2008 conference completed today. This is my third time at Connections.

The stated goal of the show is networking and dealmaking. The former is difficult to measure, but I suppose I did some of that. I know the latter occurred, and the work will occupy me for a while.

BroadSoft spent some time advertising the "xtend" platform. These are API's to control the BroadWorks call control. Their heart is in the right place: they want anybody to be able to use the BroadWorks platform for fun and profit. They want developers to harness the tool and do amazing things new things.

However, the results are mostly disappointing. We saw 1,001 ways to enable and disable Call Forwarding. And we saw a few variations on the "call this number from that address book" ; while it's true that initiating a call with an eleven-digit phone number is tedious, we haven't seen that click-to-dial is a serious enabler. I'm willing to be convinced, though.

Speaking as an opinionated engineer/developer: BroadSoft has some impediments --

1. BroadWorks is closed source; many developers don't want to spend their free time on closed-source commercial systems.

2. Most of the BroadWorks documentation is locked away in the "Broad-oft Boulevard", so many outsiders can hardly understand what it does.

3. There are open-source alternatives to BroadWorks, like Asterisk and OpenSER. The former has many d�vot�es.

4. Voice calling is an intrinsically demanding medium: it requires a lot of attention to participate in a conversation. Many people intentionally avoid being reached so they can work or recover from it.

5. Voice calling has been widely abused by telemarketers. many developers currently feel like text messaging -- SMS and Instant Messaging -- is the best way to contact someone.

Among the exhibitors, ESNAtech did demonstrate some neat integration of BroadWorks, mobile phones, and Microsoft OCS. But all the pieces -- especially the latter -- are very expensive. If OCS was $100/seat, this tool would likely see more adopters.

Scott Hoffpauir, BroadSoft's CTO, showed a lot of interest in IMS. BroadSoft has a lot of apparently-true IMS capabilities and interfaces. (Somehow, IMS reminds me of Jabba the Hut. It's this massive, foreign, demanding monster.)

Hoffpauir also mentioned that BroadSoft intends to implement Call Recording, outgoing fax, and SMS functions (maybe SIP interfaces to an SMS gateway). This isn't good news for some of BroadSoft's "partners" who do these already. It's likely planners and engineers would rather wait for BroadSoft to provide and support those features.

The speaker lineup was a little disappointing. We were promised Tim O'Reilly, but he didn't show. Instead we got a soi-disant "futurist," from the "Global Futures Group". I didn't get a lot from him. The first strike against him was that he had been in business for 18 years, but he didn't offer any examples of technology he had predicted.

He talked about a global everywhere web. He mentioned clothes having their own IP address. He chastised attendees for not using a wiki to get free product ideas from customers. Yeah, OK. He was trying to inspire excitement.

His best point was about companies focusing too much on their "core competencies". It was basically a call to learn more and apply your learning.

Walter Mossberg, writer for the Wall Street Journal, lead a roundtable, including a couple of guys and Emily Green from the Yankee Group.

Mossberg made a good point. Why, he asked, can't carriers be happy with just providing the access and the transport? Instead, they try to force you to their services and their email system and their web site.

Carriers' marketing and product-management folks are afraid of being, "just a dumb pipe". What these good people misunderstand is how hard, and how important, it is to provide reliable, high-speed, low-latency Internet transport. Just do that job really well, people.

I suppose the perceived risk is of being a commodity. With LNP, you can use any "dumb pipe" for your phone conversations that you want. But one "dumb pipe" might sound better than another, or drop your calls less, or have superior customer service. It's really not a commodity.

Emily Greene of the Yankee Group made a good point, not often made: Carriers retain control of applications and features so they can make them reliable. People are willing to accept the fabulous mobility of cell phones in exchange for dropped calls and unintelligible speech. This reduced standard for quality has helped VoIP in some ways. But BroadSoft's customers are primarily selling to business users, for whom the desk phone is supposed to be very reliable and consistent.

Timothy Ferriss, author of "The 4-Hour Workweek" , also spoke. He made the greatest number of rhetorical points per minute of speaking. He talked about spending your time on the right things; i.e., eliminating low-value tasks from your work life.

He's pretty hard on email. He talked about how people lose so much time checking email, and staying informed, and trying to be involved in decisions. He suggests checking your email at set times each day, then do actual work the rest of the time.

He pointed out that human multitasking is really just task switching. Amen, brother. (Anybody familiar with Operating System schedulers already knew this. On the other hand, when I make breakfast, and have food in both the oven and on the stove, does that count as multitasking?) Switching tasks takes a lot of time and energy, and we humans usually aren't aware of the time we spend on it.

Ferriss also argued for thinking of your non-work time as necessary recovery time. If you're always checking email and continuing to work on the weekend, then you're never fully productive or fully relaxed. You and I need time to rebound from work.

ln grad school, I started taking Sundays off from work. I noticed early on that this forced me to work much harder through the week and on Saturday.

Email and computer-mediated busyness are like addictions. We keep doing the same things, frustrated by the results. We have to keep ourselves from personal harmful addictions to these machines. As the ancient Jewish Christian Sha'ul said, if we present ourselves as slaves to someone, then we effectively are slaves.

How important is a lab system?

How important is it to have a lab replicating your production (VoIP)
environment?

Conventional wisdom says that everybody has a lab: some people just
host their production users on it.

Having a lab incurs a lot of additional cost and work:

-- You have to buy the lab equipment, and the software.

-- You have to install and integrate the lab system.

-- You have to keep it up to date and secure.

-- To use the lab, you have to do everything twice: first on the lab,
then once it's proven, on the production system.

It's often easier to buy the lab gear in the first place than it is to
do all the work to actually use the lab.

This CDR has been brought to you from the letters B, F, and the symbol #

TWICE in the past month, I've bumped into CDRs from SS7 equipment in
Atlanta that include alphabetic characters and pound signs in the
calling party number (ANI) field of the CDR.

One of the calls was from 5176#0B600. (That's a phone number.)

What's going on here?

Voicemail peak-hour oversubscription ratio (48:1)

If I'm a phone company and I have 100 subscribers, and every one of them has voicemail, how many people will be calling into the voicemail system at any one time?

Back in the old days, they'd provision trunks into the voicemail system between the Class-5 switch and the voicemail system. They'd have to know how many trunks to provision to let all the subscribers both receive voicemails, and call in to check the voicemails.

I don't have any good answers here. But I did a quick study using data from logs on a couple of service providers.

These are new-fangled VoIP service providers. So for them, a media/application server supports:

• Voice mail
  
• Automated Attendant
  
• Music on Hold
  
• Call Center (hold queues)
  

However, the server can email voicemails somewhere else. So when people check their voicemail, they may not be checking over the phone. This is different than the old-school voicemail systems of 2001, and affects the utilization.

Just as an example, one service provider I checked had a 48:1 oversubscription ratio: i.e., for each 48 subscribers, there's one RTP stream at peak. They also had about 0.0001 calls-per-second-per-subscriber.

(I say "at peak" because I'm interested in building a network that supports the peak, "busy-hour" load. If I design for the average, I'll have problems when the load goes above that average.)

Does this ratio apply to you? All of these things could affect it:

• How many of your users actually buy your voicemail service?
  
• ...check their voicemail over the phone?
  
• ...get all their voicemail via email, and listen to it that way?
  
• ...use automated attendant?
  
• ...put callers on hold, and get on hold music?
  
• ...use call centers? And how long are callers waiting in queue?
  

As another note, it appears that the average duration of voicemail is in the 25-30 second range. This is true at two service providers, both in NFL cities.

Update -- A residential service provider reports around a 160:1 oversubscription ratio just for basic voicemail. (I.e., One concurrent call for every 160 subscribers.) A few of them do get voicemail via email.

Someone familiar with voicemail software development told me they expect over 200 subscribers per concurrent call for traditional voicemail.

The cat in Aastra 2.2 SIP Software

The Aastra 57i 2.2 software has this ASCII art embedded:

|                ("`-''-/").___..--''"`-._         
|   Line         (`6_ 6  )   `-.  (     ).`-.__.`) 
|  Manager      (_Y_.)'  ._   )  `._ `. ``-..-'   
|               _..`--'_..-_/  /--'_.' ,'              
|              (il),-''  (li),'  ((!.-'        

Is it a Cat? Or is it a Pig?

On Having an Opinion: Goodness/Badness vs. Pros/Cons

There are two ways of giving recommendations on technical issues: 

(a) To say that something is "good" or that something is "bad". 

(b) To list Pros and Cons, or else Advantages and Limitations.

The most popular, and natural way is just to say what's good, or often, what's best. In my opinion Linux servers are better than Windows servers; I'm ascribing goodness to Linux servers.

But I've noticed that more mature Computer Scientists seem to talk about advantages and limitations. My first memory of this came in my Comp 243 "Distributed Systems" class at UNC-CH. F. Don Smith was the professor. It was 2001. We was talking about network file systems. He introduced the section on the Sprite file system saying that it was one of the most interesting. Then he talked about it, and at the end of the section I asked: "Why is it that you say this is the best?" (*)

He corrected my question: "I didn't say that this was the best -- only that it was the most interesting."

My error took a few years to soak in. I was accustomed to thinking about goodness and badness of technical matters. But Don didn't use those terms. In his classes, he only talked about features, limitations, and therefore advantages.

It's human tendency to say that something we don't like is bad. In a technical context, what we usually mean is that I don't think that will work. 

There's something funny going on here; goodness and badness in this way exist in a total ordering. In my example above, Linux servers are closer to good than Windows servers are. If I had to add Solaris servers to the mix, I might put it somewhere between the two, but certainly closer to Linux servers.

But what if I need features that only the Solaris LVM offers? Is the "goodness" of Linux enough to outweigh my need for the Solaris LVM?

Goodness is only one dimension. In reality, technical decisions should be based on factors like features and constraints. Maybe Solaris's LVM does something I need, but perhaps I need much more expensive hardware. The cost of the hardware is a constraint, while the LVM is a feature.

The advantages and limitations of Solaris, Linux, and Windows servers for any one application can be listed. But, ultimately, a recommendation has to be made.  What server type fits the application the best?

So all the complicated evaluation of advantages and limitations has to be projected onto a single-dimensional line: fitness.

Is fitness  the same as goodness? 

(*) All quotations are approximated, and based on my memory. They're not guaranteed to be word-for-word exact.

Interop Lab Testing for VoIP Devices (2008 edition)

In an Interoperation (Interop) Lab, devices are made to work together. When they seem to, the vendors of the devices claim that they "interop with" each other. This is necessary, but not sufficient, to know things will work together.

Background

Suppose you make telephone soft-switch or application server, such as BroadSoft BroadWorks, Sylantro, MetaSwitch, or the Alcatel-Lucent Network Gateway (aka Lucent Compact Switch). The customers of your several-hundred-thousand-dollar device want to use VoIP telephones with it, but they've tried a few cheap ones and had trouble. And they had another one they liked, but when they upgraded it, call-hold stopped working. After opening several tickets with you and with the phone vendor, they give up and realize that neither of you did anything wrong.

The phone was always doing something reasonable -- though not quite what they hoped -- and the softswitch was doing something reasonable -- though not quite what they hoped. This is the curse of the VoIP Implementor: everybody can follow the rules, and working together, get nothing done.

After complaints from customers about this difficulty, you (the softswitch vendor) launch an interop testing program. Depending on the sort of technical staff you have, you might do it in-house, or outsource it.

Your goal is to confirm that specific products "work with" your product. You want your customers to be able to choose products confidently. It's a laudable goal.

A common interop lab setup might look like this, for a softswitch vendor's lab, when testing a new VoIP phone:



You've got the new phone (the Device Under Test, or DUT), and your piece of gear. Plus you connect them with an ordinary Ethernet switch. And you have another gold phone that you know to work with your platform, because phone calls require two phones.

Advantages and Limitations of Interop Testing

Interop testing of this variety can be very useful. SIP VoIP telephony is an evolving body of standards. There are more than one way to do something (such as put a call on hold, or make one phone number appear on two different phone (shared call appearance (SCA))). It is very useful to identify fundamental incompatibilities. Often, there are certain settings required on both sides: to use this device, The "rfc2543_hold" option must be turned on. Somebody has to figure this out: it had might as well be two the vendors involved.

This testing is only as good as the test plan. If the test plan doesn't include something that you need to do, then the testing isn't quite complete for you.

In addition, no two devices have all the same features. The vendor may test for T.38, but if the DUT doesn't do T.38, they don't want to mark it as failed for T.38. Instead, the testing engineer is likely just to mark that feature as Not Supported. Depending on policy, or haste, the testing engineer may just mark anything that doesn't work as Not Supported. There is no "one true standard" for interoperability. Ultimately, if the DUT and his softswitch work together at all, the two are "certified" as interoperable. It is to neither vendor's advantage to anger the other by claiming they're not interoperable.

Finally, Interop lab testing only tests the interoperation between a pair of device. Devices X and Y work together. But real systems are made of devices A through Z.

Real Integration Testing

A network for a VoIP Service Provider using SIP peering might look like this:

A real network for an SS7-connected CLEC might look more like this:

There are lots of devices. And many of them may affect the success you'll have with the DUT.

Take, for example, a simple SIP phone. It's easily possible that the signaling path for a PSTN call, in a CLEC case, is

• SIP Phone
  
• Customer Premise ALG
  
• Session Border Controller
  
• Softswitch
  
• PSTN Gateway
  
• SS7 STP networks
  
• PSTN Class 5 Telephone Switch
  
• PBX
  
• PBX handset
  

It's definitely possible that the SIP phone could work with the softswitch, but irritate one of the other components. For example,

• The SIP phone and the softswitch may use Requires: a specific SIP feature package, but the SBC doesn't allow it or support it.
  
• Or perhaps yet-another form of caller ID is dreamed up, and the PSTN gateway can't support it, so that all calls from this phone appear to have no caller ID.
  
• Or maybe the phone signals some sort of ISUP calling party category in SIP that the PSTN gateway passes through, and confuses a downstream PSTN telephone switch.
  
• What if the phone needs SIP over TCP, but the ALG doesn't support it properly?
  
• Or the DUT signals RFC2833 support properly, but the PSTN gateway expects telephone-events to have a specific codec number (i.e., tickling a bug that was already present)?
  
• Perhaps it works fine if the SBC is configured with a "classic" configuration, but breaks miserably if you switch to the "new" configuration.
  
• Maybe it can download configurations from the lab FTP server, but chokes if the FTP server is a little slow
  

My point is that we have to test components an in and-to-end environment to really get confidence that they work. This may change, but only if the complexity here is accidental (a side-effect of today's state of things) and not essential complexity (fundamental to the job). With all the new features that VoIP telephony systems try to support, and the upgradability the protocol designers intend, I'm not sure I can distinguish which it is now.


Show me a "simple SIP phone", and I'll show you a phone that nobody likes because it's not flexible enough to be configured to do crazy things.

Why won't anybody build far-end echo cancellation into their VoIP phones and ATAs?

Nobody VoIP Phone or ATA on the market offers talker far-end echo cancellation. They should.

Background

Some background: echo is when you hear yourself talking. It's usually the caused by the device on the other end of the call, but it's exacerbated in VoIP networks because they have long delays.

Suppose you have a phone call that includes a VoIP device (such as a PolyCom SoundPoint 650 IP SIP Phone, or a Cisco 7260 SIP phone, or a Cisco/LinkSys/Sipura ATA, or an Aastra 57i).



There's a VoIP phone plugged into some sort of IP network. In many cases, it's an Ethernet network with a DS1 (T1) connection to the VoIP service provider / ISP. There's a VoIP-PSTN gateway connected there, such as a MetaSwitch VP3510, or a Lucent Compact Switch (LCS), or an AudioCodes, or a Cisco AS5400. It has an Ethernet interface and TDM interfaces, such as DS3, DS1. Maybe it does ISUP and has SS7 A-Links, or ISDN PRI. Then there's the PSTN, which includes traditional telephone switches -- and actually a lot of VoIP hidden in there too. Finally, there's a PSTN phone.

When the VoIP phone user says something, he may hear an echo of his own voice coming back.



Cisco has a nice WAV file demonstrating what echo sounds like. (This page is known to some insiders as the Cisco Duck Quack page.)

Why does C3P0 hear his own voice coming back?

The PSTN Phone isn't perfect. Some of the electrical voice signal that enters it may be "reflected" back. This is sometimes called the "Two-wire-to-four-wire" conversion: a normal phone line is a two-wire electrical circuit with both sides of the conversation on it, but the handset itself has two wires for the speaker, and two wires for the microphone. If this isn't built just perfectly, some of the sound signal that's sent to the speaker will be reflected back down the wire. Many phones are not perfect.

The PSTN Phone may be on speakerphone, or pick up other acoustic echo. Normal room walls echo sound back to us. It's there, but we don't normally notice it.

The VoIP Network is long.We won't normally notice echo in a room, unless we're in a big room. Our brain is pretty good at filtering out sound that's echoed back very quickly; if we hear an echo less than 100 [ms] or so after we say it, we don't even notice it normally. So if the room is big enough that sound takes a long time to echo back, then we might notice it.

(How big would a room need to be? Around 55 feet / 17 meters across would work nicely to reflect off the far wall. Sound travels 340 meters / second, and we'll hear an echo if our voice takes around 100 [ms] to get back to us. That means it needs to be 0.1 * 340 meters from my mouth to my ear, or half that from one side of the room to the other because the sound travels that distance twice.)

It can take a long time for a sound signal carried via VoIP to get from the talker C3P0 back to his ear. 100 millisecond round-trip-time is easily achievable. Why? Because packets sit in buffers along the way. In traditional non-VoIP TDM networks, digital voice data is rarely "buffered" to any extent. But in VoIP networks, buffering always happens. This "buffering" means that packets are sitting inside devices -- phones, routers, switches -- effectively adding to the delay between the talker and his echo.

Put echo cancellation in the VoIP Phone.

This is all very annoying for the talker who hears himself. VoIP users suffer from this much more than PSTN users, because of this delay. Technically, the PSTN phone is creating the echo, it seems silly to just point blame. Technically, VoIP networks should be free of jitter (variation in packet transit delay) -- but networks aren't perfect, so VoIP phones have jitter buffers built in. It's standard equipment, like seat belts in cars.

But echo cancellation -- the sort that's really needed, that would cancel out the echo received back across the VoIP network -- just is not put into VoIP phones normally. Some phones, such as Polycom, have limit acoustic echo cancellation capability to prevent the VoIP phone itself from echoing back. That's nice, but that's not the main problem.

It seems VoIP phone vendors are spending their time building in G.722 wideband codec support ("HD Voice") , so conversations within your office building will sound nice. Whoop-ee.

VoIP Echo Cancellation is possible. Vendors are routinely building limited echo-cancellation ability into the PSTN-VoIP gateway device. (E.g., General Bandwidth G6, MetaSwitch). But it can't always be used, and it's not always effective. For example, if you connect to the PSTN through Level(3), then you don't get a gateway with echo cancellation. And on some gateways, using echo cancellation limits the number of calls you can make through the box.

But is echo cancellation in the VoIP gateway sufficient? Apparently not. If it were, I wouldn't hear about many complaints from my client base. And Ditech would have no reason to make a VoIP-only echo cancellation box. But as it is, VoIP carriers have lots of trouble with echo.

And, in general, centralizing the echo-cancellation capability doesn't seem optimal either. Putting lots of work and intelligence in one place tends to make one really-expensive place.

VoIP Phone and ATA Vendors should add echo cancellation to their devices. Polycom, Cisco, Aastra, Linksys, Snom, Adtran, etc. listen up: your device should cancel out the echo received back in the RTP stream from the VoIP network. It can't be that hard! I know there are DSPs that can help with this. Add this feature to your top-of-the-line phone! Charge more for it! Make it a premium add-on license if you must!

Some people just want a super-cheap VoIP phone. But some people are trying to replicate the services of traditional phone systems, but using VoIP. For them, the echo problem is real, and serious. It can't always be solved with a gateway. And an extra 25 percent in the cost of the CPE might be hard to swallow, but not as hard as having no solution at all.

Call Transfer Scenarios

The IETF Call Control - Transfer draft seems to have the best and latest info on call transfer scenarios. But I can't find a good summary of the cases involving transfer-like scenarios. Here's my stab at a complete list, although I haven't taken the time to try to prove this is comprehensive.

-- Blind transfer, original recipient is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Bob transfers to Charles,

Charles answers,

Charles and Alice talk.

-- Blind transfer, originator is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Alice transfers to Charles,

Charles answers,

Charles and Bob talk.

-- Blind transfer before call answer, original recipient is facilitator:
Alice calls Bob,

Bob's phone rings,

Bob diverts the call to Charles,

Charles answers,

Charles and Alice talk.

-- Blind transfer before call answer, originator is facilitator:

Alice calls Bob,

Bob's phone rings,

Alice diverts the call to Charles,

Charles answers,

Charles and Bob talk.

-- Attended transfer, original recipient is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Bob puts Alice on hold,

Bob makes call to Charles

Charles answers,

Bob and Charles talk,

Bob completes transfer,

Charles and Alice talk.

-- Attended transfer, originator is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Alice puts Bob on hold,

Alice makes call to Charles

Charles answers,

Alice and Charles talk,

Alice completes transfer,

Charles and Bob talk.

--  Busy-Failed Blind transfer, original recipient is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Bob transfers to Charles,

Charles is busy,

Alice and Bob talk.

-- Busy-Failed Blind transfer, originator is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Alice transfers to Charles,

Charles is busy,

Alice and Bob talk.

--  No-Answer-Failed Blind transfer, original recipient is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Bob transfers to Charles,

Charles never answers,

Alice and Bob talk.

-- No-Answer-Failed Blind transfer, originator is facilitator:

Alice calls Bob,

Bob answers,

Alice and Bob talk,

Alice transfers to Charles,

Charles never answers,

Alice and Bob talk.

I don't list the busy-failed attended call case, or the no-answer-failed attended call case, because those never get to the transfer (REFER message) case. So those just look like a call that gets put on hold, the facilitator makes another call, it fails, then the original call is retrieved from hold.

AT&T "will not block or degrade [Internet] traffic."

At the metaswitch user forum, AT&T VP Joe Weinman was asked about
whether AT&T would block internet traffic that they disagreed with. He
said: "AT&T is a common carrier. We will not block or degrade traffic."