2021.02.17 - CopperCom, Asterisk, SIP!

After several weeks of stuggling with strange issues with SIP calls between our old CopperCom telephone switch and our old Asterisk 11 install (anyone sensing a pattern?) I finally found a fix.

The Problem

So we use a CopperCom telephone switch. Telephone switches are not cheap. Shortly after we bought this switch, before we were even done setting it up CopperCom announced that it was getting out of the softswitch business (I don't remeber in they got bought out, went bankrupt or what, but they weren't going to be making phone switches anymore). That was not awesome considering we couldn't exactly return this switch, but whatever, this is before the rise of the cloud so there was nothing wrong with the switch, it just meant that manufacturer support would be limited moving forward and there wouldn't be any new versions or hardware upgrades. The documentation provided with the switch was pretty good, we also found a CopperCom usergroup with lots of good people and were able to get 2 more complete CopperCom switches and tons of spare parts so I feel pretty confident we'll be able to continue to use this switch for quite a while.

One of the options CopperCom offered but we didn't take was a SIP interface. At the time Asterisk was still in it's infancy and really (I think at least) SIP was mostly used as a long cost long distance transport. We found it was much more cost effective to just buy a SIP to T1 converter and use that instead of paying CopperCom for native SIP support. However now most customer PBXes are now SIP based and we are wasting alot of T1 resources on these so we wanted to add SIP support. We already had the SIP cards, we just needed to add them. Thankfully there are a couple of ex-CopperCom employees that offer their expertise and we were able to get one to help us get the SIP cards installed and configured.

After installing the cards I noticed on small problem durring testing... on the SIP PBX if I would transfer a call from one extention to another, or transfer the call to a Parking slot I would loose all audio from the PBX to the CopperCom (audio in the other direction worked so it was just a one-way audio loss). One-way audio is a common problem with SIP... unfortunately for me the common cause of one-way audio with SIP is NAT issues. In this case there is no NAT (for testing I have the PBX directly connected to the CopperCom), so that couldn't possibly be the problem, however it's so common that searching google for One-Way audio Asterisk basicly only returns that.

Research

The CopperCom employee said he has never run into this issue, and a fresh install with a modern verison of Asterisk worked as expected (Asterisk 11 was EOLed in 2017). Some hunting on the internet finally pointed me to these bug reports:

In particular the issue seemed to be that the SIP standard is vague on certain things and when the audio source changed Asterisk was acting in a way that broke the jitter buffers. I did a packet capture and confirmed that I was seeing stupidly huge jitter being generated from Asterisk when I transfered a call. I'm not a programmer but the fix seemed simple, in order to avoid generating this stupid huge (fake!) jitter Asterisk just needed to tell the CopperCom that the audio source had changed. In looking through the suggested patches on the bug posts I found this:

Index: main/channel.c =================================================================== --- main/channel.c (revision 281649) +++ main/channel.c (working copy) @@ -4679,8 +4679,8 @@ ast_set_flag(c0, AST_FLAG_END_DTMF_ONLY); /* Before we enter in and bridge these two together tell them both the source of audio has changed */ - ast_indicate(c0, AST_CONTROL_SRCUPDATE); - ast_indicate(c1, AST_CONTROL_SRCUPDATE); + ast_indicate(c0, AST_CONTROL_SRCCHANGE); + ast_indicate(c1, AST_CONTROL_SRCCHANGE);

Seem simple enough, change two lines that say AST_CONTROL_SRCUPDATE to AST_CONTROL_SRCCHANGE. I made the change, recompiled and tested. The fake jitter was now gone, so my fix had worked... however the one-way audio problem still happened. In fact it was worse, now even just putting a call on hold would cause audio loss, before only transferring a call that would cause one-way audio.

The Fix

I went on a bunch of unproductive searching but most led me to fixes put in really really old verisons of Asterisk that couldn't possible apply to me. I also tried to make adjustments to the CopperCom, but the actaul RTP stream is all done with hardware and there were no options I could find to adjust the jitter buffer or even turn on any useful debugging. Finally it hit me, changing the AST_CONTROL_SRCUPDATE to AST_CONTROL_SRCCHANGE made things worse, so what if I looked through the code and changed all the instances of AST_CONTROL_SRCCHANGE to AST_CONTROL_SRCUPDATE instead? This is a horrible idea, making blind changes to code you don't understand. Don't do it.

I did at least leave a note in the source code where I made the change so one day maybe if I figure out what the differance is between AST_CONTROL_SRCUPDATE and AST_CONTROL_SRCCHANGE I can go back and try to fix or at least understand it. In the meantime however the one way audio issue is gone and the old Asterisk 11 boxes are talking to the CopperCom over SIP perfectly. Yay for stupid fixes!

-Nick