Wednesday, October 28, 2009

Using mod_xml_curl In FreeSWITCH - How Hard Can It Be? (Part 2)

In my previous post I talked about my desire to go through the exercise of setting up xml_curl to pull user directory information out of a database. I left off having just verified that I was receiving proper CGI parameters. I first tested a dialplan call using portaudio to call the MOH extension:

pa call 9999

This yielded good results: my Perl script dumped the CGI params to a text file and I got a good feel for what data looks like coming from an xml_curl request.

Next I needed a way to send information back to xml_curl/FreeSWITCH. I recalled some talk about sending a generic "no found" response. Sure enough, the wiki confirms it. I added a simple function to my perl script that would reply to xml_curl and would dump the "no found" message to my log file also:

sub return_not_found() {
print "Content-Type: text/xml\n
<?xml version=\"1.0\"?>
<document type=\"freeswitch/xml\">
<section name=\"result\">
<result status=\"not found\" />

## dump to file for checking
print FILEOUT "Content-Type: text/xml\n
<?xml version=\"1.0\"?>
<document type=\"freeswitch/xml\">
<section name=\"result\">
<result status=\"not found\" />

A simple call to &return_not_found() will gracefully handle the situation where my CGI script has no useful information to send back. At this point I did a quick test and had the CGI script send the no found info back on any xml_curl request just to make sure it was working. It did.

Next up: having a phone register so that I could see what a 'directory' request looks like. In this case you could use any SIP device that tries to REGISTER, even another FS box with a gateway defined. I found a Snom 300 that I have in production and added a second identity. (The first identity is being used in production so I didn't want to mess with it.) I added "Tony Romo" as the second identity on the phone, with the account and auth. user as "1100" and the appropriate password from my database file. I watched on the FS CLI to make sure that the phone tried to register, then I went to my script's log file to see what the request looked like. It was a bit shorter than the 'dialplan' request but it definitely had useful information:

".charset" => "ISO-8859-1",
".fieldnames" => {},
".parameters" => [
"Core-UUID" => ["9b4e5fe9-9bc9-4bed-915a-3c277cbd760b"],
"Event-Calling-File" => ["mod_voicemail.c"],
"Event-Calling-Function" => ["resolve_id"],
"Event-Calling-Line-Number" => [1278],
"Event-Date-GMT" => ["Thu, 29 Oct 2009 17:00:15 GMT"],
"Event-Date-Local" => ["2009-10-29 10:00:15"],
"Event-Date-Timestamp" => ["1256835615546212"],
"Event-Name" => ["GENERAL"],
"FreeSWITCH-Hostname" => ["michael-collinss-macbook-pro.local"],
"FreeSWITCH-IPv4" => [""],
"FreeSWITCH-IPv6" => ["::1"],
action => ["message-count"],
domain => [""],
escape => 1,
hostname => ["michael-collinss-macbook-pro.local"],
key => ["id"],
key_name => ["name"],
key_value => [""],
section => ["directory"],
tag_name => ["domain"],
user => [1100],
}, "CGI")

The values I was particularly interested in were 'section' and 'user' which had "directory" and "1100" as their respective values. I knew I was in business! I added a little bit of logic to my script so that I could differentiate between a 'dialplan' and 'directory' request. The relevant tidbits look like this:

my $q = CGI->new;
my $section = $q->{'section'}->[0];

if ( $section eq 'directory' ) {
} elsif ( $section eq 'dialplan' ) {
} else {
## We don't anything other than dialplan or directory, so if we're here then return a no found
print FILEOUT "\n\n--- UNKNOWN REQUEST ---\n";
print FILEOUT "\nSection: $section\n\n";
print FILEOUT dump($q);
print FILEOUT "\n = = = = = = = = = = = = = = = = = = = = = = =\n";

Okay, so at this point I have a basic script that knows the difference between a 'directory' and a 'dialplan' request and can return a no found message if need be. Now I needed to tackle the database stuff.

DBI: Perl's Handy Dandy DataBase Interface
When you have a Perl question there's always one place you can go: the CPAN. I had used DBI before, although I can't recall exactly when/where/how. But I did remember that it wasn't really all that difficult for simple stuff. After some digging around I found this code:

use DBI;
my $dbh;
$dbh = DBI->connect('dbi:Pg:dbname=xml_curl_test;host=', 'xmlcurl', 'freeswitch',
{ RaiseError => 1, AutoCommit => 1});

In short, this creates a db handler object ($dbh) upon which you can do SQL operations. Let's take a quick look at part of the connection string:


The connect uses the format of DSN, user, password. My DSN is for 'Pg' (PostgreSQL) with a dbname of xml_curl and the local host. My user is 'xmlcurl' with a password of 'freeswitch'. The other stuff - { RaiseError => 1, AutoCommit => 1} - is recommended in the DBI docs so I roll with it.

My SQL call looks like this:

my $sql = "SELECT lastname, firstname, mailbox, password, context FROM users WHERE username = '$user'";
my $row = $dbh->selectrow_hashref($sql);

This makes the SQL query and puts a single row of results into a hash reference $row. From here I can access the results like this:


I like this because I don't have to remember what column number each field is, instead I simply reference the name of the column. In my case I'm assuming that I have only a single row of results; you probably want to do some error checking on a production system.

The last order of business is to format some XML and give it back. Since I'm just doing a POC I decided to cheat and use a string with some placeholders and then swap them out. Here's my template definition:

## User template variable; replace the ALL CAPS VALUES with stuff pulled from database
use constant USER_TEMPLATE =>
' <user id="USER_ID">
<param name="password" value="USER_PASS"/>
<param name="vm-password" value="USER_PASS"/>
<variable name="toll_allow" value="domestic,international,local"/>
<variable name="accountcode" value="USER_ID"/>
<variable name="user_context" value="USER_CONTEXT"/>
<variable name="effective_caller_id_name" value="USER_FIRSTNAME USER_LASTNAME"/>
<variable name="effective_caller_id_number" value="USER_ID"/>
<variable name="outbound_caller_id_name" value="$${outbound_caller_name}"/>
<variable name="outbound_caller_id_number" value="$${outbound_caller_id}"/>
<variable name="callgroup" value="techsupport"/>

The items in all caps will get replaced with some simple regular expressions. Stay tuned for Part 3 where I show you the entire script. It's not pretty but it is functional and it took me all of an hour to get everything together. It's taken much longer to blog it all and figure out how to format XML code so that it looks right. ;)

Using mod_xml_curl In FreeSWITCH - How Hard Can It Be? (Part 1)

I've been playing with FreeSWITCH for more than three years. In all that time I've never really had a good reason to use mod_xml_curl. Until now, that is. The good reason is nothing more than my own edification. I like to answer questions on the mailing list and finally I got sick of telling people to use xml_curl without having used it my self.

MOD_XML_CURL: The Magic Module For Dynamic FreeSWITCH Configuration
Okay, here's a quick overview of mod_xml_curl: it's a module in FreeSWITCH that allows parts of the FreeSWITCH configuration to be fetched from a web server that can, in turn, grab the configuration information from a database (or anywhere else for that matter). The primary ways to use xml_curl are for the dialplan and the user directory, but you can also use it for configuration options and for phrase macros. The specific question on the mailing list was about pulling user information from a database, so that's where I decided to start.

I've read about it, I've heard people talk about it, I know lots of people who use it, but I've never done it myself, so the first question I had to answer is this: what do I do first? Well, the specific question was about pulling user information from a database, so I decided the first thing to do would be to look at the sample XML files in the default configuration to get an idea of what kind of XML config stuff I would need to generate and send back to FreeSWITCH. I also poked around in the contrib folder to see what others had done. Finally, I consulted the oft-maligned but generally underappreciated FreeSWITCH wiki. Amazingly, with just these meager resources I was able to get a POC working in just a few hours. Now mind you, I did this in my spare time while avoiding my wife and kids, and with limited resources. (I don't have a stack of VoIP phones on my home LAN like some of you people do.)

BTW, I'm not a Web-programming guru, nor am I a database admin. I'm familiar with these from having tinkered with some LAMP setups. I'm decent with Linux, decent with Apache, decent with MySQL, and somewhat advanced with Perl. Small problem: I don't have MySQL on my laptop which is where I needed to run everything. However, I had installed PostgreSQL version 8.3 for another project. I know almost nothing about PG. Fortunately, I had Mr. William King (quentusrex on IRC - #freeswitch on around and he gave me a few pointers. ("In MySQL I did it this way, how do I do that in PG?") A tip o' the cap to him. So at this point I was all set.

My configuration at the start looked like this:
MacBook Pro with OS X Leopard
Latest SVN of FreeSWITCH
Apache 2.2
Perl 5.8.8
PostgreSQL 8.3

So the first thing I decided to do was create a new database in PG, then create a users table. With William's help and the handy pgadmin3 program I was in business. For kicks, I went back to my table and added an extra column: context. I didn't need this column but I thought it would be a nice little exercise to add this value for each user even though I would usually just want "default" for the context. My table ended up looking like this:

uid bigserial NOT NULL,

lastname character varying(15),
firstname character varying(15),
username character varying(11) NOT NULL,
mailbox character varying(11) NOT NULL,
"password" character varying(11) NOT NULL,
context character varying(11) DEFAULT 'default'::character varying,

Here's a nice little pic of the data I put in there thanks to the "edit data" widget in pgadmin3:

I put first and last names in separate fields because I figured that would be common in a real world scenario. I have the username field which is what I'm kinda using for everything: userid, auth. username and username. (Yeah, I'm cheating, get over it.) I have also a password and a context field. That's it. At this point I was feeling confident. I had the database, a users table, a login role for accessing the database, and a little momentum on my side.

Now I needed to decide which task to handle next: handling the CGI call or figuring out how to query the database. I decided to tackle what I felt was the tougher of the two: handling the xml_curl call and returning information that FreeSWITCH could use. It turns out that it wasn't actually all that tough: I opened up conf/autoload_configs/xml_curl.conf.xml and found this near the top:

<!-- The url to a gateway cgi that can generate xml similar to what's in this file only on-the-fly (leave it commented if you dont need it) -->
<!-- one or more |-delim of configuration|directory|dialplan -->
<!-- <param name="gateway-url" value="" bindings="dialplan"/> -->

I'm no genius but to me that's pretty clear information: I needed a 'binding' for directory. For kicks I left in the 'directory' binding as well. I ended up with this:

<param name="gateway-url" value="" bindings="directory|dialplan"/>

I saved everything, started up FS and promptly realized that I didn't actually build mod_xml_curl. No worries. A quick trip to the FS source directory did the trick:

cd /usr/src/freeswitch.trunk
make mod_xml_curl-install

Gotta love FreeSWITCH - way easy. (I added mod_xml_curl to modules.conf and modules.conf.xml so that it would auto build and auto load, respectively.) I loaded up mod_xml_curl and saw no errors or warnings. I saw that it added a new app named "xml_curl" so I looked at the syntax. (Just type 'xml_curl help' to see the syntax.) Basically it just allows you to turn on debug for xml_curl, which dumps a copy of the information returned from the webserver when xml_curl makes a call. The FS CLI shows the file name and path when it creates this debug file.

The next order of business was to observe what my CGI script would see in terms of information coming from xml_curl. Ah Perl, my lovely Swiss-army chainsaw. That was easy:

use strict;
use warnings;
use Data::Dump qw(dump); # I prefer this to Data::Dumper
use CGI.

my $q = CGI->new;
open(FILEOUT,'>','/tmp/cgidump.txt'); ## look in this file to see what's up
print FILEOUT dump($q);

But where to put this lovely file? I poked around to find my default Apache configs and it turns out that there's an alias for "cgi-bin" so that's where I put my Perl script to handle the CGI call. That's it. Now I can see what's coming in. All I need to do is make a phone call or have a user try to register to FS. (Recall that I set up an xml_curl binding for both 'directory' and 'dialplan'.) Dialing out was easier since I didn't have a phone handy. Naturally since my script didn't return anything useful there were errors on the FS CLI, but I was more interested in what was in my cgidump.txt file:

".charset" => "ISO-8859-1",
".fieldnames" => {},
".parameters" => [
"Answer-State" => ["answered"],
"Call-Direction" => ["inbound"],
"Caller-Caller-ID-Name" => ["FreeSWITCH"],
"Caller-Caller-ID-Number" => ["0000000000"],
"Caller-Channel-Name" => ["portaudio/9999"],
"Caller-Context" => ["features"],
"Caller-Destination-Number" => ["is_zrtp_secure"],
"Caller-Dialplan" => ["XML"],
"Caller-Network-Addr" => [""],
"Caller-Privacy-Hide-Name" => ["false"],
"Caller-Privacy-Hide-Number" => ["false"],
"Caller-Screen-Bit" => ["true"],
"Caller-Source" => ["mod_portaudio"],
"Caller-Unique-ID" => ["718c69cd-48cf-4701-90b1-ab2e471cfb6d"],
"Channel-Name" => ["portaudio/9999"],
"Channel-Read-Codec-Name" => ["L16"],
"Channel-Read-Codec-Rate" => [8000],
"Channel-State" => ["CS_EXECUTE"],
"Channel-State-Number" => [4],
"Channel-Write-Codec-Name" => ["L16"],
"Channel-Write-Codec-Rate" => [8000],
"Core-UUID" => ["9b4e5fe9-9bc9-4bed-915a-3c277cbd760b"],
"Event-Calling-File" => ["mod_dialplan_xml.c"],
"Event-Calling-Function" => ["dialplan_xml_locate"],
"Event-Calling-Line-Number" => [368],
"Event-Date-GMT" => ["Thu, 29 Oct 2009 05:18:05 GMT"],
"Event-Date-Local" => ["2009-10-28 22:18:05"],
"Event-Date-Timestamp" => ["1256793485008872"],
"Event-Name" => ["REQUEST_PARAMS"],
"FreeSWITCH-Hostname" => ["michael-collinss-macbook-pro.local"],
"FreeSWITCH-IPv4" => [""],
"FreeSWITCH-IPv6" => ["::1"],
"Hunt-Caller-ID-Name" => ["FreeSWITCH"],
"Hunt-Caller-ID-Number" => ["0000000000"],
"Hunt-Channel-Name" => ["portaudio/9999"],
"Hunt-Context" => ["features"],
"Hunt-Destination-Number" => ["is_zrtp_secure"],
"Hunt-Dialplan" => ["XML"],
"Hunt-Network-Addr" => [""],
"Hunt-Privacy-Hide-Name" => ["false"],
"Hunt-Privacy-Hide-Number" => ["false"],
"Hunt-Screen-Bit" => ["true"],
"Hunt-Source" => ["mod_portaudio"],
"Hunt-Unique-ID" => ["718c69cd-48cf-4701-90b1-ab2e471cfb6d"],
"Presence-Call-Direction" => ["inbound"],
"Unique-ID" => ["718c69cd-48cf-4701-90b1-ab2e471cfb6d"],
escape => 1,
hostname => ["michael-collinss-macbook-pro.local"],
key_name => [""],
key_value => [""],
section => ["dialplan"],
tag_name => [""],
"variable_channel_name" => ["portaudio/9999"],
"variable_current_application" => ["execute_extension"],
"variable_current_application_data" => ["is_zrtp_secure XML features"],
"variable_endpoint_disposition" => ["ANSWER"],
variable_playback_ms => [0],
"variable_playback_samples" => [0],
"variable_playback_seconds" => [0],
variable_read_codec => ["L16"],
variable_read_rate => [8000],
variable_write_codec => ["L16"],
variable_write_rate => [8000],
"variable_zrtp_secure_media" => ["true"],
}, "CGI")

Looking closely you can probably guess that I just used the pa (portaudio) command to dial 9999. I was in business! That's all for part 1 of this subject.

To recap: I thought I would try setting up a proof-of-concept script for using mod_xml_curl in FreeSWITCH. I already had Apache, Perl, and PostgreSQL installed on my laptop, and the latest FreeSWITCH SVN. I configured xml_curl.conf.xml to call my CGI script: I did a very simple data dump of the params sent into the script from xml_curl. This allowed me to verify that communications from xml_curl to my CGI script were functional.

In part 2 I'll talk about sending a response back to the server and setting up the DBI to query a database.

Sunday, October 25, 2009

VoIP: Why You Should Try FreeSWITCH

VoIP is here to stay. Do a google search on "PBX" or "telephone system" and you'll see there's no going back to the TDM-only days. I'm an old-school PBX guy and I can definitely see the writing on the wall. I actually like TDM, but the advantages of VoIP are just too hard to ignore or dismiss.

So, you want to get into VoIP, but you aren't sure where to begin. There are lots of products out there claiming to be "what you need." Of course, you may not know exactly what you need. Do you need a PBX? A soft-switch? An application server? Something else? It's hard to know when you're just getting started. Even if you've already started down the path with a particular piece of software (or hardware for that matter) there's something you need to know about: FreeSWITCH.

Before I continue, let me just say that I am a bit of a fanboy. I am certainly biased. If you can look past my obvious homerism and think about what FreeSWITCH can do for you then I think you might be pleasantly surprised. FreeSWITCH is extremely scalable and versatile. It is designed to meet many different needs. So, what exactly is FreeSWITCH?

FreeSWITCH is a cross-platform, scalable soft-switching library. In other words, it can be embedded in other software, like .so or .dll files are. Yes, it runs on Windows. It runs reasonably well, I might add. Most people don't embed FreeSWITCH in their own software but instead run the "freeswitch" application. This is like the difference between cURL (the program) and libcurl (the underlying software library). The default FreeSWITCH install is a simple but powerful PBX. However, it can scale up or down. For example, you can use it as a soft-phone or a soft-switch. Since it is a telephony switching engine that can scale you can use it in many forms. I use it as a soft-phone on my laptop. (I find it to be much more stable than Ekiga.) There are others who use it as a carrier class soft-switch serving many thousands of customers. FreeSWITCH will scale with your hardware, so you can use it as you see fit.

The scalable nature of FreeSWITCH helps you. You want to get into VoIP but don't know exactly what you need? FreeSWITCH can scale to meet your needs. It's a great choice for a telephony platform. So if you want to build a SOHO PBX or a large-scale enterprise PBX or a VoIP-PSTN gateway then FreeSWITCH can help. So what can't it do? FreeSWITCH operates as a back-to-back user agent (B2BUA) which means that it is not designed to act as a SIP proxy. Instead, it is meant to complement a SIP proxy. (If you need a SIP proxy then stick with OpenSIPS or one of its cousins.)

One thing that FreeSWITCH does very well is handling media. It can handle audio streams at 8kHz, 16kHz, 32kHz, and 48kHz. It can handle numerous codecs, including some really nice HD codecs like Polycom's Siren(tm) codecs or the fantastic CELT. Best of all, it can bridge audio streams from any codec/sampling rate to any other codec/sampling. I know that may not sound like a big deal, but just try it with other systems. Try having 20 people in a single conference where they are all calling in with different codecs and sampling rates and see how well they can hear each other. I don't know of any other system that can handle sampling rates above 16kHz. FreeSWITCH will go up to 48kHz. If you have a good headset and ear phones then the high definition codecs sound incredible. (Once you go HD voice you will never want to go back to standard 8kHz analog phone lines or worse, GSM cell phone quality calls.)

FreeSWITCH is extremely well-designed and is very stable. It can scale far beyond its competitors on the same hardware. While 10,000 call legs on Asterisk was prize-worthy (see the Asterisk S-Prize) the FS devs were playing around with 20,000 call legs just for fun. They long ago found out the levels where the number of UDP ports runs out as well as how easily they could saturate a 100MB Ethernet connection. Bragging rights aside, the number of concurrent calls on a single box isn't that great a measurement of how awesome your telephony platform really is. Just because you can run 16,000 call legs on one box does not mean you should, at least not in production. One hiccup taking out 8,000 calls is probably not good for business.

FreeSWITCH is configured with XML files, or with a back-end database that serves XML configuration information. This is very scalable - one can have multiple FreeSWITCH boxes receiving configuration information from a redundant db cluster. FreeSWITCH is very modular, and in the XML configuration you can enable or disable various modules. Don't need a module? Disable it. You enable only the modules that you need.

One other great thing about FreeSWITCH: the hooks. You can enable the built-in HTTP server and send commands and receive the output as text, XML or HTML markup. FreeSWITCH can output CDRs in CSV files or with the very powerful cdr_xml module. CDRs in XML are amazing as it is, but in addition to dumping XML to a file, FreeSWITCH can POST XML CDRs to a Web server that can in turn handle the db back-end. This allows for easy off-line handling of CDRs. The power of the XML CDRs cannot be overstated. If you've ever dealt with the drama of CDRs on an Asterisk system then you know what I'm talking about. When it comes to CDRs, it's better to have too much information rather than not enough. Single-record CSVs are okay, but XML CDRs give you a complete call history. You can log the A leg, B leg, or both legs. Each leg's XML CDR file has a field to identify the "other" leg so that you can logically associate two XML CDR files and create a complete call record. The greatest "hook" in FreeSWITCH, though, is the event socket.

FreeSWITCH has an amazing eventing system where modules send event messages to the core and receive messages from the core. This allows for agnosticism in the design of the system. Disparate modules don't have to know anything about each other. All they have to do is send messages to the core and act upon messages from the core. The event socket is an extension of this eventing system. An external process can connect to FreeSWITCH and subscribe to these events and act accordingly. The process can control calls, record stats, monitor the system, etc. Any programming language that supports socket programming can be used to control FreeSWITCH. It really is amazing.

Try FreeSWITCH, I promise you will be impressed. Even if you are totally satisfied with your existing system, be it OSS like Asterisk and YATE, or proprietary like 3CX, I recommend giving FreeSWITCH a test drive. Some have done so and immediately decided to add FreeSWITCH to their repertoire. Others needed functionality or stability that Asterisk or other systems could not provide. What can FreeSWITCH do for you? Among other things it will give you one of the most powerful telephony engines in the world and you won't have to spend a dime on software. At the very least you will have an ace up your sleeve the next time you need to build a VoIP system.