Cisco Communications Manager – Going Mixed Mode

Introduction

It’s with great enthusiasm I’m coming to you today to talk about Communications Manager encryption! Collaboration and encryption are two of my favorite subjects so bringing them together is exciting. For many years engineers have steered clear of enabling encryption on Communications Manager, but I think it’s time it becomes the default. The position I’m taking isn’t a favored one and I’m hoping the rest of this post may change your mind. Let’s face it — It’s 2016 and security is and has become a really big deal. Ask any CIO or CSO and they’ll tell you encrypted telecommunications inside or outside of their business is important. The ugly truth (and the reason I’m writing this) is that most Communications Manager systems are running completely in the clear.

The Ugly

I’m not going to get into the details of all surface attack areas in the Cisco CUCM solution. The basics are very well understood that the signaling and media is in the clear. It’s easy to access conversations using a simple packet capture in Wireshark and Vomit. Since nearly all Cisco infrastructure devices support some type of packet capture it is a trivial task to record any phone conversation of your choice. If you want to look at some details of VoIP penetration testing I suggest taking a look at Viproy. You’ll find a wealth of information already generated by people who do this for a living. For example – the Tapberry Pi IP Phone is a fairly intrusive device, but would you ever suspect a tampered IP phone?

Now I’m not going to come out saying that enabling CUCM mixed-mode is the end all solution to this problem. It’s merely there to build a foundation on which all other endpoints, trunks, and associated services can be encrypted. Will mixed-mode prevent all of the surface attack areas that’s a part of penetration testing? Nope, not even close. You have to look and think about security within the solution as a concept, not a feature. When you’re bolting on new services like Expressway, Conductor, or Contact Center is it your mode of operation to ignore SSL/TLS, set all SIP to TCP 5060, and just “make it work”? That’s the mode of operation I’d like to see changed among collaboration engineers. Failing back to un-encrypted means something is wrong and I don’t think it’s OK as a long term mode of operation.

The more this is implemented the more it’ll become a standard.

The Good

The primary fear that’s been instilled in collaboration engineers is the thought of a lost security token. Most of us have been there doing an upgrade or certificate management and completely forgot to check if it’s mixed mode. I have news for you – that isn’t the products fault, but it was really bad prior to 10.x. I believe it’s just a lingering fear from everything prior to 10.x that makes us glaze over mixed-mode. When it does go wrong it goes wrong very dramatically. If the phone trusting keychain breaks you’ll have an operational CUCM with zero phones. Understanding the recovery keys that are now in place is fundamental to restoring operations.

Communications Manager 10.x has helped alleviate the concerns when it comes to losing a security token. It’s now a simple operation to enable services, run the commands, and mixed-mode is operational. The focus here is enabling CTL (Certificate Trust Lists) on Communications Manager to show there isn’t anything to worry about. Were also talking about using the embedded “utils ctl” operations and not using hardware eTokens. It’s also good to note that phones no longer talk to the call manager service for certificate operations so it’s an “out-of-band” certificate management.

How does Communications Manager prevent security token loss? The tokenless CTL approach has been enhanced by including an additional SAST (Site Administrator Security Token) in the CTL file for recovery operations. It just so happens that this additional SAST is the ITLRecovery certificate. The benefit should be clear if you’re familiar with ITL operations. The ITLRecovery certificate does not change during hostname or certificate operations, and there is a procedure for manually backing up this key. So you can take this secondary token offline independent of the disaster recovery system. It’s a virtual to physical operation so you can burn the key to CD or save it in a key vault.

Just to reiterate – performing the CTL operation using the command line the CTL file gets signed by the Callmanager.pem private key and includes the ITLRecovery certificate as a second SAST. This should remove your fears of lost security tokens using this new tokenless CTL method. The ITLRecovery key has very clear guidance how to backup this key and I’ll demonstrate it below.

The Activation

I’m going to run through a few things to check before getting to mixed-mode activation. The activation of CTL is based on the assumption all secure by default TVS and ITL operations are working normally. If you want to check all ITL operations this link will walk you through a detailed process. Unified Communications Manager ITL Enhancements in Version 10.0(1)

Certificate management is super important and the last thing you want to happen is a certificate expiring without warning. My prerequisite to any and all certificate operations is to ensure that the certificate monitor is in operation. (I’m making the assumption your CUCM can send e-mails. It better.) My recommendation is to configure certificate monitor to start notifications 90 days before expiration. The default 29 days is to short especially three years later when installers are gone and you’re scrambling to figure out what to do.

I want this thing to annoy all the right people every 7 days. Make it noisy.

Example:

CertMonitor

Now that we have that out of the way it’s important to now check the validity of all current CUCM certificates. Enabling mixed-mode turns on CTL operations and inside that CTL file is the Callmanager.pem. I generally have a rule of thumb that if I’m on a cluster and Callmanager.pem has less than a year to go just spend the extra cycle and reissue it.

So let’s take a look at my lab callmanager.pem and see if it needs any action. If the validity period is less than a year go ahead and reissue Callmanager.pem ensuring its SHA256 and X509v3 compliant. This is an entirely different process so I’m counting on everything is good with Callmanager.pem and ITL operations.

admin: show cert list own

CallManager/CallManager.pem: Certificate Signed by warcop-server1-ca

admin:show cert own CallManager/CallManager.pem

Validity From: Tue Jul 22 16:45:34 EDT 2014

To:   Thu Jul 21 16:45:34 EDT 2019

Great! My lab is still good for a few more years. Let’s backup the ITLRecovery key and you’ll enter your SFTP details to the file.

admin:file get tftp ITLRecovery.p12

Please wait while the system is gathering files info …done.
Sub-directories were not traversed.
Number of files affected: 1
Total size in Bytes: 1717
Total size in Kbytes: 1.6767578
Would you like to proceed [y/n]? y
SFTP server IP: 172.31.255.99
SFTP server port [22]:
User ID: cisco
Password: *****
Download directory: /

Go ahead and save this file to a key vault or burn to CD as this is your primary recovery token for both ITL operations and CTL operations.

Time to activate some services.

Activate the Cisco CTL Provider service on each Cisco Unified Communications Manager server in the cluster.

Activate the Cisco Certificate Authority Proxy service only on the first node in the cluster.

Make sure to activate the CAPF services and CTL provider service before performing any CTL operations. This ensures you do not have to update the CTL again after CAPF is enabled.

Do you have a CTL file already? It’s possible that long ago the cluster was running mixed mode and was changed back to non-secure mode. If this was the case the CTL file left on the TFTP servers may be old and needs deleting. You can check for a ctl file with “show ctl”.

admin:show ctl

Length of CTL file: 0
CTL File not found. Please run CTLClient plugin or run the CLI – utils ctl.. to generate the CTL file.
Error parsing the CTL File.

So we’ve determined that we do not have a CTL file and the “show ctl” command above hints us how to generate it. If you do have a CTL file first make sure you’re completely positive you’ll never need the old one and run “file delete tftp CTLFile.tlv”

The following are the only options for “utils ctl” operations.

  • utils ctl set-cluster mixed-mode
    • Updates the CTL file and sets the cluster to mixed mode.
  • utils ctl set-cluster non-secure-mode
    • Updates the CTL file and sets the cluster to non-secure mode.
  • utils ctl update CTLFile
    • Updates the CTL file on each node in the cluster.

At this point the last remaining set is to activate CTL operation. You’ll need to be prepared to reboot the cluster and all phones will begin downloading the CTL file.

admin:utils ctl set-cluster mixed-mode

This operation will set the cluster to Mixed mode. Do you want to continue? (y/n): Y
Moving Cluster to Mixed Mode
Cluster set to Mixed Mode
Please Restart the TFTP and Cisco CallManager services on all nodes in the cluster that run these services

Now you can verify that the CTL file has been generated and exists in the TFTP directory.

admin:show ctl

The checksum value of the CTL file:
a0352774cac687c17a253497dd521b89(MD5)
133f24b90202b3b744b754f9b9aceb6007d82223(SHA1)
Length of CTL file: 7200
The CTL File was last modified on Tue Mar 15 21:51:16 EDT 2016

(take a look at the end of this post for a full CTL file output example)

The CTL file is going to contain four records and two of which are the SAST tokens with the first record showing the Callmanager.pem signing record and the last showing the ITLRecovery record. At this point you have a valid CTL and you’re ready to reboot the cluster. The command indicates that you only need to restart services, but I favor rebooting the cluster to get everything off to a fresh start.

So this what should display after reboot when you check the Cluster Security Mode Enterprise Parameter. (Ignore LBM Security Mode – it’s just in the screenshot)

ClusterSecure

Congratulations! You’ve put down the foundation to enable encryption across a variety of devices, trunks, and services.

With version 11 it’s important to note that the ITLRecovery key validity period has been extended from 5 years to 20 years. If you perform an upgrade to version 11 this key is not automatically extended because that would break things. Once you’re on version 11 you should go through the ITLRecovery key re-issuance procedure. Once the new key is generated it will be 20 years and then you should perform “utils ctl update CTLfile”.

Hint: Version 11 also introduced something different when you regenerate a Callmanager, ECDSA, or Tomcat certificate. Now that HTTPS configuration downloads are available the TFTP service need be deactivated and reactivated. A service restart isn’t enough to have TFTP pull that new certificate and have the service on port 6972 running with the latest signature.

From an operations standpoint when do I need to run “utils ctl update CTLfile”? Any time you’re performing cluster changing operations such as adding or removing nodes, after restoring cluster nodes, changing IPs or hostnames, and especially anytime you upload or change certificates on the system. I’ve just worked it into my routine to keep the CTL file updated during all certificate operations.

I hope I’ve dispelled some myths about tokenless CTL and you’ll take a second look at enabling mixed-mode. It’s highly unlikely security engineers are going to become familiar with all CUCM security operations so I believe it’s the responsibility of the collaboration engineer to step up. We’re talking a basic understanding of PKI and nothing more than a few keys within the system.

Please feel free to hit me up on Twitter @Warcop if you see something that needs further clarification. Also, feel free to disagree with me!

Important References:

Security Guide for Cisco Unified Communications Manager , Release 11.0(1)

Unified Communications Manager ITL Enhancements in Version 10.0(1)

IP Phone Security and CTL (Certificate Trust List)

Viproy VoIP Penetration Testing and Exploitation Kit

vomit – voice over misconfigured[1] internet telephones

Full CTL File:

admin:show ctl
The checksum value of the CTL file:
f52754c4c647ae3a4f25c4bd24726213(MD5)
296d6a7990cc80ad9bcfa1d747e0a18e34a18df4(SHA1)

Length of CTL file: 7200
The CTL File was last modified on Sat Feb 20 20:08:59 EST 2016

Parse CTL File
—————-

Version:        1.2
HeaderLength:   420 (BYTES)

BYTEPOS TAG             LENGTH  VALUE
——- —             ——  —–
3       SIGNERID        2       105
4       SIGNERNAME      56      CN=cucm1.warcop.net;OU=IT;O=Warcop;L=Atlanta;ST=GA;C=US
5       SERIALNUMBER    19      76:00:00:00:07:F2:F3:96:30:3A:DA:D8:27:00:00:00:00:00:07
6       CANAME          21      CN=warcop-server1-ca
7       SIGNATUREINFO   2       15
8       DIGESTALGORTITHM        1
9       SIGNATUREALGOINFO       2       8
10      SIGNATUREALGORTITHM     1
11      SIGNATUREMODULUS        1
12      SIGNATURE       256
9e  8b  d8  25  78  8f  5d  c0
49  7e  f3  ac  4f  9b  ae  4f
b5  98  98  d  c1  86  30  32
6c  65  78  9b  3a  88  c9  5f
63  5  a2  a6  2d  b9  de  f1
5e  8a  8e  29  cd  84  48  a6
a9  71  86  40  39  20  b2  21
d0  5d  35  e4  9c  69  3b  56
15  c4  cd  f7  93  3c  3  87
eb  56  ba  e1  93  4  14  b3
15  83  69  23  ba  73  e5  20
95  fd  7  21  a4  53  8e  a6
10  3a  3c  e8  85  f0  fc  ee
62  c3  8a  a8  c1  df  e  45
f1  4  8f  1d  ae  46  39  91
d9  2d  c5  6a  8f  5d  3d  e8
54  65  cf  cd  56  d7  16  89
d6  d3  d3  74  91  3d  5c  2a
92  23  3e  d0  a6  40  d2  eb
0  5b  f8  c5  9  e4  aa  3d
39  39  9d  14  6c  7  d7  20
cb  d8  74  64  53  17  2d  3d
ad  8b  e3  c8  fd  b3  63  70
50  a4  15  69  97  c5  e4  a0
f3  bf  78  7c  91  30  fc  41
3e  2e  dd  be  4c  50  3b  60
72  4a  de  76  ee  99  ff  b1
ae  69  c3  b  21  13  f7  b6
94  2  88  fa  d2  e  3b  58
8d  d2  71  f3  d3  93  78  9
9b  98  fe  a2  f5  b9  80  8e
42  44  1b  b  54  88  ea  49
14      FILENAME        12
15      TIMESTAMP       4

CTL Record #:1
—-
BYTEPOS TAG             LENGTH  VALUE
——- —             ——  —–
1       RECORDLENGTH    2       2260
2       DNSNAME         6       cucm1
3       SUBJECTNAME     56      CN=cucm1.warcop.net;OU=IT;O=Warcop;L=Atlanta;ST=GA;C=US
4       FUNCTION        2       System Administrator Security Token
5       ISSUERNAME      21      CN=warcop-server1-ca
6       SERIALNUMBER    19      76:00:00:00:07:F2:F3:96:30:3A:DA:D8:27:00:00:00:00:00:07
7       PUBLICKEY       270
8       SIGNATURE       256
9       CERTIFICATE     1594    0D D1 77 CB E1 C4 64 B3 CB 6E F0 24 69 DE C8 7B DF 47 63 67 (SHA1 Hash HEX)
10      IPADDRESS       4
This etoken was used to sign the CTL file.

CTL Record #:2
—-
BYTEPOS TAG             LENGTH  VALUE
——- —             ——  —–
1       RECORDLENGTH    2       2260
2       DNSNAME         6       cucm1
3       SUBJECTNAME     56      CN=cucm1.warcop.net;OU=IT;O=Warcop;L=Atlanta;ST=GA;C=US
4       FUNCTION        2       CCM+TFTP
5       ISSUERNAME      21      CN=warcop-server1-ca
6       SERIALNUMBER    19      76:00:00:00:07:F2:F3:96:30:3A:DA:D8:27:00:00:00:00:00:07
7       PUBLICKEY       270
8       SIGNATURE       256
9       CERTIFICATE     1594    0D D1 77 CB E1 C4 64 B3 CB 6E F0 24 69 DE C8 7B DF 47 63 67 (SHA1 Hash HEX)
10      IPADDRESS       4

CTL Record #:3
—-
BYTEPOS TAG             LENGTH  VALUE
——- —             ——  —–
1       RECORDLENGTH    2       1100
2       DNSNAME         6       cucm1
3       SUBJECTNAME     53      CN=CAPF-3ca8c4ee;OU=IT;O=Warcop;L=Atlanta;ST=GA;C=US
4       FUNCTION        2       CAPF
5       ISSUERNAME      53      CN=CAPF-3ca8c4ee;OU=IT;O=Warcop;L=Atlanta;ST=GA;C=US
6       SERIALNUMBER    16      4C:F5:F5:88:A1:DB:CB:58:BC:2A:8C:4F:6C:6B:3C:8E
7       PUBLICKEY       140
8       SIGNATURE       128
9       CERTIFICATE     666     D2 BF B8 54 05 0D 62 30 F6 BA 12 98 E0 37 7C 08 E5 CB 64 70 (SHA1 Hash HEX)
10      IPADDRESS       4

CTL Record #:4
—-
BYTEPOS TAG             LENGTH  VALUE
——- —             ——  —–
1       RECORDLENGTH    2       1160
2       DNSNAME         6       cucm1
3       SUBJECTNAME     68      CN=ITLRECOVERY_cucm1.warcop.net;OU=IT;O=Warcop;L=Atlanta;ST=GA;C=US
4       FUNCTION        2       System Administrator Security Token
5       ISSUERNAME      68      CN=ITLRECOVERY_cucm1.warcop.net;OU=IT;O=Warcop;L=Atlanta;ST=GA;C=US
6       SERIALNUMBER    16      69:62:6A:28:E3:76:37:C7:23:75:C2:D9:80:3C:D7:82
7       PUBLICKEY       140
8       SIGNATURE       128
9       CERTIFICATE     696     43 BA 4D 06 24 8D 5A FA 5D FB 6C 28 7A 7F 50 33 BB 72 D8 D2 (SHA1 Hash HEX)
10      IPADDRESS       4
This etoken was not used to sign the CTL file.

The CTL file was verified successfully.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cisco Collaboration Certificates and Security

A lot has changed with CUCM certificate management and requirements. I think it’s only starting to sink in that CUCM is an application and should be treated like any other application that requires certificates. Continuous development has extended to on-prem apps. 

PKI isn’t new and certificates are not new. We’re not talking about bleeding edge technology and the issues that I’m seeing are related to misunderstanding fundamental technologies. Subtract CUCM from the equation and SSL/TLS/PKI principles apply regardless of the product underneath. 

Just because we as voice engineers haven’t “done it this way” before doesn’t mean we should “keep doing it this way”.

I usually start with an assessment of the client and their ability to manage internal PKI. If my assessment is that they haven’t done their PKI correctly or do not understand the concepts I immediately divert to a managed PKI discussion. For example, if I see a single root CA that’s SHA1 and it’s also the issuing CA that’s a thumbs down. If I also see they are lacking MDM or ISE it’s pretty evident adding a single management CA isn’t going to accomplish the end goal.

What is the end goal? Easy.. You don’t want anyone to click through a certificate warning regardless of device. If you as an administrator are clicking through certificate warnings you should seriously consider fixing that. 

Do PKI right or let the professionals do it for you. 

You can save yourself a lot of time, effort, and management headache if you’ll use 3rd party verified certificates internally and externally. The reality is you’re going to spend a fraction of the cost of internal PKI management. 

This also means that your servers should be operating in a domain that is resolvable in the context of the Internet. The service domain and server fields of the collaboration applications should be within an FQDN of a domain that you own. This is a strong recommendation from Cisco and has been a personal recommendation for years. Microsoft has even been advocating this forever but it’s been ignored in a large way. 

Let’s keep it real and stop using IP addresses to define connection points in and out of applications. We’ve all complained to developers before as network engineers if they’ve hard coded IP addresses. However we as voice engineers are doing the same thing. Use fully qualified names and SRV records to connect services together. TLS relies on the naming context of each service connection point and IP addresses in certificates are not acceptable.

Also it’s not reasonable to accept wide use of wild card certificates. What wild cards are intended for and what they’re being used for these days are two drastically different things. If your issuing authority will re-key a wildcard for a single issue SAN that is a step in a better direction but I’m not a fan of that either. You’re putting all your eggs in one basket against that wildcard. 

My final soap box is about secure clusters. A large majority of CUCM clusters are non-secure meaning they use unencrypted communications. Yes the token management hasn’t always been the easiest but this should be a default configuration change moving forward. Version 11 of CUCM has introduced a lot of enhancements in this area. It’s still PKI we are talking but it’s a different key ring. SIP signaling and media encryption inside the network should be just as important as outside the network. More and more we are doing Ethernet handoffs to carriers for the WAN. In reality your voice communications are exposed on those connections unless you’re also running an encrypted WAN. How many of us are running an encrypted WAN? 

Encryption for telecommunications should be a high priority for all enterprises and engineers responsible for telecommunications. It’s been a high exposure area for quite a long time and the conversation needs to shift. We’re not behind the firewall anymore. 

  

Cisco UCM 11 and LDAP Group Filtering

Introduced in UCM version 11 is the ability to synchronize groups from Active Directory. The primary driver is to have Active Directory groups available in the Cisco Jabber contact list.

One problem this brings up is that if you’re synchronizing from the base DN you’ll import all security groups. So you’ll need an effective filter to only get the groups you want. Generally speaking distribution groups are an ideal target for what you want represented in UCM and Jabber. The more granular you get will require more administration in Active Directory.

I’ll first outline the LDAP filters that will look for security groups and filter different types of groups. These are the bit values for each group and you’ll end up using the bitwise value.

All Security Groups with a type of Global
(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483650))

All Security Groups with a type of Domain Local
(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483652))

All Security Groups with a type of Universal
(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2147483656))

Values for the different group types:
Global = 2
Domain Local = 4
Universal = 8
Security Group = 2147483648
Distribution Group = no value

Using the above information we can then build LDAP filters to only import distribution groups. Since a distribution group doesn’t have a value you have to add the NOT operator to the query.

All Global Distribution Groups
(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=2)(!(groupType:1.2.840.113556.1.4.803:=2147483648)))

All Domain Local Distribution Groups
(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=4)(!(groupType:1.2.840.113556.1.4.803:=2147483648)))

All Universal Distribution Groups
(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=8)(!(groupType:1.2.840.113556.1.4.803:=2147483648)))

Now what if you want distribution groups and security groups and want to select certain groups? Depending on how much additional administration you want in Active Directory you can pick a custom attribute. At this point all you have to do is look for the custom attribute not in use and populate it. Exchange 2010 SP2 and higher introduces 5 new multivalued attributes, but in this example we’re still using a custom attribute.

I recommend running some queries to determine if you have any custom attributes currently in use and then picking the next available value. You can use whatever value you would like just as long as it’s descriptive enough why it’s being used. In the examples below I’ve used “CiscoUCM” as a value to indicate the system thats using it in a query.

All Groups with Custom Attribute 1 – (Note the LDAP property is named extensionAttribute1)
(&(objectCategory=group)(extensionAttribute1=CiscoUCM))

Only Universal Distribution Groups with Custom Attribute 1
(&(objectCategory=group)(groupType:1.2.840.113556.1.4.803:=8)(extensionAttribute1=CiscoUCM)(!(groupType:1.2.840.113556.1.4.803:=2147483648)))

I personally prefer the highest granular approach based on universal distribution groups and a custom attribute. This way the control is based on the source information and synchronized Cisco UCM/IMP information is kept to a minimum.

Happy filtering!

Cisco Jabber 11 Links and Videos

Cisco Jabber version 11 is significant in many ways and it’s difficult to outline all of the details in a single post. First and foremost Cisco is continuing to show that cross-platform feature parity, mobile first, and secure first is a leading part of the strategy.

Enterprise groups, persistent chat, and WebEx CMR escalation are three things significant enough to mention. There are some future integrations that will be better shown than blogging about.

There is a lot of work happening back at Cisco and I’m sure the Jabber team backlog is growing. In the meantime I highly recommend taking a look at the videos below to become a little more familiar with Cisco Jabber 11.

Check out these new Jabber 11 Videos

The release notes detailing the new features:

Jabber_Blue

Cisco UC and VMware Latency Sensitivity

With VMware 5.5 showing up everywhere including Cisco collaboration specific hosts I found it was time to look into the use of latency sensitivity. Please don’t run off and turn this on without a proper understanding of what you’re changing on your UC cluster especially if you’re not follow co-residency guidelines. (That was a disclaimer and why wouldn’t you follow co-res guidelines anyway?)

When scoping or designing a UC cluster that involves Unity Connection there was always an extra core reservation for the VMware scheduler. With VMware 5.5 and the ability to specify latency sensitivity we don’t need to have one core sitting idle.

The reason this is important was due to core oversubscription and the co-residency guidelines published by Cisco. For example; if your host has 12 cores you can now safely lay out the VM reservations across all 12 cores and not 11 cores. This only applies if you have an active Unity Connection VM on the host machine. All other VMs on the host must have the latency sensitivity set to “normal”. Only the Unity Connection VMs will be set to “high”.

So now you’re wondering how do I cut it on? First it has to be modified while the machine is powered off, and if you have vSphere Web Client it is simple as “VM Options | Advanced settings | Edit | Latency Sensitivity = High”.

vsphere_web_latency_sens

OK – I see this yellow exlaimation point and what is about to happen? The warning is to let you know this should be performed on a VM that has CPU reservation. If you used the OVA to deploy Unity Connection and have not changed vCPU amounts or reservations you’re in the clear to proceed.

Since most UC clusters and ESXi hosts specific for collaboration are not joined to vSphere you need a way to turn on latency sensitivity without vSphere Web.

  • Using the vSphere client edit the settings of the Unity Connection VM.
  • Click Options | Advanced-General | Configuration
  • First scan to see if “sched.cpu.latencySensitivity” is in the configuration parameters.
  • If the setting does not exist click “Add Row”
    • Name = sched.cpu.latencySensitivity
    • Value = high
  • Click OK | OK

vsphere_gui_latency_sens

Don’t forget to power it back on. 🙂

Cisco Jabber Certificate Warning Again?

There has been a flurry of activity lately when it comes to Jabber and certificate warnings. Jabber adoption is growing and being exposed to a lot of different installations of Communications Manager and IM & Presence.

Not only do we have the adoption rate skyrocketing but we’re also seeing the development cycle becoming extremely fast. In the last 2 months I believe I’ve counted 4 minor releases to just 1 train of Jabber. This is presenting several challenges between the development and the deployment of Cisco Jabber. The technical information, deployment recommendations, and caveats are not being consumed fast enough in the field to understand the nuances of getting it to work. All of this is a great effort by Cisco but I do believe there could be a better way getting technical notes out to the field.

Certificate warnings are something that just isn’t acceptable. If your workaround is telling someone to ‘Click Accept’ then you’re not doing something right. We’re dealing with a client application and parts of the CUCM system that in the past have rarely needed to be touched. If communications manager and presence were installed for telephony only there are many things lacking to get these certificate warnings to go away.

Here is the latest information straight from Cisco regarding the certificates you need to adjust depending on the products you have deployed.

Required Certificates for On-Premises Servers On-premises servers present the following certificates to establish a secure connection with Cisco Jabber:

Server Certificate
Cisco Unified Presence or Cisco Unified Communications Manager IM and Presence Service HTTP (Tomcat) XMPP (cup, cup-xmpp, cup-xmpp-s2s, tomcat)
Cisco Unified Communications Manager HTTP (Tomcat) and CallManager certificate (secure SIP call signalling for secure phone & CTI connection validation) (callmanager, tomcat)
Cisco Unity Connection HTTP (Tomcat)
Cisco WebEx Meetings Server HTTP (Tomcat)
Cisco VCS Expressway Cisco Expressway-E Server certificate (used for HTTP, XMPP and SIP call signaling)

So let’s break this table down into something a little more understandable. On Communications Manager this involves uploaded valid CA certificates to callmanager-trust and tomcat-trust. Once those are uploaded you need to generate a CSR for callmanager.pem and tomcat.pem. Even though you can sign all your CUCM nodes with one SAN certificate there are several caveats with this and my personal recommendation is to continue signing each CUCM node with its own certificate.

On IM & Presence you need to upload valid CA certificates to cup-trust, cup-xmpp-trust, and tomcat trust. Once that is complete you need to generate the CSR and sign cup, cup-xmpp, cup-xmpp-s2s, and tomcat.

On Unity Connection you’ll only need to upload and sign the tomcat certificate.

One Expressway-CORE you’ll need to upload the valid CA certificate and generate a server certificate. On Expressway-EDGE you’ll 100% need a public signed CA certificate for the server certificate. If you’re not going to use Expressway for B2B communications and every device using Jabber MRA has managed certificates you MIGHT get away with a private CA signed certificate, but I still wouldn’t recommend it.

These certificates need to be x509v3 compliant for server authentication, client authentication, and ipsec. If you’re looking for the exact settings needed I have the certificate template EKUs below. All of the certificates can all be signed by a Microsoft CA with an x509v3 compliant template.

Version: V3
SignatureAlgorithm: SHA1withRSA (1.2.840.113549.1.1.5)

Extension: ExtKeyUsageSyntax (OID.2.5.29.37)
Critical: false
Usage oids: 1.3.6.1.5.5.7.3.1, 1.3.6.1.5.5.7.3.5, 1.3.6.1.5.5.7.3.2,

Extension: KeyUsage (OID.2.5.29.15)
Critical: false
Usages: digitalSignature, nonRepudiation,keyEncipherment,dataEncipherment

The callmanager certificate also has keyAgreement usage but this is mutually exclusive of keyEncipherment per RFC 3280. The keyCertSign is also not needed as an EKU because this isn’t a self-signed certificate. Your callmanger certificate is perfectly valid without these two EKUs. My x509v3 template typically includes nonRepudiation to allow the certificate template to be used for VMware servers.

A final reminder is if you truly want to support all client types without doing managed PKI go ahead and save yourself a lot of headache and get public CA signed certificates for all of this.

So you’ve done all of that and you still get a certificate warning?

There is a good possibility you have certificates correct but you’re still getting these prompts. Let’s understand this issue a little deeper before I get to the long term fix. SSL certificate validation is based on the client and server agreeing to the subject name or SAN name. For example, if you have a valid and trusted certificate on a web server and you browse to https://10.1.1.0 you’re going to get a certificate warning from your browser. This is because you told the browser you wanted a secure connection to 10.1.1.0 but the server sends back a certificate the the name webserver.domain.com. The same thing is happening underneath Jabber between CUCM, CUPS and Expressway. When Jabber connects to CUCM via UDS or CTI the server may be responding with an IP address even though you’ve requested the FQDN. SO WAIT – isn’t that a reverse of what you just said? Absolutely it is but the same principle applies. The CUCM and CUPS “System > Server” setting is giving you this problem. So while Jabber tries to connect to CUCM via cucm-node1.domain.com the UDS/CTI is responding with https://10.1.1.20 which will be the IP address of the node.

So now what – I can’t just go and change that “System > Server” value without breaking the whole cluster! This is where some additional pain points may come into play. The “System > Server” value controls quite a few communications between cluster nodes and endpoints. This value is essentially put into the phone configuration files that are downloaded. Over the years Cisco SRND and guides have flip flopped with the recommendation here. There have also been several bugs along the way using hostname or FQDN. Now we’re to a point where FQDN is the standard and any bugs will need to be fixed to support FQDN. My first recommendation is make sure you’re on the latest versions of CUCM and CUPS and this is super important if you’re serious about the Jabber deployment.

Some have opted to put the hostname or IP address of the “System > Server” values into the SAN certificate fields that are uploaded to the servers. While this sounds like an OK workaround that particular SSL certficate it still invalid per the CA/Browser forum. It may “work” but at this point in time it is more of a hack. You can read the full details of this CA/Browser change at https://www.digicert.com/internal-names.htm

I know this post got a little long but it’s really important stuff to get your deployment up to the new standards, using valid certificates, and keeping certificate warning prompts away from your users.

Thanks and feel free to comment or reach me on Twitter!

Expressway 8.5 and Unity Connection

Expressway 8.5 adds the capability to add Unity Connection directly to the Expressway Unified Communications configuration. This gives Jabber mobile and remote access clients the ability to manage voicemail messages via HTTPS to the VMREST(CUMI) API.

In previous versions of Expressway we had to manually add the Unity Connection servers to the HTTP allow list. With this version once we add Unity Connection it automatically updates the HTTP allow list.

A caveat to be aware of similar to the CUCM and IM&P caveat is if you’re using host names or IP addresses for the “System -> Server” setting. In many cases your Unity Connection server is going to be defined by hostname which is default. The thing to note is if your Unity Connection is hostname only and you add this to Expressway your HTTP allow list is going to be incorrect. Expressway will detect hostname only and add only the hostname to the HTTP allow list.

So what’s the fix? Change your Unity Connection server value to FQDN. If you’re running a cluster this does have a little impact where you’ll need to restart services but if it is a single node you can make this change without impact.

Once Unity Connection server value is FQDN go ahead and add or refresh the server on Expressway and the HTTP allow list will update correctly. Just one of the little nuances if you head down this road on Expressway and Jabber MRA cannot connect to voice mail.

Thanks!

Jabber and Group Configuration Files

Jabber 10.6 is bringing a wealth of new features and functionality to the desktop. However we’re getting to a point you might not want to enable all of these slick features for your entire enterprise. So I’ve recently fielded a question how to ‘group’ enable features.

Let’s get a little context with configuration files so you’ll understand when and when not to have group configuration files. If you’d like to spend time reading the deployment and installation guide you can get more details.

Global configuration files applies to all users. This file (jabber-config.xml) is downloaded by the client from your TFTP nodes. (cough.. UDS via HTTPS). Group configuration files applies to subset of users and take priority over global configuration files.

The ‘take priority’ piece is what you need to logically plan for. You may even think of the global configuration file as a baseline for the environment and have features added by a group configuration file only. This may get to be a lot of work so don’t go overboard with group configuration files. You’ll end up creating a lot of work maintaining the CSF devices.

The group configuration file is uploaded and maintained just like the global one. You can create a group naming convention that suits your needs. Some have used something similar such as ‘jabber-config-group1.xml’ or feature specific as ‘jabber-config-hunt.xml’ that takes the global config and adds in the hunt group tab.

So where do you specify this group configuration file? On the CSF, BOT, TCT, or TAB “Cisco Support Field”. Yes I know this isn’t the greatest place to maintain this and it’s already been suggested to break this out into another field but that isn’t the direction we’re going. This is really only a stop gap or you have a specific group of people you need to add a feature or turn off a feature. Executives and Contact Centers come to mind.

So scroll down to “Cisco Support Field” and put in something like this.
configurationfile=jabber-config-group1.xml

You can even get creative with TFTP directories if you already have this going on CUCM. So configurationfile=/folder/jabber-config-group1.xml

If you’re uploading the file for the first time you’ll need to restart your TFTP service. If the file already exists and you’re simply making changes you do not need to restart the TFTP service. Send me a note on Twitter if you’d like me to post an explanation of this.

Another thing to note is that the Jabber client will be notified of this change and if it determines it needs a restart it will prompt the user to log out and sign back in.

I’d recommend creating a test XML file for your deployment so you can test the enabling and disabling of features without affecting the enterprise population. Once you get the hang of policies and working with different files you’ll be able to test and roll out new features very quickly.

Thanks and happy Jabbering!

Unity Connection 10.5 SAML SSO with ADFS

Having a functional SAML SSO product in place is important before attempting to connect any Cisco collaboration applications to it. I’m focusing on Microsoft ADFS because it seems to be the current go-to product for Microsoft centric organizations.

You need to collect and check your federation metadata file before starting any Unity Connection configuration. This file gives you the value you need to configure the custom claim rule on ADFS. You can download the file from https://adfs.domain.com/FederationMetadata/2007-06/FederationMetadata.xml

Open this file and get your entityID and in my configuration this looks like https://adfs.domain.com/adfs/services/trust

Go ahead and collect the file from Unity Connection on the SAML SSO configuration page by clicking “Export All Metadata”. You’ll need this file available to your ADFS management application so copy it to your server.

Now we have everything we need and it’s time to configure ADFS and Unity Connection.

  1. Open the ADFS Management application
  2. Select Add Relying Party Trust.
  3. From Add Relying Party Trust Wizard Welcome page, click Start.
  4. From the Select Data Source screen, click the Import data about the relying party from a file radio button and browse to the Fedlet metadata XML file which you downloaded from the SAML Single Sign-on configuration pages. Click Next.
  5. From the Specify Display Name screen, enter a name in the Display name field. Click Next.
  6. From the Choose Issuance Authorization Rules screen, choose Permit All Users to access this relying party. Click Next.
  7. Review the settings on the Ready to Add trust screen. Click Next.
  8. To add the Claim Rules, from the Finish screen, check the Open the Edit Claim Rules dialog for this relying party trust when the wizard closes check box. Click Close.
  9. From the Edit Claim Rules screen, click Add Rule.
  10. Select the default Claim Rule template Send LDAP Attributes as Claims. Click Next.
  11. From the Configure Claim Rule screen, enter a claim rule name (Example: “Send uid attribute”) in the Configure rule name field.
  12. From the Attribute store drop-down list, choose Active Directory.
  13. From the LDAP Attribute drop-down list, choose the attribute in the directory that the Cisco Unified Communications application end users are synchronized with (typically “sAMAccountName” or “userPrincipalName”).
  14. From the Outgoing Claim Type drop-down list, enter “uid”. Note: “uid” will not appear in the list of drop-down items, you must manually enter it.
  15. Click Finish.
  16. To add a second rule, click Add Rule.
  17. From the Claim rule template drop-down list, select Send Claims Using a Custom Rule.
  18. In Claim rule name field, enter a name (Example: “Send additional attributes”).
  19. See the next paragraph and code and after pasting click OK and Apply.

So now you can paste your custom claim rule to this ADFS text input box. There is a bit of an issue copy and pasting this correctly from the Cisco documentation and the fact it’s plain wrong in Cisco documentation. Note the two modifications below for entityID and FQDN for Unity Connection on the last line. You should be able to copy below, modify two lines and be able to paste into ADFS management.

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"]

=> issue(Type = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier", Issuer =

c.Issuer, OriginalIssuer = c.OriginalIssuer, Value = c.Value, ValueType = c.ValueType,

Properties["http://schemas.xmlsoap.org/ws/2005/05/identity/claimproperties/format"] =

"urn:oasis:names:tc:SAML:2.0:nameid-format:transient",

Properties["http://schemas.xmlsoap.org/ws/2005/05/identity/claimproperties/namequalifier"] =

"https://adfs.domain.com/adfs/services/trust",

Properties["http://schemas.xmlsoap.org/ws/2005/05/identity/claimproperties/spnamequalifier"] =

"unityconnection.domain.com");

Now that all of this has been done you can configure Unity Connection by clicking Enable SAML SSO and following the dialogs. Once you enable SAML SSO obviously you’ll need LDAP users synchronized with administrative privileges on the Unity Connection server. The recovery URL will be available on the start web page just in case ADFS is not functioning.

If everything is working click “Run SSO Test…” and select your administrator account that is synchronized via LDAP. If the redirect works and your browser automatically signs into ADFS you’ll get the wonderful pop-up below.

SSOWorks

Configuring Cisco Jabber Problem Reporting

Cisco Jabber has an administrative feature allowing users to submit problem reports directly from the user interface. When troubleshooting issues with Jabber this is usually one of the first steps. I consider this a better method instead of having the user save the PRT zip file, subsequently lose it on their computer, and then clogging your inbox.

The nuts and bolts of how this works is briefly described in the Cisco Jabber installation and configuration guide. You need a method to submit the problem report and a method to retrieve the problem report. This post is purely focusing on how to submit the problem report to a server and it’s up to you on how to retrieve it. If you’re anticipating using this frequently or in an enterprise you’re going to need to do some light file handling on the server side. The Cisco problem reporting upload mechanism doesn’t randomize file names during the upload.

I’m working with my FreeBSD server specialized for Jabber functions so underneath I have Apache and PHP. The disclaimer I’m going to insert is that is up to YOU to secure your installation. Creating a free-for-all upload location on your network poses a big security risk. My goal here is to give you the basics with Jabber and what it is doing.

Cisco Jabber does an HTTP POST directly to the URL you specified in your jabber-config.xml. This HTTP POST contains the following information so this is merely informational if you would like use a different upload method. The “ciscoID” portion of the POST is generated by the Cisco JID.

POST /uploadprt.php HTTP/1.1
Request Method: POST
Host: yuengling.brewery.com
Full request URI: http://yuengling.brewery.com/uploadprt.php
Type: multipart/form-data
Content-Disposition: form-data; name="zipFileName"; filename="PROBLEM_FEEDBACK_Cisco_Jabber_ciscoID.zip"

Now that you know how Cisco Jabber uploads to your server you need a corresponding catch-all script to accept the upload. Cisco gives you an example HTML form that you can put on your server. This form is optional and not necessary to have Cisco Jabber automatically upload PRTs. The goal was to provide you an example of how you need to accept the ‘zipFileName’ form data.

uploadprt.html

<form name="uploadPrt" action="http://yuengling.brewery.com/uploadprt.php" method="post" enctype="multipart/form-data">
<input type="file" name="zipFileName" id="zipFileName" />

<input type="submit" name="submitBtn" id="submitBtn" value="Upload File" />
</form>

The next piece of the puzzle is the uploadprt.php file and how it interacts with the incoming HTTP POST from Jabber. Jabber uploads a file name that is exactly “PROBLEM_FEEDBACK_Cisco_Jabber_ciscoID.zip” and if you do nothing else to the file name the user will continually overwrite the same file. So I decided to go ahead and add a time-stamp which you can see below in the $uploadfile variable.

I do not recommend you use this code without adding in proper file handling controls. I’m working in a sealed environment so this is the bare essential method to accept the upload. Obviously the $uploaddir needs to be set to your server location you want to store the files.

uploadprt.php

<?php
$uploaddir = '/usr/local/www/apache22/data/uploads/';
$uploadfile = $uploaddir . date('Y_m_d_H_i_s') . basename($_FILES['zipFileName']['name']);
move_uploaded_file($_FILES['zipFileName']['tmp_name'], $uploadfile);
?>

So you’ve put all this together and as always I’m ready to tell Jabber via the jabber-config.xml file where to upload problem reports. It’s important to note that the PrtLogServerUrl parameter points to your upload PHP script and not the HTML form provided in the documentation.

jabber-config.xml

<?xml version="1.0" encoding="utf-8"?>
<config version="1.0">
<Client>
<PrtLogServerUrl>http://yuengling.brewery.com/uploadprt.php</PrtLogServerUrl>
</Client>
</config>

The end result is I have a directory on the server that houses the Jabber problem reports as “2014_12_10_20_19_50PROBLEM_FEEDBACK_Cisco_Jabber_ciscoID.zip”. From here you can work this into any further automation methods such as e-mailing or FTP. The first goal has been met to get the file from the client to the server and from there further automate.

If everything is working you’ll end up with a little window that looks like this. Happy Jabber-ing!

Jabber_PRT_success