cucm

Cisco Collab and Open VMware Tools

Warcop collaboration , , ,

Hi! I’m back with a quick take on Cisco Collaboration UCOS 12.5 and switching to open VMware Tools.

There is a long and bumpy history with native VMware tools on Cisco UCOS collaboration applications.  If you have had your UC solution for many years, then it is likely you have bumped into issues along with the way.  Keeping the native VMware Tools upgraded, installed, and working can be a challenge during upgrades.  So I was very excited in 12.5 you now have the option to move straight to Open VMware Tools.

What are these Open VMware Tools? In short – the better VMware Tools.  The open-vm-tools package is 100% supported by both Cisco and VMware. Moving to open-vm-tools will not take you out of any ‘compliance’ or get you yelled at by TAC.

The first advantage is that you’re de-coupling the ESXi version of tools from the UCOS application.  This means you’ll no longer need the “Check and upgrade VMware Tools before each power on” setting on the guest machine.  The open-vm-tools package is built into CentOS6/7 (and many others) by default, so you’ll no longer rely on the ESXi side of the house to get this right.  For example, if the systems team upgrades ESXi underneath your collaboration application you won’t have the additional worry of VMware Tools staying in sync.

For Cisco this means they simply keep the open-vm-tools package in CentOS6/7 UCOS and can keep it in line during application maintenance. I think it’s a win-win for both sides.

So how do we get there? EASY – but you do require a REBOOT. WARNING! You need to ensure that the native VMware Tools are operational prior to switching to open-vm-tools.  You can check the status of VMware Tools from ESXi. If there is a warning about version or operating system selection you need to fix that first. Also, please be sure you’re working with the latest patched version of the UCOS software release if you’re doing VMware Tools maintenance. There are some bugs and field notices that may get you stuck. Patch stuff!

This is primarily geared for 12.5 so as of this post I’m assuming you’re working with 12.5 SU2. VMware will give you the installed and running status for the tools.

Check that you have native VMware Tools operational

admin:utils vmtools status

Version: 10.3.10.10540
Type: native VMware Tools

Now prior to making the switch and the reboot make sure you or someone else has UN-checked VM Options “Check and upgrade VMware Tools during each power on”.

Move the system to permissive.  This will relax Linux with a setenforce command.  This isn’t called out as required in all locations, but I’d certainly put it in the “recommended” category when making this switch.  You’ll easily be able to move the system back to enforcing.

admin:utils os secure status
OS Security status: enabled
Current mode: enforcingadmin:utils os secure permissive
OS security mode changed to Permissive

Make the switch to open-vm-tools package which will remove the native.

admin:utils vmtools switch open

This will uninstall the native VMware Tools and install the open-vm-tools.
The system will be rebooted automatically.
Do you want to proceed (yes/no) ? yes

The UCOS server will reboot and switch out the native for the open-vm-tools package. VMware will now show both installed and status, but it will read “VMware Tools is not managed by vSphere”.  You can also check at the UCOS CLI again with

admin:utils vmtools status

Version: 10.1.5.59732
Type: open-vm-tools

Don’t forget to change back to enforcing!

admin:utils os secure enforce

In conclusion I think this was a good move by Cisco to bring the VMware Tools swap exposed natively with a UCOS CLI.  Getting away from those native tools and just getting it managed within CentOS is great.

I believe this will remain ‘Optional’ for quite a while, but it’s possible this will become a required changed for CSR14.

For more information from VMware about Open-VM-Tools check out the repo https://github.com/vmware/open-vm-tools

Hit me up @Warcop on Twitter – Thanks!

_______________________________________

Extra VMware Tools troubleshooting.

Got root? (Recovery ISO | Atl+F2) Make sure you’re working in the active partition by checking timestamps on /mnt/part1 or /mnt/part2.

/usr/bin/vmware-uninstall-tools.pl will remove the tools

If you’re in a situation where you don’t have VMware tools in the active partition, then copy them from the inactive side. It’s likely they’re still over there. Find ‘vmware-tools’ directories on the inactive side and copy them over so that you can run the /usr/bin scripts.

Did you get stuck on a reboot and the only thing on the console is “Probing EDD”. Welcome to Field Notice FN70379 where you’ll need to generate new initramfs with “/usr/bin/vmware-config-tools.pl -d”

https://www.cisco.com/c/en/us/support/docs/field-notices/703/fn70379.html

Communications Manager LDAP Groups Caveat

Warcop collaboration , , , , , , , , ,

If you’re wanting to LDAP synchronize Active Directory distribution groups for use with Cisco Jabber you’ll want to pay close attention to the ‘Synchronize’ setting. This setting is found on the ‘LDAP Directory’ configuration page. TLDR – If you’re using different synchronization agreements for users and groups the user directory synchronization must also be selected for ‘Users and Groups’.

Why and what’s happening inside the DirSync service?

If the directory synchronization agreement is set for ‘Users Only’ the LDAP search filter looks like this:
LDAPUsers

If the directory synchronization agreement is set for ‘Users and Groups’ the LDAP search filter looks like this:
LDAPUsersandGroups

You’ll notice at the bottom of the second screenshot the ‘memberof’ attribute request. This is where the user synchronization agreement requests the all of the groups the user is a member of. This also means that if this is the first time you’re setting up the agreements you’ll have to synchronize groups and then users. If you’re adding groups you have to run a full sync on both agreements.

So again — if you’re using different synchronization agreements for users and groups because you’re looking in different containers both agreements need to be set for Synchronize: ‘Users and Groups’. Obviously if you’re using one synchronization agreement to import both users and groups in a single container you wouldn’t run into this little caveat.

I got tripped up on this recently because the setting would seemingly imply it’s what you’re importing and not the search filter. It took a couple of packet captures to figure it out. PCAP or didn’t happen!

CCM packet cap:
utils network capture eth0 file packetcap1 count 100000 size all

jabber

Cisco Collaboration Certificates and Security

Warcop bit-bucket , , ,

A lot has changed with CUCM certificate management and requirements. I think it’s only starting to sink in that CUCM is an application and should be treated like any other application that requires certificates. Continuous development has extended to on-prem apps. 

PKI isn’t new and certificates are not new. We’re not talking about bleeding edge technology and the issues that I’m seeing are related to misunderstanding fundamental technologies. Subtract CUCM from the equation and SSL/TLS/PKI principles apply regardless of the product underneath. 

Just because we as voice engineers haven’t “done it this way” before doesn’t mean we should “keep doing it this way”.

I usually start with an assessment of the client and their ability to manage internal PKI. If my assessment is that they haven’t done their PKI correctly or do not understand the concepts I immediately divert to a managed PKI discussion. For example, if I see a single root CA that’s SHA1 and it’s also the issuing CA that’s a thumbs down. If I also see they are lacking MDM or ISE it’s pretty evident adding a single management CA isn’t going to accomplish the end goal.

What is the end goal? Easy.. You don’t want anyone to click through a certificate warning regardless of device. If you as an administrator are clicking through certificate warnings you should seriously consider fixing that. 

Do PKI right or let the professionals do it for you. 

You can save yourself a lot of time, effort, and management headache if you’ll use 3rd party verified certificates internally and externally. The reality is you’re going to spend a fraction of the cost of internal PKI management. 

This also means that your servers should be operating in a domain that is resolvable in the context of the Internet. The service domain and server fields of the collaboration applications should be within an FQDN of a domain that you own. This is a strong recommendation from Cisco and has been a personal recommendation for years. Microsoft has even been advocating this forever but it’s been ignored in a large way. 

Let’s keep it real and stop using IP addresses to define connection points in and out of applications. We’ve all complained to developers before as network engineers if they’ve hard coded IP addresses. However we as voice engineers are doing the same thing. Use fully qualified names and SRV records to connect services together. TLS relies on the naming context of each service connection point and IP addresses in certificates are not acceptable.

Also it’s not reasonable to accept wide use of wild card certificates. What wild cards are intended for and what they’re being used for these days are two drastically different things. If your issuing authority will re-key a wildcard for a single issue SAN that is a step in a better direction but I’m not a fan of that either. You’re putting all your eggs in one basket against that wildcard. 

My final soap box is about secure clusters. A large majority of CUCM clusters are non-secure meaning they use unencrypted communications. Yes the token management hasn’t always been the easiest but this should be a default configuration change moving forward. Version 11 of CUCM has introduced a lot of enhancements in this area. It’s still PKI we are talking but it’s a different key ring. SIP signaling and media encryption inside the network should be just as important as outside the network. More and more we are doing Ethernet handoffs to carriers for the WAN. In reality your voice communications are exposed on those connections unless you’re also running an encrypted WAN. How many of us are running an encrypted WAN? 

Encryption for telecommunications should be a high priority for all enterprises and engineers responsible for telecommunications. It’s been a high exposure area for quite a long time and the conversation needs to shift. We’re not behind the firewall anymore.