How to upgrade Cisco Expressway

It is funny how detailed the upgrade of other collaboration appliances is very detailed but there is zero official documentation on how to upgrade Cisco Expressway. It is not complicated at all! But is frustrating that they cannot have a document about it, so I hope this helps.

1. Identify and download the correct file
from the Cisco Software Central website.
There are usually two files, the *.ova file is used for new installations while the *.tar.gz file is used to upgrade existing system.

2. Login to Expressway and go to Maintenance > Upgrade.

3. Optional but highly recommended, create a backup file.

4. Select the upgrade file and start the upgrade.

Configuring AnyConnect using only the CLI

The below configuration will allow remote clients using the AnyConnect client to connect as well of having access to clientless WebVPN version to download the client in case they need to.
This configuration does not consider the use of a certificate but you could follow Cisco’s article Configure ASA: SSL Digital Certificate Installation and Renewal to do so. I’ll also assume that basic configuration and routing is already configured and working for a simple design like the following:

Download the AnyConnect packages from Cisco.com and store them on the ASA flash. At the time of writing, the latest version is 4.9 so the files we will need are:

anyconnect-win-4.9.00086-webdeploy-k9.pkg
anyconnect-macos-4.9.00086-webdeploy-k9.pkg
anyconnect-linux64-4.9.00086-webdeploy-k9.pkg

Additionally, download and modify based on your needs the default AnyConnect profile (to make the download easy, it is on *.docx format, but it must uploaded as XML to the ASA flash). This XML file can be created manually or using the Standalone Profile Editor (tools-anyconnect-win-4.9.00086-profileeditor-k9.msi). The most common settings that usually I change are below.

Now to the configuration!

conf t
! Make sure that HTTPs is preferred
http redirect outside 80
!
! DTLS 1.2 requieres ASA 9.10(1)+ and AnyConnect 4.7+
ssl server-version tlsv1.2 dtlsv1.2
ssl client-version tlsv1.2
ssl cipher default high
ssl dh-group group24
! To avoid ASDM incompatibilities problem we will keep the
! SSL cipher as default for TLS 1.2
ssl cipher tlsv1.2 medium
!
! Set the IP range for the AnyConnect VPN clients
ip local pool ANYCONNECT-POOL 192.168.100.1-192.168.100.254 mask 255.255.255.0
!
! Define an object contanting the AnyConnect VPN subnet
object network OBJ-ANYCONNECT-SUBNET
subnet 192.168.100.0 255.255.255.0
!
! Define the list of subnets that will be protected by the VPN
access-list SPLIT-TUNNEL standard permit 10.10.10.0 255.255.255.0
!
! Enable clientless WebVPN
webvpn
enable outside
anyconnect enable
! This enables the group list before authentication
tunnel-group-list enable
! list all AnyConnect images
anyconnect image disk0:/anyconnect-win-4.9.00086-webdeploy-k9.pkg
anyconnect image disk0:/anyconnect-macos-4.9.00086-webdeploy-k9.pkg
anyconnect image disk0:/anyconnect-linux64-4.9.00086-webdeploy-k9.pkg
! list the default VPN profile with a name of “DEFAULT”
anyconnect profiles DEFAULT disk0:/AnyConnect-Default-Profile.xml
!
! The Group Policy will use all the compones previously configured
group-policy GroupPolicy_ANYCONNECT-PROFILE internal
group-policy GroupPolicy_ANYCONNECT-PROFILE attributes
address-pool value ANYCONNECT-POOL
dns-server value 10.10.10.20 10.10.10.21
wins-server none
split-tunnel-policy tunnelspecified
split-tunnel-network-list value SPLIT-TUNNEL
default-domain value petenetlive.com
! SSL and DTLS is prefered in most cases
vpn-tunnel-protocol ssl-client
webvpn
anyconnect profiles value DEFAULT type user
! Post-autenthication Banner (optional)
! banner value LINE 1 — LINE 1
! banner value LINE 2 — LINE 2
! banner value LINE 3 — LINE 3
!
! The Tunnel Group will list custom elements per group
tunnel-group ANYCONNECT-PROFILE type remote-access
tunnel-group ANYCONNECT-PROFILE general-attributes
! Local autenthication
authentication-server-group LOCAL
default-group-policy GroupPolicy_ANYCONNECT-PROFILE
tunnel-group ANYCONNECT-PROFILE webvpn-attributes
! Dropdown option label as “Users”
group-alias Users enable
!
! no nat
nat (inside,outside) source static any any destination static OBJ-ANYCONNECT-SUBNET OBJ-ANYCONNECT-SUBNET no-proxy-arp route-lookup
!
! Allow ICMP inspection
policy-map global_policy
class inspection_default
inspect icmp

Fortigate – Link Monitor

Fortinet will push for the use of their SD-WAN solution on Fortigate firewalls to bundle and load balance Internet circuits using a link health monitor that you can configure from the GUI, but bundling your Internet connections is not always an option and that’s why that I often find myself setting a link health monitor. Don’t you hate when manufacturer re-use names for the similar features?

Either solution work very similar to an IP SLA on Cisco, where you define a test criteria and some actions if the test in question fails. The difference is that the link health monitor can be configured on the GUI and only applied to SD-WAN interfaces, while link monitor is only configured via the CLI.

A simple ping test can be configured with just a couple of commands, by default the result of a failed test is to remove the associated routes with that interface out of the routing table.

config system link-monitor
edit outside_primary_internet
set srcintf port1
set server 8.8.8.8
next
end

Additional settings can be defined like protocol and alert sensitivity, for more information refer to the official documentation: https://kb.fortinet.com/kb/viewContent.do?externalId=FD36151

OpenSSL Basics: Generate self-signed certificate

OpenSSL allows users to perform various SSL tasks, including private keys generation, Certificate Signing Request (CSR) and SSL certificate installation. Most of the Linux distributions come with OpenSSL pre-compiled, but if you’re on a Windows system, you can get it from here. This it is not an exhaustive list of all the things you can do with OpenSSL but the most common uses for non-developers like me, since I find myself creating/manipulating certificates for test and prod environments from time to time.

As always, preparation is the key so double check your OpenSSL version.

openssl version –a

Create an encrypted private key using RSA with a 2048-bit key.

openssl genrsa -aes256 -out private_key_example_com.pem 2048

Create a CSR using the private key generated in the previous step and complete the certificate details as needed.

openssl req -new -key private_key_example_com.pem -out example_com.csr

Sign the CSR created earlier with a valid period or 365 days.

openssl x509 -req -days 365 -in example_com.csr -signkey private_key_example_com.pem -out certificate_example_com.crt

Review the content of all the files created previously.

openssl rsa -noout -text -in private_key_example_com.pem
openssl req -noout -text -in example_com.csr
openssl x509 -noout -text -in certificate_example_com.crt

 

Looking for something easier? DigiCert offers a CSR Tool to generate the private key and CSR file in one single step using OpenSSL.

 

Other useful commands:

Check the certificate of a website

openssl s_client -showcerts -connect example.com:443

From PEM to PKCS#12

openssl pkcs12 -export -inkey private_key_example_com.pem -in certificate_example_com.crt -out bundle_example_com.pfx

To extract the Private Key

openssl pkcs12 -in bundle_example_com.pfx -nocerts -nodes -out exp_private_key_example_com.pem

To extract the SSL certificate

openssl pkcs12 -in bundle_example_com.pfx -nokeys -clcerts -out exp_certificate_example_com.crt

Finally, this is a great article explaining how to validate that a private key matches a signed certificate.

CUCM and CUC SIP integration

The SIP integration between CUCM (Cisco Unified Communication Manager) and CUC (Cisco Unity Connection) in my opinion is the best way to enable voice mail features to any environment, it is cleaner and easier.

CUCM configuration.

Create a new SIP Trunk Security Profile, very similar to the one we created for the IM&P Integration.

Create a SIP trunk pointing to ALL Unity servers.
In addition to all the required fields, pay attention to the ones marked below. The Calling Search Space (CSS) used on this trunk must have access to the partition of the Voice Mail Pilot created below.

Following best practices create a Route Group, Route List and Route Patter for Unity.
Do not forget to check “Run On All Active Unified CM Nodes” on the Route List.

Create a Voice Mail Pilot and Voice Mail Profile.
Set both to be the system default (or edit the default one already on the system).

Unity configuration.

Create a new Phone System under Telephony Integration > Phone System (or edit the default).

Save and then Add a Port Group from the Related Links menu.

Once it is saved, we can add additional CUCM servers under Edit > Servers.

Finally, under Telephony Integration > Ports.
Add the number of ports supported by the system and make sure all port behaviors are enabled.

The configuration is completed!

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Unity uses SIP NOTIFY messages to turn ON or OFF them MWI (Message Waiting Indicator) light; this is one of the reasons why SIP Trunk Security Profile settings and the Out-Of-Dialog Refer CSS are so important since CUC needs to “call” the remote endpoint and for that, it needs the correct permissions to do so.

The reverse happens once the message has been listen/deleted
Messages-Waiting = NO
lampMode = OFF

CUCM: Hunt timers and settings

I look for this reference more often that I would like to admit, so I’ll put it here as personal reminder.

RNA Reversion timeout: Number of seconds to ring at each line in the line group.

Maximum Hunt Timer: Total of seconds a call will be tried within the hunt group before using the Forward Hunt No Answer setting.
The default value specifies 1800 seconds (30 minutes). This timer cancels if either a hunt member answers the call or if the hunt list gets exhausted before the timer expires [1].

A common question is how “Use Forward Settings of Line Group Member” works and it is important to remember that Hunting ignores the Call Forward No Answer (CFNA), Call Forward Busy (CFB), or Call Forward All (CFA) configured values for the attempted party [2].

When “Use Forward Settings of Line Group Member” is used, the appropriate Forward No Coverage settings of the last phone dialed will be used.

Packet captures on a Cisco UC environment

There are multiple ways to capture packets on a network but since not all the networks are created equal, this post will focus in the two most common elements of any Cisco UC deployment; an IP Phone and CUCM.

Collecting a packet capture from a Cisco IP Phone.

1- Connect the phone to the switch and the computer doing the capture as normal.

2- Enable Span to PC port.
On Call Manager Administration go to Device > Phone > Select the appropriate phone.
Set Span to PC Port as enable.

Save and Apply Config so the phone can retrieve the new configuration file.

3- Start collecting files using Wireshark.
For more information check this video, in future posts we will dissect a packet capture step by step.

4- Disable Span to PC port.
For security reasons, disable this feature once troubleshooting and/or packet capture is completed.

Collecting a packet capture from a CUCM

Due to the huge number of packets this could collect, it is recommended to limit the capture with the correct filter.

1- Login to CUCM command line interface (CLI).
Theirs is only one command here, utils network capture but endless filter options.

2- Set the correct filters.
The below examples are not an exhaustive list of all the possible options but rather a guideline on how to use this command, please refer to the documentation for all the options.

Syntax:

  • file fname : send output to a file, platform/cli/fname.cap.
  • port num : limits capture to a specific port number (either source or destination port).
  • host prot addr : limits capture to traffic to and from a specific host. Options for prot [IP | arp | rarp | all ], and addr should be in IPv4 or hostname format. If host is used, src or dest should not be provided.

Examples:

utils network capture eth0 – captures IP packets on the specified Ethernet interface.
utils network capture eth0 file MYCAPTURE – captures on eth0 and save the capture as MYCAPTURE.pcap.
utils network capture eth0 file MYCAPTURE port 5060 – MYCAPTURE.pcap will only contain packets with port 5060 (SIP).
utils network capture eth0 file MYCAPTURE port 5060 host all 10.2.3.4 – MYCAPTURE.pcap will only contain packets from and to 10.2.3.4 with port 5060 (SIP).

NOTE: Press Control-C to stop capturing packets.

3- Download the file.
To transfer the file to a SFTP from CLI use the command: file get activelog platform/cli/MYCAPTURE.pcap

To download using RTMT go to Trace & Log Central > Collect Files > Packet Capture Logs.

NOTE: Using RTMT will download all the files available on the specific server(s) during the defined time window.

Cisco IP Phone – Critical RCE Flaw

Cisco released a new advisory on 04/16/2020 regarding a critical flaw in the web server of its IP phones. If exploited, this could could allow an unauthenticated, remote attacker to execute code with root privileges or launch a denial-of-service (DoS) attack.

Proof-of-concept (PoC) exploit code has been posted on GitHub for the vulnerability CVE-2020-3161.
The only way to prevent against this attack is by either disable the web (which is disabled by default) or upgrade the devices to a fixed software.

Below is the list of affected phones:

  • IP Phone 7811, 7821, 7841, and 7861 Desktop Phones
  • IP Phone 8811, 8841, 8845, 8851, 8861, and 8865 Desktop Phones
  • Unified IP Conference Phone 8831
  • Wireless IP Phone 8821 and 8821-EX

Patching the device is always the best option since not only fixes this particular problem, but others that you may not be aware (like CDPwn) and enhance the reliability of the device.

[Source]

CUCM – SAF and CCD Configuration

This article describes how to configure SAF and CCD.

### IOS Router Configuration

conf t
! Enable the EIGRP instance
router eigrp SAF
service-family ipv4 autonomous-system 1
exit-service-family
exit
! Enable Extensible Messaging Client Protocol (XMCP)
service-routing xmcp listen ipv4 port 5050
client username SAFUSER password SAFPASSWORD
domain 1 default

### CUCM Configuration

Advanced Features > SAF > SAF Securiy Profile

Advanced Features > SAF > SAF Forwarder

The SAF Client must match the username.
Select the CUCM Servers that will act as SAF Clients.

NOTE: IPv6 or FQDN are not supported [1].

Device > Trunk

Selecet the correct Service Type and then complete the basic configuration.
No IP or hostname is requiered for this type of trunk.

For the next steps, we pretty much want to define all the elements listed under CCD. For some of then like DN Group and Partition we just need to define the name, I’ll be listing the other ones below.

Call Routing > Call Control Discovery > Hosted DN Group

Define a name

Call Routing > Call Control Discovery > Hosted DN Pattern

This are the patterns that our cluster will advertise.

Call Routing > Call Control Discovery > Advertising Service

Complete the requiered information and make sure the features is enabled.

Call Routing > Call Control Discovery > Advertising Service

Define the partition name and add it to the correct Calling Search Space (CSS) to enable the phones to have access to the CCD learned patterns.

Call Routing > Call Control Discovery > Requesting Service

Complete the requiered information and make sure the features is enabled.

Call Routing > Call Control Discovery > Blocked Learned Pattern

Blocked Learned patterns are optional. Any match will not be added to the call-routing table.
Learned pattern = The entire length of the received pattern is checked.
Learned pattern prefix = The most significant digits are checked.
Romote call control identity = Blocke received routes based on SAF Client ID.
Remote IP = Block routes based on advertising IP address.

SAF status and Learned routes can be checked using RTMT

Sources:
https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/service-advertisement-framework-saf/whitepaper_c11-622512.html
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/saf/configuration/xe-3s/saf-xe-3s-book/saf-xmcp.html
http://kwallaceccie.mykajabi.com/ccd

Configure Cisco Expressway for MRA (12.5.4)

Installing Cisco Expressway it is not terrible difficult, but the whole process itself is extremely finicky and although the servers won’t complain if we skip a few steps they will definitely come back to bit us. I’ll try to break down all the steps, but this may vary depending on the version since more features and functionalities are constantly being added.

The key to success is preparation! Let’s review a few interesting details that most guides omit.

  • Create an Android [1] and iPhone [2] specific SIP Profile.
  • External and internal SRV record must be created [3].
  • Wildcard certificates are NOT supported [4].
  • Open the correct port in your network [5][6].
  • Request your licenses in advance [7].

Step 1 – Download and install the Expressway .ova file from cisco.com. Expressway C and E share the same image, it is the license file what determine its capabilities. If you need additional guidance, please check this video from Paul Stryer.
Interestingly enough I hit a kernel panic bug right after installation (Bug ID CSCvq39993) the fix it is super simple and faster than redeploying the whole VM again.

Condition:
Expressway X12.5.2, X12.5.3 or X12.5.4 has been freshly deployed from OVA on VMware, and has not been upgraded since initial VM installation.

kernel: #PF error: [normal kernel read fault]
kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
kernel: rcu: INFO: rcu_sched self-detected stall on CPU

Workaround:
Upgrade the VM Expressway deployment using the tar.gz file in the web admin UI, to any version, including the same version that was originally deployed. For example, you can upgrade X12.5.4 to X12.5.4 to resolve the issue. Cisco recommends upgrading to the latest available version.

Step 1.5 – Change default password for the admin (GUI) and root (CLI).

Step 2 – Configure NTP servers and time zone under System > Time.

Step 3 – Configure DNS servers, System host name and Domain name under System > DNS.

Step 4 – Configure system name under System > Administration.

Step 5 – Install the correct option keys under Maintenance > Option Keys.

  • LIC-EXP-SERIES
  • LIC-EXP-GW
  • LIC-EXP-E (only on EXP-E)
  • LIC-EXP-AN (Optional for EXP-E if using Dual-NIC)
  • LIC-EXP-RMS (Optional for B2B calls)

Step 6 – Set the Unified Communications mode to “Mobile and Remote access” under Configuration > Unified Communications > Configuration.
OAuth is the recommended authentication method [8], but what nobody tell you is that OAuth also needs to be enabled on CUCM and Unity to work.

On EXP-C it looks like this:

On CUCM and Unity under Enterprise Parameters the OAuth with Refresh Login Flow parameter must be set to Enabled.

Additionally on Unity go to Authz Server to create a new authorization server pointing to the CUCM Publisher [9].

Step 7 – Define SIP domains under Configuration > Domains.
EXP-C requires the external and internal domain defined (example.com and internal.local)

Step 8 – Link EXP-C with CUCM under Configuration > Unified Communications > Unified CM servers.
A dedicated user with Standard AXL API Access role is recommended.

Repeat the above process for IM&P and Unity.

Step 9 – Generate a Certificate signing request (CSR) under Maintenance > Security > Server certificate.
The certificate must contain at minimum the Common Name (expe.example.com) and the domain name as one of the Alternative Names (example.com) listed. [10][11].

Once the CSR is signed by a trusted CA, the following files need to be uploaded, in this order.
Root certificate – Maintenance > Security > Trusted CA certificate.
Server certificate – Maintenance > Security > Server certificate.
Revocation list – Maintenance > Security > CRL management.

Step 10 – Configure the traversal zones under Configuration > Zones > Zones.
Set the zone type as Unified Communications traversal.
Define the traversal Username and Password.
From drop down menu set the Authentication Policy to Treat as Authenticated.
EXP-E should be configured with the EXP-C cluster FQDN.
EXP-C must have the EXP-E Cluster FQDN along with all the EXP-E servers names.

Confirm connectivity by checking the newly created zone.

Step 11 – Enable SIP.
Under Configuration > Protocol > SIP
Make sure SIP and TCP modes are enable for EXP-C and SIP and TLS modes for EXP-E.

Step 12 – Set the HTTP allow list [Pending official documentation].
I had a lot of problems integrating MRA with Cisco Unity for voicemail, EXP-C rejected almost all the HTTP request towards Unity, so I had to add a few entries to the HTTP allow list.
The different variations of the URLs I saw on the logs being rejected with traffic_server[28499]: Event=”Request Failed” Detail=”Access denied” Reason=”No match in HTTP allow list” were the followings:

I created the following rules:

NOTE: Although I’m just showing the entries for my Publisher server, I had to create the same rules for all my Subscribers.

Step 13 – Once everything is ready, we can review your deployment with this online tool: https://cway.cisco.com/csa/ and using our Jabber clients.