Cisco RTMT – No Services/System Logs

The “No service/System Logs selected for trace collection” error on Cisco RTMT usually happens for two reasons:
1- The RTMT version does not match the system version.
2- There is a DNS problem.

Identify a DNS problem is quite simple if you perform a DNS lookup or a packet capture like the one below.

Another easy way to stop for DNS problems is to pay attention to the number of servers listed when collecting the logs, in my example below none of the servers are listed, the “All Servers” columns will always be present.

A simple edit to the host file of the machine will solve the problem very quickly but eventually this needs to be addressed at the DNS level of the organization.

How to upgrade Cisco Expressway

It is funny how detailed the upgrade of other collaboration appliances is very detailed but there is zero official documentation on how to upgrade Cisco Expressway. It is not complicated at all! But is frustrating that they cannot have a document about it, so I hope this helps.

1. Identify and download the correct file
from the Cisco Software Central website.
There are usually two files, the *.ova file is used for new installations while the *.tar.gz file is used to upgrade existing system.

2. Login to Expressway and go to Maintenance > Upgrade.

3. Optional but highly recommended, create a backup file.

4. Select the upgrade file and start the upgrade.

Configuring AnyConnect using only the CLI

The below configuration will allow remote clients using the AnyConnect client to connect as well of having access to clientless WebVPN version to download the client in case they need to.
This configuration does not consider the use of a certificate but you could follow Cisco’s article Configure ASA: SSL Digital Certificate Installation and Renewal to do so. I’ll also assume that basic configuration and routing is already configured and working for a simple design like the following:

Download the AnyConnect packages from Cisco.com and store them on the ASA flash. At the time of writing, the latest version is 4.9 so the files we will need are:

anyconnect-win-4.9.00086-webdeploy-k9.pkg
anyconnect-macos-4.9.00086-webdeploy-k9.pkg
anyconnect-linux64-4.9.00086-webdeploy-k9.pkg

Additionally, download and modify based on your needs the default AnyConnect profile (to make the download easy, it is on *.docx format, but it must uploaded as XML to the ASA flash). This XML file can be created manually or using the Standalone Profile Editor (tools-anyconnect-win-4.9.00086-profileeditor-k9.msi). The most common settings that usually I change are below.

Now to the configuration!

conf t
! Make sure that HTTPs is preferred
http redirect outside 80
!
! DTLS 1.2 requieres ASA 9.10(1)+ and AnyConnect 4.7+
ssl server-version tlsv1.2 dtlsv1.2
ssl client-version tlsv1.2
ssl cipher default high
ssl dh-group group24
! To avoid ASDM incompatibilities problem we will keep the
! SSL cipher as default for TLS 1.2
ssl cipher tlsv1.2 medium
!
! Set the IP range for the AnyConnect VPN clients
ip local pool ANYCONNECT-POOL 192.168.100.1-192.168.100.254 mask 255.255.255.0
!
! Define an object contanting the AnyConnect VPN subnet
object network OBJ-ANYCONNECT-SUBNET
subnet 192.168.100.0 255.255.255.0
!
! Define the list of subnets that will be protected by the VPN
access-list SPLIT-TUNNEL standard permit 10.10.10.0 255.255.255.0
!
! Enable clientless WebVPN
webvpn
enable outside
anyconnect enable
! This enables the group list before authentication
tunnel-group-list enable
! list all AnyConnect images
anyconnect image disk0:/anyconnect-win-4.9.00086-webdeploy-k9.pkg
anyconnect image disk0:/anyconnect-macos-4.9.00086-webdeploy-k9.pkg
anyconnect image disk0:/anyconnect-linux64-4.9.00086-webdeploy-k9.pkg
! list the default VPN profile with a name of “DEFAULT”
anyconnect profiles DEFAULT disk0:/AnyConnect-Default-Profile.xml
!
! The Group Policy will use all the compones previously configured
group-policy GroupPolicy_ANYCONNECT-PROFILE internal
group-policy GroupPolicy_ANYCONNECT-PROFILE attributes
address-pool value ANYCONNECT-POOL
dns-server value 10.10.10.20 10.10.10.21
wins-server none
split-tunnel-policy tunnelspecified
split-tunnel-network-list value SPLIT-TUNNEL
default-domain value petenetlive.com
! SSL and DTLS is prefered in most cases
vpn-tunnel-protocol ssl-client
webvpn
anyconnect profiles value DEFAULT type user
! Post-autenthication Banner (optional)
! banner value LINE 1 — LINE 1
! banner value LINE 2 — LINE 2
! banner value LINE 3 — LINE 3
!
! The Tunnel Group will list custom elements per group
tunnel-group ANYCONNECT-PROFILE type remote-access
tunnel-group ANYCONNECT-PROFILE general-attributes
! Local autenthication
authentication-server-group LOCAL
default-group-policy GroupPolicy_ANYCONNECT-PROFILE
tunnel-group ANYCONNECT-PROFILE webvpn-attributes
! Dropdown option label as “Users”
group-alias Users enable
!
! no nat
nat (inside,outside) source static any any destination static OBJ-ANYCONNECT-SUBNET OBJ-ANYCONNECT-SUBNET no-proxy-arp route-lookup
!
! Allow ICMP inspection
policy-map global_policy
class inspection_default
inspect icmp

Fortigate – Link Monitor

Fortinet will push for the use of their SD-WAN solution on Fortigate firewalls to bundle and load balance Internet circuits using a link health monitor that you can configure from the GUI, but bundling your Internet connections is not always an option and that’s why that I often find myself setting a link health monitor. Don’t you hate when manufacturer re-use names for the similar features?

Either solution work very similar to an IP SLA on Cisco, where you define a test criteria and some actions if the test in question fails. The difference is that the link health monitor can be configured on the GUI and only applied to SD-WAN interfaces, while link monitor is only configured via the CLI.

A simple ping test can be configured with just a couple of commands, by default the result of a failed test is to remove the associated routes with that interface out of the routing table.

config system link-monitor
edit outside_primary_internet
set srcintf port1
set server 8.8.8.8
next
end

Additional settings can be defined like protocol and alert sensitivity, for more information refer to the official documentation: https://kb.fortinet.com/kb/viewContent.do?externalId=FD36151

OpenSSL Basics: Generate self-signed certificate

OpenSSL allows users to perform various SSL tasks, including private keys generation, Certificate Signing Request (CSR) and SSL certificate installation. Most of the Linux distributions come with OpenSSL pre-compiled, but if you’re on a Windows system, you can get it from here. This it is not an exhaustive list of all the things you can do with OpenSSL but the most common uses for non-developers like me, since I find myself creating/manipulating certificates for test and prod environments from time to time.

As always, preparation is the key so double check your OpenSSL version.

openssl version –a

Create an encrypted private key using RSA with a 2048-bit key.

openssl genrsa -aes256 -out private_key_example_com.pem 2048

Create a CSR using the private key generated in the previous step and complete the certificate details as needed.

openssl req -new -key private_key_example_com.pem -out example_com.csr

Sign the CSR created earlier with a valid period or 365 days.

openssl x509 -req -days 365 -in example_com.csr -signkey private_key_example_com.pem -out certificate_example_com.crt

Review the content of all the files created previously.

openssl rsa -noout -text -in private_key_example_com.pem
openssl req -noout -text -in example_com.csr
openssl x509 -noout -text -in certificate_example_com.crt

 

Looking for something easier? DigiCert offers a CSR Tool to generate the private key and CSR file in one single step using OpenSSL.

 

Other useful commands:

Check the certificate of a website

openssl s_client -showcerts -connect example.com:443

From PEM to PKCS#12

openssl pkcs12 -export -inkey private_key_example_com.pem -in certificate_example_com.crt -out bundle_example_com.pfx

To extract the Private Key

openssl pkcs12 -in bundle_example_com.pfx -nocerts -nodes -out exp_private_key_example_com.pem

To extract the SSL certificate

openssl pkcs12 -in bundle_example_com.pfx -nokeys -clcerts -out exp_certificate_example_com.crt

Finally, this is a great article explaining how to validate that a private key matches a signed certificate.

CUCM and CUC SIP integration

The SIP integration between CUCM (Cisco Unified Communication Manager) and CUC (Cisco Unity Connection) in my opinion is the best way to enable voice mail features to any environment, it is cleaner and easier.

CUCM configuration.

Create a new SIP Trunk Security Profile, very similar to the one we created for the IM&P Integration.

Create a SIP trunk pointing to ALL Unity servers.
In addition to all the required fields, pay attention to the ones marked below. The Calling Search Space (CSS) used on this trunk must have access to the partition of the Voice Mail Pilot created below.

Following best practices create a Route Group, Route List and Route Patter for Unity.
Do not forget to check “Run On All Active Unified CM Nodes” on the Route List.

Create a Voice Mail Pilot and Voice Mail Profile.
Set both to be the system default (or edit the default one already on the system).

Unity configuration.

Create a new Phone System under Telephony Integration > Phone System (or edit the default).

Save and then Add a Port Group from the Related Links menu.

Once it is saved, we can add additional CUCM servers under Edit > Servers.

Finally, under Telephony Integration > Ports.
Add the number of ports supported by the system and make sure all port behaviors are enabled.

The configuration is completed!

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Unity uses SIP NOTIFY messages to turn ON or OFF them MWI (Message Waiting Indicator) light; this is one of the reasons why SIP Trunk Security Profile settings and the Out-Of-Dialog Refer CSS are so important since CUC needs to “call” the remote endpoint and for that, it needs the correct permissions to do so.

The reverse happens once the message has been listen/deleted
Messages-Waiting = NO
lampMode = OFF

CUCM: Hunt timers and settings

I look for this reference more often that I would like to admit, so I’ll put it here as personal reminder.

RNA Reversion timeout: Number of seconds to ring at each line in the line group.

Maximum Hunt Timer: Total of seconds a call will be tried within the hunt group before using the Forward Hunt No Answer setting.
The default value specifies 1800 seconds (30 minutes). This timer cancels if either a hunt member answers the call or if the hunt list gets exhausted before the timer expires [1].

A common question is how “Use Forward Settings of Line Group Member” works and it is important to remember that Hunting ignores the Call Forward No Answer (CFNA), Call Forward Busy (CFB), or Call Forward All (CFA) configured values for the attempted party [2].

When “Use Forward Settings of Line Group Member” is used, the appropriate Forward No Coverage settings of the last phone dialed will be used.

Packet captures on a Cisco UC environment

There are multiple ways to capture packets on a network but since not all the networks are created equal, this post will focus in the two most common elements of any Cisco UC deployment; an IP Phone and CUCM.

Collecting a packet capture from a Cisco IP Phone.

1- Connect the phone to the switch and the computer doing the capture as normal.

2- Enable Span to PC port.
On Call Manager Administration go to Device > Phone > Select the appropriate phone.
Set Span to PC Port as enable.

Save and Apply Config so the phone can retrieve the new configuration file.

3- Start collecting files using Wireshark.
For more information check this video, in future posts we will dissect a packet capture step by step.

4- Disable Span to PC port.
For security reasons, disable this feature once troubleshooting and/or packet capture is completed.

Collecting a packet capture from a CUCM

Due to the huge number of packets this could collect, it is recommended to limit the capture with the correct filter.

1- Login to CUCM command line interface (CLI).
Theirs is only one command here, utils network capture but endless filter options.

2- Set the correct filters.
The below examples are not an exhaustive list of all the possible options but rather a guideline on how to use this command, please refer to the documentation for all the options.

Syntax:

  • file fname : send output to a file, platform/cli/fname.cap.
  • port num : limits capture to a specific port number (either source or destination port).
  • host prot addr : limits capture to traffic to and from a specific host. Options for prot [IP | arp | rarp | all ], and addr should be in IPv4 or hostname format. If host is used, src or dest should not be provided.

Examples:

utils network capture eth0 – captures IP packets on the specified Ethernet interface.
utils network capture eth0 file MYCAPTURE – captures on eth0 and save the capture as MYCAPTURE.pcap.
utils network capture eth0 file MYCAPTURE port 5060 – MYCAPTURE.pcap will only contain packets with port 5060 (SIP).
utils network capture eth0 file MYCAPTURE port 5060 host all 10.2.3.4 – MYCAPTURE.pcap will only contain packets from and to 10.2.3.4 with port 5060 (SIP).

NOTE: Press Control-C to stop capturing packets.

3- Download the file.
To transfer the file to a SFTP from CLI use the command: file get activelog platform/cli/MYCAPTURE.pcap

To download using RTMT go to Trace & Log Central > Collect Files > Packet Capture Logs.

NOTE: Using RTMT will download all the files available on the specific server(s) during the defined time window.

Cisco IP Phone – Critical RCE Flaw

Cisco released a new advisory on 04/16/2020 regarding a critical flaw in the web server of its IP phones. If exploited, this could could allow an unauthenticated, remote attacker to execute code with root privileges or launch a denial-of-service (DoS) attack.

Proof-of-concept (PoC) exploit code has been posted on GitHub for the vulnerability CVE-2020-3161.
The only way to prevent against this attack is by either disable the web (which is disabled by default) or upgrade the devices to a fixed software.

Below is the list of affected phones:

  • IP Phone 7811, 7821, 7841, and 7861 Desktop Phones
  • IP Phone 8811, 8841, 8845, 8851, 8861, and 8865 Desktop Phones
  • Unified IP Conference Phone 8831
  • Wireless IP Phone 8821 and 8821-EX

Patching the device is always the best option since not only fixes this particular problem, but others that you may not be aware (like CDPwn) and enhance the reliability of the device.

[Source]

CUCM – SAF and CCD Configuration

This article describes how to configure SAF and CCD.

### IOS Router Configuration

conf t
! Enable the EIGRP instance
router eigrp SAF
service-family ipv4 autonomous-system 1
exit-service-family
exit
! Enable Extensible Messaging Client Protocol (XMCP)
service-routing xmcp listen ipv4 port 5050
client username SAFUSER password SAFPASSWORD
domain 1 default

### CUCM Configuration

Advanced Features > SAF > SAF Securiy Profile

Advanced Features > SAF > SAF Forwarder

The SAF Client must match the username.
Select the CUCM Servers that will act as SAF Clients.

NOTE: IPv6 or FQDN are not supported [1].

Device > Trunk

Selecet the correct Service Type and then complete the basic configuration.
No IP or hostname is requiered for this type of trunk.

For the next steps, we pretty much want to define all the elements listed under CCD. For some of then like DN Group and Partition we just need to define the name, I’ll be listing the other ones below.

Call Routing > Call Control Discovery > Hosted DN Group

Define a name

Call Routing > Call Control Discovery > Hosted DN Pattern

This are the patterns that our cluster will advertise.

Call Routing > Call Control Discovery > Advertising Service

Complete the requiered information and make sure the features is enabled.

Call Routing > Call Control Discovery > Advertising Service

Define the partition name and add it to the correct Calling Search Space (CSS) to enable the phones to have access to the CCD learned patterns.

Call Routing > Call Control Discovery > Requesting Service

Complete the requiered information and make sure the features is enabled.

Call Routing > Call Control Discovery > Blocked Learned Pattern

Blocked Learned patterns are optional. Any match will not be added to the call-routing table.
Learned pattern = The entire length of the received pattern is checked.
Learned pattern prefix = The most significant digits are checked.
Romote call control identity = Blocke received routes based on SAF Client ID.
Remote IP = Block routes based on advertising IP address.

SAF status and Learned routes can be checked using RTMT

Sources:
https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/service-advertisement-framework-saf/whitepaper_c11-622512.html
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/saf/configuration/xe-3s/saf-xe-3s-book/saf-xmcp.html
http://kwallaceccie.mykajabi.com/ccd