How to change Cisco CIMC Password

The procedure to change CIMC Password

1- Login CIMC GUI

2- Admin -> User Management

3- Click on the Admin Account -> Select Modify user 

4- In the pop up  window tick Chanage Password

5- Enter the new password and Save

6- You don’t need to restart any service or CIMC to effect the changes

7- You can logout and login back to check new password

Cisco UCS firmware upgrade

Recently I have upgraded UCS C240 from firmware to 4.0,



1-      Download UCS CIMC firmware from Cisco.com (Path Downloads Home >>Servers - Unified Computing>>UCS C-Series Rack-Mount Standalone Server Software>>UCS C240 M5 Rack Server Software)

2-      Login CIMC and Launch KVM (I was using Java)

3-      Activate the Virtual Devices from Virtual Media tab

4-      Map the ISO to the CD/DVD option.

5-      Then reboot the system waiting for the option to press F6

6-      select the Cisco vKVM-Mapped option (Cisco vKVM-Mapped vDVD1xx)

7-      Now the system will start copying the files and will perform some checks. This step would take 10-15 mins

8-      Host Upgrade Utility will show you list of upgrades, choose all or Specific

9-      When the upgrade completes all components will have status PASS: this would take 20-40 mins

10-   Click Exit and the server will reboot and activate the new CIMC. Wait for the server to come back up and you will see that CIMC has been upgraded

Ref below document, which talks about UCS firmware upgrade brief with screen shot.

https://www.vcloudinfo.com/2019/09/how-to-upgrade-ucs-firmware.html

CUCM DBReplication reset

When you reset dbreplication  you could check the status using 'utils dbreplication runtimestate'. you would be able to see number of of tables checked aganist the total of 706

Cluster Replication State: Replication status command started at: 2019-11-28-21-14

     Replication status command in PROGRESS. Checked 54 tables out of 706

     Last Completed Table: scratch

     No Errors or Mismatches found.

Cluster Replication State: Replication status command started at: 2019-11-28-21-14

     Replication status command in PROGRESS. Checked 118 tables out of 706

     Last Completed Table: functionrole

     No Errors or Mismatches found.

UC Data center movement

We had to shift our datacenter to another location. We had following components to shift from existing DC to new DC and business wanted to do it without any downtime and no additional cost apart from shipping the hardware.

UCS Servers
CUCM
CUC
UCCX
CUEAC
Expressway C and Expressway E
PLM

Following are the UCS and applications running on each UCS

ESXi01 C240 M4SX	ESXi02 C220-M3S	ESXi03 C220-M3S
EXPRESSWAYC01	EXPRESSWAYC02	UNITY CONNECTION 02
EXPRESSWAYE01	CUCMTFTP01	UCCX 02
CUCM01	EXPRESSWAYE02	CUCMSUB02
PLM01	CUCMSUB01
UNITY CONNECTION01	EXPRESSWAYC02
ATTENDANT CONSOLE1
UCCX01

We had multiple remote sites in the US region, so I have proposed to split it in two different DC's for geographical redundancy. We decided to ship ESXi01 initially and change the network to the new network. All the primary applications were running on ESXi01



Prerequisite


1- Update new IPs in DNS Records,

2- New TFTP IP address need to be updated on AD

3- You can clone Publishers and can use it in case any emergency. However Cisco does not support Cloning and Vmotion



This is what I did pre and post shipment of ESXI01


1- Communicated to local team that During the esxi01 movement activity, VOIP cluster would in READ ONLY mode and cannot make any changes

2- Mannual back up and total registered phone count taken. " Show Risdb query phone " would give you the total registered and unregistered details. Including firmware, phone type, IP address etc.

3- few screen shots from RTMT and CIMC console to make sure there are no major alarms

4- Before shipping changed the ESXi IP address

5-Shipped UCS device via overnight courier service

6-Connected UCS and changed CIMC ( CIMC only for UCS management, this won't impact anything)

7-started with CUCM Publiser IP change followed by Subs

8- CUCM DB didn't sync automatically so I did 'utils dbreplication reset all' then cluster REPLICATION SETUP changed to  (2) Setup Completed. Before it was showing  (3)DB Active-Dropped

9- Then CUC and PLM, CUEAC IP's changed with out issues

10- UCCX we could not change due to bug CSCve59850, I had to open a TAC case to fix this issue, TAC made changes in the root. however later changing IP address UCCX system -> server were showing as 'UNKNOWN", TAC again made changes in the intraclustercomm.ips

11- Expressway we didn't face any issues. Created a new DMZ setup and assigned new IPs and NAT IP. 

12- CUEAC  You need to change IP from Regedit and update it in TSP as well


License


We didn't changed the hostname and Domain name of servers. 


We had procured new license only for UCCX. 


CUC and CUCM license were on PLM and you don't require new license when you make network changes for CUC and CUCM


CUEAC -  Since this is running on windows license will not be void after changing IP


Major issue faced


1- Home office user's phone got unregistered, it was trying to communicate only Primary CUCM. I had created a new Device pool for home office phone and then added only active CUCM 

CCX Historical Datastore Replicate status shows UNKNOWN

I have changed UCCX IP Address to new IP, later when I tried to generate CCX reports I couldn't get any data in the report.  We have redundant UCCX nodes and the secondary node was not shown in HR Datastore. Following are the steps I followed to fix the issue

1

-

Go to 'Cisco Unified CCX Serviceability' -> Tools -> Datastore Control Center -> Replication Servers -> Related Links -> Click Go  -> from Data Datastore select "Cisco Unified CCX Historical Datastore"

I didn't see my secondary node and Replicate Status was showing as ' UNKNOWN'

2-  Click on Synchronize Data -> Now the data will be start sync, you could check the progress -> move your cursor towards Info and click on specs Icon

Once the data sync Job status would be shown as 'Completed'

If this didn't fix the issue, check utils uccx dbreplication status from CCX CLI and perform utils uccx dbreplication reset

Collect Expressway Logs

Collecting Expressway(EXP) Diagnostic Logs



Select Maintenance > Diagnostics > Diagnostic Logging

Check “Take tcpdump while logging” and select “Start new log”
Do this on both EXP E and EXP C and reproduce the problem.
Once the problem is reproduced, stop the Diagnostic logs on each EXP “Stop logging”
Click Download the logs to download it. Log will be download .tar format

The file name will be diagnostic_log_fclxpresswayx01_XXX. tar

When publisher goes down, unable to recognize DTMF for the UCCX calls

I had to shutdown my publisher server and Primary Subscriber for UCS move to different DC. Due to this CUBE media resource got unregistered and it impacted UCCX  customer calls. The system was unable to recognize DTMF. The provider uses  RFC 2833 In a band and UCCX use Out of band, due to mismatch in DTMF you need MTP resource.


Provider -  RFC 2833 In band
UCCX Support only -  Out of band

I have changed following and media resource got registered back to CUCM and UCCX issue got fixed.

sccp ccm x.x.x.x identifier 1 version 7.0

sccp ccm group 1
 associate ccm 1 priority 1