Actions

Upgrading Prime Infrastructure

From Rabbi Blog

Revision as of 15:13, 6 April 2021 by Rabbi Bob (talk | contribs) (→‎Things worth noting)

This was a rather painful journey upgrading Cisco Prime Infrastructure. Along the way the drive needed to be expanded and the love of vm snapshots was made even more solid that before. The following is not intended to be gospel, only a detail of getting from Prime 2.2.0.0.158 up to 3.1.3 that had it's "fun"

References


General Approach

  • Backup the VM \ Create a copy of VM \ turn off source
  • Working from the VM copy and do thing ala the following:
    • download and verify PI-Upgrade-3.0.0.0.78.tar.gz.zip - 678c86f1d9d36a74daafcdf483845c4b
    • extract PI-Upgrade-3.0.0.0.78.tar.gz.zip and verify checksum: 03446c34f1a4ffd5d445c070ee17bfb9
    • Copy file to VM copy
    • CLI: admin# ncs stop
    • CLI: admin# application upgrade PI-Upgrade-3.0.0.0.78.tar.gz defaultRepo
    • Get Coffee & await completion
    • CLI: admin# ncs start
    • verify Prime is running
    • double check RADIUS auth
    • Resync WLC configurations
    • Synchronize devices
    • If fail, turn off VM and revert to copy

Random

  • enable ftp
  • set ftp user password: ncs password ftpuser admin password password
  • ftp to prime, upload file
  • disable ftp
  • CLI: root
    • CLI: mv /ftp/<name of file> ../defaultRepo
    • CLI: exit
  • OR
    • Prime root does wget! (well, apparently it did do root until 3.1.x)
    • CLI: cd /localdisk/defaultRepo
    • CLI: wget http://<url>/PI_3_1_3-1.0.16.ubf
  • Drive Space - if you run into optvol drive space issues after going to 3.0x from 2.x, revert back to the snapshot, add the drive in 2.x, then go forward. A happy Prime is what we want here. In my instance, many NCS CLEANUPs were done to squeeze drive space and eventually the 3.x database appeared to eat itself to the point that TAC could not rescue it. Reverted to snapshot and all things went well following the better path.
  • Compatibility
    • Make sure you've checked your WLC code compatibility - it is possible to upgrade the WLC to the point where it is not compatible with Prime. This occurred with new 2800 APs requiring the latest WLC code and then loads of fun ensued.

Preparation

  • VMWare Copied Prime01 to Prime01_1
    • Verified 2.2 was operable
  • On Prime01_1
    • reverted to snapshot v2.2.0
    • ncs stop
    • halt
    • VMWare
    • powering on host
    • Verified 2.2 was operable
    • verify df shows new drive space
      • root df
        • /dev/mapper/smosvg-optvol 295801392 147598280 53% /opt
      • Prime UI Administration -> Appliance
        • /dev/mapper/smosvg-optvol 132937592 147598280 53
  • snapshot (VM)

Install PI-Upgrade-3.0.0.0.78.tar.gz.zip

    • Copy file to VM copy (enable ftp, et al)
    • CLI: admin# ncs stop
    • CLI: admin# application upgrade PI-Upgrade-3.0.0.0.78.tar.gz defaultRepo
    • Get Coffee & await completion
    • CLI: admin# ncs start
    • verify Prime is running
    • double check RADIUS auth
    • Resync WLC configurations
      • Step 1 Choose Configuration > Network > Network Devices, then select Device Type > Wireless Controller.
      • Step 2 Select the check box(es) of the applicable controller(s).
      • Step 3 Click Sync.
      • Step 4 Click Yes to proceed.
    • Verify operability
    • snapshot (VM)

Install pi302-16.ubf patch

    • WLC - Wireless Global Conf
      • set Login Credentials (see password vault)
      • Apply
      • Save Configuration
      • Prime - Resync WLC configurations
    • Install pi302-16.ubf patch
    • ncs stop
    • ncs start
      • est wait: 15 minutes
    • verify operability
    • snapshot (VM)

Notes

  • 3.0.0.0.78
  • PI 3.0.2
  • Prime Insight Agent 1.0.0

Install pi_technology_package-3.0.2-1.0.56.ubf patch

Notes

  • 3.0.0.0.78
  • PI 3.0.2 2.0.0
  • Prime Insight Agent 1.0.0
  • PI 3.0 TECH PACK 1.0.3

Install pi303-10.ubf patch

  • Download via PI (warning received unknown origin)
  • Install via PI
  • ncs stop
  • ncs start
    • est wait: 15 minutes
  • verify operability
  • snapshot (VM)

Install pi303_update_02-1.0.10.ubf patch

pi303_update_03-1.0.8.ubf

  • Going with pi303_update_03-1.0.8.ubf from the download list in PI
  • PI download
  • PI install
  • ncs stop
  • ncs start
    • est wait: 15 minutes
  • verify operability
  • snapshot (VM)

PI-Upgrade-3.0.X_to_3.1.0.0.132.tar.gz

  • Copy file to VM
  • CLI: admin# ncs stop
  • CLI: admin# application upgrade PI-Upgrade-3.0.X_to_3.1.0.0.132.tar.gz defaultRepo
  • Get Coffee & await completion (est wait: 7 hours)
  • CLI: admin# ncs start
  • verify Prime is running
  • double check RADIUS auth
  • Resync WLC configurations
  • Synchronize devices
  • snapshot (VM)

Notes

  • Have patience with 3.1 installing - power cycling the VM can result in a damaged system (ex: grub menu appeared after one such attempt). Snapshot at each successful point.
  • There appears to be no 'root' enabled in 3.1.x, not that it was entirely needed but it had come in handy prior.

Install PI_3_1_3-1.0.16.ubf patch

Install Device-Pack-3-PI3.1-14.ubf

Configure NTP

  • PRIME01/admin(config)# ntp server 10.10.2.170
  • PRIME01/admin(config)# exit
  • PRIME01/admin# wri me
  • ncs stop
  • reload
  • Verify NTP & Time

Resync

  • Resync WLC configurations
    • Step 1 Choose Configuration > Network > Network Devices, then select Device Type > Wireless Controller.
    • Step 2 Select the check box(es) of the applicable controller(s).
    • Step 3 Click Sync.
    • Step 4 Click Yes to proceed.
  • Synchronize devices (devices will not show the correct software until this patch has been applied)
    • Step 1 Choose Inventory > Device Management > Network Devices.
    • Step 2 Select the device whose configuration you want synchronized with the configuration stored in the Prime Infrastructure database.
      • Select WLC first, then do APs
    • Step 3 Click Sync.

Roll Up

Had a fun time with the upgrade, even TAC had problems at some of the issues, in the end I kept reverting back to my VM snapshot and trying things out, eventually making it up to the latest version. The 2800 APs are now showing up in inventory and Prime actually seems a little snappier in places (though initial map AP loading is a little slower).

Things worth noting

  • 2.x to 3.0 upgrade took 7 hours for the database phase to finish
  • 3.0 to 3.1 upgrade took 4 hours for the database phase to finish
  • Root does not appear to be accessible once you get to 3.1 (you still have it in 3.0.x). It was helpful having that there and getting out of v2.x.x. Firing off a wget from root and being in defaultRepo/ was handy.
  • If OptVol drive space gets filled up when going to 3.x.x, my suggestion is to revert back to 2.x.x snapshot and then add a disk drive of desired size to the VM. * The OS will see it on boot up and absorb it into it's drive config.
    • Note: root level "df" and Prime's disk usage appeared to be off in the drive size read out (df showed the correct size), however both agreed on the % in use.
    • IIRC, you can add two extra drives max over time.
    • Don't waste a lot of time using NCS CLEANUP if 2.x.x disk usage is 85% or more. We ran into a lot of issues with the database and TAC spent a couple of hours attempting to resurrect the db after the attempt to upgrade was done immediately after an NCS CLEANUP. Reverting to snapshot and adding disk space saved the day.
  • Prior to 3.1, the routine of "install patch\upgrade\etc" and then do an NCS STOP\START from CLI was the way to go. In 3.1, the GUI now alerts you that it will be doing the restart for you. Notes:
    • Maintenance patches appear to reboot the entire OS
    • Have patience. With NCS STOP, you can watch Prime go down systematically. When the GUI does it, you do not see any progress and that is discerning. I waited about 30 minutes and then did a power cycle, which ended drastically at the GRUB screen. Revert to Snapshot and then did the following:
      • Issued patch update through GUI
      • Logged into VSphere and then into Prime via CLI
      • Left the screen up in eye view. Some 20 minutes later, I started seeing OS level (linux) services being halted in the usual linux fashion. It paused on NCS *** Service for a long time (10 minutes), then exited, rebooted the OS, came up, Prime started.
    • DevicePatches did not appear to reload or reboot, waited awhile (30 minutes) and then did the NCS STOP\START out of prior course of habit. All was well.

Will this be your experience? Probably (hopefully) not. Hopefully the above helps identify the slight changes (like the OS reboot via GUI) that sometimes leaves one wondering if the system is doing anything.

3.2 to 3.4

  • Somewhere along the way, via the Software Upgrade interface, we got up to 3.4.

3.4 to 3.6

  • Downloaded PI-Upgrade-3.4_to_3.6.0.0.172.tar.gz from Cisco and got it over to defaultrepo/
  • Cleaned out older files from defaultRepo (it will error and tell you want to remove if needed)
admin#shell
<password>
sudo -s
cd /localdisk/defaultRepo
wget http:// someapacheserver/PI-Upgrade-3.4_to_3.6.0.0.172.tar.gz
exit
exit
admin# ncs stop
admin# application upgrade PI-Upgrade-3.4_to_3.6.0.0.172.tar.gz defaultRepo
Retrieved from "https://www.rabbibob.com/index.php?title=Upgrading_Prime_Infrastructure&oldid=1642"