I ran into an interesting issue where our backup ACE appliance finally bit the dust. Rebooting the guy didn’t do anything (it would keep crapping out after 15 minutes of uptime) and consoling to the appliance got me just a non responsive putty session. In  the front and the back of the unit there was a blinking amber light (never a good light to see!) Good thing that it was the standby ACE and not the active so production traffic wasnt hit. After a call to TAC to get the appliance replaced, it was time to get to configuring it. Got in a new ACE but how to install this new guy in without affecting production traffic? Luckly the ACE would be up for long enough for me to pull out the certificates and the keys from the system. A quick #show crypto files showed me what files were on the particular context I was in. Note that ACE (similar to an ASA) commands are on a per context basis. So just because you do #show run or whatever other commands you do, make sure your doing it for ALL contexts on the appliance. Last thing you want to do is blow out the config after returining it back to the factory to find out “oops”!

ACE-01/Admin# sh cr f
Filename                                 File  File    Expor      Key/
                                         Size  Type    table      Cert
———————————————————————–
cisco-sample-cert                        1082  PEM     Yes        CERT
cisco-sample-key                         887   PEM     Yes         KEY

This here is just the sample keys Cisco gives us but any other keys/certs you see on here do a #crypto export cisco-sample-cert to get the certificate and do a copy and paste to notepad. Then I went to the new ACE and did a #crypto import terminal cisco-sample-cert  and paste in the cert/key. These are essential to have in the other ACE as failure to do this will not just mess up policy map commands but also give your users that “untrusted webpage” message on their browsers.Remember, just like in my CCNA Security Part 3 YouTube video, make sure to copy the whole thing, including —-BEGIN CERTIF… portion.

ACE-01/Admin# crypto export cisco-sample-cert
—–BEGIN CERTIFICATE—–
MIIC8jCCAlugAwIBAgIJAK3k4vFQt869MA0GCSqGSIb3DQEBBQUAMFoxCzAJBgNV
BAYTAklOMQswCQYDVQQIEwJLQTEMMAoGA1UEBxMDQkxSMQ4wDAYDVQQKEwVDSVND
TzENMAsGA1UECxMEQURCVTERMA8GA1UEAxMIU1NMLVRFU1QwHhcNMDkwNDAzMDk1
MDU1WhcNMTkwNDAxMDk1MDU1WjBaMQswCQYDVQQGEwJJTjELMAkGA1UECBMCS0Ex
DDAKBgNVBAcTA0JMUjEOMAwGA1UEChMFQ0lTQ08xDTALBgNVBAsTBEFEQlUxETAP
BgNVBAMTCFNTTC1URVNUMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDPomBm
W862OG+U3w0cYUFsSIIj5muGASI695qkYF6yWlBdQMqamhOxixaVmrlhWf/hO325
4KXqNuprIY94qdGlnu6ulrmWYlPv9Rxe/n+qeq/nMdwmN8c9q3CwHKNNM/jhNLv8
1ukD5ymICWiVbnL8rQWNKedfVSavegVJ7Y2TOwIDAQABo4G/MIG8MB0GA1UdDgQW
BBSheuJQVJ2ChqUB9RR7eA2uEhgM2TCBjAYDVR0jBIGEMIGBgBSheuJQVJ2ChqUB
9RR7eA2uEhgM2aFepFwwWjELMAkGA1UEBhMCSU4xCzAJBgNVBAgTAktBMQwwCgYD
VQQHEwNCTFIxDjAMBgNVBAoTBUNJU0NPMQ0wCwYDVQQLEwRBREJVMREwDwYDVQQD
EwhTU0wtVEVTVIIJAK3k4vFQt869MAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEF
BQADgYEAfK46lgPphuhAb9TRKoj9tWB6kAfi3p6ZuOke9KrBthYK36HWuHMSCLQz
uiF7l2BMHNGizeDd+e/WFdUMnAUxevSIduZ92XOMfBHMwwR5pG8aRDc5GvDVoHm6
mYRWwRORKIZviTCwDpb8oNGSxH1EAwuTCm9AZ5nOoRzUX0Ci+eA=
—–END CERTIFICATE—–

I’ve done on the Admin context (which is the default context you get out of the box), but do a #changeto ? to see if any other contexts are configured on the ACE.  Luckily here the ACSW is nice enough to auto-fill for you the name of the context so a #changeto UA works fine — hit TAB and it autofill it in for you!

ACE-01/Admin# changeto ?
  Admin         
  UAT-DEV
  INSIDE    
  INTRANET

After you’ve done this for all contexts, make sure you have the SAME version on both ACEs. Very similar to an ASA for fault tolerance. If its not the same version, grab it off Cisco.com downloads and you can either TFTP the image to the image0: drive or easier, jump into the GUI of the ACE and go to ADMIN > TOOLS > FILE BROWSER > IMAGE0: > UPLOAD

ace

BTW, if you get an error about the disk being “read-only” you need to format the flash first before being able to upload the image onto the flash. The usual commands are the same to set the image as the main one to boot off of, checkout the boot system image:c4710ace-t1k9-mz.A4_1_0.bin command from the #sh run to verify.

Once that is set, checkout if any licenses need to be moved over. #sh license usage but in my case I didn’t have any to transfer over so I can’t write too much about that.

ACE-01/Admin# sh license usage
License                      Ins   Lic    Status   Expiry Date   Comments
                                  Count
——————————————————————————–
ACE-AP-C-UP1                  No     –    Unused                 –
ACE-AP-C-UP2                  No     –    Unused                 –
ACE-AP-C-UP3                  No     –    Unused                 –
ACE-AP-01-LIC                 No     –    Unused                 –
ACE-AP-01-UP1                 No     –    Unused                 –
ACE-AP-02-LIC                 No     –    Unused                 –
ACE-AP-02-UP1                 No     –    Unused                 –
ACE-AP-04-LIC                 No     –    Unused                 –
ACE-AP-04-UP1                 No     –    Unused                 –
ACE-AP-04-UP2                 No     –    Unused                 –
ACE-AP-VIRT-5                 No     –    Unused                 –
ACE-AP-500M-LIC               No     –    Unused                 –
ACE-AP-VIRT-020               No     –    Unused                 –
ACE-AP-C-100-LIC              No     –    Unused                 –
ACE-AP-C-500-LIC              No     –    Unused                 –
ACE-AP-C-500-UP1              No     –    Unused                 –
ACE-AP-OPT-50-K9              No     –    Unused                 –
ACE-AP-C-1000-LIC             No     –    Unused                 –
ACE-AP-C-2000-LIC             No     –    Unused                 –
ACE-AP-OPT-LIC-K9             No     –    Unused                 –
ACE-AP-OPT-UP1-K9             No     –    Unused                 –
ACE-AP-SSL-05K-K9             No     –    Unused                 –
ACE-AP-SSL-07K-K9             No     –    Unused                 –
ACE-AP-SSL-100-K9             No     –    Unused                 –
ACE-AP-SSL-UP1-K9             No     –    Unused                 –
ACE-AP-SSLUP-5K-K9            No     –    Unused                 –
ACE-AP-VIRT-020-UP            No     –    Unused                 –

Hopefully you have a backup config copy of the Standby ACE. If not, you’ll need to do some more legwork. Specifically, for the Admin context and for each context you’ll need to configure the ft group settings to point to the other ACE and of course basic layer 3 connectivity. The higher priority of the two ACEs becomes the master/active ACE for the context. You can also optionally opt for the preempt command to take over if the other ACE is down (like our FHRP) but its not listed below. On the other ACE, just make sure the priority is lower so that it becomes the slave/standby for the pair.

ft group 2
  peer 1
  priority 210
  peer priority 200
  associate-context UAT-DEV
  inservice
ft group 3
  peer 1
  priority 210
  peer priority 200
  associate-context INSIDE
  inservice
ft group 4
  peer 1
  priority 210
  peer priority 200
  associate-context INTRANET

Once you got that in and you restored a backup #sh startup to the standby ACE, time to rack it up and connect it up!

Before you actually connect the LAN cables, I wanted to make sure I could manage it via SSH in the event that for some reason the standby ACE  tries to push its “bad” config to the active ACE — causing an outage.

For each context, I isused the following two commands.

no ft auto-sync running-config
no ft auto-sync startup-c0nfig

These would stop the ACE from trying to push down its config to its peer. After I hooked up the cables and was able to ping/SSH to the new ACE from my desk I issued the following show commands to verify ft group settings.

#show ft group summary
!I ONLY ADDED 1 GROUP IN HERE, NOT ALL FT GROUPS ARE SHOWN
FT Group                     : 2
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_HOT
My Config Priority           : 210
My Net Priority              : 210
My Preempt                   : Enabled
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority         : 200
Peer Net Priority            : 200
Peer Preempt                 : Enabled
Peer Id                      : 1
No. of Contexts              : 1

Or a quicker output would be the
#show ft group brief
FT Group ID: 1  My State:FSM_FT_STATE_ACTIVE    Peer State:FSM_FT_STATE_STANDBY_HOT
                Context Name: Admin     Context Id: 0   Running Cfg Sync Status:Successful

#show ft group detail

FT Group                     : 1
No. of Contexts              : 1
Context Name                 : Admin
Context Id                   : 0
Configured Status            : in-service
Maintenance mode             : MAINT_MODE_OFF
My State                     : FSM_FT_STATE_ACTIVE
My Config Priority           : 210
My Net Priority              : 210
My Preempt                   : Enabled
Peer State                   : FSM_FT_STATE_STANDBY_HOT
Peer Config Priority         : 200
Peer Net Priority            : 200
Peer Preempt                 : Enabled
Peer Id                      : 1
Last State Change time       : Thu Aug  4 15:31:21 2011
Running cfg sync enabled     : Enabled
Running cfg sync status      : Running configuration sync has completed
Startup cfg sync enabled     : Enabled
Startup cfg sync status      : Startup configuration sync has completed
Bulk sync done for ARP: 0
Bulk sync done for LB: 0
Bulk sync done for ICM: 0

I just wanted to see if the active ACE recongized  the peer and  that the cf sync status was set NOT set. The output shown above is the “final product” with the states showing as HOT and Running config sync has completed.

Finally as the last step, I went to the Active ACE and did a re-enabled “ft auto-sync running-config” and “ft auto-sync startup-config”. After a few minutes all the group states went to HOT and the “running config sync has completed” appeared. Good to go!

As a FINAL step, to verify failover I’ve created a window to test failover to the new ACE and test some services riding off each context. On the Admin context,  issue the #ft switchover all command to failover all FT groups, or do it one by one specifying the ft group # via #ft switchover # command. Or to see all the options, issue the ? command

ACE-01/Admin# ft switchover ?
  all      Switchover all ft-ids
  force    Manually set the local HA state when FT Vlan is down
  <1-255>  Specify FT Group ID
  <cr>     Carriage return.

Window is waiting for approval, so when I can i’ll write up a Part2 to this!

William Zambrano

William Zambrano

NYC networkers is run by William Zambrano, a passionate network engineer who has been in the IT industry for eight years who posts up blog articles, YouTube videos, and holds meetup.com events in the NYC area. He lives in Queens, New York and has consulted in various different companies in the NY area. Previously William worked as a Cisco Certified Systems Instructor (CCSI) but now currently works for Arista Networks serving as a Systems Engineer. William can be reached by email at willzambrano@gmail.com

More Posts - Website

Follow Me:
Twitter