a new york city based group for IT professionals

Cisco ASA %Error opening disk0:/ (too many open files)

Ran into a strange issue today with a pair of Cisco ASA 5525x series routers. The FW are configured in a A/S setup, and we ran into a bug that required us to upgrade the code. Problem was, one of the ASAs in the pair didn’t even let me get to the disk0 via #dr while the other ASA constantly failed  to upload the image via FTP.

FW01/pri/act# dir

Directory of disk0:/

%Error opening disk0:/ (Too many open files)
8238202880 bytes total (4793798656 bytes free)
Googling around I found a few articles stating that the flash of the ASA could of gone bad and one could try to reformat the flash.

http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113266-asa-flash-error-ts.html

Not a good look when the ASA is in ANOTHER country in a datacenter with NO ONE around!  Trying other commands to get to the flash/disk resulted in similar outputs

FW01/pri/act# show flash: filesys

%Error show flash: (No such device)
FW01/pri/act# show flash: all

%Error show flash: (No such device)
FW01/pri/act# show disk0: all

%Error show disk0: (No such device)
FW01/pri/act# show disk0: controller

%Error show disk0: (No such device)

So to me it looks like the whole ASA is running off RAM! Real bad! I could either risk loosing the config and rebooting the box or hoping it’ll clear out the disk issue once it reboots and it comes back online working OK. I also couldnt upload a new .bin file to the disk0 of the device, seeing how the ASA itself couldn’t even see it!

Aftering attempting to run the checkdisk, it all showed failed.

FW01/sec/act# fsck disk0:

fsck of disk0: complete
WARNING: Restoring security context mode failed.
WARNING: Restoring cluster interface mode failed.

Looked to me like rebooting this guy would cause an RMA call to TAC. And sure enough — it did. Turns out we hit a bug CSCub40805 — which in my case the fix WASNT to reboot the ASA since we lost connection to the box, but to simply RMA a new ASA since the flash/disk0 was basically shot.

 

So whats the moral of the story?

I’m NYer, and NYers are skeptical….sometimes. I didnt fully trust when the bug ID said “reboot and its fixed”. I made sure to do a FULL backup of the #sh run using the #more system:running-config command and of course perform this during the proper maintenance window.

Its times like these maintenance is where people loose their jobs. A upgrade or maintenance goes sour which causes big company downtime and SOMEONE needs to take the blame. Unfortunately sometimes that someone is you. Make sure you take all precautions before doing something like this as it won’t just affect your connection to the device but the company as a whole. Once again — layer 8 skills: use your best judgement to keep all managers in the loop, set the proper expectations, and at the end of the day: cover your ass. Maybe this is why not everyone likes to do this kind of job, your job is almost on the line each time you do a maintenance.

 

Don’t  forget ARP!

Remember back in CCNA? What does the ARP table do for us? Map the IP to the MAC adddress right? We’ll since the ASA failover cluster uses the vIP and vMAC, this will change when we introduce a new ASA. Thus, devices will loose connecticty due to being unable to reach the IP address due to having a new MAC. Thus, you can either wait for the old APR entry to time out and just clear out the ARP table from the devices to speed up the process.

 

 Copy those files over!

Although the ASA failover is pretty good in copying most of the config from the primary to standby, two big important things it doesnt copy over is there Anyconnect Image files and any VPN XML files you are using. Ensure that you copy these files over when preparing the ASA.

Once you copy over the files, simply configure the interfaces and configure the failover portion of the config.

 

Ending

Once the new ASA finally came in, I had the remote hands find out what message was on the “bad” ASA couldnt find the startup config due to the disks being shot. During bootup, I found this message appear…

Reading from flash…

Flash read failed
ERROR: MIGRATION – Could not get the startup configuration.

Cryptochecksum (changed): d41d8cd9 8f00b204 e9800998 ecf8427e

INFO: MIGRATION – Saving the startup errors to file ‘flash:upgrade_startup_errors_201409101414.log’
Pre-configure Firewall now through interactive prompts [yes]?

Lovely…I had the remote hands swap out the ASA and luckly with the same config was able to bring the ASA back  up.

 

 

William Zambrano

William Zambrano

NYC networkers is run by William Zambrano, a passionate network engineer who has been in the IT industry for eight years who posts up blog articles, YouTube videos, and holds meetup.com events in the NYC area. He lives in Queens, New York and has consulted in various different companies in the NY area. Previously William worked as a Cisco Certified Systems Instructor (CCSI) but now currently works for Arista Networks serving as a Systems Engineer. William can be reached by email at willzambrano@gmail.com

More Posts - Website

Trackbacks/Pingbacks

  1. Cisco Bug ID CSCub40805 After some time disk operations fail and power supply info is missing - NYC Networkers - […] 8.6 code, I found it applied to my 9.1 code.  This fix says simply to reset the ASA, but …

Submit a Comment

Your email address will not be published. Required fields are marked *


− 1 = four

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>