A quick  tip for those out in the field tshooting latency issues.

When troubleshooting a latency issue, many run straight for QoS. But there was one find that ended up NOT being QoS.

Debugs.

Long story short — someone forgot to turn debugging off on a certain device. When users complained of issues with their resources, QoS was looked at. No drops on the interface, but a quick look at the cpu via #sh proc cpu history showed high CPU usage.

 

100                                                             
  90              *                                              
  80              *       *                *  *     *            
  70              *  **** ***** ** *** ****** ******* *    ** ***
  60              **#############################################
  50              **#############################################
  40              **#############################################
  30        *     *##############################################
  20        *    *###############################################
  10 *******#****################################################
    0….5….1….1….2….2….3….3….4….4….5….5….6
              0    5    0    5    0    5    0    5    0    5    0
              CPU% per minute (last 60 minutes)
             * = maximum CPU%   # = average CPU%

 

:O

Turned up someone left some chatty debugs on the device which after turning off…

 100                         *                                               
  90   *  *                  *                                               
  80 *** *****  * * ** **  ***                                               
  70 ********************* ***                                               
  60 ########################*                                               
  50 ########################*                                               
  40 ########################*  *  **                    *                   
  30 #########################  *  **                    *                   
  20 #########################  *  **            *       *                   
  10 #########################****#**            ****#* ******               
    0….5….1….1….2….2….3….3….4….4….5….5….6….6….7..
              0    5    0    5    0    5    0    5    0    5    0    5    0 

 

And magically the latency issue went away. I took this one for granted, but someone we should try to keep in mind when tshooting latency issues.

 

William Zambrano

William Zambrano

NYC networkers is run by William Zambrano, a passionate network engineer who has been in the IT industry for eight years who posts up blog articles, YouTube videos, and holds meetup.com events in the NYC area. He lives in Queens, New York and has consulted in various different companies in the NY area. Previously William worked as a Cisco Certified Systems Instructor (CCSI) but now currently works for Arista Networks serving as a Systems Engineer. William can be reached by email at willzambrano@gmail.com

More Posts - Website

Follow Me:
Twitter