Output drops due to QoS on 2960/3560/3750 switches

Source, big thanks to author

Do you see incrementing output drops on some interfaces after configuring QoS on your 2960/3560/3750 switch?
Common Scenarios
• Some of the interfaces start experiencing output drops once QoS is configured on the switch.
• Specific applications may experience degraded performance after configuring QoS on the switch. Say IP phones start experiencing choppy calls.
Possible Reason
Once you enable QoS on the switch, some traffic may start getting lesser resources than before (bandwidth or buffer) and hence may get dropped on the switch.


Troubleshooting steps

Step1> Identify the interfaces which carry outgoing data for the affected application or are seeing incrementing output drops. Compare the interface output rate and the interface speed and ensure that the drops are not due to overutilization of the link.

Switch#sh int gi1/0/1
<some output ommitted>
GigabitEthernet0/1 is up, line protocol is up (connected)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

!interface speed is 1000 mbps

  input flow-control is off, output flow-control is unsupported
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1089  <<---

!ensure these drops are incrementing

  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 4000 bits/sec, 6 packets/sec
  5 minute output rate 3009880 bits/sec, 963 packets/sec

!output rate is about 3 mbps while the interface speed is 1000 mbps.
Step2> Ensure that QoS is enabled on the switch. If it is not enabled, output drops are not related to QoS and hence further steps mentioned here are irrelevant.

Switch#sh mls qos
QoS is enabled  <<----
QoS ip packet dscp rewrite is enabled

Step3> Identify the marking of the outgoing traffic that is getting dropped on the interface.

Switch#sh mls qos int gi1/0/1 statistics

GigabitEthernet1/0/1 (All statistics are in packets)

  dscp: incoming
-------------------------------

0 -  4 :           0            0            0            0            0
5 -  9 :           0            0            0            0            0
10 - 14 :           0            0            0            0            0
15 - 19 :           0            0            0            0            0
20 - 24 :           0            0            0            0            0
25 - 29 :           0            0            0            0            0
30 - 34 :           0            0            0            0            0
35 - 39 :           0            0            0            0            0
40 - 44 :           0            0            0            0            0
45 - 49 :           0       198910            0            0            0
50 - 54 :           0            0            0            0            0
55 - 59 :           0            0            0            0            0
60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------

0 -  4 :           0            0            0            0            0
5 -  9 :           0            0            0            0            0
10 - 14 :           0            0            0            0            0
15 - 19 :           0            0            0            0            0
20 - 24 :           0            0            0            0            0
25 - 29 :           0            0            0            0            0
30 - 34 :           0            0            0            0            0
35 - 39 :           0            0            0            0            0
40 - 44 :           0            0            0            0            0
45 - 49 :           0      248484            0            0            0
50 - 54 :           0            0            0            0            0
55 - 59 :           0            0            0            0            0
60 - 64 :           0            0            0            0
  cos: incoming
-------------------------------

  0 -  4 :           2            0            0            0            0
  5 -  7 :           0            0            0
  cos: outgoing
-------------------------------

  0 -  4 :           0            0            0            0            0
  5 -  7 :           0            0            0
  output queues enqueued:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           248484      0           0
queue 1:           0           0           0
queue 2:           0           0           0
queue 3:           0           0           0

  output queues dropped:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:       1089           0           0
queue 1:           0           0           0
queue 2:           0           0           0
queue 3:           0           0           0

Policer: Inprofile:            0 OutofProfile:            0

Note: Though you see queue 0-threshold 1 dropping packets, this actually will be queue 1 in further troubleshooting as queue numbering is 1 to 4 in further outputs.


Step4> Check the marking to output-q map on the switch to determine which queue-threshold pair
maps to the marking getting dropped.

In this scenario, queue1-threshold1 is mapped to dscp 46, which is getting dropped on the interface. This means that dscp 46 traffic is being sent to queue1 and is getting dropped because that queue has insufficient buffer or lesser CPU cycles.

Switch#sh mls qos maps dscp-output-q

   Dscp-outputq-threshold map:
     d1 :d2    0     1     2     3     4     5     6     7     8     9
     ------------------------------------------------------------
      0 :    02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01
      1 :    02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01
      2 :    03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01
      3 :    03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      4 :    01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01
      5 :    04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      6 :    04-01 04-01 04-01 04-01


Step5> There are two ways to tackle these drops. First method is by changing the buffer and threshold values for the queue dropping packets. second method is to configure the scheduler so that the queue dropping packets is serviced more often than rest of the queues.
First let us see how we can change the buffer and threshold for the affected queues. Let us check the buffer and threshold values associated with the queue identified in step 4.
Note: Each queue set has the option to configure the buffer size and threshold value for the four egress queues. Then, you can apply any one of the queue sets to any of the ports. By default, all interfaces use queue-set 1 for output queues unless explicitly configured to use queue-set 2.

In this scenario queue 1 in queue-set 1 has 25% of the total buffer space and threshold 1 is set to 100%

Switch#sh mls qos queue-set
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     100     200     100     100
threshold2:     100     200     100     100
reserved  :      50      50      50      50
maximum   :     400     400     400     400
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     100     200     100     100
threshold2:     100     200     100     100
reserved  :      50      50      50      50
maximum   :     400     400     400     400


Step6> If you wish to change the buffer and threshold values just for the affected interface, please change queue-set 2 and configure the affected interface to use queue-set 2.

Note: You can change queue-set 1 also but as all the interfaces by default use queue-set 1, the change will be reflected to all the interfaces.

In the next step I am changing queue-set 2 so that queue 1 gets 70% of total buffer.
Switch(config)#mls qos queue-set output 2 buffers 70 10 10 10

In next step i am changing queue-set 2, queue 1 thresholds. Both Threshold 1 and Threshold 2 are mapped to 3100 so that they can pull buffer from the reserved pool if required.
Switch(config)#mls qos queue-set output 2 threshold 1 3100 3100 100 3200
Step7> Check if the changes reflect under correct queue and queue-set.

Switch#sh mls qos queue-set
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     100     200     100     100
threshold2:     100     200     100     100
reserved  :      50      50      50      50
maximum   :     400     400     400     400
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      70      10      10      10
threshold1:    3100     100     100     100
threshold2:    3100     100     100     100
reserved  :     100      50      50      50
maximum   :    3200     400     400     400


Step8> Make the affected interface use queue-set 2 so that the changes take effect on this interface.

Switch(config)#int gi1/0/1
Switch(config-if)#queue-set 2
Switch(config-if)#end

Confirm if the interface is mapped to queue-set 2
Switch#sh run int gi1/0/1
interface GigabitEthernet1/0/1
switchport mode access
mls qos trust dscp
queue-set 2
end
Check if the interface is still dropping packets.
Step9> We can also configure the scheduler to increase the rate at which queue 1 will be serviced using the share and shape options. In this example Queue 1 alone will get 50% of the total CPU cycles and rest three queues will collectively get 50% of the CPU cycles.
Switch(config-if)#srr-queue bandwidth share 1 75 25 5

Switch(config-if)#srr-queue bandwidth shape  2  0  0  0
Check if the interface is still dropping packets.
Step10> If the packets are still getting dropped, as a last resort we can enable priority queue on this interface. This will ensure that all the traffic in the priority queue to gets processed before any other queue.
Note:Priority queue is serviced until empty before the other queues are serviced.By default on 2960/3560/3750 switches, queue 1 is the priority queue.

Switch(config)#int gi1/0/1
Switch(config-if)#priority-queue out
Switch(config-if)#end
The marking getting dropped on the interface can be mapped so that it goes to queue 1, which is now the priority queue . In this way we ensure that traffic with this marking always gets processed before anything else.
Switch(config)#mls qos srr-queue output dscp-map queue 1 threshold 1 ?

Update 1 (From official documentation).

Queue Map Configuration:

Rack1SW1(config)#mls qos srr-queue output cos-map queue 1
 threshold 3 5
Rack1SW1(config)#mls qos srr-queue output cos-map queue 1
 threshold 1 2 4
Rack1SW1(config)#mls qos srr-queue output cos-map queue 2 
 threshold 2 3
Rack1SW1(config)#mls qos srr-queue output cos-map queue 2
 threshold 3 6 7
Rack1SW1(config)#mls qos srr-queue output cos-map queue 3
 threshold 3 0
Rack1SW1(config)#mls qos srr-queue output cos-map queue 4
 threshold 3 1

Rack1SW1(config)#mls qos srr-queue output dscp-map queue 1
 threshold 3  46
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 2
 threshold 1  16
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 2
 threshold 1  18 20 22
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 2
 threshold 1  25
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 2
 threshold 1  32
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 2
 threshold 1  34 36 38
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 2
 threshold 2  24 26
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 2
 threshold 3  48 56
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 3
 threshold 3  0
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 4
 threshold 1  8
Rack1SW1(config)#mls qos srr-queue output dscp-map queue 4
 threshold 3  10 12 14