95th percentile: why is it loved by both Internet service providers and their customers and what is its value for protection against DDoS attacks
Although customers of Internet service providers (ISPs) purchase communication channels with a precisely defined bandwidth, they are often not charged for the entire port capacity, but only for the bandwidth actually consumed. For ISP providers, this method is known as burstable billing. Moreover, this actually consumed bandwidth is usually taken into account not according to the highest of the indicators recorded during traffic measurements, but by subtracting 5% of the maximum - according to the largest of 95% of the remaining values. This method is called the 95th percentile. This article provides an example of calculating the 95th percentile in StormWall. Сalculation algorithm may differ from different providers.
For example, the measurement of the actual bandwidth can be performed every 5 minutes for one month: The traffic volume recorded for these minutes is divided by 300 seconds - thus obtaining the average value of the actual bandwidth for these 5 minutes. Then the statistics of these values are analyzed, 5% of the highest values are discarded, and the customer is charged for the maximum of the remaining 95%. How the incoming and outgoing traffic is taken into account in this case, individually or together, depends on the calculation method of the particular ISP.
Of course, the volume of traffic consumed is unevenly distributed over time: It varies during the day (traffic at night is usually not as high as during the day), during the month (it is not as high on weekends, before vacations, and during vacations as it is on other days), and during the year (since seasonal variations occur in many businesses). However, if we generalize and analyze the values of the bandwidth actually used during the month, it turns out that their statistics are close to the Gaussian normal distribution. Incidentally, the 50th percentile coincides with the median - the middle of the Gaussian curve.
Similar estimates using percentiles can be found in various industries and applications: The Nth percentile indicates that the proportion of cases (measurements, situations, etc.) in which the values of the estimated indicator do not exceed a certain value is N/100.
Why the 95th percentile is beneficial for both ISP providers and their customers
Why is the 95th percentile so popular in telecommunications? In fact, ISPs sometimes consider 90% of the values of the actual bandwidth consumed, less often 98%, while 10% or 2% of the maximum is truncated. In most cases, however, the 95% or 95th percentile is chosen as a kind of reasonable compromise between the interests of the ISP provider and its customers.
Both providers and their customers consider it economically justified and quite fair to charge 95% of the actual bandwidth consumed: customers pay for the actual bandwidth consumed, moreover, such a calculation truncates peaks caused by short-term random factors (including DDoS attacks), and values that are systematically collected are retained for most of the measurement period. Thus, the 95th percentile becomes a kind of insurance for the customer against additional costs caused by an unforeseen short-term increase in traffic.
On the ISP's side, the 95th percentile represents a perfectly acceptable bonus for the customer, since 5% of the peak values for one of the customers will certainly be compensated for by lower traffic for other customers and will not require any additional costs for the provider.
Practical significance of the 95th percentile for customers
The actual 95th percentile value means that 5% of the time (i.e., 36 hours per month), the actual bandwidth consumed may exceed the amount taken into account by the ISP when billing. During these 5% of the time, the customer can send or receive much more traffic without incurring additional charges.
Let us assume that a certain e-commerce company purchased Internet access with a bandwidth of 1000 Mbps at a price of $2700/month, while according to the tariff the excess is to be paid at $2.7/Mbit. The following month, the company held a seasonal sale, as a result of which load peaks occurred for several hours, and updates were downloaded to the company's servers at the end of the month. Due to these events, short-term bandwidth increases were observed several times during the month, sometimes reaching 9500 Mbps (DDoS attacks were not included). At the end of the month, all values of actual bandwidth consumed, measured on average every 5 minutes, were tabulated by the provider with N=8640 values (for 30 days in the month), then sorted in descending order, M=432 (5%) of the upper values truncated, and the maximum of the remaining values (433th) taken as the X value. This X is the 95th Let us assume X=1135 Mbps. In this case, the surcharge is (1135 - 1000) * $2.7 = 135 * $2.7 = $364.5.
If the Internet provider based its calculations not on 95%, but simply on the maximum value of the bandwidth actually consumed, the company would have to pay an additional (9500 - 1000) * $2.7 = $22950, which is a very large amount for an online business that is not exactly large. Or bandwidth would have to be capped at 1000 Mbps, resulting in slow operation or even partial unavailability of resources during peak load hours - exactly when availability is most important.
DDoS attacks can also be the cause of a sharp short-term spike in traffic. And if their total duration exceeds the 5th percentile of 36 hours and at the same time the Internet resource is not connected to DDoS protection or the anti-DDoS services are not able to filter out the attacks 100%, then the volume of unauthorized traffic caused by DDoS attacks will affect the overall statistics of the values of actually consumed bandwidth and will lead to an increase in the cost of communication channels.
The values of the 95th percentile can also be used when planning network capacity. For example, if the value of the 95th percentile is 20-30% of the bandwidth of the network devices, it is quite possible to increase their utilization by another 10-15% if desired. If the value of the 95th percentile is 65-70% of their capacity, then it is worth thinking about improving the performance of the network devices. Similarly, the 95th percentile helps in planning the bandwidth of leased communication channels.
Note that performing such assessments is very important to increase the resilience of Internet resources to DDoS risks: Both the performance of network devices and the width of the leased bandwidth of communication channels should be increased in advance, otherwise there is a risk that the illegal traffic of a sufficiently strong DDoS attack will exhaust them completely and it will not be possible to skip at least part of the legitimate one.
