Nginx之tcp_nodelay

目录

一、知识准备

● 在nginx优化中有个经常需要设置的参数,tcp_nodelay
● 该参数最核心的功能,就是把小包组成成大包,提高带宽利用率也就是著名的nagle算法
● tcp协议中,有一个现象:应用层数据可能很低(比如1个字节),而传输层开销有40字节(20字节的IP头+20字节的TCP头)。这种情况下大部分都是控制包的传输,既加大了带宽的消耗,带宽利用率也不高
● nagle算法就是为了解决这个问题。在发出去的数据还未被确认之前,或者说还没有收到对端的ack之前,新生成的小包是不允许被发送的,必须要凑满一个MSS或者等到收到确认后再发送,直至超时

二、环境准备

组件 版本
OS Ubuntu 18.04.1 LTS
docker 18.06.0-ce

客户端 : 192.168.17.171
服务端 : 192.168.17.173

三、打开nagle算法

192.168.17.173,先准备一个nginx配置文件,并且打开nagle算法,设置tcp_nodelay off;

 

root<span class="token metadata symbol">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">/</span>tmp# more nginx<span class="token punctuation">.</span>conf
user  nginx<span class="token punctuation">;</span>
worker_processes  <span class="token number">1</span><span class="token punctuation">;</span>

error_log  <span class="token operator">/</span><span class="token keyword">var</span><span class="token operator">/</span>log<span class="token operator">/</span>nginx<span class="token operator">/</span>error<span class="token punctuation">.</span>log warn<span class="token punctuation">;</span>
pid        <span class="token operator">/</span><span class="token keyword">var</span><span class="token operator">/</span>run<span class="token operator">/</span>nginx<span class="token punctuation">.</span>pid<span class="token punctuation">;</span>


events <span class="token punctuation">{</span>
    worker_connections  <span class="token number">1024</span><span class="token punctuation">;</span>
<span class="token punctuation">}</span>


http <span class="token punctuation">{</span>
    include       <span class="token operator">/</span>etc<span class="token operator">/</span>nginx<span class="token operator">/</span>mime<span class="token punctuation">.</span>types<span class="token punctuation">;</span>
    default_type  application<span class="token operator">/</span>octet<span class="token operator">-</span>stream<span class="token punctuation">;</span>

    log_format  main  <span class="token string">'$remote_addr - $remote_user [$time_local] "$request" '</span>
                      <span class="token string">'$status $body_bytes_sent "$http_referer" '</span>
                      <span class="token string">'"$http_user_agent" "$http_x_forwarded_for"'</span><span class="token punctuation">;</span>

    access_log  <span class="token operator">/</span><span class="token keyword">var</span><span class="token operator">/</span>log<span class="token operator">/</span>nginx<span class="token operator">/</span>access<span class="token punctuation">.</span>log  main<span class="token punctuation">;</span>

    sendfile        on<span class="token punctuation">;</span>
    tcp_nodelay off<span class="token punctuation">;</span>

    keepalive_timeout  <span class="token number">65</span><span class="token punctuation">;</span>

    include <span class="token operator">/</span>etc<span class="token operator">/</span>nginx<span class="token operator">/</span>conf<span class="token punctuation">.</span>d<span class="token comment">/*.conf;
}
</span>

启动容器

 

root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">/</span>tmp<span class="token comment"># docker run -d --name nginx_delay -v /tmp/nginx.conf:/etc/nginx/nginx.conf -p 80:80 nginx:latest</span>
<span class="token number">6</span>b7d5a5d3c3ed021fed6847d138837754c5732979d1c829ec62107ec80440db8
root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">/</span>tmp<span class="token comment"># docker ps | grep nginx_delay</span>
<span class="token number">6</span>b7d5a5d3c3e        nginx<span class="token symbol">:latest</span>                                                        <span class="token string">"nginx -g 'daemon of…"</span>   <span class="token number">7</span> seconds ago       <span class="token constant">Up</span> <span class="token number">6</span> seconds        <span class="token number">0.0</span><span class="token number">.0</span><span class="token number">.0</span><span class="token punctuation">:</span><span class="token number">80</span><span class="token operator">-</span><span class="token operator">&gt;</span><span class="token number">80</span><span class="token operator">/</span>tcp   nginx_delay

首先使用tcpdump抓取本机的80端口的流量:

 

root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">/</span>tmp<span class="token comment"># tcpdump -i ens3 port 80 -afexnnvv -w nginx_ab.cap</span>

在192.168.17.171,使用ab压测工具对该端口进行放量

注意:必须使用 -k 参数,使用keepalived模式下才能模拟出nagle算法

 

root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">~</span><span class="token comment"># ab -n 1000 -c 100 -k http://127.0.0.1/</span>
<span class="token constant">This</span> is <span class="token constant">ApacheBench</span><span class="token punctuation">,</span> <span class="token constant">Version</span> <span class="token number">2.3</span> <span class="token operator">&lt;</span><span class="token variable">$Revision</span><span class="token punctuation">:</span> <span class="token number">1807734</span> $<span class="token operator">&gt;</span>
<span class="token constant">Copyright</span> <span class="token number">1996</span> <span class="token constant">Adam</span> <span class="token constant">Twiss</span><span class="token punctuation">,</span> <span class="token constant">Zeus</span> <span class="token constant">Technology</span> <span class="token constant">Ltd</span><span class="token punctuation">,</span> http<span class="token punctuation">:</span><span class="token operator">/</span><span class="token operator">/</span>www<span class="token punctuation">.</span>zeustech<span class="token punctuation">.</span>net<span class="token operator">/</span>
<span class="token constant">Licensed</span> to <span class="token constant">The</span> <span class="token constant">Apache</span> <span class="token constant">Software</span> <span class="token constant">Foundation</span><span class="token punctuation">,</span> http<span class="token punctuation">:</span><span class="token operator">/</span><span class="token operator">/</span>www<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>org<span class="token operator">/</span>

<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>
<span class="token builtin">Time</span> per request<span class="token punctuation">:</span>       <span class="token number">44.619</span> <span class="token punctuation">[</span>ms<span class="token punctuation">]</span> <span class="token punctuation">(</span>mean<span class="token punctuation">)</span>
<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>

过滤掉大量信息,我们来到这个指标Time per request,无论怎么测试,平均延时总是在40ms左右

我们来看下抓包信息,使用wireshark打开

在大量的数据包中,我们先处理一下数据包,随便选取一个syn,选取与该syn对应的tcp流

选取一个片段来分析

● 在Linux中,默认打开了延迟确认,所谓延迟确认,就是不是收到每个请求都发送一次ack,而是等待一段时间,如果这段时间正好有包需要发送,就坐着“顺风车”一起发出,否则超时后单独发送。所以客户端会等待40ms,再发送这个ack
● 由于nginx也设置了nagle算法,如果没有收到ack,它会等着包的到来,所以就会呈现这个样子
(1)192.168.17.171首先发送一个http get请求(677号包)
(2)192.167.17.173发送PSH,ACK(999号包)
(3)此时由于Linux默认打开延迟确认,192.168.17.171会等待40ms,看看有没有“顺风车”;而192.168.17.173上的nginx由于关闭了tcp_nodelay,它也会等待着ack的到来再回应
(4)40ms过后,192.168.17.171没有等到“顺风车”,此时发送ack(1109号包)
(5)192.168.17.173收到ack后发送了http 200(1118号包)
(6)192.168.17.171收到数据之后发送确认ack(1127号包)

 

<span class="token number">192.168</span><span class="token number">.17</span><span class="token number">.171</span><span class="token punctuation">:</span><span class="token number">47388</span>                        <span class="token number">192.168</span><span class="token number">.17</span><span class="token number">.173</span><span class="token punctuation">:</span><span class="token number">80</span>
     <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>                                 <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">+</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">677</span>  http get          <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">&gt;</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">999</span>  <span class="token constant">PSH</span><span class="token punctuation">,</span><span class="token constant">ACK</span>           <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">&lt;</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>          delay <span class="token number">40</span>ms             <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">1109</span>  <span class="token constant">ACK</span>              <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">&gt;</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">1118</span>  http <span class="token number">200</span>         <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">&lt;</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">1127</span>  <span class="token constant">ACK</span>              <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">&gt;</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>                                 <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">+</span>

四、关闭nagle

只需要设置tcp_nodelay on;

 

root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">/</span>tmp<span class="token comment"># sed -i '/tcp_nodelay/s/off/on/g' nginx.conf</span>
root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">/</span>tmp<span class="token comment"># docker rm -f nginx_delay</span>
nginx_delay
root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">/</span>tmp<span class="token comment"># docker run -d --name nginx_delay -v /tmp/nginx.conf:/etc/nginx/nginx.conf -p 80:80 nginx:latest</span>
bac9bcf7a6e392a7a07afae165c3d5b4e3fb2fc43d3470f35802e12d1e7ae70d

再用ab测试:

 

root<span class="token variable">@k8s</span><span class="token operator">-</span>node2<span class="token punctuation">:</span><span class="token operator">~</span><span class="token comment"># ab -n 1000 -c 100 -k http://127.0.0.1/</span>
<span class="token constant">This</span> is <span class="token constant">ApacheBench</span><span class="token punctuation">,</span> <span class="token constant">Version</span> <span class="token number">2.3</span> <span class="token operator">&lt;</span><span class="token variable">$Revision</span><span class="token punctuation">:</span> <span class="token number">1807734</span> $<span class="token operator">&gt;</span>
<span class="token constant">Copyright</span> <span class="token number">1996</span> <span class="token constant">Adam</span> <span class="token constant">Twiss</span><span class="token punctuation">,</span> <span class="token constant">Zeus</span> <span class="token constant">Technology</span> <span class="token constant">Ltd</span><span class="token punctuation">,</span> http<span class="token punctuation">:</span><span class="token operator">/</span><span class="token operator">/</span>www<span class="token punctuation">.</span>zeustech<span class="token punctuation">.</span>net<span class="token operator">/</span>
<span class="token constant">Licensed</span> to <span class="token constant">The</span> <span class="token constant">Apache</span> <span class="token constant">Software</span> <span class="token constant">Foundation</span><span class="token punctuation">,</span> http<span class="token punctuation">:</span><span class="token operator">/</span><span class="token operator">/</span>www<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>org<span class="token operator">/</span>

<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>
<span class="token builtin">Time</span> per request<span class="token punctuation">:</span>       <span class="token number">14.285</span> <span class="token punctuation">[</span>ms<span class="token punctuation">]</span> <span class="token punctuation">(</span>mean<span class="token punctuation">)</span>
<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>

再来观察抓包的结果:

● 由于客户端依然打开了延迟确认,所以192.168.17.171收到数据包之后依然没有及时回应
● 但是nginx,tcp_nodelay on,所以192.168.17.173收到数据包后会立即响应ack
● 192.168.17.171收到之后,已经有2个没有确认的数据包了,所以会立即发送ack进行确认:
(1)192.168.17.171首先发送一个http get请求(447号包)
(2)192.168.17.173收到之后立即响应psh,ack(740号包)
(3)192.168.17.173发送http 200(741号包)
(4)192.168.17.171回应ack(742号包)

 

<span class="token number">192.168</span><span class="token number">.17</span><span class="token number">.171</span><span class="token punctuation">:</span><span class="token number">49718</span>                        <span class="token number">192.168</span><span class="token number">.17</span><span class="token number">.173</span><span class="token punctuation">:</span><span class="token number">80</span>
     <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>                                 <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">+</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">447</span>  http get          <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">&gt;</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">740</span>  <span class="token constant">PSH</span><span class="token punctuation">,</span><span class="token constant">ACK</span>           <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">&lt;</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">741</span>  http <span class="token number">200</span>          <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">&lt;</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>       no<span class="token punctuation">.</span><span class="token number">742</span>  <span class="token constant">ACK</span>               <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">&gt;</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">|</span>       <span class="token operator">|</span>                                 <span class="token operator">|</span>        <span class="token operator">|</span>
     <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">-</span><span class="token operator">+</span>                                 <span class="token operator">+</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">--</span><span class="token operator">+</span>

五、小结

● 本文复现了经典的40ms问题
● 本文中提到了2个名词,nagle算法与延迟确认,它们看上去很相似,但是并不一样。nagle算法是需要等到对端ack来临,或者凑满一个mss之后才发送数据包;而延迟确认针对的是ack,ack会等待“顺风车”,如果有,就乘坐顺风车发送,否则等待超时之后单独发送
● 本文中延迟确认是Linux默认打开的功能,所以在实验中,客户端都会有延时确认的情况,要关闭客户端延迟确认,需要设置setsockopt中的TCP_QUICKACK
● 本文中主要讨论的是nginx的nagle算法,nagle算法完全由tcp协议的ack机制决定,如果对端ACK回复很快的话,nagle事实上不会拼接太多的数据包,虽然避免了网络拥塞,网络总体的利用率依然很低
● nagle算法在与延迟确认互相作用的情况下,会产生严重的延时效果,这是需要警惕的
● nginx中是否打开nagle算法,要取决于业务场景。比如在实验中看到:
(1)tcp_nodelay off,会增加通信的延时,但是会提高带宽利用率。在高延时、数据量大的通信场景中应该会有不错的效果
(2)tcp_nodelay on,会增加小包的数量,但是可以提高响应速度。在及时性高的通信场景中应该会有不错的效果

ozabc