tcpdump 网络排障

一、tcpdump作用。1、网络层是否通。通过抓icmp包进行排查。2、传输层的端口是否通。通过抓端口数据包进行排查端口服务是否有问题。二、tcpdump用法。# 基本语法 tcpdump [选项] [过滤规则]# 常用选项-i 网卡名 # 指定抓包的网卡-n # 不解析域名加快速度-c 数量 # 抓 N 个包后自动退出port 端口号 # 只抓指定端口的数据包host IP # 只抓指定 IP 的数据包三、理解tcpdump抓icmp包。icmp理解ICMPInternet Control Message Protocol互联网控制报文协议 是 TCP/IP 协议族里的一个“信使”协议。它不负责传输用户数据比如网页、文件而是专门用来传递网络状态信息和错误报告。类比 如果 TCP/UDP 是“货车”负责拉货那 ICMP 就是“对讲机”负责问路和报错。理解操作1在交换机ping服务器进行抓包。1、发起 Ping在交换机上执行 ping 172.27.101.198因为要模拟“从交换机到服务器的流量”2、抓包tcpdump在服务器上执行 tcpdump -i eno1 icmp -n因为服务器是“终点”要看这个 Ping 包有没有到达服务器网卡。理解操作2在PC机ping服务器进行抓包。1、发起 Ping在我的PC机上执行 ping 172.27.101.198因为要模拟“从PC机到服务器的流量”2、抓包tcpdump在服务器上执行 tcpdump -i eno1 icmp -n因为服务器是“终点”要看这个 Ping 包有没有到达服务器网卡。PC机ping服务器C:\Users\Johnping 172.27.101.198正在 Ping 172.27.101.198 具有 32 字节的数据:来自 172.27.101.198 的回复: 字节32 时间1ms TTL62来自 172.27.101.198 的回复: 字节32 时间1ms TTL62来自 172.27.101.198 的回复: 字节32 时间1ms TTL62来自 172.27.101.198 的回复: 字节32 时间1ms TTL62172.27.101.198 的 Ping 统计信息:数据包: 已发送 4已接收 4丢失 0 (0% 丢失)往返行程的估计时间(以毫秒为单位):最短 0ms最长 0ms平均 0msC:\Users\John服务器接受到的数据包[rootrac1 ~]# tcpdump -i eno1 icmptcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes10:29:53.316983 IP 172.17.8.242 rac1: ICMP echo request, id 1, seq 54244, length 4010:29:53.317003 IP rac1 172.17.8.242: ICMP echo reply, id 1, seq 54244, length 4010:29:54.328872 IP 172.17.8.242 rac1: ICMP echo request, id 1, seq 54245, length 4010:29:54.328899 IP rac1 172.17.8.242: ICMP echo reply, id 1, seq 54245, length 4010:29:55.348825 IP 172.17.8.242 rac1: ICMP echo request, id 1, seq 54246, length 4010:29:55.348838 IP rac1 172.17.8.242: ICMP echo reply, id 1, seq 54246, length 4010:29:56.352372 IP 172.17.8.242 rac1: ICMP echo request, id 1, seq 54247, length 4010:29:56.352399 IP rac1 172.17.8.242: ICMP echo reply, id 1, seq 54247, length 40^C8 packets captured----------实际输出到屏幕的包数受 -c 限制8 packets received by filter ----------在内核中匹配过滤规则的总包数不受 -c 限制0 packets dropped by kernel--------全部成功交给了内核协议栈一个都没弄丢。这句是重点。注意Ping 要“从客户端发起”抓包要“在服务端抓”。tcpdump使用icmp排查案例场景一只有一个默认网关172.27.0.1并且在网口eno1上的时候。操作步骤1、在终端1中直接ping他的网关。ping 172.27.0.12、在终端2中对eno1网口进行抓包。tcpdump -i eno1 icmp -n -c 5,输出如下tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes14:23:11.123456 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 1, length 64 14:23:11.123478 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 12345, seq 1, length 64 14:23:12.124567 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 2, length 64 14:23:12.124589 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 12345, seq 2, length 64 14:23:13.125678 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 3, length 64 14:23:13.125700 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 12345, seq 3, length 645 packets captured5 packets received by filter0 packets dropped by kernel3、在终端3中对一个业务口eno2进行抓包。tcpdump -i eno2 icmp -n -c 5输出如下tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eno2, link-type EN10MB (Ethernet), capture size 262144 bytes0 packets captured0 packets received by filter0 packets dropped by kerneleno2 上没有任何 ICMP 包 说明流量只走 eno1正常场景二两个默认网关并且在网口eno1和eno2上的时候。操作步骤1、终端1中ping 172.27.0.1。2、终端2对eno1进行tcpdump抓包打印如下tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes14:23:11.123456 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 1, length 64 14:23:11.123478 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 12345, seq 1, length 6414:23:13.125678 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 3, length 64 14:23:13.125700 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 12345, seq 3, length 6414:23:15.127890 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 5, length 64 14:23:15.127912 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 12345, seq 5, length 643、终端3对eno2进行tcpdump抓包打印如下tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eno2, link-type EN10MB (Ethernet), capture size 262144 bytes14:23:12.124567 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 2, length 6414:23:14.126789 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 4, length 6414:23:16.128901 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 12345, seq 6, length 64seq 1, 3, 5 → 从 eno1 发出奇数序列号seq 2, 4, 6 → 从 eno2 发出偶数序列号结论系统在 eno1 和 eno2 之间轮询发送 ICMP 请求。eno2 是业务网口它的网关 10.0.0.1 不认识管理网段所以 eno2 发出的包永远收不到回复。这就是看到的 50% 丢包 延迟尖刺 的根本原因。疑惑自己ping自己的管理口/网关/IP自己捕捉tcpdump的时候为什么什么都没有抓到[rootrac1 ~]# tcpdump -i eno1 icmp -n -c 5tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes^C0 packets captured16 packets received by filter0 packets dropped by kernel原因Linux内核存在一个高效的机制当ping自己的管理IP时内核会把数据包扔给lo环接口处理然后立即返回不会经过eno1自然不能抓到数据包。此时要抓包也是对lo接口抓包。反正就是自己ping自己管理IP就是不行注意只要是目标 IP 为本机无论哪个网口还是其他管理口也走lo数据包都不经过物理网卡都走 lo 环回接口所以任何物理网口都抓不到。正确对管理网口进行抓包操作1、ping管理网口网关就可以。[rootrac1 ~]# ping 172.27.0.1PING 172.27.0.1 (172.27.0.1) 56(84) bytes of data.64 bytes from 172.27.0.1: icmp_seq1 ttl255 time0.856 ms64 bytes from 172.27.0.1: icmp_seq2 ttl255 time0.872 ms64 bytes from 172.27.0.1: icmp_seq3 ttl255 time0.858 ms64 bytes from 172.27.0.1: icmp_seq4 ttl255 time1.62 ms64 bytes from 172.27.0.1: icmp_seq5 ttl255 time0.900 ms^C--- 172.27.0.1 ping statistics ---10 packets transmitted, 10 received, 0% packet loss, time 133msrtt min/avg/max/mdev 0.856/1.038/1.623/0.253 ms2、对管理网口进行抓包。[rootrac1 ~]# tcpdump -i eno1 icmp -n -c 5tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes14:26:55.455379 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 54245, seq 1, length 6414:26:55.456209 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 54245, seq 1, length 6414:26:56.497230 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 54245, seq 2, length 6414:26:56.498069 IP 172.27.0.1 172.27.101.198: ICMP echo reply, id 54245, seq 2, length 6414:26:57.520231 IP 172.27.101.198 172.27.0.1: ICMP echo request, id 54245, seq 3, length 645 packets captured19 packets received by filter0 packets dropped by kernel 点。四、理解tcpdump抓端口包。环境在ESXI服务器虚拟化出来的一台服务器没有登录过这次要抓到登录进去的端口号数据包信息。管理端口正常开发SSH能连上确认三次握手过程。理解操作1、 sudo ss -tlnp | grep ssh查看端口。命令确认服务器端口号服务器里运行的一个编号用来区分不同的软件服务前面grep过滤出ssh。[adminlocalhost ~]$ sudo ss -tlnp | grep ssh[sudo] password for admin:LISTEN 0 128 *:22 *:* users:((sshd,pid1345,fd3))LISTEN 0 128 [::]:22 [::]:* users:((sshd,pid1345,fd4))[adminlocalhost ~]$由*:22判断其端口号是22,SSH服务监听在本机所有IP地址的22端口上。2、启用抓包抓管理网口ens192的22端口。提前在服务器上输入sudo tcpdump -i ens192 port 22 -n -c 10[adminlocalhost ~]$ sudo tcpdump -i ens192 port 22 -n -c 10tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes05:50:09.195672 IP 172.17.8.242.63234 172.27.101.128.ssh: Flags [S], seq 1153149008, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 ---------第一次握手客户端172.27.8.242向服务端172.27.101.198的22端口发SYNC请求连接。05:50:09.195766 IP 172.27.101.128.ssh 172.17.8.242.63234: Flags [S.], seq 491607821, ack 1153149009, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0 ---------第二次握手服务端回复SYN-ACK同意连接。05:50:09.195946 IP 172.17.8.242.63234 172.27.101.128.ssh: Flags [.], ack 1, win 8212, length 0 ---------第三次握手客户端回复ACK连接建立。05:50:09.197033 IP 172.17.8.242.63234 172.27.101.128.ssh: Flags [P.], seq 1:34, ack 1, win 8212, length 33 ---------服务器开始推动数据05:50:09.197054 IP 172.27.101.128.ssh 172.17.8.242.63234: Flags [.], ack 34, win 229, length 0 ---------客户端确认收到数据。05:50:09.223819 IP 172.27.101.128.ssh 172.17.8.242.63234: Flags [P.], seq 1:22, ack 34, win 229, length 2105:50:09.225924 IP 172.17.8.242.63234 172.27.101.128.ssh: Flags [P.], seq 34:1466, ack 22, win 8212, length 143205:50:09.246570 IP 172.27.101.128.ssh 172.17.8.242.63234: Flags [P.], seq 22:1302, ack 1466, win 251, length 128005:50:09.248588 IP 172.17.8.242.63234 172.27.101.128.ssh: Flags [P.], seq 1466:1514, ack 1302, win 8207, length 4805:50:09.256171 IP 172.27.101.128.ssh 172.17.8.242.63234: Flags [P.], seq 1302:1594, ack 1514, win 251, length 29210 packets captured10 packets received by filter0 packets dropped by kernel注意这个第1-3次握手都没显示执行3的时候就开始抓上面的包了。3、发起ssh连接。C:\Users\Johnssh admin172.27.101.128The authenticity of host 172.27.101.128 (172.27.101.128) cant be established.ED25519 key fingerprint is SHA256:qRwXoIvu6dDXQEuqsA23Z0zPRaeNxFhcyo4mfNVM4.This key is not known by any other names.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesWarning: Permanently added 172.27.101.128 (ED25519) to the list of known hosts.admin172.27.101.128s password:Last login: Fri Jul 10 05:43:17 2026[adminlocalhost ~]$[adminlocalhost ~]$五、解惑。为什么这次使用端口号进行tcpdump抓包没有使用ICMP进行抓包以下是区别