Analyze and Troubleshoot your network
Table of Contents
Bandwidth, Throughput and Latency #
Bandwidth tells you how much data can be sent or transferred from Point A to Point B at a given time. Bandwidth is also referred to your internet speed provided by your ISP. It is measured in bits, megabits, and gigabits per second.
You can determine how fast is your internet speed by running a test on speedtest.net. On the CLI, the most commonly used tool is sppedtest-cli.
Throughput is the measure of how much data is successfully transferred from source to destination.
Latency is how much time a packet took or how fast it takes to get from Point A to Point B.
Higher bandwidth doesn’t technically increase your internet speed. For example, if your bandwidth is 20Mbps and the destination has a speed of 5Mbps, increasing your bandwidth won’t make the communication faster.
If there’re more than 15 people using a 20Mbps bandwidth intensively, then increasing it will definitely smooth things out.
Latency also plays a big role in network communication. Regardless of your bandwidth, if latency is higher, then communication will seems slower. If it took you 30ms with a speed of 10 Mbps to ping a game server, and someone closer to the server with a speed of 2 Mbps took 12 ms to ping the server, increasing your bandwidth won’t change anything. Even if you have a 1 Gbps bandwidth, someone with a lower latency and lower response time is gonna have a better experience than you.
Network Connectivity Tools #
There are so many tools out there. Here are some of the commands that I use on a daily basis in my Home Lab.
Ping #
I always use ping to check is a system is up. If you’re getting a Request timeout, maybe the system is down. Or maybe it’s not accepting ICMP request:
MacBook-Pro:~ kavish$ ping 192.168.100.129
PING 192.168.100.129 (192.168.100.129): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
You can try ssh even if you don’t have access just to verify if you get a response back. Nmap is also great. To probe a system without ping, use nmap -Pn(not reliable).
Pinging your own host or localhost only tells you that the TCP/IP stack of your OS is working correctly. It doesn’t tell you if your network card is properly configured or has a problem. Even misconfigured, damaged network cards or cards with incorrect drivers can still reply selfping.
If you can’t reach a host on another subnet or internet, maybe your router/switch or your host’s network configuration is badly configured, or there’s a problem with Ethernet cable or wireless access point.
If everything seems fine, try running ping for long time with -i(for interval), to the destination host or your network:
ping -i 3 -c 300 x.x.x.x | tee ping-results-x.x.x.x.txt
fping #
With fping you can ping a subnet and get a report about which hosts are alive:
root@debian:~# fping -gaadqAs -i 1 -r 0 192.168.100.0/24
192.168.100.1 (192.168.100.1)
192.168.100.121 (192.168.100.121)
192.168.100.95 (192.168.100.95)
254 targets
3 alive
251 unreachable
0 unknown addresses
251 timeouts (waiting for response)
254 ICMP Echos sent
3 ICMP Echo Replies received
0 other ICMP received
0.079 ms (min round trip time)
131 ms (avg round trip time)
391 ms (max round trip time)
3.059 sec (elapsed real time)
Only hosts that allow ICMP requests will respond.
Traceroute and Tracepath #
traceroute is very important tool to look at the path a packet is taking to reach its destination:
MacBook-Pro:~ kavish$ traceroute debian.com
traceroute: Warning: debian.com has multiple addresses; using 149.20.4.15
traceroute to debian.com (149.20.4.15), 64 hops max, 52 byte packets
1 192.168.100.1 (192.168.100.1) 2.441 ms 1.565 ms 1.267 ms
2 192.168.227.1 (192.168.227.1) 3.640 ms 4.349 ms 4.063 ms
3 196.20.225.43 (196.20.225.43) 204.781 ms 205.879 ms 205.969 ms
4 197.226.230.12 (197.226.230.12) 211.801 ms 211.779 ms 214.471 ms
5 if-ae-26-2.tcore2.ldn-london.as6453.net (80.231.62.57) 372.131 ms 371.869 ms 369.466 ms
6 if-ae-32-2.tcore3.nto-newyork.as6453.net (63.243.216.22) 347.301 ms 337.069 ms 346.392 ms
7 if-ae-22-2.tcore1.nto-newyork.as6453.net (63.243.128.17) 333.214 ms 330.403 ms 326.341 ms
8 if-ae-0-2.tcore1.sqn-sanjose.as6453.net (63.243.128.31) 339.461 ms 339.687 ms
if-ae-12-2.tcore1.sqn-sanjose.as6453.net (63.243.128.29) 322.194 ms
9 if-ae-1-2.tcore2.sqn-sanjose.as6453.net (63.243.205.2) 342.399 ms 340.810 ms 342.774 ms
10 if-ae-5-4.tcore2.pdi-paloalto.as6453.net (64.86.21.41) 386.252 ms 341.316 ms
if-ae-5-2.tcore2.pdi-paloalto.as6453.net (64.86.21.2) 343.736 ms
11 if-ae-2-2.tcore1.pdi-paloalto.as6453.net (66.198.127.1) 349.563 ms 340.262 ms 338.812 ms
12 int-0-0-0-4.r1.pao1.isc.org (149.20.65.145) 364.905 ms
int-0-0-0-3.r1.pao1.isc.org (149.20.65.29) 361.785 ms
int-0-0-0-4.r1.pao1.isc.org (149.20.65.145) 355.652 ms
13 mirror-isc3.debian.org (149.20.4.15) 332.558 ms 344.476 ms 332.610 ms
If there drops or timeouts on a public router, it’s out of your control. But it it’s happening on the network that you’re on, you’ll know where the problem is.
One more tool I should mentioned is tracepath. It does the same thing as traceroute, but uses UDP and only UDP and does not require root privileges. Some routers have a limit on how many ICMP request can be sent. If that’s your case, try tracepath.
More info here.
Netstat/SS #
To get a look on how many ports are open or what services are currently listening on your host, use
ss -ntulp
netstat -ntulp
netstat -ntulp4 #for IPV4
The above command will list all UDP/TCP ports that are open along with the services. You can also use nmap to scan your host or hosts on the network.
Throughput test with iperf #
Start the server with iperf3 -s:
[root@rockylinux ~]# nmcli | grep inet4
inet4 192.168.100.125/24
[root@rockylinux ~]#
[root@rockylinux ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
I’m testing from localhost with an interval of 5 seconds:
[root@rockylinux ~]# iperf3 -c localhost -t 5
Connecting to host localhost, port 5201
[ 5] local ::1 port 43974 connected to ::1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 740 MBytes 6.19 Gbits/sec 4 2.93 MBytes
[ 5] 1.00-2.00 sec 795 MBytes 6.69 Gbits/sec 5 2.93 MBytes
[ 5] 2.00-3.00 sec 679 MBytes 5.69 Gbits/sec 7 2.93 MBytes
[ 5] 3.00-4.00 sec 792 MBytes 6.66 Gbits/sec 2 2.93 MBytes
[ 5] 4.00-5.00 sec 780 MBytes 6.54 Gbits/sec 6 2.93 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 3.70 GBytes 6.35 Gbits/sec 24 sender
[ 5] 0.00-5.02 sec 3.69 GBytes 6.32 Gbits/sec receiver
iperf Done.
This is a very basic test. Other useful options for sysadmins:
- -P parallel connections
- -R reverse mode: the server will send and the client will receive
- -b indicate a limit bandwidth (for example 10k, 5mb, 1gb)
- -J output in json
- -B bind to another network interface
mtr #
mtr is also a very powerful tool that combines the functionality of ping and traceroute. Once all the address of each hop is determined, mtr will send ICMP ECHO requests to each one of them:
rockylinux (192.168.100.125) 2021-10-29T18:17:46+0400
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. _gateway 0.0% 34 2.4 4.6 2.4 20.9 3.5
2. 192.168.227.1 0.0% 34 6.0 6.9 4.9 18.6 2.6
3. 196.20.225.43 0.0% 34 229.8 229.5 227.2 242.7 2.7
4. 197.226.230.12 0.0% 34 232.1 231.8 229.0 248.5 4.0
5. ve990.core1.lon7.he.net 0.0% 34 229.7 228.2 225.8 236.6 2.3
6. swp26.annegrit.canonical.com 0.0% 33 220.8 223.3 220.8 232.2 3.3
7. cactuar.canonical.com 0.0% 33 248.0 249.7 247.4 259.3 2.7
Around 34 ICMP ECHO requests were sent. There’s no sign of packet loss. Last, Avg, Best, Worst are latencies in milliseconds. The first 4 hops belong to my router and ISP. The command will run forever. Just like ping. You can run it in report mode with -r and -c for count:
[root@rockylinux ~]# mtr -rc 10 ubuntu.com
Start: 2021-10-29T18:21:30+0400
HOST: rockylinux Loss% Snt Last Avg Best Wrst StDev
1.|-- _gateway 0.0% 10 3.0 3.2 2.6 4.7 0.6
2.|-- 192.168.227.1 0.0% 10 5.9 5.6 5.1 6.0 0.3
3.|-- 196.20.225.43 0.0% 10 227.4 252.1 226.8 302.0 32.9
4.|-- 197.226.230.12 0.0% 10 237.4 231.0 228.6 237.4 2.8
5.|-- ve956.core2.lon2.he.net 0.0% 10 221.9 222.9 221.4 225.5 1.6
6.|-- swp9.il3-core1.canonical. 10.0% 10 248.0 248.5 247.7 249.2 0.6
7.|-- website-content-cache-3.c 0.0% 10 248.3 251.3 247.7 278.7 9.6
TCP can used if hosts on your LAN doesn’t allow ICMP request. TCP mode will use SYN packets just like nmap:
[root@rockylinux ~]# mtr --tcp --port 443 -rc 10 google.com
Start: 2021-10-29T18:29:56+0400
HOST: rockylinux Loss% Snt Last Avg Best Wrst StDev
1.|-- _gateway 0.0% 10 3.0 3.1 2.6 3.9 0.5
2.|-- 192.168.227.1 0.0% 10 5.6 6.4 5.2 10.7 1.7
3.|-- 196.20.225.43 0.0% 10 5.9 7.8 5.5 19.0 4.1
4.|-- 197.226.230.12 0.0% 10 6.3 6.4 5.6 7.3 0.6
5.|-- 72.14.221.88 0.0% 10 48.6 45.0 43.4 48.6 1.7
6.|-- 108.170.253.145 0.0% 10 44.1 44.2 43.5 45.4 0.6
7.|-- 108.170.226.181 0.0% 10 43.8 44.4 42.7 46.7 1.2
8.|-- mba01s07-in-f14.1e100.net 0.0% 10 44.3 46.5 43.3 60.2 5.0
“Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover.” ― H. Jackson Brown Jr., P.S. I Love You