Skip to main content

Analyze and Troubleshoot your network

·8 mins

Bandwidth, Throughput and Latency #

Bandwidth tells you how much data can be sent or transferred from Point A to Point B at a given time. Bandwidth is also referred to your internet speed provided by your ISP. It is measured in bits, megabits, and gigabits per second.

You can determine how fast is your internet speed by running a test on speedtest.net. On the CLI, the most commonly used tool is sppedtest-cli.

Throughput is the measure of how much data is successfully transferred from source to destination.

Latency is how much time a packet took or how fast it takes to get from Point A to Point B.

Higher bandwidth doesn’t technically increase your internet speed. For example, if your bandwidth is 20Mbps and the destination has a speed of 5Mbps, increasing your bandwidth won’t make the communication faster.

If there’re more than 15 people using a 20Mbps bandwidth intensively, then increasing it will definitely smooth things out.

Latency also plays a big role in network communication. Regardless of your bandwidth, if latency is higher, then communication will seems slower. If it took you 30ms with a speed of 10 Mbps to ping a game server, and someone closer to the server with a speed of 2 Mbps took 12 ms to ping the server, increasing your bandwidth won’t change anything. Even if you have a 1 Gbps bandwidth, someone with a lower latency and lower response time is gonna have a better experience than you.

Network Connectivity Tools #

There are so many tools out there. Here are some of the commands that I use on a daily basis in my Home Lab.

Ping #

I always use ping to check is a system is up. If you’re getting a Request timeout, maybe the system is down. Or maybe it’s not accepting ICMP request:

MacBook-Pro:~ kavish$ ping 192.168.100.129
PING 192.168.100.129 (192.168.100.129): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

You can try ssh even if you don’t have access just to verify if you get a response back. Nmap is also great. To probe a system without ping, use nmap -Pn(not reliable).

Pinging your own host or localhost only tells you that the TCP/IP stack of your OS is working correctly. It doesn’t tell you if your network card is properly configured or has a problem. Even misconfigured, damaged network cards or cards with incorrect drivers can still reply selfping.

If you can’t reach a host on another subnet or internet, maybe your router/switch or your host’s network configuration is badly configured, or there’s a problem with Ethernet cable or wireless access point.

If everything seems fine, try running ping for long time with -i(for interval), to the destination host or your network:

ping -i 3 -c 300 x.x.x.x | tee ping-results-x.x.x.x.txt

fping #

With fping you can ping a subnet and get a report about which hosts are alive:

root@debian:~# fping -gaadqAs -i 1 -r 0 192.168.100.0/24
192.168.100.1 (192.168.100.1)
192.168.100.121 (192.168.100.121)
192.168.100.95 (192.168.100.95)

     254 targets
       3 alive
     251 unreachable
       0 unknown addresses

     251 timeouts (waiting for response)
     254 ICMP Echos sent
       3 ICMP Echo Replies received
       0 other ICMP received

 0.079 ms (min round trip time)
 131 ms (avg round trip time)
 391 ms (max round trip time)
        3.059 sec (elapsed real time)

Only hosts that allow ICMP requests will respond.

Traceroute and Tracepath #

traceroute is very important tool to look at the path a packet is taking to reach its destination:

MacBook-Pro:~ kavish$ traceroute debian.com
traceroute: Warning: debian.com has multiple addresses; using 149.20.4.15
traceroute to debian.com (149.20.4.15), 64 hops max, 52 byte packets
 1  192.168.100.1 (192.168.100.1)  2.441 ms  1.565 ms  1.267 ms
 2  192.168.227.1 (192.168.227.1)  3.640 ms  4.349 ms  4.063 ms
 3  196.20.225.43 (196.20.225.43)  204.781 ms  205.879 ms  205.969 ms
 4  197.226.230.12 (197.226.230.12)  211.801 ms  211.779 ms  214.471 ms
 5  if-ae-26-2.tcore2.ldn-london.as6453.net (80.231.62.57)  372.131 ms  371.869 ms  369.466 ms
 6  if-ae-32-2.tcore3.nto-newyork.as6453.net (63.243.216.22)  347.301 ms  337.069 ms  346.392 ms
 7  if-ae-22-2.tcore1.nto-newyork.as6453.net (63.243.128.17)  333.214 ms  330.403 ms  326.341 ms
 8  if-ae-0-2.tcore1.sqn-sanjose.as6453.net (63.243.128.31)  339.461 ms  339.687 ms
    if-ae-12-2.tcore1.sqn-sanjose.as6453.net (63.243.128.29)  322.194 ms
 9  if-ae-1-2.tcore2.sqn-sanjose.as6453.net (63.243.205.2)  342.399 ms  340.810 ms  342.774 ms
10  if-ae-5-4.tcore2.pdi-paloalto.as6453.net (64.86.21.41)  386.252 ms  341.316 ms
    if-ae-5-2.tcore2.pdi-paloalto.as6453.net (64.86.21.2)  343.736 ms
11  if-ae-2-2.tcore1.pdi-paloalto.as6453.net (66.198.127.1)  349.563 ms  340.262 ms  338.812 ms
12  int-0-0-0-4.r1.pao1.isc.org (149.20.65.145)  364.905 ms
    int-0-0-0-3.r1.pao1.isc.org (149.20.65.29)  361.785 ms
    int-0-0-0-4.r1.pao1.isc.org (149.20.65.145)  355.652 ms
13  mirror-isc3.debian.org (149.20.4.15)  332.558 ms  344.476 ms  332.610 ms

If there drops or timeouts on a public router, it’s out of your control. But it it’s happening on the network that you’re on, you’ll know where the problem is.

One more tool I should mentioned is tracepath. It does the same thing as traceroute, but uses UDP and only UDP and does not require root privileges. Some routers have a limit on how many ICMP request can be sent. If that’s your case, try tracepath.

More info here.

Netstat/SS #

To get a look on how many ports are open or what services are currently listening on your host, use

ss -ntulp
netstat -ntulp
netstat -ntulp4 #for IPV4

The above command will list all UDP/TCP ports that are open along with the services. You can also use nmap to scan your host or hosts on the network.

Throughput test with iperf #

Start the server with iperf3 -s:

[root@rockylinux ~]# nmcli | grep inet4
	inet4 192.168.100.125/24
[root@rockylinux ~]#
[root@rockylinux ~]# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

I’m testing from localhost with an interval of 5 seconds:

[root@rockylinux ~]# iperf3 -c localhost -t 5
Connecting to host localhost, port 5201
[  5] local ::1 port 43974 connected to ::1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   740 MBytes  6.19 Gbits/sec    4   2.93 MBytes
[  5]   1.00-2.00   sec   795 MBytes  6.69 Gbits/sec    5   2.93 MBytes
[  5]   2.00-3.00   sec   679 MBytes  5.69 Gbits/sec    7   2.93 MBytes
[  5]   3.00-4.00   sec   792 MBytes  6.66 Gbits/sec    2   2.93 MBytes
[  5]   4.00-5.00   sec   780 MBytes  6.54 Gbits/sec    6   2.93 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec  3.70 GBytes  6.35 Gbits/sec   24             sender
[  5]   0.00-5.02   sec  3.69 GBytes  6.32 Gbits/sec                  receiver

iperf Done.

This is a very basic test. Other useful options for sysadmins:

  • -P parallel connections
  • -R reverse mode: the server will send and the client will receive
  • -b indicate a limit bandwidth (for example 10k, 5mb, 1gb)
  • -J output in json
  • -B bind to another network interface

mtr #

mtr is also a very powerful tool that combines the functionality of ping and traceroute. Once all the address of each hop is determined, mtr will send ICMP ECHO requests to each one of them:

rockylinux (192.168.100.125)                                               2021-10-29T18:17:46+0400
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                           Packets               Pings
 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. _gateway                                              0.0%    34    2.4   4.6   2.4  20.9   3.5
 2. 192.168.227.1                                         0.0%    34    6.0   6.9   4.9  18.6   2.6
 3. 196.20.225.43                                         0.0%    34  229.8 229.5 227.2 242.7   2.7
 4. 197.226.230.12                                        0.0%    34  232.1 231.8 229.0 248.5   4.0
 5. ve990.core1.lon7.he.net                               0.0%    34  229.7 228.2 225.8 236.6   2.3
 6. swp26.annegrit.canonical.com                          0.0%    33  220.8 223.3 220.8 232.2   3.3
 7. cactuar.canonical.com                                 0.0%    33  248.0 249.7 247.4 259.3   2.7

Around 34 ICMP ECHO requests were sent. There’s no sign of packet loss. Last, Avg, Best, Worst are latencies in milliseconds. The first 4 hops belong to my router and ISP. The command will run forever. Just like ping. You can run it in report mode with -r and -c for count:

[root@rockylinux ~]# mtr -rc 10 ubuntu.com
Start: 2021-10-29T18:21:30+0400
HOST: rockylinux                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                   0.0%    10    3.0   3.2   2.6   4.7   0.6
  2.|-- 192.168.227.1              0.0%    10    5.9   5.6   5.1   6.0   0.3
  3.|-- 196.20.225.43              0.0%    10  227.4 252.1 226.8 302.0  32.9
  4.|-- 197.226.230.12             0.0%    10  237.4 231.0 228.6 237.4   2.8
  5.|-- ve956.core2.lon2.he.net    0.0%    10  221.9 222.9 221.4 225.5   1.6
  6.|-- swp9.il3-core1.canonical. 10.0%    10  248.0 248.5 247.7 249.2   0.6
  7.|-- website-content-cache-3.c  0.0%    10  248.3 251.3 247.7 278.7   9.6

TCP can used if hosts on your LAN doesn’t allow ICMP request. TCP mode will use SYN packets just like nmap:

[root@rockylinux ~]# mtr --tcp --port 443 -rc 10 google.com
Start: 2021-10-29T18:29:56+0400
HOST: rockylinux                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                   0.0%    10    3.0   3.1   2.6   3.9   0.5
  2.|-- 192.168.227.1              0.0%    10    5.6   6.4   5.2  10.7   1.7
  3.|-- 196.20.225.43              0.0%    10    5.9   7.8   5.5  19.0   4.1
  4.|-- 197.226.230.12             0.0%    10    6.3   6.4   5.6   7.3   0.6
  5.|-- 72.14.221.88               0.0%    10   48.6  45.0  43.4  48.6   1.7
  6.|-- 108.170.253.145            0.0%    10   44.1  44.2  43.5  45.4   0.6
  7.|-- 108.170.226.181            0.0%    10   43.8  44.4  42.7  46.7   1.2
  8.|-- mba01s07-in-f14.1e100.net  0.0%    10   44.3  46.5  43.3  60.2   5.0

“Twenty years from now you will be more disappointed by the things that you didn't do than by the ones you did do. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover.”H. Jackson Brown Jr., P.S. I Love You