Lesson 13 - Layer 2 Connectivity Troubleshooting Part 3

Lesson 13 - Layer 2 Connectivity Troubleshooting Part 3

This lesson is the last one in the series on how to troubleshoot connectivity issues at the layer 2 of OSI model. Bear in mind, that these do not involve layer 2 technologies such as VLANs or Spanning-Tree Protocol, since we have not talked about those yet.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 3
New installation as per Pic. 1 shows lack of connectivity between the two computers. Initial diagnostics performed revealed the following facts:

• Switches are connected via fibre optics cable and the ports show proper status (interface is up, line protocol is up).
• Computers have proper addresses and subnet masks assigned.
• Cables connecting computers with switches have been tested and proved to be working correctly.
• Computers reply to echo requests packets (firewalls disabled).

The technician who set up this new network calls you for help.

Pic. 1 - New design with connectivity problem

Icons designed by: Andrzej Szoblik - http://www.newo.pl

Dealing with this trouble ticket we are going to collect the tools we used in the previous lessons trying to resolve this issue.

Step 1

First let's try to 'divide and conquer' (concept mentioned in lesson 11) by sending ping packets from PC1 to PC2. Before we do that though, we need to purge the existing ARP cache on PC1 and PC2 to have a fresh information. We do that by opening Command Line Interface window and typing: arp -d host-address (linux), or arp -d (MS Windows).

Test results:

• The pings timed out. No reply from PC2.

Step 2

Since the ARP cache entries age out relatively quickly (depending on which operating system you use), we need to quickly check what they contain Alternatively we can send large series of ping packets.

Test results:

• PC1 does NOT contain expected mac-to-ip mapping. We expected to see
  192.168.1.2 at 00:1e:4f:b0:b2:fc. It is not there, though.
• PC2 DOES contain the the right mac-to-ip mapping. It shows the following:
  192.168.1.1 at 00:50:bf:9c:45:6a.

These are very interesting results, don't you think? Before we take the next step let's gather what we know so far.

Since the ping was initiated by PC1, it sent its ARP request broadcast message and that query must have been delivered to PC2. We can conclude that, based on the fact that we have cleared PC2's ARP cache in step 1, and it has the proper mapping now. PC2 did receive ARP request from PC1 and learned what MAC and IP address it uses. 

Step 3

We could omit that step, but we're curious if PC2 replies to the ARP request from PC1. We launch our 'wireshark' tool on PC2, ping again from PC1 to PC2 and capture all packets on PC2. What we discover in this packet trace is that PC2 has replied to ARP request with proper ARP reply unicast message back to PC1.

Step 4

Clearly, something between the computers (switches) does not work properly. It seems that we have some sort of unidirectional communication. What we need to establish is, where this unidirectional communication is taking place. We login to the SW1 and SW2 and issue the following commands ('x' here stands for switch number in Pic. 1):

SWx#show mac address-table interface f0/1

SWx#show mac address-table interface f0/24

Test results:

• SW1 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/1 interface. This is expected.
• SW1 does NOT learn MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/24 interface. This is unexpected. It should learn it from the ARP reply sent by PC2.
• SW2 learns source MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/1 interface. This is expected.
• SW2 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/24 interface. This is expected.

This way, we have discovered that SW1 has unidirectional link towards SW2 (SW2 sends frames towards SW1 but the latter does not seem to receive those). Probably, the fiber optics connection does not work properly (grease, dirt, a strand is broken etc.).

One more time, this lesson illustrates how useful the commands and knowledge described in the previous posts, can be in real life scenarios. 

In my next post, I will show you how to log system messages so they can be analyzed later. System messages are invaluable pieces information in the process of troubleshooting networking issues.

Lesson 12 - Layer 2 Connectivity Troubleshooting Part 2

Lesson 12 - Layer 2 Connectivity Troubleshooting Part 2

In the previous lesson I attempted to show you how to go about a performance problems we might experience due to the duplex mismatch. It was our first trouble ticket in the series on layer 2 problems.

In this lesson I am going to create another issue to show you how your current skills can be practically useful in diagnosing network connectivity problems.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 2
Some client computers have problems accessing FTP server located in the same network 192.168.1.0/24. This behavior is very intermittent.

We need to collect a bit more data related to this problem. A good starting point might be to take down the following pieces of information:


FTP Server
IP address  = 192.168.1.4
MAC address = 00:10:5a:d3:e4:e0

FTP Client1 (with no connectivity to FTP server)
IP address = 192.168.1.2
MAC address = 00:10:4f:b0:b2:fc

Here is one way of doing basic diagnostics.

Step 1
We want to make sure we have layer 1-3 connectivity between the Client1 and the server first (remember 'divide and conquer' method from the previous lesson?).

jr@mandala:~$ ping 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=0.372 ms
64 bytes from 192.168.1.4: icmp_seq=2 ttl=64 time=0.349 ms
64 bytes from 192.168.1.4: icmp_seq=3 ttl=64 time=0.371 ms
64 bytes from 192.168.1.4: icmp_seq=4 ttl=64 time=0.374 ms

It seems we have layer 3 connectivity working just fine.

Here's an interesting question for you. How do you know, you have received the replies from the FTP server in question (192.168.1.4)?

Now, you may have very confused expression on you face as in 'what do you mean?'. Have I not just gotten the reply from it?

Well, let me show you something to address this question.

Step 2
In order to be absolutely sure I got the reply from the FTP server (192.168.1.4) I need to verify my computer's ARP cache (Client1). Recall, that every time a host sends the packets, it encapsulates them in layer 2 frames (here Ethernet ones). In order to do that, the Client must have a valid destination MAC address (00:10:5a:d3:e4:e0) mapped to its IP address (192.168.1.4). If this mapping entry is not found in the APR cache, the Client1 will send ARP request to learn MAC address of 192.168.1.4. Knowing it let's check the ARP cache after we have sent ping packets. Here's what we find in (Pic. 1):

Pic. 1 - ARP Cache Entries

Pay attention to the highlighted entry. Is the MAC address mapped to our FTP server's IP address correct? 

NO! 

This MAC address does not belong to the server. FTP server's MAC address is: 00:10:5a:d3:e4:e0.

That test proved that getting echo replies to our echo packets (ping) must also be verified in the sender's ARP cache.

So, which device does this MAC address 00:10:0f:a3:3b:e6 belong to? And how on earth did it end up in our Client's ARP cache?

Well, in order to answer the second question, the reasons for this wrong mapping might be different. There could be the device with the duplicate address (same as the FTP server). Then, when the clients send ARP request for the MAC address of the server, this 'rouge' device (not the legitimate FTP server), also responds to the query. And if its answers arrives later than from the legitimate server, the override the legitimate MAC address mapping in the ARP cache. If some other clients get the reply for their ARP query from the 'rouge' device first and then from the FTP server, they create the mapping correctly. That would explain why some computers can still FTP to the server and others can't. Another reason for this wrong mac-to-ip mappings might be ARP poisoning attack in the network (eg. using Ettercap tool).

As for the answer to the first question, if this is not an ARP poisoning attack, you can use your knowledge from the lesson 9 about switching which helps you understand CAM table creation. Then, send uninterrupted ping towards the 'rouge' device and try to trace the mac addresses from switch to switch to find out where the device with duplicate IP address is located. MAC address table should help you find it relatively quickly.

There might be one more question I would like to clarify. Why the duplicate address was not detected by the 'rouge' device? Answer to that question may surprise you. The duplicate address detection uses so called 'gratuitous arp'. This is an unsolicited advertising of the MAC address upon computer startup. If some computers use the same MAC or IP, the newly computer cannot use its IP in the network. Unfortunately, some operating systems may not use this mechanism. They 'trust' what the operator does. If she or he wants this address, there will be no protest on their part.

I hope you are now beginning to see how much you already know!

In my next lesson you are going to analyze the third trouble ticket for layer 2 connectivity. This is going to be the last troubleshooting lesson. The upcoming lessons will describe other tools and interesting technologies such as VLANs, Spanning-Tree Protocol. Once we finish with foundations related to layer 2, we'll start discussing layer 3 technologies and concepts such as IP addressing, subnetting, routing etc.

Lesson 11 - Layer 2 Connectivity Troubleshooting Part 1

Lesson 11 - Layer 2 Connectivity Troubleshooting Part 1

The last two lessons I tried to explain the foundations related to layer 2 operation. I discussed very important switching process and CDP protocol which comes in handy at times. If you also have watched the video in the lesson 10, you got a glimpse of few commands explained in the theory earlier on.

In this lesson, we'll focus on practical application of the layer 1 and layer 2 concepts which could be helpful in troubleshooting networking issues.

Even though we're capable of creating lots of great things we're still human beings. And 'to err is human' adage is as conspicuous in the networking field as anywhere else. This means, that occasionally things won't work as expected. In such situations we need to be able to isolate and fix the problems quickly.

Reactive troubleshooting almost always uses the following work flow:

1. A problem is reported.
2. Data and facts are collected.
3. Data analysis must be performed.
4. Potential causes are eliminated.
5. Hypothesis is drawn.
6. Hypothesis is verified. 

In some situations the steps 3 and 4 could be omitted. But it depends on the nature of the problem reported, severity of issue, skills of the technician etc.

As we have yet to learn layer 2 technologies such as Vlans and Spannig-Tree Protocol which add complexity to troubleshooting process, let us, for now, focus on simple cases. This way we're going to build our troubleshooting skills step by step given the knowledge we posses.

Recall the process of moving data from one computer to another. Everything sent from the application layer goes down to the physical layer. That teaches us one important thing: if the layer 1 is not operational, nothing else will work!

So, we could start diagnosing the networking problems by checking the layer 1 connectivity first. Then, we could move on to the layer 2 and work our way up till we isolate the problem. This makes perfect sense. However, a lot of technicians use other method known as 'divide and conquer'.  In networking diagnostics that might be checking the layers in the middle. For instance using the 'ping' utility we can check the layer 1 through layer 3 status as a starting point. Of course, you already know how the 'ping' works, don't you? 'Ping' uses ICMP protocol which is a layer 3 messaging mechanism encapsulated directly in IP headers.

---

NOTICE
Ping utility is a layer 3 reachability checking mechanism. It sends a number of ICMP echo messages to the destination host. If the destination host receives echo messages, it sends ICMP echo reply messages back to the sender. Of course, assuming that there are no mechanisms implemented that would filter those messages along the path (like local firewall for instance), the sender should receive those replies and thus effectively checking layer 1 through layer 3 reachability. Thus, the path is verified in BOTH directions.

---

If you aim at the layer 3 reachability using 'ping' utility, you may get one of the two results:

1. You get the reply from the destination. This leads to a conclusion that all layers 1 through 3 are working properly. And the connectivity problems might be related to upper layers (layer 4 upward).
2. You do not get the reply from the destination. In that case, this step is not enough to determine the nature of the issue. In this case, you must perform some additional diagnostics.

Now, I am going to show you a few of such steps.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 1
Data transfer from PC1 toward PC2 is very slow (refer to Pic 1).

Pic. 1 - Simple Topology

 

Icons designed by: Andrzej Szoblik - http://www.newo.pl

As you see, without the topology diagram, it is much more challenging to do diagnostics. If we know the topology and the path between the source and destination devices in question, we can focus on all components that participate in the data transmission and isolate the issue.

First, let us determine what we know.


The transmission succeeds, but is slow. Verify that yourself. Do not assume it's true or properly tested by the person who reported that. Most users cannot properly describe the nature of the problems.

So, you have done the tests and the transfer proves to be slow indeed.


Some questions you might ask:

• Has this problem occurred recently?
• Is this a new client computer or destination server?
• Has any configuration/update/cable replacement/or any other changes been done on any devices in the path before the problem manifested itself?
• Are other computers experiencing the same problem, or is this individual case?
• etc.

Further data and fact collection may depend on the answers to those questions. For the argument's sake let's assume, that the server is a brand new computer installed the day before the problem occurred. All clients sending files to the server suffer from slow data transfer. When you connected the same client to the server directly, the transfer is very fast (computer-to-server through cross over cable).


Given those facts, we're going to take a quick look at the status of the interface where our server computer is connected. The command that is very useful to check both the layer 1 and layer 2 status is:

SW2#show int f0/2 

Pic. 2 - show interface command

This output deserves some explanation.

FastEthernet is up,

This is the status of the layer 1 connectivity (your cable seems to be attached right?).

line protocol is up

This is the status of the layer 2. It looks like the keepalive packets sent every 10 seconds are working back and forth (do not trust it entirely though; look at the trouble ticket 2 in the next lesson).

half-duplex, 100Mb/s

Duplex negotiated is HALF duplex. Most network adapters used in the computers use AUTO negotiation. This means, that the NIC (Network Interface Card aka network adapter), sends special signals to the port of the switch trying to negotiate FULL duplex and the highest speed supported. Unfortunately, some NIC manufacturers do not follow the specification regarding this signaling. This causes some "misunderstanding" between the port of the switch and the NIC. Switch typically drops down the duplex from FULL to HALF. That causes the switch port to enable the Carrier Sense, Multiple Access with Collision Detection mechanism (CSMA/CD) which is used on SHARED not dedicated connections (for details look at the lesson 8).

What we end up having is the NIC working in FULL duplex but the port of the switch runs HALF duplex. It 'thinks' it can either send or receive data but not do both at the same time. When server begins to 'push' data across the network, the port of the switch cannot send anything out towards the server. When it finally 'thinks' it can (medium free) and sends data towards the server, the latter begins to send data down towards the port as it is allowed in FULL duplex connections. The switch must stop immediately as the frames are experiencing collision. At least that is how the switch works under the circumstances. Then it waits till the carrier is free again (no data from the server down the port). This problem is known as: DUPLEX MISMATCH. That results in great number of collisions and late collisions recorded on the port of the switch like shown in the above output.

The solution to that problem could be the following actions:

1. Try to upgrade the NIC driver using your server/computer vendor's web site.
2. If the problem persists, you may try to hard code speed and duplex. You have to do this on both ends of the connections. This disables the AUTO NEGOTIATION feature (do not listen to people who say you should do this on one end of the connection).
3. Sometimes, though very unlikely in our situation, the cable can cause that sort of behavior. Replacing it to the one that is proven to be good, might help. Again, typically we would see some other layer 1 errors (CRC errors, carrier loss).
4. Replace the NIC on the server to the one that you are sure is working well.

Hard Coding Speed and Duplex

SW2#configure terminal

SW2(config)#interface fastethernet0/2

SW2(config-if)#speed 100

SW2(config-if)#duplex full

As for the computer, you have to refer to the manual of your operating system  how to set speed and duplex on the NIC manually. Perhaps google this. Google are the best!

In the next lessons, we will resolve two more connectivity issues given the skills we obtained in previous lessons.

Lesson 13 - Layer 2 Connectivity Troubleshooting Part 3

This lesson is the last one in the series on how to troubleshoot connectivity issues at the layer 2 of OSI model. Bear in mind, that these do not involve layer 2 technologies such as VLANs or Spanning-Tree Protocol, since we have not talked about those yet.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 3
New installation as per Pic. 1 shows lack of connectivity between the two computers. Initial diagnostics performed revealed the following facts:

• Switches are connected via fibre optics cable and the ports show proper status (interface is up, line protocol is up).
• Computers have proper addresses and subnet masks assigned.
• Cables connecting computers with switches have been tested and proved to be working correctly.
• Computers reply to echo requests packets (firewalls disabled).

The technician who set up this new network calls you for help.

Pic. 1 - New design with connectivity problem

Icons designed by: Andrzej Szoblik - http://www.newo.pl

Dealing with this trouble ticket we are going to collect the tools we used in the previous lessons trying to resolve this issue.

Step 1

First let's try to 'divide and conquer' (concept mentioned in lesson 11) by sending ping packets from PC1 to PC2. Before we do that though, we need to purge the existing ARP cache on PC1 and PC2 to have a fresh information. We do that by opening Command Line Interface window and typing: arp -d host-address (linux), or arp -d (MS Windows).

Test results:

• The pings timed out. No reply from PC2.

Step 2

Since the ARP cache entries age out relatively quickly (depending on which operating system you use), we need to quickly check what they contain Alternatively we can send large series of ping packets.

Test results:

• PC1 does NOT contain expected mac-to-ip mapping. We expected to see
  192.168.1.2 at 00:1e:4f:b0:b2:fc. It is not there, though.
• PC2 DOES contain the the right mac-to-ip mapping. It shows the following:
  192.168.1.1 at 00:50:bf:9c:45:6a.

These are very interesting results, don't you think? Before we take the next step let's gather what we know so far.

Since the ping was initiated by PC1, it sent its ARP request broadcast message and that query must have been delivered to PC2. We can conclude that, based on the fact that we have cleared PC2's ARP cache in step 1, and it has the proper mapping now. PC2 did receive ARP request from PC1 and learned what MAC and IP address it uses. 

Step 3

We could omit that step, but we're curious if PC2 replies to the ARP request from PC1. We launch our 'wireshark' tool on PC2, ping again from PC1 to PC2 and capture all packets on PC2. What we discover in this packet trace is that PC2 has replied to ARP request with proper ARP reply unicast message back to PC1.

Step 4

Clearly, something between the computers (switches) does not work properly. It seems that we have some sort of unidirectional communication. What we need to establish is, where this unidirectional communication is taking place. We login to the SW1 and SW2 and issue the following commands ('x' here stands for switch number in Pic. 1):

SWx#show mac address-table interface f0/1

SWx#show mac address-table interface f0/24

Test results:

• SW1 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/1 interface. This is expected.
• SW1 does NOT learn MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/24 interface. This is unexpected. It should learn it from the ARP reply sent by PC2.
• SW2 learns source MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/1 interface. This is expected.
• SW2 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/24 interface. This is expected.

This way, we have discovered that SW1 has unidirectional link towards SW2 (SW2 sends frames towards SW1 but the latter does not seem to receive those). Probably, the fiber optics connection does not work properly (grease, dirt, a strand is broken etc.).

One more time, this lesson illustrates how useful the commands and knowledge described in the previous posts, can be in real life scenarios. 

In my next post, I will show you how to log system messages so they can be analyzed later. System messages are invaluable pieces information in the process of troubleshooting networking issues.