Lesson 14 - NTP and Syslog Services

Lesson 14 - NTP and Syslog Services

My previous three posts were a humble attempt to show you some real life networking issues and how to go about them using the skills described so far.

In this lesson I would like to present two services that are extremely important in management of your switches and routers: Network Time Protocol, and Syslog Services. Even though you will not find them in CCNA curriculum, it is good idea to know what is their role and how to quickly configure them on your devices.

System Messages
If you work as a network admin, it is critical that you collect and analyze system messages sent by switches and routers. IOS can send those important messages to the console port 0 by default. You can store them in the switch or router's memory but they will be purged if you have power outage or reboot your device. Also, memory will store as many of them, and then it will begin to overwrite the oldest ones. We need to redirect them to an external server. One of the popular services used to collect system messages is called: Syslog Server. If you are Window user you must probably pay for such server software (although KIWI server used to be freeware, but I don't know if it still free software). Unix and Linux have this service installed by default. All you have to do is to set it up correctly, so it accepts messages from external clients.

If you want to check how to do it using Ubuntu Linux distribution, please refer to my small Ubuntu notepad at: http://ubuntu-garage.blogspot.com/2010/09/ubuntu-syslog-server.html


System messages have the different levels of severity as shown below.

0 - Emergency - System-unusable messages
1 - Alert - Take immediate action
2 - Critical - Critical condition
3 - Error - error message
4 - Warning - warning message
5 - Notice - normal but significant condition
6 - Informational - information message
7 - Debug - debug messages and log FTP commands and WWW URLs

As you see, the lower the number the higher severity the level is. I'm sure I don't have to tell you that the levels 0-3 will need your special attention, do I?

System Logging Message takes the following format:

timestamp%<facility>-<severity>-<mnemonic>: <message-text>

Take a look at such message as sent by IOS (Pic. 1)

Pic. 1 - IOS Syslog Message Example.

Network Time Protocol (NTP)
All messages should carry a time stamp. The time of an event allows administrator to see when things went hairy and correlate them with other events that might follow. The problem is that low-end Cisco devices do not keep the date and time like computers do. In order for them to keep the track of time you must either manually set the clock with 'clock' command or synchronize their time with some external sources. The first method is not recommended as after reboot, a router or switch loses its time. That is why the second method is recommended using NTP protocol.

It is not my intention to give you an in-depth description of NTP and syslog services. Instead, I would like to draw your attention to those services and show you how to set it up quickly.

NTP server information:

NTP Server IP = 10.1.1.1
NTP Password = S3cr3t!!!
NTP MD5 Key = 1

Step 1
Create MD5 key 1 to authenticate with the NTP server.

R1(config)#ntp authentication-key 1 md5 S3cr3t!!!

Step 2
Enable authentication for NTP.

R1(config)#ntp authenticate

Step 3
Tell the router which key our router trusts (we have only one but may use more in the future). We do not want to accidentally synchronize the time with same 'fake' server.

R1(config)#ntp trusted-key 1

Step 4
Finally, configure IP address of the NTP server and specify which key to use for authentication.

R1(config)#ntp server 10.1.1.1 key 1

In case you did not use authentication (not recommended), you would be typing in the step 4 line without the 'key 1' argument.


Verification

---

Notice!
It is recommended that you initially set the clock manually before you allow NTP synchronization. Big time gap between your router and the NTP server clocks, will make synchronization extremely long process.

---

Step 1 - Check the status of NTP

R1#show ntp status

Clock is synchronized, stratum 5, reference is 10.1.1.1
nominal freq is 250.0000 Hz, actual freq is 250.0001 Hz, precision is 2**18
reference time is D04096A6.9715EE2B (14:03:18.590 UTC Sun Sep 19 2010)
clock offset is -7.9613 msec, root delay is 3.83 msec
root dispersion is 14.74 msec, peer dispersion is 6.74 msec
R1#

Step 2 (optional) - Check NTP association.

R1#show ntp association

      address         ref clock     st  when  poll reach  delay  offset    disp
*~10.1.1.1         127.127.7.1       4     9    64  377     5.6    4.22    13.4
 * master (synced), # master (unsynced), + selected, - candidate, ~ configured
R1#

Step 3 (optional) - Check NTP association details.

R1#show ntp association detail

10.1.1.1 configured, authenticated, our_master, sane, valid, stratum 4
ref ID 127.127.7.1, time D04097CC.0209500C (14:08:12.007 UTC Sun Sep 19 2010)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay 0.00 msec, root disp 0.03, reach 377, sync dist 10.239
delay 7.72 msec, offset 5.1799 msec, dispersion 6.35
precision 2**24, version 3
org time D04097E6.97A012CA (14:08:38.592 UTC Sun Sep 19 2010)
rcv time D04097E6.98D73524 (14:08:38.597 UTC Sun Sep 19 2010)
xmt time D04097E6.905799B4 (14:08:38.563 UTC Sun Sep 19 2010)
filtdelay =    33.02    7.72   15.73   22.32    5.65   27.62   23.62   15.66
filtoffset =   11.77    5.18   13.89   23.08    4.22    7.94   12.65    3.32
filterror =     0.02    0.99    1.97    2.94    3.92    4.90    5.87    6.85

Syslog Server Configuration


Syslog Server Information:
IP address = 192.168.1.2
Facility = Local7

R1 Configuration:

R1(config)#logging host 192.168.1.2 

R1(config)#logging facility local7

From now on, all system messages are going to be sent to syslog server with ip address 192.168.1.2.

In my next lesson, I'm going to introduce another layer 2 technology: Virtual LANs (VLANs)

Lesson 13 - Layer 2 Connectivity Troubleshooting Part 3

Lesson 13 - Layer 2 Connectivity Troubleshooting Part 3

This lesson is the last one in the series on how to troubleshoot connectivity issues at the layer 2 of OSI model. Bear in mind, that these do not involve layer 2 technologies such as VLANs or Spanning-Tree Protocol, since we have not talked about those yet.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 3
New installation as per Pic. 1 shows lack of connectivity between the two computers. Initial diagnostics performed revealed the following facts:

• Switches are connected via fibre optics cable and the ports show proper status (interface is up, line protocol is up).
• Computers have proper addresses and subnet masks assigned.
• Cables connecting computers with switches have been tested and proved to be working correctly.
• Computers reply to echo requests packets (firewalls disabled).

The technician who set up this new network calls you for help.

Pic. 1 - New design with connectivity problem

Icons designed by: Andrzej Szoblik - http://www.newo.pl

Dealing with this trouble ticket we are going to collect the tools we used in the previous lessons trying to resolve this issue.

Step 1

First let's try to 'divide and conquer' (concept mentioned in lesson 11) by sending ping packets from PC1 to PC2. Before we do that though, we need to purge the existing ARP cache on PC1 and PC2 to have a fresh information. We do that by opening Command Line Interface window and typing: arp -d host-address (linux), or arp -d (MS Windows).

Test results:

• The pings timed out. No reply from PC2.

Step 2

Since the ARP cache entries age out relatively quickly (depending on which operating system you use), we need to quickly check what they contain Alternatively we can send large series of ping packets.

Test results:

• PC1 does NOT contain expected mac-to-ip mapping. We expected to see
  192.168.1.2 at 00:1e:4f:b0:b2:fc. It is not there, though.
• PC2 DOES contain the the right mac-to-ip mapping. It shows the following:
  192.168.1.1 at 00:50:bf:9c:45:6a.

These are very interesting results, don't you think? Before we take the next step let's gather what we know so far.

Since the ping was initiated by PC1, it sent its ARP request broadcast message and that query must have been delivered to PC2. We can conclude that, based on the fact that we have cleared PC2's ARP cache in step 1, and it has the proper mapping now. PC2 did receive ARP request from PC1 and learned what MAC and IP address it uses. 

Step 3

We could omit that step, but we're curious if PC2 replies to the ARP request from PC1. We launch our 'wireshark' tool on PC2, ping again from PC1 to PC2 and capture all packets on PC2. What we discover in this packet trace is that PC2 has replied to ARP request with proper ARP reply unicast message back to PC1.

Step 4

Clearly, something between the computers (switches) does not work properly. It seems that we have some sort of unidirectional communication. What we need to establish is, where this unidirectional communication is taking place. We login to the SW1 and SW2 and issue the following commands ('x' here stands for switch number in Pic. 1):

SWx#show mac address-table interface f0/1

SWx#show mac address-table interface f0/24

Test results:

• SW1 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/1 interface. This is expected.
• SW1 does NOT learn MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/24 interface. This is unexpected. It should learn it from the ARP reply sent by PC2.
• SW2 learns source MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/1 interface. This is expected.
• SW2 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/24 interface. This is expected.

This way, we have discovered that SW1 has unidirectional link towards SW2 (SW2 sends frames towards SW1 but the latter does not seem to receive those). Probably, the fiber optics connection does not work properly (grease, dirt, a strand is broken etc.).

One more time, this lesson illustrates how useful the commands and knowledge described in the previous posts, can be in real life scenarios. 

In my next post, I will show you how to log system messages so they can be analyzed later. System messages are invaluable pieces information in the process of troubleshooting networking issues.

Lesson 12 - Layer 2 Connectivity Troubleshooting Part 2

Lesson 12 - Layer 2 Connectivity Troubleshooting Part 2

In the previous lesson I attempted to show you how to go about a performance problems we might experience due to the duplex mismatch. It was our first trouble ticket in the series on layer 2 problems.

In this lesson I am going to create another issue to show you how your current skills can be practically useful in diagnosing network connectivity problems.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 2
Some client computers have problems accessing FTP server located in the same network 192.168.1.0/24. This behavior is very intermittent.

We need to collect a bit more data related to this problem. A good starting point might be to take down the following pieces of information:


FTP Server
IP address  = 192.168.1.4
MAC address = 00:10:5a:d3:e4:e0

FTP Client1 (with no connectivity to FTP server)
IP address = 192.168.1.2
MAC address = 00:10:4f:b0:b2:fc

Here is one way of doing basic diagnostics.

Step 1
We want to make sure we have layer 1-3 connectivity between the Client1 and the server first (remember 'divide and conquer' method from the previous lesson?).

jr@mandala:~$ ping 192.168.1.4
PING 192.168.1.4 (192.168.1.4) 56(84) bytes of data.
64 bytes from 192.168.1.4: icmp_seq=1 ttl=64 time=0.372 ms
64 bytes from 192.168.1.4: icmp_seq=2 ttl=64 time=0.349 ms
64 bytes from 192.168.1.4: icmp_seq=3 ttl=64 time=0.371 ms
64 bytes from 192.168.1.4: icmp_seq=4 ttl=64 time=0.374 ms

It seems we have layer 3 connectivity working just fine.

Here's an interesting question for you. How do you know, you have received the replies from the FTP server in question (192.168.1.4)?

Now, you may have very confused expression on you face as in 'what do you mean?'. Have I not just gotten the reply from it?

Well, let me show you something to address this question.

Step 2
In order to be absolutely sure I got the reply from the FTP server (192.168.1.4) I need to verify my computer's ARP cache (Client1). Recall, that every time a host sends the packets, it encapsulates them in layer 2 frames (here Ethernet ones). In order to do that, the Client must have a valid destination MAC address (00:10:5a:d3:e4:e0) mapped to its IP address (192.168.1.4). If this mapping entry is not found in the APR cache, the Client1 will send ARP request to learn MAC address of 192.168.1.4. Knowing it let's check the ARP cache after we have sent ping packets. Here's what we find in (Pic. 1):

Pic. 1 - ARP Cache Entries

Pay attention to the highlighted entry. Is the MAC address mapped to our FTP server's IP address correct? 

NO! 

This MAC address does not belong to the server. FTP server's MAC address is: 00:10:5a:d3:e4:e0.

That test proved that getting echo replies to our echo packets (ping) must also be verified in the sender's ARP cache.

So, which device does this MAC address 00:10:0f:a3:3b:e6 belong to? And how on earth did it end up in our Client's ARP cache?

Well, in order to answer the second question, the reasons for this wrong mapping might be different. There could be the device with the duplicate address (same as the FTP server). Then, when the clients send ARP request for the MAC address of the server, this 'rouge' device (not the legitimate FTP server), also responds to the query. And if its answers arrives later than from the legitimate server, the override the legitimate MAC address mapping in the ARP cache. If some other clients get the reply for their ARP query from the 'rouge' device first and then from the FTP server, they create the mapping correctly. That would explain why some computers can still FTP to the server and others can't. Another reason for this wrong mac-to-ip mappings might be ARP poisoning attack in the network (eg. using Ettercap tool).

As for the answer to the first question, if this is not an ARP poisoning attack, you can use your knowledge from the lesson 9 about switching which helps you understand CAM table creation. Then, send uninterrupted ping towards the 'rouge' device and try to trace the mac addresses from switch to switch to find out where the device with duplicate IP address is located. MAC address table should help you find it relatively quickly.

There might be one more question I would like to clarify. Why the duplicate address was not detected by the 'rouge' device? Answer to that question may surprise you. The duplicate address detection uses so called 'gratuitous arp'. This is an unsolicited advertising of the MAC address upon computer startup. If some computers use the same MAC or IP, the newly computer cannot use its IP in the network. Unfortunately, some operating systems may not use this mechanism. They 'trust' what the operator does. If she or he wants this address, there will be no protest on their part.

I hope you are now beginning to see how much you already know!

In my next lesson you are going to analyze the third trouble ticket for layer 2 connectivity. This is going to be the last troubleshooting lesson. The upcoming lessons will describe other tools and interesting technologies such as VLANs, Spanning-Tree Protocol. Once we finish with foundations related to layer 2, we'll start discussing layer 3 technologies and concepts such as IP addressing, subnetting, routing etc.

Lesson 11 - Layer 2 Connectivity Troubleshooting Part 1

Lesson 11 - Layer 2 Connectivity Troubleshooting Part 1

The last two lessons I tried to explain the foundations related to layer 2 operation. I discussed very important switching process and CDP protocol which comes in handy at times. If you also have watched the video in the lesson 10, you got a glimpse of few commands explained in the theory earlier on.

In this lesson, we'll focus on practical application of the layer 1 and layer 2 concepts which could be helpful in troubleshooting networking issues.

Even though we're capable of creating lots of great things we're still human beings. And 'to err is human' adage is as conspicuous in the networking field as anywhere else. This means, that occasionally things won't work as expected. In such situations we need to be able to isolate and fix the problems quickly.

Reactive troubleshooting almost always uses the following work flow:

1. A problem is reported.
2. Data and facts are collected.
3. Data analysis must be performed.
4. Potential causes are eliminated.
5. Hypothesis is drawn.
6. Hypothesis is verified. 

In some situations the steps 3 and 4 could be omitted. But it depends on the nature of the problem reported, severity of issue, skills of the technician etc.

As we have yet to learn layer 2 technologies such as Vlans and Spannig-Tree Protocol which add complexity to troubleshooting process, let us, for now, focus on simple cases. This way we're going to build our troubleshooting skills step by step given the knowledge we posses.

Recall the process of moving data from one computer to another. Everything sent from the application layer goes down to the physical layer. That teaches us one important thing: if the layer 1 is not operational, nothing else will work!

So, we could start diagnosing the networking problems by checking the layer 1 connectivity first. Then, we could move on to the layer 2 and work our way up till we isolate the problem. This makes perfect sense. However, a lot of technicians use other method known as 'divide and conquer'.  In networking diagnostics that might be checking the layers in the middle. For instance using the 'ping' utility we can check the layer 1 through layer 3 status as a starting point. Of course, you already know how the 'ping' works, don't you? 'Ping' uses ICMP protocol which is a layer 3 messaging mechanism encapsulated directly in IP headers.

---

NOTICE
Ping utility is a layer 3 reachability checking mechanism. It sends a number of ICMP echo messages to the destination host. If the destination host receives echo messages, it sends ICMP echo reply messages back to the sender. Of course, assuming that there are no mechanisms implemented that would filter those messages along the path (like local firewall for instance), the sender should receive those replies and thus effectively checking layer 1 through layer 3 reachability. Thus, the path is verified in BOTH directions.

---

If you aim at the layer 3 reachability using 'ping' utility, you may get one of the two results:

1. You get the reply from the destination. This leads to a conclusion that all layers 1 through 3 are working properly. And the connectivity problems might be related to upper layers (layer 4 upward).
2. You do not get the reply from the destination. In that case, this step is not enough to determine the nature of the issue. In this case, you must perform some additional diagnostics.

Now, I am going to show you a few of such steps.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 1
Data transfer from PC1 toward PC2 is very slow (refer to Pic 1).

Pic. 1 - Simple Topology

 

Icons designed by: Andrzej Szoblik - http://www.newo.pl

As you see, without the topology diagram, it is much more challenging to do diagnostics. If we know the topology and the path between the source and destination devices in question, we can focus on all components that participate in the data transmission and isolate the issue.

First, let us determine what we know.


The transmission succeeds, but is slow. Verify that yourself. Do not assume it's true or properly tested by the person who reported that. Most users cannot properly describe the nature of the problems.

So, you have done the tests and the transfer proves to be slow indeed.


Some questions you might ask:

• Has this problem occurred recently?
• Is this a new client computer or destination server?
• Has any configuration/update/cable replacement/or any other changes been done on any devices in the path before the problem manifested itself?
• Are other computers experiencing the same problem, or is this individual case?
• etc.

Further data and fact collection may depend on the answers to those questions. For the argument's sake let's assume, that the server is a brand new computer installed the day before the problem occurred. All clients sending files to the server suffer from slow data transfer. When you connected the same client to the server directly, the transfer is very fast (computer-to-server through cross over cable).


Given those facts, we're going to take a quick look at the status of the interface where our server computer is connected. The command that is very useful to check both the layer 1 and layer 2 status is:

SW2#show int f0/2 

Pic. 2 - show interface command

This output deserves some explanation.

FastEthernet is up,

This is the status of the layer 1 connectivity (your cable seems to be attached right?).

line protocol is up

This is the status of the layer 2. It looks like the keepalive packets sent every 10 seconds are working back and forth (do not trust it entirely though; look at the trouble ticket 2 in the next lesson).

half-duplex, 100Mb/s

Duplex negotiated is HALF duplex. Most network adapters used in the computers use AUTO negotiation. This means, that the NIC (Network Interface Card aka network adapter), sends special signals to the port of the switch trying to negotiate FULL duplex and the highest speed supported. Unfortunately, some NIC manufacturers do not follow the specification regarding this signaling. This causes some "misunderstanding" between the port of the switch and the NIC. Switch typically drops down the duplex from FULL to HALF. That causes the switch port to enable the Carrier Sense, Multiple Access with Collision Detection mechanism (CSMA/CD) which is used on SHARED not dedicated connections (for details look at the lesson 8).

What we end up having is the NIC working in FULL duplex but the port of the switch runs HALF duplex. It 'thinks' it can either send or receive data but not do both at the same time. When server begins to 'push' data across the network, the port of the switch cannot send anything out towards the server. When it finally 'thinks' it can (medium free) and sends data towards the server, the latter begins to send data down towards the port as it is allowed in FULL duplex connections. The switch must stop immediately as the frames are experiencing collision. At least that is how the switch works under the circumstances. Then it waits till the carrier is free again (no data from the server down the port). This problem is known as: DUPLEX MISMATCH. That results in great number of collisions and late collisions recorded on the port of the switch like shown in the above output.

The solution to that problem could be the following actions:

1. Try to upgrade the NIC driver using your server/computer vendor's web site.
2. If the problem persists, you may try to hard code speed and duplex. You have to do this on both ends of the connections. This disables the AUTO NEGOTIATION feature (do not listen to people who say you should do this on one end of the connection).
3. Sometimes, though very unlikely in our situation, the cable can cause that sort of behavior. Replacing it to the one that is proven to be good, might help. Again, typically we would see some other layer 1 errors (CRC errors, carrier loss).
4. Replace the NIC on the server to the one that you are sure is working well.

Hard Coding Speed and Duplex

SW2#configure terminal

SW2(config)#interface fastethernet0/2

SW2(config-if)#speed 100

SW2(config-if)#duplex full

As for the computer, you have to refer to the manual of your operating system  how to set speed and duplex on the NIC manually. Perhaps google this. Google are the best!

In the next lessons, we will resolve two more connectivity issues given the skills we obtained in previous lessons.

Lesson 10 - Cisco Discovery Protocol

Lesson 10 - Cisco Discovery Protocol

In the previous lesson we have explored how switches build their mac address table (aka Content Addressable Memory). It is critical to understand those concepts in order to perform troubleshooting related to connectivity issues.

In this lesson we'll continue studying layer 2 technologies. Today's theme is Cisco Discovery Protocol. This protocol comes in handy in many situations (trust boundary for Cisco IP Phones, auto qos and others).

What is Cisco Discovery Protocol?
CDP is Cisco proprietary layer 2 protocol. It is enabled by default on majority of Cisco devices including IP phones. It can work on any connections supporting SNAP (such as LANs, but also ATM and Frame-Relay). The only time you'll see the CDP turned off by default, is when you use frame-relay configured interfaces.

What does CDP do?

Every Cisco device using this protocol, reports information about itself by advertising special packets out of its all active interfaces. The important pieces of information it advertise include its:

• Hostname
• Platform
• Ports where CDP packets are advertised
• IOS version
• IP address

CDP can help administrator discover Cisco devices connected and create a  topology diagram or prepare an inventory of the gear used. It can also be an additional tool in troubleshooting problems in the network. Working as a support technician, I found it useful numerous times.

Let's take a quick tour through the CLI (command line interface) and see what major commands CDP allows us to use and what they show.

I'm connected to my Cisco switch SW1 and in the privileged mode type in the following command:

Pic. 1

Using our best friend '?' we can see the CDP options. The last line '' stands for: 'carriage return'. A good, old-fashioned terminal lingo for 'press enter'. Let's try this first.

Pic. 2

Based on the output (Pic. 2) we see that CDP version 2 is enabled, the information packets (though technically should be called frames) are sent every 60 seconds. We also learn, that SW1 will keep CDP information it receives from its Cisco neighboring devices for 180 seconds (holdtime). Let's look at another CDP command:

Pic. 3

In the Pic. 3 we can see the traffic statistics such as CDP packets sent, received, any CDP encapsulation problems etc.

The below command (output in Pic. 4) will tell you which interfaces CDP is running on.

SW1#show cdp interface  

 
You can disable CDP on a specific interface or group of interfaces. For instance, if you do not want to run CDP on Fas0/1 interface, you could use the following command:

SW1#configure terminal

SW1(config)#interface Fas0/1

SW1(config-if)#no cdp enable

If you want to disable CDP on a group of interfaces you can use 'interface range' command. For instance, disabling CDP on Fas0/1, Fas0/2, Fas0/5 and Fas0/8 would look like this:

SW1#configure terminal

SW1(config)#interface range fas0/1 - 2 , fas0/5 , fas0/8

SW1(config-if-range)#no cdp enable

---

NOTICE

If you use the 'interface range' command, the consecutive ports can be specified with '-' but make sure your use 'space' before and after '-' (fas0/1 - 2). The same applies to non-consecutive ports (fas0/5 , fas0/8). There is 'space' before and after the comma character ','.

---

As you will see later, CDP discloses some vital information (e.g. IOS version), so for the security reasons you may decide to turn off CDP altogether. Be careful before you do that though, as some application may rely on this protocol. Disabling CDP can cause cascading problems in your network. The command which disables CDP completely (on all interfaces) is:

SW1#configure terminal

SW1(config)#no cdp run

Pic. 4

Now, let's see what neighboring devices SW1 discovered by listening to their CDP packets (Pic. 5).

Pic. 5 

Dissecting the Pic. 5 output we learn the following:

• SW1 received CDP packets from the device named 'R1' (hostname).
• This CDP packet was sent from R1's Fas0/0 interface (the last column 'Port ID').
• SW1 received this CDP information packet on its Fas0/1 local interface ('Local Intrfce').
• This leads us to a conclusion that R1's Fas0/0 interface is directly connected to SW1's Fas0/1 interface.
• R1 neighbor is a router as the capability list shows 'R S I' (R=router, S=switching capability, I=IGMP support).
• R1 is 2611XM platform.

That's not all by any means. There is another command we can use to obtain more information about R1. Click at the Pic. 6

Pic. 6

It shows you information about IOS version running on R1, as well as its IP address 192.168.10.254 configured on Fas0/0 interface. Now you understand why you might consider turning off CDP on some interfaces. You do not want to show such details to a third party company (like your service provider), that connect to your devices.

Instead of using 'show cdp entry R1', you can also use the following command that displays detailed (similar to Pic. 6) output about ALL discovered neighbors :

SW1#show cdp neighbor detail

In lesson 11, you will learn the commands related to switch mac-address-table which was covered in theory only (lesson 9). Also, we'll hone all our skills we have obtained so far. Things will begin to fall into place. At least that's my hope.

A practical application of the two last lessons below.

Lesson 9 - Bridging/Switching Learning Process

Lesson 9 - Bridging/Switching Learning Process

In the previous lesson we looked at the Ethernet and a hub operation. We classified hub to be a layer 1 device as it does not understand any headers used by upper layers of our networking model. It simply forwards the bits it receives out all remaining ports. Even though, they do provide basic connectivity to our hosts, they also reveal a lot of weaknesses. For details look at the lesson 8.

A more intelligent and robust device that can replace a hub is a layer 2 switch. You've already learned how to navigate in IOS, and provide the switch with a simple configuration. With this lesson we begin a more serious exploration of layer 2 through layer 4 functions starting with Layer 2. This is good enough excuse to brush upon encapsulation/de-encapsulation process, and the structure of the headers.

Enapsulation Process Re-Visited
This is a quick review of encapsulation process I would present to my nine year old son (if he wanted to learn more).

1. Assuming that TCP session is already complete, the application is forming the request (data) which is sent down to the layer 4. Layer 4 process places source and destination port numbers in the header (Pic. 1).

Pic. 1 - Application sends data to the Transport layer.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

2. Transport layer sends the segment down to the layer 3 for processing. This payload ends up encapsulated in an IP header with the source and destination IP addresses added in the header (Pic. 2).

Pic. 2 - Transport layer sends the segment to the Internet layer.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

3. The layer 3 sends the packet down to the layer 2 (Data-Link) for processing. Layer 2 adds its own header (here the Ethernet header) with the source and the destination MAC addresses (Pic. 3).

Pic. 3 - Internet layer sends packet down to Data-Link layer.

Then, this whole 'thing' is converted into bits and put onto the wire.

Now, we can see what happens when the bits are sent to the port of the switch to reach the destination computer. Let's bring back the Ethernet header to see what we find in it (Pic. 4).

Pic. 4 - Ethernet Header

Icons designed by: Andrzej Szoblik - http://www.newo.pl

In this header there are two pieces of information that switches use to build their mac-address-table (CAM) and make forwarding decisions. Those are: source and destination MAC addresses. And here is how it works.

Initially, the mac-address-table (aka CAM) is completely empty (Pic. 5).

Pic. 5 - Content Addressable Memory (CAM) initially is empty.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

Sooner or later, some computers begin to transmit something across the network. In my example, the PC1 (source MAC address: 1111.1111.1111) begins transmission to PC3 (destination MAC address: 3333.3333.3333). Below is the sequence of events.

1. PC1 with the source MAC address of 1111.1111.1111 sends the Ethernet frame to the destination MAC address of 3333.3333.3333 (PC3).

2. SW1 receives the frame on port F0/1. It 'reads' the source MAC address and maps it to the receiving port in its CAM (like shown in the Pic. 6).

---

NOTICE
Switches learn MAC addresses by reading the SOURCE MAC from the INCOMING frames (going towards the switch) only. They do not learn anything when the frame leaves the switch.

---

Pic. 6 - SW1 learns dynamically 1111.1111.1111 by reading the source MAC address from the incoming frame.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

3. SW1 then, reads the destination MAC address and will try to find the outbound port for that destination. Since 3333.3333.3333 has not yet been mapped to any port, the switch will perform flooding (Pic. 7).

Flooding is the act of sending a frame out of all active ports except the port where the frame arrived.

There are few reasons why switch decides to flood a frame:

• Switch does not know where the destination host is = unknown MAC address.
• The destination MAC address is broadcast: FFFF.FFFF.FFFF.
• The destination MAC address is multicast.

 Pic. 7 - SW1 Floods unknown destination MAC address 3333.3333.3333.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

4. Computer with MAC address other than 3333.3333.3333 drop the incoming frames. PC3 is the destination of the frame so it further processes it (de-encapsulation). Meanwhile, the SW2 learns the source MAC address on the receiving port F0/12 and maps it in its CAM. Since, as of right now, it does not know where 3333.3333.3333 resides, it also floods the frame (Pic. 8).

Pic. 8 - SW2 is learning 1111.1111.1111 on F0/12 and flooding the frame.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

5. PC3 (3333.3333.3333) responds the PC1 (1111.1111.1111). SW2 receives the frame sourced with 3333.3333.3333 on port F0/2. It puts this in its mac-address-table (CAM), mapping it to the inbound port F0/2. Next, it will read the destination MAC address in the frame (1111.1111.1111) and consults it with its CAM entries. It finds the outbound port F0/12 where this address has already been learned from the incoming frame. This time, the transmission is not flooded as SW2 has the mapping in the table. SW1 receives the frame on its port F0/12. It reads the source MAC address (3333.3333.3333) and maps it to the receiving port F0/12 in its CAM table. Then it looks at the destination MAC address (1111.1111.1111) and finds the outbound port which is F0/1. NO FLOODING THIS TIME on either switch! All illustrated in the Pic. 9.

Pic. 9 - Port-to-Port transmission in the reply as all MAC addresses in question have already been learned.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

Since all computers 'speak' now and then, the switches will learn their MAC addresses from the incoming frames by reading the source MAC address field in the headers. They will populate the CAM and like depicted in Pic. 10.

---

NOTICE
All entries in the CAM table (mac-address-table) have a default aging timer which is 300 seconds (5 minutes). If the host do not refresh those entries by sending frames toward the port, the entries will be removed after 5 minutes. Of course, if the host transmits the frames again the incoming ports will map them again.

---

Pic. 10 - Content Addressable Memory (CAM) full.

Icons designed by: Andrzej Szoblik - http://www.newo.pl

In the next lesson we will look at Cisco Discovery Protocol.In lesson 11, we're going to put all the pieces together to show you the practical applications of what you have learned.

Lesson 8 - Ethernet and Hub Operations

Lesson 8 - Ethernet and Hub Operations

After my last conversation with my son about building the home network I promised I'd explain to him how a hub worked. And now, he's showing at the door... . So, like in the previous lessons I'll focus on the most important aspects regarding CCNA level.

In order to explain how hub works, let me talk about the Ethernet technology first. Ethernet sets the rules regarding how data coming down from the upper layers (Layers 5 through 3) should be placed on the media. It defines such transmission aspects like the maximum speed, cable type and length, connector types, how frame should be structured etc. In other words, Ethernet defines both the layer 2 and the layer 1 of our OSI model. The reason the layer 3 cannot send data directly onto the media is that there are different, disparate media types and technological limitations related to them. For instance, there will be different way of transmitting data between Ethernet switches and two computers using modems to communicate. That is why, we need the layer 2 as a "translator" so the layer 3 protocol can use different types of media without re-writing its functions. Besides Ethernet, there were other protocols in use such as Token-Ring, but today Ethernet seems to be the most commonly used solutions in local area networks (LANs).

Initially, Ethernet used coaxial cable that was terminated on both ends, and the computers were attached to the cable using so called 'T-connectors'. They shared the media, and as a result of that, they shared the available bandwidth. Today, coaxial cable is not used that often. Instead, we use UTP cable (details in the Lesson 7). But the logic of operation in both cases is quite similar.

Ethernet, using a coaxial or a UTP cable (with the hub as concentrator connecting hosts), is based on the idea that only ONE computer can transmit something at a time. Other computers will ALL receive this signal and have to wait for their turn if they have something they want to transmit (Pic 1).

Pic1. Ethernet transmissions in bus (coaxial cable) and star topology (UTP cable + hub).

Switch/router/computer icons designed by: Andrzej Szoblik - http://www.newo.pl

There is no arbiter to decide who can talk in a given time. Instead, the devices have a mechanism allowing them to 'sense' the wire to see if there is any transmission in progress. If not, they can use the media. However, if a host is already transmitting and, some other hosts "thinks" the medium is not used and starts transmitting as well, this phenomenon is called a collision. In that case, both senders will stop their transmission and wait a specified by the algorithm time before they try again. How long they have to wait is determined by so called back off algorithm. Needles to say, the more collisions occur the longer the hosts have to wait before they retransmit corrupted bits. The mechanism of checking the wire before transmission, detecting and reacting to collisions is known as CSMA/CD (carrier sense, multiaccess with collision detection).

• Carrier Sense - "listen" to the wire to see if you can transmit.
• Multiple Access - multiple hosts share the bandwidth and only one computer can send or receive bits but not both at the same time (half-duplex).
• Collision Detection - in case other computer(s) started transmitting while other transmission is in progress, detect it and use back off algorithm.

The major downsides of this approach are the security (everyone "hears" everyone else), and the fact that only one computer can send data across the network at a time. What's worse, it can either transmit or receive bits (half duplex). The more computers you throw in the network, the harder is for them to get a chance to send something. Also, the chances for collisions are higher. In fact, collisions are imminent and make the system perform poorly.

HUB
A hub is a simple type of concentrator used to connect multiple computers, servers, printers etc., together. It is considered a layer 1 device due to the fact that it does not understand any headers. All it recognizes are electrical signals representing 1s and 0s. Hub operation is very simple. If signals appears on one port (interface), it is replicated on to all other active interfaces like depicted in picture 1 (right hand side).

Since hubs follow Ethernet rules in which there can be only one sender at a time, and because they transmit bits out of all remaining ports, they maintain one single collision domain.  

Pic. 2 - Hubs maintain one single collision domain.

Switch/router/computer icons designed by: Andrzej Szoblik - http://www.newo.pl

A collision domain could be defined as any group of devices which can receive transmissions from other hosts. So, hubs are not seriously considered to be a proper solution in medium/large networks. The problem with the collisions, typical in hub-based networks, has been solved by introducing another type of concentrators called bridge (old, software-based device) and switch (modern, hardware-based device) equipment used in the layer 2 of OSI mode.

Discussion on bridges and switches though, are the topic I reserve for the next lesson.

Lesson 7 - Building a Home Network

 Lesson 7 - Building a Home Network

"Dad! What are you doing?" - My son Mattie is looking over my shoulder while I'm connecting three old computers to the hub I'v salvaged from a certain death. "I'm building a small, home network." - I reply surprised at his sudden interest. "What do you need to build a small network?" - There's this little sparkle in his eyes only kids his age have. "And how do you do it?". There's nothing I can do but explain to him how it's done!

We will need few components to build this network. First of all, we'll need computers, running some operating system that can talk across the network (like Mac OS, Windows, Linux).

Pic. 1 - Computers.

Computers are already equipped with NICs (Network Interface Cards) by the manufacturer.

"What does the NIC look like dad?" - He interrupts. Nine years old kids can be very tenacious, almost obsessive. They are not easily dismissed. They won't stop unless they're fully satisfied with the answer. So, I'm taking one NIC out of the computer and explain that it is plugged in to the PCI slot in order to work. "PCI slot is connected to the motherboard of the computer and the operating system uses a special piece of software to talk to the NIC. It's called a driver. The driver translates between an operating system and the NICs hardware. The NIC sends bits down to the wire and knows what to do when bits come back from the network". Bits are small pieces of information (1s and 0s) which in software make data like music, pictures, text documents etc.

Pic. 2 - NIC (Network Interface Card).

"Okay." - Mattie says, "But what else will you need?

"I will need Unshielded Twisted Pair cables (UTP). One, for each computer. The cables (sometimes referred to as Ethernet cables), use RJ-45 connectors that the most commonly used nowadays." - I reply. "Here's the cable without the connector:

Pic. 3 - UTP cable without RJ-45 termination

"Mattie's looking at the cable and I know what's going to happen next. "Dad, why does it have so many wires and why are those twisted like that?"

One pair of the wires (two wires) are used to transmit data. Another pair, is used to receive data from the network. Other wires can be used to carry the power to some types of the devices (PoE devices) or to accomplish faster speeds (1Gbps etc.). They are twisted like that on purpose. The guy who invented that concept was Graham Bell. He invented it for the telephony purposes and patented that in 1881. He discovered that twisting wires (conductors) minimized or canceled Electromagnetic Interference (EMI) from external sources and, so called, cross talk from the neighboring wires.

The cable must be terminated at both ends with RJ-45 type of connector, like the one depicted below:

Pic. 4 - RJ-45 Connector.

Below is the cable with the connectors.

Pic. 5 - UTP cable with RJ-45 connectors.

The UTP cables can have different category numbers (CAT 1-6). The higher the category number is, the better quality of the cable, the faster, and better transmissions are going to be. Also, the UTP cables can be terminated in two different ways like explained below.

Straight-Through Cable
In straight through cable the transmitting pair of wires are 1 and 2, the receiving pair of are wires 3 and 6. There are two major standards (ways) of using the colored wires, but important thing is, that the colors on the both ends of the cable are terminated identically. Please, look at the picture below.

Pic. 6 - Straight Through Cable.

Cross-over Cable
In the cross-over cable, the position of the wires is changed such that the sending pair is terminated at the receiving pair on the other side of the cable. It is illustrated below.

Pic. 7 - Cross-over Cable

My son Mattie's holding both types of cable, looking at them and I know I cannot dismiss him with that explanation. So, I continue.

If you connected two computers together and the NICs are wired identically, you would connect the sending pair (pins 1 and 2) to the sending pair on the other end. This obviously would not work. You must connect sending pair on one end (pins 1 and 2) to the receiving pair (pins 3 and 6) on the other end. For instance, if you connect the following devices together, you'll need cross-over cable:

• computer-to-computer
• switch-to-switch
• hub-to-hub
• computer-to-router (directly)

---

NOTICE!
Modern NICs can 'sense' the type of cable and adjust the operation regardless of the cable used. But this is not always the case.

---

Devices such as hubs and switches, are designed such way they can use straight-through cables. The cross between the transmitting and receiving pairs is done in their port controllers. So, the following device connections will use straight-through cable:

• computer-to-hub
• computer-to-switch
• router-to-hub
• router-to-switch

Now, all we have to do is to connect the cables to the the hub, configure IP addresses on the NICs and voila! They can talk to one another.

Ethernet Hub
It is a simple device that allows to connect a few computers together. Look at the typical, cheap hub you can buy for home purposes:

Pic. 8 - An Ethernet Hub.

"How does the hub work then?" - I can tell my son got some interest by now.

"Well, that is the topic of our next lesson." - I say smiling.
;)

Lesson 13 - Layer 2 Connectivity Troubleshooting Part 3

This lesson is the last one in the series on how to troubleshoot connectivity issues at the layer 2 of OSI model. Bear in mind, that these do not involve layer 2 technologies such as VLANs or Spanning-Tree Protocol, since we have not talked about those yet.

---

NOTICE
The steps presented in this lesson are not the ALL possible diagnostics you can do. And they do not have to be done in this specific order. I am merely listing some logical steps which might be useful in order to 'nail down' the root cause of the problem.

---

Trouble Ticket 3
New installation as per Pic. 1 shows lack of connectivity between the two computers. Initial diagnostics performed revealed the following facts:

• Switches are connected via fibre optics cable and the ports show proper status (interface is up, line protocol is up).
• Computers have proper addresses and subnet masks assigned.
• Cables connecting computers with switches have been tested and proved to be working correctly.
• Computers reply to echo requests packets (firewalls disabled).

The technician who set up this new network calls you for help.

Pic. 1 - New design with connectivity problem

Icons designed by: Andrzej Szoblik - http://www.newo.pl

Dealing with this trouble ticket we are going to collect the tools we used in the previous lessons trying to resolve this issue.

Step 1

First let's try to 'divide and conquer' (concept mentioned in lesson 11) by sending ping packets from PC1 to PC2. Before we do that though, we need to purge the existing ARP cache on PC1 and PC2 to have a fresh information. We do that by opening Command Line Interface window and typing: arp -d host-address (linux), or arp -d (MS Windows).

Test results:

• The pings timed out. No reply from PC2.

Step 2

Since the ARP cache entries age out relatively quickly (depending on which operating system you use), we need to quickly check what they contain Alternatively we can send large series of ping packets.

Test results:

• PC1 does NOT contain expected mac-to-ip mapping. We expected to see
  192.168.1.2 at 00:1e:4f:b0:b2:fc. It is not there, though.
• PC2 DOES contain the the right mac-to-ip mapping. It shows the following:
  192.168.1.1 at 00:50:bf:9c:45:6a.

These are very interesting results, don't you think? Before we take the next step let's gather what we know so far.

Since the ping was initiated by PC1, it sent its ARP request broadcast message and that query must have been delivered to PC2. We can conclude that, based on the fact that we have cleared PC2's ARP cache in step 1, and it has the proper mapping now. PC2 did receive ARP request from PC1 and learned what MAC and IP address it uses. 

Step 3

We could omit that step, but we're curious if PC2 replies to the ARP request from PC1. We launch our 'wireshark' tool on PC2, ping again from PC1 to PC2 and capture all packets on PC2. What we discover in this packet trace is that PC2 has replied to ARP request with proper ARP reply unicast message back to PC1.

Step 4

Clearly, something between the computers (switches) does not work properly. It seems that we have some sort of unidirectional communication. What we need to establish is, where this unidirectional communication is taking place. We login to the SW1 and SW2 and issue the following commands ('x' here stands for switch number in Pic. 1):

SWx#show mac address-table interface f0/1

SWx#show mac address-table interface f0/24

Test results:

• SW1 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/1 interface. This is expected.
• SW1 does NOT learn MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/24 interface. This is unexpected. It should learn it from the ARP reply sent by PC2.
• SW2 learns source MAC address of PC2 (00:1e:4f:b0:b2:fc) on its Fa0/1 interface. This is expected.
• SW2 learns source MAC address of PC1 (00:50:bf:9c:45:6a) on its Fa0/24 interface. This is expected.

This way, we have discovered that SW1 has unidirectional link towards SW2 (SW2 sends frames towards SW1 but the latter does not seem to receive those). Probably, the fiber optics connection does not work properly (grease, dirt, a strand is broken etc.).

One more time, this lesson illustrates how useful the commands and knowledge described in the previous posts, can be in real life scenarios. 

In my next post, I will show you how to log system messages so they can be analyzed later. System messages are invaluable pieces information in the process of troubleshooting networking issues.