OSPF Troubleshooting

Scenario:

After months of hard work, studying and doing labs you have the feeling you truly know how routing protocols work. Your boss decides to upgrade your salary and your title from Trainee to Junior network engineer and you will be responsible for daily maintenance for the network. Your senior network colleague (who is a true Guru but also very chaotic) is away for the the day and you are unable to reach him. Some users called you and complained about unreachable servers and slow links. It’s 8:00am and they need to have access to the network by 12:00am. Time to show your boss who the new Guru is and perhaps get another salary upgrade!

Goal:

  • All IP addresses have been preconfigured for you as following:
    192.168.XY.X /24 where X = router1 and Y = router2.
    for example: 192.168.58.5 between router5 and router8.
  • Every router has 1 or 2 loopback interfaces as following:
    Loopback0: x.x.x.x /24 for example: 1.1.1.1 for router1.
    Loopback1: xx.xx.xx.xx /24 for example: 11.11.11.11 for router1.
  • OSPF is preconfigured with the areas as specified in the topology picture.
  • Do not use show run! (this will spoil the fun πŸ™‚ use the appropiate ‘show’ and ‘debug’ commands. This will teach you the skills needed to become a true troubleshooting master.
  • The network engineer responsible for the backbone area has some problems. R1 was replaced and the most recent config was unavailable so he used an older config. The OSPF adjancency between R1, R2 and R3 is not working now. You don’t have access to R2 and R3.
  • After fixing the problem on R1 you notice traffic from R1 towards R2 uses the fastethernet link. All traffic should be sent through the serial interface, when the frame-relay link fails it should switch to the fastethernet.
  • Area 51 seems to have problems, you can’t get any of the OSPF adjancencies up…make sure Area 51 has adjancencies between R1, R4 and R5.
  • R4 is the most powerful router in Area 51, make sure it will become the DR.
  • There used to be a link between R1 and R4 which failed…replacing the cable is impossible so you need to find another solution to fix the separated backbone.
  • With Area 1 and 51 up and running everything should be ok. however when you look in the routing table of R1 you still don’t see all of the networks. For example the 4.4.4.0 network is not available.
  • RIP networks are configured to be redistributed into OSPF, however you don’t see and of the 172.16.X.X networks that are behind R7.
  • The security officer made some changes in Area 56, the OSPF neighborship between R5 and R6 is not working anymore.
  • For some reason R6 doesn’t see any of the networks from the backbone area, see if you can find and solve the issue.
  • R6 is configured to redistribute RIP into OSPF, however you don’t see any of the networks coming from R7 in any of the OSPF routers.
  • The network engineer of R6 has configured summarization of the RIP routes. however you still see 4 different 172.16.0.0 /24 networks.Configure the correct summary and make sure you don’t advertise networks that you don’t have.
  • You notice that the 6.6.6.0 network is advertised in OSPF as a /24, make sure it’s advertised as a /32 without changing the subnet mask.
  • The Redistributed routes in OSPF have the same cost, no matter which router you look at. Change this on R6 so the cost increases throughout the network.

It took me 1000s of hours reading books and doing labs, making mistakes over and over again until I mastered all the troubleshooting issues for CCNP.

Would you like to be a master of troubleshooting too? In a short time without having to read 900 page books or google the answers to your questions and browsing through forums?

I collected all my knowledge and created a single ebook for you that has everything you need to know to become a master of troubleshooting.

You will learn all the secrets about OSPF troubleshooting and more.

Does this sound interesting to you? Take a look here and let me show you how to Master CCNP TSHOOT

IOS:

c3640-jk9s-mz.124-16.bin

Topology:

You need to register to be able to download the GNS3 Topology File. (Registration is Free!)

Video Solution:

Configuration Files

You need to register to download the GNS3 topology file. (Registration is free!)

Once you are logged in you will find the configuration files right here.

Opt In Image
Do you want your CCNA or CCNP Certificate?

The How to Master series helps you to understand complex topics like spanning-tree, VLANs, trunks, OSPF, EIGRP, BGP and more.

Written by RenΓ© Molenaar - CCIE #41726

You May Also Like

About the Author: Rene Molenaar

RenΓ© - CCIE #41726 is the creator of GNS3Vault.com where he shares CCNA, CCNP and CCIE R&S labs. He also blogs about networking on http://networklessons.com

79 Comments

  1. Am I the only one having trouble with this lab? In the beginning of the lab it is stated:

    # The network engineer responsible for the backbone area has some problems. R1 was replaced and the most recent config was unavailable so he used an older config. The OSPF adjancency between R1, R2 and R3 is not working now. You don’t have access to R2 and R3.
    # After fixing the problem on R1 you notice traffic from R1 towards R2 uses the fastethernet link. All traffic should be sent through the serial interface, when the frame-relay link fails it should switch to the fastethernet.

    I tried and tried and could not even get ospf routes to show up int he routing table. After trying all I knew to do I recreated Area 0 on my own and worked through the config. I was able to get everything to come up and by costing the traffic away from F0/0 on router 1 was able to get the preferred path to Lo0 on R2 to be the serial interface. Not sure if it is a lab problem or a GNS3 problem, or my problem, but again when I recreate the scenario on my own it works fine.

    My configs are:
    Router1
    interface Loopback0
    ip address 1.1.1.1 255.255.255.0
    !
    interface FastEthernet0/0
    bandwidth 100000
    ip address 192.168.12.1 255.255.255.0
    ip ospf cost 5000
    duplex auto
    speed auto
    !
    interface Serial1/0
    no ip address
    encapsulation frame-relay
    serial restart-delay 0
    no frame-relay inverse-arp
    !
    interface Serial1/0.123 multipoint
    ip address 192.168.123.1 255.255.255.0
    ip ospf network broadcast
    frame-relay map ip 192.168.123.2 102 broadcast
    frame-relay map ip 192.168.123.3 103 broadcast
    !
    interface Serial1/1
    no ip address
    shutdown
    serial restart-delay 0
    !
    interface Serial1/2
    no ip address
    shutdown
    serial restart-delay 0
    !
    interface Serial1/3
    no ip address
    shutdown
    serial restart-delay 0
    !
    router ospf 1
    router-id 1.1.1.1
    log-adjacency-changes
    network 1.1.1.0 0.0.0.255 area 0
    network 192.168.12.0 0.0.0.255 area 0
    network 192.168.123.0 0.0.0.255 area 0

    Router 2
    interface Loopback0
    ip address 2.2.2.2 255.255.255.0
    !
    interface FastEthernet0/0
    ip address 192.168.12.2 255.255.255.0
    duplex auto
    speed auto
    !
    interface Serial1/0
    no ip address
    encapsulation frame-relay
    serial restart-delay 0
    no frame-relay inverse-arp
    !
    interface Serial1/0.123 point-to-point
    ip address 192.168.123.2 255.255.255.0
    ip ospf network broadcast
    frame-relay interface-dlci 201
    !
    interface Serial1/1
    no ip address
    shutdown
    serial restart-delay 0
    !
    interface Serial1/2
    no ip address
    shutdown
    serial restart-delay 0
    !
    interface Serial1/3
    no ip address
    shutdown
    serial restart-delay 0
    !
    router ospf 1
    router-id 2.2.2.2
    log-adjacency-changes
    network 2.2.2.0 0.0.0.255 area 0
    network 192.168.12.0 0.0.0.255 area 0
    network 192.168.123.0 0.0.0.255 area 0

    Router 3
    interface Loopback0
    ip address 3.3.3.3 255.255.255.0
    !
    interface FastEthernet0/0
    no ip address
    shutdown
    duplex auto
    speed auto
    !
    interface Serial1/0
    no ip address
    encapsulation frame-relay
    serial restart-delay 0
    no frame-relay inverse-arp
    !
    interface Serial1/0.123 point-to-point
    ip address 192.168.123.3 255.255.255.0
    ip ospf network broadcast
    frame-relay interface-dlci 301
    !
    interface Serial1/1
    no ip address
    shutdown
    serial restart-delay 0
    !
    interface Serial1/2
    no ip address
    shutdown
    serial restart-delay 0
    !
    interface Serial1/3
    no ip address
    shutdown
    serial restart-delay 0
    !
    router ospf 1
    router-id 3.3.3.3
    log-adjacency-changes
    network 3.3.3.0 0.0.0.255 area 0
    network 192.168.123.0 0.0.0.255 area 0

  2. So what went wrong Andy? Something you missed?

    I like this lab, it’ll help you truly understand OSPF…troubleshooting is nothing more but understanding the technology completely.

  3. indeed its a great lab took 6 hours to troubleshoot it. but i am stuck on the last task..

    * The Redistributed routes in OSPF have the same cost, no matter which router you look at. Change this on R5 so the cost increases throughout the network. You are not allowed to make changes on R6.

    i don’t know how to fulfill this task. >:( help would be really appreciated. πŸ™‚

    Thanks to the Owner of this lab it was great troubleshooting this lab.:)

  4. Hi Bala,

    I’m glad you liked it…I created this lab sometime ago for a CCNP TSHOOT course i was teaching.

    Let me give you a little hint for the redistributed routes….Google a bit for “OSPF E1” or “OSPF E2” and you’ll find the answer.

    Good luck!

    Rene

  5. Rene,

    Thanks for an excellent lab i do know about E1 & E2 types the problem is that tha task said u need to make change on R5 don’t make any changes on R6 ( Redistribution is taking place on R6) so how can i change routes that are coming as E2 and i have to convert it E1 on R5. i don’t know about this. If i have to make E1 on R6 that is a piece of cake for me but on R5 I don’t know what to do there.>:(>:(>:(

    Please help.

  6. Hi Bala,

    I think i took the wrong router number lol…Normally you would do this on R6. Actually I don’t think it’s possible to do it on R5 since it’s already in the routing table.

    Perhaps (this is just theoretical) you could redistribute the routes into another OSPF process and redistribute it again…or something like that. Don’t think it’s possible though…or perhaps make it even more funky and use different VRF’s or something.

    I’m going to test this, as i’m writing this i’m getting another idea for a funky lab πŸ˜‰

    Rene

  7. REne

    hahahahahah ;D

    i tried to redistribute ospf into ospf to do the same thing the error message was can’t redistribute ospf into ospf process id was same. yeah i think different process id may be redistributed into diff ospf instance.

    It’s great that u r getting another idea for another lab that will be fun again to trouble shoot.

    Thanx for all the help.

  8. glad you like it πŸ˜‰ I”m thinking of some VRF into VRF lab…not sure. I keep on thinking if it would be possible to change the external type on R5 instead of R6…

    Will be sometime when i’m home, not sitting in a hotel room with slow internet =)

  9. Well frankly speaking don’t have a idea what is a VRF may be i’ll read them in a future video. i am self studying CCNP ROUTE from CBT NUGGETS & planning to read the book after i finish this video series.

    looking forward to see more great labs from you. keep up the good work. πŸ™‚

  10. If you want to read a really good book…get “Routing TCP Vol1 and Vol2” from Jeff Doyle. Those two are the best (and most fun) books i know.

    VRF’s are for routers what a VLAN is for a switch…(well kind of..)

    a VRF is a virtual routing table, which means you can have multiple routing tables on a single router and they are seperate from each other. An ISP could use this to connect to multiple customers, exchange routing information and make sure information from Customer A does not mix with Customer B. We use this for MPLS as well.

    I got a small VRF lab in the MPLS section, check it out πŸ™‚

    PS – Thanks for the comment, that’s what keeps me going!

  11. Thanks for the book i will definitely going to find that book i heard that book along time ago but didn’t remember until u remind me. I was told this book is very awesome to learn routing.

    Well keep up the good work. Thanks for all the great labs.

  12. Hi folks,
    i m getting an error while loading this lab can you check & feedback..
    Below are the errors.
    Local configuration OSPF Troubleshooting_configs/R7.cfg cannot be found for router ROUTER R7, use configuration C:DOCUME~1ADMINI~1LOCALS~1Temp2Temporary Directory 1 for OSPFTroubleshooting.zipOSPF Troubleshooting_configsR7.cfg instead
    *** Warning: Connecting FR1 port 1 to R1 s0/0 resulted in:
    ‘FRSW’ object has no attribute ‘set_port’
    *** Warning: Connecting FR1 port 2 to R2 s0/0 resulted in:
    ‘FRSW’ object has no attribute ‘set_port’
    *** Warning: Connecting FR1 port 3 to R3 s0/0 resulted in:
    ‘FRSW’ object has no attribute ‘set_port’
    *** Warning: Connecting SW1 port 1 to access 1 R4 f0/0 resulted in:
    invalid Ethernet switchport config
    *** Warning: Connecting SW1 port 2 to access 1 R1 f2/0 resulted in:
    invalid Ethernet switchport config
    *** Warning: Connecting SW1 port 3 to access 1 R5 f0/0 resulted in:
    invalid Ethernet switchport config

    *** Error: errors during loading of the topology file, please correct them

  13. As the message stated that it can’t find config file for R7 router. look again may be the error is on your end.

  14. Hi,

    I’m stucked in the last goal. How can I change the external OSPF without changing R6 config?
    I know about external type 1/2 OSPF, but it would work only when redistributing RIP to OSPF in R6…

    thanks in advance, Bruno

  15. I passed ROUTE with help from your lab and prepared for job interviews.. I used this lab and the eigrp troubleshooting lab.

    Thanks!

  16. Hey guys…

    I’m totally stuck on area 0. I’ve got OSPF to become at a full state with R3 and R3 is the elected DR for the area 0 frame relay topology. I shutdown the Fa1/0 link on R1 to only find out that there is an issue with communication between R2 and R1.

    I get the hello packets from R2 but it’s stuck at a 2WAY/DROTHER state.. you would think R2 would be elected the BDR if R1 isn’t competing in the OSPF election process..

    I’ll do a #ip ospf priority 1 command and i’ll get full state with R2 but still no route to 2.2.2.2!!

    DON’T TELL ME THE ANSWER! Can you guys maybe point me to some reading material that’ll help me understand whats going on here?? or a couple proper debug commands??

    I feel like i’m getting somewhere!

  17. Hi Matthew,

    You doing ok on this one? I never give away easy answers πŸ˜›

    If OSPF gets stuck during the neighbor adjacency you should try a debug to see what is going on. Afterwards you can look at the show commands to check the differences between neighbors.

    If I am not entirely sure which debug/show commands to use, I just work through the list….

    [b]debug ip ospf ?
    show ip ospf ?[/b]

    and make sure to use [b]”show ip ospf interface”[/b] commands to see OSPF information per interface.

    Good luck!

  18. I’m making progress… It looks like R3 has issues communicating with R2 also. Which leads me to beleive there is an pvc issue between the two…

    I’m also receiving hellos from both of them with the multicast ospf address.

    Am I on the right track? or cold?
    haha!

    I’m going to try and set up the interface as a ospf point-to-multipoint /w broadcast and see if I get anywhere…

  19. alright I got it… I had to cheat though and look at the configs on R2 and R3.. I learned a ton though

    I don’t want to spoil any thing for anyone else πŸ™‚
    Time to move on!! I’ve been on this thing for a week lol

  20. I’m kind of surprised though… I feel like my previous configs should have worked. I must have been missing one thing..

    *shrug*

  21. Hi Matthew,

    Not sure what you missed but OSPF and frame-relay is a recipe for trouble. You might want to look at my “OSPF network type” labs in the OSPF section, maybe that’ll be helpful.

    Seems you learned a lot from this one ;D

    Rene

  22. okay from my understanding.. the only way to get the point-to-multipoint area 0 network working correctly is if R1 is the DR.

    *clear ip ospf process
    isn’t enough because R3 still won’t release its DR status and sense I have no access to it. I’m SOL

    The only way I can get R1 to become the DR is to set its priority, save my configs, and restart the lab.

    For some reason I feel like this isn’t the fix you had in mind

    am I wrong?

  23. Hi Matthew,

    R1 has a manual configured router-ID (4.4.4.4) so it should become the DR. There’s a PVC between R1 and R3 so if you reset the neighbor adjacency R1 should be the DR. Did R3 become the DR in your example? If so can you check why that happened? πŸ™‚

    Rene

  24. I did a #clear ip OSPF process on R1 to reset the neighbor adjacency?

    I remember reading something about OSPF routers remembering who is the DR incase it goes down and comes back up…

    So the only way to make R1 become the DR would be to enter R3 and reset its OSPF process!

    Maybe I’m wrong.. but I’ve looked and looked and I’m stuck on this

    help me out :

  25. Hi Matthew,

    I think you are right on this. OSPF is “non-preemptive” which means it will keep it’s status. Once you do a clear ip ospf process on R3, does R1 become the DR?

    Rene

  26. You don’t need the passwords, you should be able to troubleshoot this lab without having access to R2 or R3. If you get stuck you can always peek at their configuration in their config files.

  27. thanks for this lab.
    This topology is driving me to confusion. But when my mind is clear, i will understand.
    The good Idea is in redistribution LAB.

    I hope you are OK

    My Regards
    Omar

    1. Hi Omar,

      Your welcome. It’s not an easy lab…you’ll need to know everything about OSPF on CCNP level. Just take your time and if you get stuck just watch my video and you’ll see how to progress to the next task.

      Good luck!

      Rene

  28. The OSPF Troubleshoot LAB was AWESOME! I really got some practice solving it, and learned new things, if your going or your thinking of taking CCNP TSHOOT or even ROUTE I really recomend you trying it.

    (took me 2 hours to solve it! O.o)

    I’ll look fordward to solving some more of your Troubleshoot labs, Thanx a ton!

    1. Glad to hear you enjoyed it πŸ™‚ This lab will really nail down your OSPF knowledge…

  29. This was very good practice! thank you Rene. I managed to resolve the lab in 1 hour 30 mins. Some of the implementations such as MTU I was not familar with but debugs helped here. The virtual links were great practice. I am still learning CCNP route and may do the exam soon. Is TSHOOT harder or easier than Route?

    1. Glad that you liked it πŸ™‚ TSHOOT is alot easier than ROUTE if you ask me…

  30. This lab is awesome, I do it at least once a month to stay sharp. It keeps me reminded of what OSPF needs to create neighbors.

  31. Hi, I am new to GNS3vault, could you please tell me, from where can I download the topology & config files, I have registered today only

  32. Fuf… I’ve done. Well, gr8 lab but I see some mistakes in the solution that has been suggested by gns3vault.com. Look at this task:
    "After fixing the problem on R1 you notice traffic from R1 towards R2 uses the fastethernet link. All traffic should be sent through the serial interface, when the frame-relay link fails it should switch to the fastethernet."
    In the final_cofig we see cost changes:

    interface FastEthernet1/0
    ip address 192.168.12.1 255.255.255.0
    [b]ip ospf cost 5000[/b]

    But there are no such command in the R2 config, so when you look at R1’s routing table after that configuration you’ll see correct route via Serial, but if you look at R2’s routing table you’ll see the route via FastEthernet. So, suggested config is not quite correct.
    I made another router ospf proccess on R1 (router ospf 2), add put ethernet link into that proccess in area 0, delete network 1.1.1.1 statement from both processes and made redistribute connected with different metrics into both processes.

    router ospf 1
    router-id 1.1.1.1
    log-adjacency-changes
    area 51 virtual-link 5.5.5.5
    area 51 virtual-link 4.4.4.4
    redistribute connected metric 10 subnets tag 10
    network 192.168.123.1 0.0.0.0 area 0
    network 192.168.145.1 0.0.0.0 area 51

    router ospf 2
    router-id 192.168.12.1
    log-adjacency-changes
    redistribute connected metric 20 subnets tag 20
    network 192.168.12.1 0.0.0.0 area 0

    1. True I didn’t include the cost on R2 so only R1 will use the serial link. You did came up with a nice solution though πŸ™‚

  33. hello rene

    If final configs.zip file is the solved file then why have you enabled password in it.

  34. Wow! This troubleshoot lab was awesome! It took about 3 hours but I did it just with a little help πŸ™‚

  35. Fantastic! Muchas gracias!
    Thank you very much for the lab. Though challenging and i got to through the end without braking the rules ;).
    Many thanks.

  36. Enjoyed doing this lab in preparation for my Tshoot exam next week. I definitely got stuck on a few places but this was a good ospf refresher as I’ve been doing nothing but switching for the past few months.

    Thanks a lot!

  37. Dear Renne

    Another "beautiful" and challenging lab ……..

    I did lab without help from video or show run command apart from sorting out the adjacency between R5 & R6 …..never thought of mtu issue …….. but learned a new thing

    All credit to Renne

    1. Yes that was a tough one.
      I noticed the mtu when I exhausted all "common" adjacency issues
      and switched to debug.

      Fixed it with [b]ip ospf mtu-ignore[/b] on interface. Little bit different solution than the one Rene used.

  38. Awesomessss!! ^^ I solved it!! OMG : )) toke some time but… anyway very happy.

    Wow .. Solved without run the [show run].

  39. This one was really tough one.

    Really good job – did not know how many things can be messed up on this topology πŸ™‚

    Loopback6 and DR/BDR election on FR (but I was close πŸ˜‰ were "stronger".

    But as in the TSHOOT book, lesson learned.

    Thanks for this lab.

    Martin

  40. What a very nice LAB Rene, I have done it, T-shoot neighborship in area 56 was pretty hard, there were some nasty stuff πŸ˜€

    Thanx a lot

  41. Rene,

    The final configuration file holds the start-up configs. Can you please upload the correct final configurations? Great lab !

      1. Hi Rene,

        Thanks, preparing for the CCNP Tshoot in 2 weeks.
        Your labs and books have really helped me with CCNP Switch and Route in past!

        Glenn

  42. Very Nice!!! a tough lab, very tricky!.

    sh ip protocols is an awesome command!
    took me a few hours and learning along the way, great lab!
    thanks!

  43. This is probably the greatest lab on the whole site, it covers almost everything needed for OSPF for CCNP Route! Thank you Rene! πŸ˜€

  44. Hi Rene

    i am not able to find the .net file for the above ospf topology on this page. Can u please help me.

  45. Man, The EIGRP T-shoot lab I ran through it with very little hassle but this really killed me haha. Got it at the end (Didn’t cheat with any commands) only really used the vid for the silly no point-to-point on the Loopback0 R6. Guess I forgot that..Ill need to read over the OSPF stuff again, taking the CCNP route in March.

    I also got stuck for a bit on the neighbour between R5 and R6 it was stuck in EXSTART…then ran a debug and saw the issue, don’t want to give it away, but I thought the MTU check was done during 2-way state when it sends the Hello’s…(More Reading is needed clearly, haha)

    Thanks for the Labs bud!

Comments are closed.