I have learned a LOT in the last 3 nights (mostly sleepless nights hehe) through this exercise, and really wanted to share it with the world. Maybe through this somebody out there will learn something as well, or perhaps avoid some of the pitfalls I encountered along the way.

Let’s begin with the basic scenario: You have an enterprise network (or CCIE lab) running some IGP. Your network is peering via BGP to various backbone routers in the lab, which all give you different unique routes. You want to be able to achieve full reachability to all of these routes from any router in your network. Here is the catch: You can only run BGP on your edge routers, and you may not redistribute into your IGP (Not that it would ever be a good idea anyways).

Here is a network diagram that shows the overall picture. Note, this diagram is of my personal home lab, which is basically a standard “IPexpert topology.” Because I am using their material for study, and basically because I am too lazy to go redo all the backbone routers and addressing, all the addressing and such is going to be pretty much exactly what IPexpert products use.

My goal for this tutorial is to take you through step by step exactly what I did to achieve this, and try to explain things as I go along. I warn you now, this will not be a short post : )

Configuring Your IGP for internal reachability

OK, lets get started then shall we? Alright, first things first, lets get our IGP going. I choose OSPF for whatever reason this time around. If you take a look at our network diagram you will see that every router has a loopback of 200.0.0.x and all of the internal networks are in the 150.100.0.0/16 range, with the exception of the ethernet links that go to the backbone routers.

On all routers:

router ospf 1
network 200.0.0.0 0.0.0.255 area 0
network 150.100.0.0 0.0.255.255 area 0

We see some neighbors start to come up, which is all good, but notice there are some things we are not expecting on R2:

R2(config-router)#do sh ip ospf neigh

Neighbor ID Pri State Dead Time Address Interface
200.0.0.5 0 FULL/ – 00:00:32 150.100.25.5 Serial2/1
200.0.0.4 0 FULL/ – 00:00:36 150.100.24.4 Serial2/0.24
200.0.0.1 1 FULL/BDR 00:00:35 150.100.12.1 Ethernet0/0

OK so we have a neighbor to R5…but it is over the PPP serial link, not the frame-relay. R6 is nowhere to be found! R2 is the hub of the frame-relay topology. It has a point-to-multipoint sub-interface, so it’s default OSPF network type is non-broadcast (requires a DR and neighbor commands). The R2/R4 link is point-to-point on both sides, so no issues there. R5 and R6 use physical links, so their default ospf network type will also be non-broadcast. Let’s go ahead and make sure R2 becomes the DR and that it can communicate with the spokes! Remember, we really only are required to have neighbor commands on 1 end.

On R2:

router ospf 1
neighbor 150.100.100.5
neighbor 150.100.100.6

On R5/R6:

int s2/0
ip ospf priority 0

R2(config-router)#neighbor 150.100.100.5
R2(config-router)# neighbor 150.100.100.6
R2(config-router)#
*Sep 10 00:02:54.472: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.5 on Serial2/0.256 from LOADING to FULL, Loading Done
*Sep 10 00:02:54.508: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.6 on Serial2/0.256 from LOADING to FULL, Loading Done
R2(config-router)#do sh ip ospf neigh

Neighbor ID Pri State Dead Time Address Interface
200.0.0.5 0 FULL/ – 00:00:39 150.100.25.5 Serial2/1
200.0.0.5 0 FULL/DROTHER 00:01:59 150.100.100.5 Serial2/0.256
200.0.0.6 0 FULL/DROTHER 00:01:50 150.100.100.6 Serial2/0.256
200.0.0.4 0 FULL/ – 00:00:34 150.100.24.4 Serial2/0.24
200.0.0.1 1 FULL/BDR 00:00:33 150.100.12.1 Ethernet0/0

MUCH better! Let’s check out the routing table on R1, one of our furthest points…

R1#show ip route ospf
200.0.0.0/32 is subnetted, 8 subnets
O 200.0.0.8 [110/140] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.9 [110/139] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.2 [110/11] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.4 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.5 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.6 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.7 [110/76] via 150.100.12.2, 00:02:02, Ethernet0/0
150.100.0.0/24 is subnetted, 12 subnets
O 150.100.220.0 [110/76] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.221.0 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.100.0 [110/74] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.81.0 [110/149] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.91.0 [110/139] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.69.0 [110/138] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.78.0 [110/139] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.40.0 [110/84] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.41.0 [110/84] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.24.0 [110/74] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.25.0 [110/74] via 150.100.12.2, 00:02:03, Ethernet0/0

OK, that is looking pretty damn good, but do we have reachability? Lets try all the loopbacks.

R1#ping 200.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R1#ping 200.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/67/68 ms
R1#ping 200.0.0.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/69 ms
R1#ping 200.0.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/68 ms
R1#ping 200.0.0.7

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.7, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/69 ms
R1#ping 200.0.0.8

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 80/82/84 ms
R1#ping 200.0.0.9

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.9, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 100/100/100 ms

Excellent!!!

Configuring BGP Peerings

OK, now that we have our IGP up and running, and we have full reachability inside our own AS (Oh yeah, we will be AS 1001 today! … Apologies to whoever actually owns AS 1009 and the rest of these examples hehe). Now it is time to set up our BGP peers. We will be peering from R1 to BB1 AS 7018 , R8 to BB2 AS 701 , and R9 to BB3 AS 3561. We will run iBGP between routers R1,R8,R9. Since we are only running iBGP between 3 routers we will go ahead and make it a full mesh.

We will go ahead and set up our iBGP peerings first before we dive into eBGP.

On R1:

router bgp 1001
neighbor 200.0.0.8 remote-as 1001
neighbor 200.0.0.8 update-source Loop0
neighbor 200.0.0.9 remote-as 1001
neighbor 200.0.0.9 update-source Loop0

On R8:

router bgp 1001
neighbor 200.0.0.1 remote-as 1001
neighbor 200.0.0.1 update-source Loop0
neighbor 200.0.0.9 remote-as 1001
neighbor 200.0.0.9 update-source Loop0

On R9:

router bgp 1001
neighbor 200.0.0.1 remote-as 1001
neighbor 200.0.0.1 update-source Loop0
neighbor 200.0.0.8 remote-as 1001
neighbor 200.0.0.8 update-source Loop0

Lets see how we did on R1…

R1(config-router)#do sh ip bgp sum
BGP router identifier 200.0.0.1, local AS number 1001
BGP table version is 1, main routing table version 1

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
200.0.0.8 4 1001 3 4 1 0 0 00:00:34 0
200.0.0.9 4 1001 4 4 1 0 0 00:00:10 0

Excellent! We don’t see any learned prefixes because, well…we have not brought anything into BGP yet. Onward to our eBGP!

OK, so again R1 will peer to AS 7018, R8 to AS 701 , and R9 to AS 3561. Let’s also redistribute our OSPF stuff into BGP so that these AS’ know about our internal AS routes.

On R1:

router bgp 1001
neighbor 100.100.100.100 remote-as 7018
redistribute ospf 1
1d06h: %BGP-5-ADJCHANGE: neighbor 100.100.100.100 Up

On R8:

router bgp 1001
neighbor 100.100.200.200 remote-as 701
redistribute ospf 1
*Mar 22 01:40:45.099: %BGP-5-ADJCHANGE: neighbor 100.100.200.200 Up

On R9:

router bgp 1001
neighbor 100.100.250.250 remote-as 3561
redistribute ospf 1
*Nov 21 01:45:40.389: %BGP-5-ADJCHANGE: neighbor 100.100.250.250 Up

OK excellent…lets do some checking to make sure we are receiving prefixes.

R1(config-router)#do sh ip bgp sum | i 100.100.100.100|200.0.0.8|200.0.0.9
100.100.100.100 4 7018 11 8 117 0 0 00:04:35 116
200.0.0.8 4 1001 13 15 117 0 0 00:09:27 119
200.0.0.9 4 1001 14 15 117 0 0 00:09:03 122

R8(config-router)#do sh ip bgp sum | i 100.100.200.200|200.0.0.1|200.0.0.9
100.100.200.200 4 701 9 7 120 0 0 00:03:41 119
200.0.0.1 4 1001 16 14 120 0 0 00:10:01 116
200.0.0.9 4 1001 13 13 120 0 0 00:09:59 122

R9(config-router)#do sh ip bgp sum | i 100.100.250.250|200.0.0.1|200.0.0.8
100.100.250.250 4 3561 8 6 123 0 0 00:02:56 122
200.0.0.1 4 1001 15 14 123 0 0 00:10:00 116
200.0.0.8 4 1001 14 14 123 0 0 00:10:21 119

Excellent, so we are getting a ton of routes from our backbone routers. Before we get too deep, lets make sure we solve that whole don’t become a transit AS problem. There are a number of ways to accomplish that, including distribute-lists, prefix-lists, filter-lists, and communities. I choose communities, because frankly it is easy to do.

First, we need to set the routers to use the new community format. If we don’t do this, we’ll see some large strange 32 bit number in the form of an integer for our communities, and that is no fun. Then we have to tag our incoming routes from the BB routers as no-export.

On R1,R8,R9:

ip bgp new-format
route-map NoTransit permit 10
set community no-export

Now apply the route-map to our eBGP peerings, and make sure we send the no-export community to our iBGP peers…

On R1:

router bgp 1001
neighbor 100.100.100.100 route-map NoTransit in
neighbor 200.0.0.8 send-community
neighbor 200.0.0.9 send-community

On R8:

router bgp 1001
neighbor 100.100.200.200 route-map NoTransit in
neighbor 200.0.0.1 send-community
neighbor 200.0.0.9 send-community

On R9:

router bgp 1001
neighbor 100.100.250.250 route-map NoTransit in
neighbor 200.0.0.1 send-community
neighbor 200.0.0.8 send-community

Let’s go ahead and clear our BGP peers on all our router’s just in case…

On R1,R8,R9:
do clear ip bgp * soft

OK, let’s make sure this is working…here is the beginning of our BGP table on R1. Lets just grab the first route there we are learning from BB1 to make sure it is tagged end to end.

R1(config-router)#do sho ip bgp
BGP table version is 233, local router ID is 200.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete

Network Next Hop Metric LocPrf Weight Path
*> 64.58.0.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.58.16.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.59.128.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.59.144.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.63.160.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.63.176.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.69.0.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.69.16.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.70.96.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.70.112.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.75.64.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.75.112.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.76.32.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.76.160.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.77.176.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.77.224.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.77.240.0/20 100.100.100.100 0 0 7018 20340 ?
–More–

R1(config-router)#do sh ip bgp 64.58.0.0
BGP routing table entry for 64.58.0.0/20, version 123
Paths: (1 available, best #1, table Default-IP-Routing-Table, not advertised to EBGP peer)
Advertised to non peer-group peers:
200.0.0.8 200.0.0.9
7018 20340
100.100.100.100 from 100.100.100.100 (209.202.63.200)
Origin incomplete, metric 0, localpref 100, valid, external, best
Community: no-export

OK, it is good on R1…now let’s make sure we see it tagged as no-export way over on R9.

R9(config-router)#do sh ip bgp 64.58.0.0
BGP routing table entry for 64.58.0.0/20, version 0
Paths: (1 available, no best path)
Not advertised to any peer
7018 20340
100.100.100.100 (inaccessible) from 200.0.0.1 (200.0.0.1)
Origin incomplete, metric 0, localpref 100, valid, internal
Community: no-export

OK so bittersweet…our no-export community went through, so we are good. You would definitely want to go ahead and check at least a few routes from each peer end to end, but for sake of the length of this post I am not going to show that today. The bad part is that on R9 we see a problem. A big problem. Basically none of the routes that we learned from R1 are getting put into the routing table. If we do a little more digging we will see the same problem on all the routers. Any routes that are learned via iBGP are showing up as valid, but not best and as inaccessible. hmmmmmmm

So let’s take a close look at that again:

R9(config-router)#do sh ip bgp 64.58.0.0
BGP routing table entry for 64.58.0.0/20, version 0
Paths: (1 available, no best path)
Not advertised to any peer
7018 20340
100.100.100.100 (inaccessible) from 200.0.0.1 (200.0.0.1)
Origin incomplete, metric 0, localpref 100, valid, internal
Community: no-export

So what’s happening here? BB1 adveritses 64.58.0.0/22 to R1, next hop 100.100.100.100. R1 passes the route to R8 and R9 via iBGP. Remember in iBGP the next hop will not change!!! So, now at R8/R9 we have 64.58.0.0/22 next hop 100.100.100.100…OK no problem…wait. Who the hell is 100.100.100.100?!!!

R9(config-router)#do sh ip route 100.100.100.100
% Subnet not in table

Ahhhhhhhh, good old next-hop reachability. So R8 and R9 have the route, but they have no idea how to reach the next hop to get there and therefore, the routes are not best, and will not be placed in the routing table. We will see the same exact problem on R1 for routes eBGP learned routes at R8,R9 and on R8 for eBGP learned routes at R1,R9 … Let’s go ahead and fix this business by bringing those backbone connected links into OSPF on all 3 of our routers.

R1(config)#router ospf 1
R1(config-router)#network 100.100.100.0 0.0.0.255 area 0

R8(config-router)#router ospf 1
R8(config-router)#network 100.100.200.0 0.0.0.255 area 0

R9(config-router)#router ospf 1
R9(config-router)#network 100.100.250.0 0.0.0.255 area 0

NOW let’s take a look at that same route on R9!

R9(config-router)#do sh ip bgp | i 64.58.0.0
*>i64.58.0.0/20 100.100.100.100 0 100 0 7018 20340 ?

R9(config-router)#do sho ip route | i 64.58.0.0
B 64.58.0.0 [200/0] via 100.100.100.100, 00:01:56

Excellent! So now we have the route in the BGP table as valid AND best, and therefore it has been placed into our routing table! So we should have reachability now! Lets try this out. All of the BB injected routes have an address of .200.

R9(config-router)#do ping 64.58.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 64.58.0.200, timeout is 2 seconds:
U.U.U
Success rate is 0 percent (0/5)

uhhhhh…hmmmm….uhhoh! sooo…we have the route…we have a route to the next hop…WTF is going on here?! Let’s do a trace

R9(config-router)#do trace 64.58.0.200

Type escape sequence to abort.
Tracing the route to 64.58.0.200

1 150.100.69.6 16 msec 20 msec 16 msec
2 150.100.69.6 !H * !H

So our ping goes from R9 over to R6 and immediately takes a dump. Let’s think this through. So R9 wants to ping 64.58.0.200. R9 check’s it’s routing table and finds a BGP route to 64.58.0.0/22 with a next hop of 100.100.100.100. OK, so now it has to figure out how to get to 100.100.100.100. We check our routing table again and see a perfectly good OSPF route to 100.100.100.0/24 through R6. R9 encapsulates our ping packet into a PPP frame and sends it on over to R6. Keep in mind our source and destination IP addresses have NOT changed. R6 gets the packet, source: 150.100.69.9 destination: 64.58.0.200. R6 checks it’s routing table for 64.58.0.200. Bzzzzzzzzzzzzzz, thanks for playing!!!

R6(config-router)#do sh ip route 64.58.0.0
% Network not in table

So, the long and short of the problem is that we do indeed have the route…and we have a valid route to the next hop…but the next hop (R6 in this case) does not have a route to the destination.

What to do, what to do… we could redistribute our BGP into OSPF at each edge router! Bah..well for one we aren’t allowed to do that here. For 2, if we did that in the real world it would be very very bad…especially if we were taking full internet tables from our eBGP peers. We’d likely crash our OSPF. So, imagine you are sitting in the CCIE lab and find yourself here. You have perfectly valid routes to all these external networks on your BGP routers, but cannot ping them. You ask the proctor if you need to just have valid and best paths or if you need actual reachability. He smiles at you and says you need to have reachability….yet your lab says you cannot redistribute either. Shit…

So what is another solution…what if we pump out a default route into OSPF from each edge router? Well that would sort of work, but we’d selectively blackhole a bunch of routes, and have a bunch of routing loops. So that is out. So we need a way for our next hop router to have a route to the final destination. Take R9 for example. When it looks up 100.100.100.100 because it is the next hop, and sends it along, whatever router gets that next should have an actual route to the end destination…so in other words the next hop should probably be R1 in this case. GRE Tunnels!

What we want to do is set up some tunnels between R1,R8,R9 and put them into OSPF. So lets get going on that.

R1(config)#int tunnel0
R1(config-if)#ip address 19
R1(config-if)#ip address 192.168.0.1 255.255.255.0
R1(config-if)#tunnel source 200.0.0.1
R1(config-if)#tunnel destination 200.0.0.8
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

R1(config-if)#int tunnel1
R1(config-if)#ip address 192.1
R1(config-if)#ip address 192.168.1.1 255.255.255.0
R1(config-if)#tunnel source 200.0.0.1
R1(config-if)#tunnel destination 200.0.0.9
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up

R1(config-if)#router ospf 1
R1(config-router)#network 192.168.0.0 0.0.255.255 area 0

R8(config)#int tunnel0
R8(config-if)#ip address 1
R8(config-if)#ip address 192.168.0.8 255.255.255.0
R8(config-if)#tunnel source 200.0.0.8
R8(config-if)#tunnel destination 200.0.0.1
*Mar 22 02:46:51.893: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

R8(config-if)#int tunnel 1
R8(config-if)#ip address 1
R8(config-if)#ip address 192.168.2.8 255.255.255.0
R8(config-if)#tunnel source 200.0.0.8
R8(config-if)#tunnel destination 200.0.0.9
R8(config-if)#
*Mar 22 02:47:24.707: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up

R8(config-if)#router ospf 1
R8(config-router)#network 192.168.0.0 0.0.255.255 area 0

R9(config-router)#interface tunnel0
R9(config-if)#ip address 192.1
R9(config-if)#ip address 192.168.1.9 255.255.255.0
R9(config-if)#tunnel source 200.0.0.9
R9(config-if)#tunnel dest 200.0.0.1
*Nov 21 02:52:38.628: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

R9(config-if)#interface tun1
R9(config-if)#ip address 192.16
R9(config-if)#ip address 192.168.2.9 255.255.255.0
R9(config-if)#tunnel source 200.0.0.9
R9(config-if)#tunnel dest 200.0.0.8
*Nov 21 02:53:00.119: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up

R9(config-if)#router ospf 1
R9(config-router)#network 192.168.0.0 0.0.255.255 area 0

OK, so here is a peek at our routers after setting all this up…

R1(config-router)#do sh ip ospf neigh | i Tunnel
200.0.0.9 0 FULL/ – 00:00:36 192.168.1.9 Tunnel1
200.0.0.8 0 FULL/ – 00:00:39 192.168.0.8 Tunnel0

R8(config-router)#do sh ip ospf neigh | i Tunnel
200.0.0.9 0 FULL/ – 00:00:36 192.168.2.9 Tunnel1
200.0.0.1 0 FULL/ – 00:00:29 192.168.0.1 Tunnel0

R9(config-router)#do sh ip ospf neigh | i Tunnel
200.0.0.8 0 FULL/ – 00:00:38 192.168.2.8 Tunnel1
200.0.0.1 0 FULL/ – 00:00:38 192.168.1.1 Tunnel0

So, the $25,000 question right now is did this help us? Lets check out R9 again

R9(config)#do sh ip route 100.100.100.100
Routing entry for 100.100.100.0/24
Known via “ospf 1”, distance 110, metric 148, type intra area
Last update from 150.100.69.6 on Serial0/0, 00:02:49 ago
Routing Descriptor Blocks:
* 150.100.69.6, from 200.0.0.1, 00:02:49 ago, via Serial0/0
Route metric is 148, traffic share count is 1

Well, we are STILL learning the next hop through OSPF via R6 … our cost is 148. So, lets make the path via our tunnels a better cost. We’ll be safe and say how about….. 25 ?

On R1,R8,R9:

int tun0
ip ospf cost 25

int tun1
ip ospf cost 25

1d07h: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.8 on Tunnel0 from FULL to DOWN, Neighbor Down: Interface down or detached

1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.9 on Tunnel1 from FULL to DOWN, Neighbor Down: Dead timer expired
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.9 on Tunnel1 from LOADING to FULL, Loading Done
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.8 on Tunnel0 from LOADING to FULL, Loading Done
1d07h: %TUN-5-RECURDOWN: Tunnel1 temporarily disabled due to recursive routing
1d07h: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to down
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.9 on Tunnel1 from FULL to DOWN, Neighbor Down: Interface down or detached
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.8 on Tunnel0 from FULL to DOWN, Neighbor Down: Interface down or detached

Ohhhhh boy….we’ll see this on each of the routers. So we set the cost of our tunnels pretty damn low…what’s happening is that we are learning the route to the destination of our tunnels THROUGH THE TUNNELS…that would be bad. Therefore, we see the tunnel go up, down, up, down, forever. Bah!

To fix this issue, we’ll make sure all the tunnel endpoints are reachable by some other means…namely static routes.

R1(config)#ip route 200.0.0.8 255.255.255.255 150.100.12.2
R1(config)#ip route 200.0.0.9 255.255.255.255 150.100.12.2

R8(config)#ip route 200.0.0.1 255.255.255.255 150.100.78.7
R8(config)#ip route 200.0.0.9 255.255.255.255 150.100.78.7

R9(config)#ip route 200.0.0.1 255.255.255.255 150.100.69.6
R9(config)#ip route 200.0.0.8 255.255.255.255 150.100.69.6

OK how about NOW? Lets check out R9 again

R9(config)#
*Nov 21 03:10:30.481: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.1 on Tunnel0 from FULL to DOWN, Neighbor Down: Dead timer expired
R9(config)#
*Nov 21 03:10:37.528: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.1 on Tunnel0 from LOADING to FULL, Loading Done
R9(config)#do sh ip route 200.0.0.1
Routing entry for 200.0.0.1/32
Known via “static”, distance 1, metric 0
Routing Descriptor Blocks:
* 150.100.69.6
Route metric is 0, traffic share count is 1

R9(config)#do ping 200.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

Ugh…so we aren’t getting the recursive routing anymore, but our tunnels are still flapping and we can’t ping the tunnel end point. Well we kind of set that ospf cost of the tunnels too low…what is happening is that R9 is trying to get to the tunnel end point at 200.0.0.1…it has a static route to 200.0.0.1 pointing to R6 so it sends it there. R6 looks up 200.0.0.1 and because the cost of the tunnel was so low, the path back to R9 and over the tunnel is actually still LOWER than the regular path through the network..so what we end up with is a routing loop. We need to VERY carefully choose our tunnel ospf costs to make this work. So lets figure out the costs through the regular network, then set the tunnels just slightly lower.

Cost of R1 –> R8: R1 –> R2 via Ethernet = 10
R2 –> R6 via Frame = 64
R6 –> R7 via FastEthernet = 1
R7 –> R8 via Serial = 64
R8 Loopback = 1
——————————-
Total Cost = 140

Cost of R1 –> R9: R1 –> R2 Ethernet = 10
R2 –> R6 Frame = 64
R6 –> R9 PPP = 64
R9 Loopback = 1
————————
Total Cost = 139

Cost of R8 –> R9: R8 –> R7 via Serial = 64
R7 –> R6 via FastEthernet = 1
R6 –> R9 via Serial = 64
R9 Loopback = 1
————————-
Total Cost = 130

So lets set our tunnel costs just slightly lower:

R1(config)#int tun0
R1(config-if)#ip ospf cost 135
R1(config-if)#int tun1
R1(config-if)#ip ospf cost 135

R8(config)#int tun0
R8(config-if)#ip ospf cost 135
R8(config-if)#int tun1
R8(config-if)#ip ospf cost 120

R9(config)#int tun0
R9(config-if)#ip ospf cost 135
R9(config-if)#int tun1
R9(config-if)#ip ospf cost 120

Finally…let’s check it out on R9.

R9(config-router)#do sh ip route 100.100.100.0
Routing entry for 100.100.100.0/24
Known via “ospf 1”, distance 110, metric 145, type intra area
Redistributing via bgp 1001
Advertised by bgp 1001
Last update from 192.168.1.1 on Tunnel0, 00:06:41 ago
Routing Descriptor Blocks:
* 192.168.1.1, from 200.0.0.1, 00:06:41 ago, via Tunnel0
Route metric is 145, traffic share count is 1

R9(config-router)#do ping 64.58.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 64.58.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 120/123/125 ms
R9(config-router)#

BOOOOOOOOOYAH!

Now, lets just make it work from EVERY router in our AS. We’ll pump out a default route from our 3 edge routers. Now it won’t cause serious loop issues or blackholes, because no matter which one of the edge routers the packet gets to, each of them have all of the BGP routes via iBGP. Excellent

R1(config-router)#router ospf 1
R1(config-router)#default-information originate always

R8(config-router)#router ospf 1
R8(config-router)#default-information originate always

R9(config-router)#router ospf 1
R9(config-router)#default-information originate always

Moment of truth. Here are 3 routes we learned via our outside AS’ … one from each. We’ll go to a random internal router that is only speaking OSPF and try each one.

From BB1: 64.58.0.200
From BB2: 202.0.0.200
From BB3: 102.6.0.200

R4#ping 64.58.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 64.58.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/72 ms
R4#
R4#ping 202.0.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 202.0.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 176/177/184 ms

R4#ping 102.6.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 102.6.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 176/176/180 ms

Bickety Bam…full reachability…no BGP –> OSPF redistribution

– Joe A