So I’ve been sitting here trying to figure something out…then all the sudden it dawns on me … duhhhhhhh, I’m an idiot!

Here is the situation. R7—R8 …pretty basic. R7 connects to R8 direct via Serial0/0 on both sides, also there is a tunnel, tunnel0 on both sides. R8 has a loopback of 200.0.0.8 that it is redistributing into OSPF. I am sitting on R7 looking at the route for 200.0.0.8 trying to figure out why I am learning it via Tun0 instead of S0/0…I mean after all S0/0 has a much better metric right?

The serial link is 150.100.78.0/24 and the tunnel is 150.100.178.0/24. The serial links are area 78 and the tunnel is area 0.

R7#sh run | beg ospf 1

router ospf 1
log-adjacency-changes
area 78 stub
network 150.100.78.0 0.0.0.255 area 78
network 150.100.178.0 0.0.0.255 area 0

R8#sh run | beg router ospf 1
router ospf 1
log-adjacency-changes
area 78 stub
redistribute connected subnets
network 150.100.78.0 0.0.0.255 area 78
network 150.100.81.0 0.0.0.255 area 78
network 150.100.178.0 0.0.0.255 area 0

R7#sh ip route | i 200.0.0.8
O E2 200.0.0.8 [110/20] via 150.100.178.8, 00:15:25, Tunnel0

Can you guys see WHY I am getting this route learned via the tunnel? It’s so painfully obvious now that I look at it….

– Joe

I have learned a LOT in the last 3 nights (mostly sleepless nights hehe) through this exercise, and really wanted to share it with the world. Maybe through this somebody out there will learn something as well, or perhaps avoid some of the pitfalls I encountered along the way.

Let’s begin with the basic scenario: You have an enterprise network (or CCIE lab) running some IGP. Your network is peering via BGP to various backbone routers in the lab, which all give you different unique routes. You want to be able to achieve full reachability to all of these routes from any router in your network. Here is the catch: You can only run BGP on your edge routers, and you may not redistribute into your IGP (Not that it would ever be a good idea anyways).

Here is a network diagram that shows the overall picture. Note, this diagram is of my personal home lab, which is basically a standard “IPexpert topology.” Because I am using their material for study, and basically because I am too lazy to go redo all the backbone routers and addressing, all the addressing and such is going to be pretty much exactly what IPexpert products use.

My goal for this tutorial is to take you through step by step exactly what I did to achieve this, and try to explain things as I go along. I warn you now, this will not be a short post : )

Configuring Your IGP for internal reachability

OK, lets get started then shall we? Alright, first things first, lets get our IGP going. I choose OSPF for whatever reason this time around. If you take a look at our network diagram you will see that every router has a loopback of 200.0.0.x and all of the internal networks are in the 150.100.0.0/16 range, with the exception of the ethernet links that go to the backbone routers.

On all routers:

router ospf 1
network 200.0.0.0 0.0.0.255 area 0
network 150.100.0.0 0.0.255.255 area 0

We see some neighbors start to come up, which is all good, but notice there are some things we are not expecting on R2:

R2(config-router)#do sh ip ospf neigh

Neighbor ID Pri State Dead Time Address Interface
200.0.0.5 0 FULL/ – 00:00:32 150.100.25.5 Serial2/1
200.0.0.4 0 FULL/ – 00:00:36 150.100.24.4 Serial2/0.24
200.0.0.1 1 FULL/BDR 00:00:35 150.100.12.1 Ethernet0/0

OK so we have a neighbor to R5…but it is over the PPP serial link, not the frame-relay. R6 is nowhere to be found! R2 is the hub of the frame-relay topology. It has a point-to-multipoint sub-interface, so it’s default OSPF network type is non-broadcast (requires a DR and neighbor commands). The R2/R4 link is point-to-point on both sides, so no issues there. R5 and R6 use physical links, so their default ospf network type will also be non-broadcast. Let’s go ahead and make sure R2 becomes the DR and that it can communicate with the spokes! Remember, we really only are required to have neighbor commands on 1 end.

On R2:

router ospf 1
neighbor 150.100.100.5
neighbor 150.100.100.6

On R5/R6:

int s2/0
ip ospf priority 0

R2(config-router)#neighbor 150.100.100.5
R2(config-router)# neighbor 150.100.100.6
R2(config-router)#
*Sep 10 00:02:54.472: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.5 on Serial2/0.256 from LOADING to FULL, Loading Done
*Sep 10 00:02:54.508: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.6 on Serial2/0.256 from LOADING to FULL, Loading Done
R2(config-router)#do sh ip ospf neigh

Neighbor ID Pri State Dead Time Address Interface
200.0.0.5 0 FULL/ – 00:00:39 150.100.25.5 Serial2/1
200.0.0.5 0 FULL/DROTHER 00:01:59 150.100.100.5 Serial2/0.256
200.0.0.6 0 FULL/DROTHER 00:01:50 150.100.100.6 Serial2/0.256
200.0.0.4 0 FULL/ – 00:00:34 150.100.24.4 Serial2/0.24
200.0.0.1 1 FULL/BDR 00:00:33 150.100.12.1 Ethernet0/0

MUCH better! Let’s check out the routing table on R1, one of our furthest points…

R1#show ip route ospf
200.0.0.0/32 is subnetted, 8 subnets
O 200.0.0.8 [110/140] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.9 [110/139] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.2 [110/11] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.4 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.5 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.6 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 200.0.0.7 [110/76] via 150.100.12.2, 00:02:02, Ethernet0/0
150.100.0.0/24 is subnetted, 12 subnets
O 150.100.220.0 [110/76] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.221.0 [110/75] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.100.0 [110/74] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.81.0 [110/149] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.91.0 [110/139] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.69.0 [110/138] via 150.100.12.2, 00:02:02, Ethernet0/0
O 150.100.78.0 [110/139] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.40.0 [110/84] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.41.0 [110/84] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.24.0 [110/74] via 150.100.12.2, 00:02:03, Ethernet0/0
O 150.100.25.0 [110/74] via 150.100.12.2, 00:02:03, Ethernet0/0

OK, that is looking pretty damn good, but do we have reachability? Lets try all the loopbacks.

R1#ping 200.0.0.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R1#ping 200.0.0.4

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/67/68 ms
R1#ping 200.0.0.5

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.5, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/69 ms
R1#ping 200.0.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/68 ms
R1#ping 200.0.0.7

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.7, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/69 ms
R1#ping 200.0.0.8

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 80/82/84 ms
R1#ping 200.0.0.9

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.9, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 100/100/100 ms

Excellent!!!

Configuring BGP Peerings

OK, now that we have our IGP up and running, and we have full reachability inside our own AS (Oh yeah, we will be AS 1001 today! … Apologies to whoever actually owns AS 1009 and the rest of these examples hehe). Now it is time to set up our BGP peers. We will be peering from R1 to BB1 AS 7018 , R8 to BB2 AS 701 , and R9 to BB3 AS 3561. We will run iBGP between routers R1,R8,R9. Since we are only running iBGP between 3 routers we will go ahead and make it a full mesh.

We will go ahead and set up our iBGP peerings first before we dive into eBGP.

On R1:

router bgp 1001
neighbor 200.0.0.8 remote-as 1001
neighbor 200.0.0.8 update-source Loop0
neighbor 200.0.0.9 remote-as 1001
neighbor 200.0.0.9 update-source Loop0

On R8:

router bgp 1001
neighbor 200.0.0.1 remote-as 1001
neighbor 200.0.0.1 update-source Loop0
neighbor 200.0.0.9 remote-as 1001
neighbor 200.0.0.9 update-source Loop0

On R9:

router bgp 1001
neighbor 200.0.0.1 remote-as 1001
neighbor 200.0.0.1 update-source Loop0
neighbor 200.0.0.8 remote-as 1001
neighbor 200.0.0.8 update-source Loop0

Lets see how we did on R1…

R1(config-router)#do sh ip bgp sum
BGP router identifier 200.0.0.1, local AS number 1001
BGP table version is 1, main routing table version 1

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
200.0.0.8 4 1001 3 4 1 0 0 00:00:34 0
200.0.0.9 4 1001 4 4 1 0 0 00:00:10 0

Excellent! We don’t see any learned prefixes because, well…we have not brought anything into BGP yet. Onward to our eBGP!

OK, so again R1 will peer to AS 7018, R8 to AS 701 , and R9 to AS 3561. Let’s also redistribute our OSPF stuff into BGP so that these AS’ know about our internal AS routes.

On R1:

router bgp 1001
neighbor 100.100.100.100 remote-as 7018
redistribute ospf 1
1d06h: %BGP-5-ADJCHANGE: neighbor 100.100.100.100 Up

On R8:

router bgp 1001
neighbor 100.100.200.200 remote-as 701
redistribute ospf 1
*Mar 22 01:40:45.099: %BGP-5-ADJCHANGE: neighbor 100.100.200.200 Up

On R9:

router bgp 1001
neighbor 100.100.250.250 remote-as 3561
redistribute ospf 1
*Nov 21 01:45:40.389: %BGP-5-ADJCHANGE: neighbor 100.100.250.250 Up

OK excellent…lets do some checking to make sure we are receiving prefixes.

R1(config-router)#do sh ip bgp sum | i 100.100.100.100|200.0.0.8|200.0.0.9
100.100.100.100 4 7018 11 8 117 0 0 00:04:35 116
200.0.0.8 4 1001 13 15 117 0 0 00:09:27 119
200.0.0.9 4 1001 14 15 117 0 0 00:09:03 122

R8(config-router)#do sh ip bgp sum | i 100.100.200.200|200.0.0.1|200.0.0.9
100.100.200.200 4 701 9 7 120 0 0 00:03:41 119
200.0.0.1 4 1001 16 14 120 0 0 00:10:01 116
200.0.0.9 4 1001 13 13 120 0 0 00:09:59 122

R9(config-router)#do sh ip bgp sum | i 100.100.250.250|200.0.0.1|200.0.0.8
100.100.250.250 4 3561 8 6 123 0 0 00:02:56 122
200.0.0.1 4 1001 15 14 123 0 0 00:10:00 116
200.0.0.8 4 1001 14 14 123 0 0 00:10:21 119

Excellent, so we are getting a ton of routes from our backbone routers. Before we get too deep, lets make sure we solve that whole don’t become a transit AS problem. There are a number of ways to accomplish that, including distribute-lists, prefix-lists, filter-lists, and communities. I choose communities, because frankly it is easy to do.

First, we need to set the routers to use the new community format. If we don’t do this, we’ll see some large strange 32 bit number in the form of an integer for our communities, and that is no fun. Then we have to tag our incoming routes from the BB routers as no-export.

On R1,R8,R9:

ip bgp new-format
route-map NoTransit permit 10
set community no-export

Now apply the route-map to our eBGP peerings, and make sure we send the no-export community to our iBGP peers…

On R1:

router bgp 1001
neighbor 100.100.100.100 route-map NoTransit in
neighbor 200.0.0.8 send-community
neighbor 200.0.0.9 send-community

On R8:

router bgp 1001
neighbor 100.100.200.200 route-map NoTransit in
neighbor 200.0.0.1 send-community
neighbor 200.0.0.9 send-community

On R9:

router bgp 1001
neighbor 100.100.250.250 route-map NoTransit in
neighbor 200.0.0.1 send-community
neighbor 200.0.0.8 send-community

Let’s go ahead and clear our BGP peers on all our router’s just in case…

On R1,R8,R9:
do clear ip bgp * soft

OK, let’s make sure this is working…here is the beginning of our BGP table on R1. Lets just grab the first route there we are learning from BB1 to make sure it is tagged end to end.

R1(config-router)#do sho ip bgp
BGP table version is 233, local router ID is 200.0.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete

Network Next Hop Metric LocPrf Weight Path
*> 64.58.0.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.58.16.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.59.128.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.59.144.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.63.160.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.63.176.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.69.0.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.69.16.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.70.96.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.70.112.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.75.64.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.75.112.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.76.32.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.76.160.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.77.176.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.77.224.0/20 100.100.100.100 0 0 7018 20340 ?
*> 64.77.240.0/20 100.100.100.100 0 0 7018 20340 ?
–More–

R1(config-router)#do sh ip bgp 64.58.0.0
BGP routing table entry for 64.58.0.0/20, version 123
Paths: (1 available, best #1, table Default-IP-Routing-Table, not advertised to EBGP peer)
Advertised to non peer-group peers:
200.0.0.8 200.0.0.9
7018 20340
100.100.100.100 from 100.100.100.100 (209.202.63.200)
Origin incomplete, metric 0, localpref 100, valid, external, best
Community: no-export

OK, it is good on R1…now let’s make sure we see it tagged as no-export way over on R9.

R9(config-router)#do sh ip bgp 64.58.0.0
BGP routing table entry for 64.58.0.0/20, version 0
Paths: (1 available, no best path)
Not advertised to any peer
7018 20340
100.100.100.100 (inaccessible) from 200.0.0.1 (200.0.0.1)
Origin incomplete, metric 0, localpref 100, valid, internal
Community: no-export

OK so bittersweet…our no-export community went through, so we are good. You would definitely want to go ahead and check at least a few routes from each peer end to end, but for sake of the length of this post I am not going to show that today. The bad part is that on R9 we see a problem. A big problem. Basically none of the routes that we learned from R1 are getting put into the routing table. If we do a little more digging we will see the same problem on all the routers. Any routes that are learned via iBGP are showing up as valid, but not best and as inaccessible. hmmmmmmm

So let’s take a close look at that again:

R9(config-router)#do sh ip bgp 64.58.0.0
BGP routing table entry for 64.58.0.0/20, version 0
Paths: (1 available, no best path)
Not advertised to any peer
7018 20340
100.100.100.100 (inaccessible) from 200.0.0.1 (200.0.0.1)
Origin incomplete, metric 0, localpref 100, valid, internal
Community: no-export

So what’s happening here? BB1 adveritses 64.58.0.0/22 to R1, next hop 100.100.100.100. R1 passes the route to R8 and R9 via iBGP. Remember in iBGP the next hop will not change!!! So, now at R8/R9 we have 64.58.0.0/22 next hop 100.100.100.100…OK no problem…wait. Who the hell is 100.100.100.100?!!!

R9(config-router)#do sh ip route 100.100.100.100
% Subnet not in table

Ahhhhhhhh, good old next-hop reachability. So R8 and R9 have the route, but they have no idea how to reach the next hop to get there and therefore, the routes are not best, and will not be placed in the routing table. We will see the same exact problem on R1 for routes eBGP learned routes at R8,R9 and on R8 for eBGP learned routes at R1,R9 … Let’s go ahead and fix this business by bringing those backbone connected links into OSPF on all 3 of our routers.

R1(config)#router ospf 1
R1(config-router)#network 100.100.100.0 0.0.0.255 area 0

R8(config-router)#router ospf 1
R8(config-router)#network 100.100.200.0 0.0.0.255 area 0

R9(config-router)#router ospf 1
R9(config-router)#network 100.100.250.0 0.0.0.255 area 0

NOW let’s take a look at that same route on R9!

R9(config-router)#do sh ip bgp | i 64.58.0.0
*>i64.58.0.0/20 100.100.100.100 0 100 0 7018 20340 ?

R9(config-router)#do sho ip route | i 64.58.0.0
B 64.58.0.0 [200/0] via 100.100.100.100, 00:01:56

Excellent! So now we have the route in the BGP table as valid AND best, and therefore it has been placed into our routing table! So we should have reachability now! Lets try this out. All of the BB injected routes have an address of .200.

R9(config-router)#do ping 64.58.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 64.58.0.200, timeout is 2 seconds:
U.U.U
Success rate is 0 percent (0/5)

uhhhhh…hmmmm….uhhoh! sooo…we have the route…we have a route to the next hop…WTF is going on here?! Let’s do a trace

R9(config-router)#do trace 64.58.0.200

Type escape sequence to abort.
Tracing the route to 64.58.0.200

1 150.100.69.6 16 msec 20 msec 16 msec
2 150.100.69.6 !H * !H

So our ping goes from R9 over to R6 and immediately takes a dump. Let’s think this through. So R9 wants to ping 64.58.0.200. R9 check’s it’s routing table and finds a BGP route to 64.58.0.0/22 with a next hop of 100.100.100.100. OK, so now it has to figure out how to get to 100.100.100.100. We check our routing table again and see a perfectly good OSPF route to 100.100.100.0/24 through R6. R9 encapsulates our ping packet into a PPP frame and sends it on over to R6. Keep in mind our source and destination IP addresses have NOT changed. R6 gets the packet, source: 150.100.69.9 destination: 64.58.0.200. R6 checks it’s routing table for 64.58.0.200. Bzzzzzzzzzzzzzz, thanks for playing!!!

R6(config-router)#do sh ip route 64.58.0.0
% Network not in table

So, the long and short of the problem is that we do indeed have the route…and we have a valid route to the next hop…but the next hop (R6 in this case) does not have a route to the destination.

What to do, what to do… we could redistribute our BGP into OSPF at each edge router! Bah..well for one we aren’t allowed to do that here. For 2, if we did that in the real world it would be very very bad…especially if we were taking full internet tables from our eBGP peers. We’d likely crash our OSPF. So, imagine you are sitting in the CCIE lab and find yourself here. You have perfectly valid routes to all these external networks on your BGP routers, but cannot ping them. You ask the proctor if you need to just have valid and best paths or if you need actual reachability. He smiles at you and says you need to have reachability….yet your lab says you cannot redistribute either. Shit…

So what is another solution…what if we pump out a default route into OSPF from each edge router? Well that would sort of work, but we’d selectively blackhole a bunch of routes, and have a bunch of routing loops. So that is out. So we need a way for our next hop router to have a route to the final destination. Take R9 for example. When it looks up 100.100.100.100 because it is the next hop, and sends it along, whatever router gets that next should have an actual route to the end destination…so in other words the next hop should probably be R1 in this case. GRE Tunnels!

What we want to do is set up some tunnels between R1,R8,R9 and put them into OSPF. So lets get going on that.

R1(config)#int tunnel0
R1(config-if)#ip address 19
R1(config-if)#ip address 192.168.0.1 255.255.255.0
R1(config-if)#tunnel source 200.0.0.1
R1(config-if)#tunnel destination 200.0.0.8
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

R1(config-if)#int tunnel1
R1(config-if)#ip address 192.1
R1(config-if)#ip address 192.168.1.1 255.255.255.0
R1(config-if)#tunnel source 200.0.0.1
R1(config-if)#tunnel destination 200.0.0.9
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up

R1(config-if)#router ospf 1
R1(config-router)#network 192.168.0.0 0.0.255.255 area 0

R8(config)#int tunnel0
R8(config-if)#ip address 1
R8(config-if)#ip address 192.168.0.8 255.255.255.0
R8(config-if)#tunnel source 200.0.0.8
R8(config-if)#tunnel destination 200.0.0.1
*Mar 22 02:46:51.893: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

R8(config-if)#int tunnel 1
R8(config-if)#ip address 1
R8(config-if)#ip address 192.168.2.8 255.255.255.0
R8(config-if)#tunnel source 200.0.0.8
R8(config-if)#tunnel destination 200.0.0.9
R8(config-if)#
*Mar 22 02:47:24.707: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up

R8(config-if)#router ospf 1
R8(config-router)#network 192.168.0.0 0.0.255.255 area 0

R9(config-router)#interface tunnel0
R9(config-if)#ip address 192.1
R9(config-if)#ip address 192.168.1.9 255.255.255.0
R9(config-if)#tunnel source 200.0.0.9
R9(config-if)#tunnel dest 200.0.0.1
*Nov 21 02:52:38.628: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up

R9(config-if)#interface tun1
R9(config-if)#ip address 192.16
R9(config-if)#ip address 192.168.2.9 255.255.255.0
R9(config-if)#tunnel source 200.0.0.9
R9(config-if)#tunnel dest 200.0.0.8
*Nov 21 02:53:00.119: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up

R9(config-if)#router ospf 1
R9(config-router)#network 192.168.0.0 0.0.255.255 area 0

OK, so here is a peek at our routers after setting all this up…

R1(config-router)#do sh ip ospf neigh | i Tunnel
200.0.0.9 0 FULL/ – 00:00:36 192.168.1.9 Tunnel1
200.0.0.8 0 FULL/ – 00:00:39 192.168.0.8 Tunnel0

R8(config-router)#do sh ip ospf neigh | i Tunnel
200.0.0.9 0 FULL/ – 00:00:36 192.168.2.9 Tunnel1
200.0.0.1 0 FULL/ – 00:00:29 192.168.0.1 Tunnel0

R9(config-router)#do sh ip ospf neigh | i Tunnel
200.0.0.8 0 FULL/ – 00:00:38 192.168.2.8 Tunnel1
200.0.0.1 0 FULL/ – 00:00:38 192.168.1.1 Tunnel0

So, the $25,000 question right now is did this help us? Lets check out R9 again

R9(config)#do sh ip route 100.100.100.100
Routing entry for 100.100.100.0/24
Known via “ospf 1”, distance 110, metric 148, type intra area
Last update from 150.100.69.6 on Serial0/0, 00:02:49 ago
Routing Descriptor Blocks:
* 150.100.69.6, from 200.0.0.1, 00:02:49 ago, via Serial0/0
Route metric is 148, traffic share count is 1

Well, we are STILL learning the next hop through OSPF via R6 … our cost is 148. So, lets make the path via our tunnels a better cost. We’ll be safe and say how about….. 25 ?

On R1,R8,R9:

int tun0
ip ospf cost 25

int tun1
ip ospf cost 25

1d07h: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.8 on Tunnel0 from FULL to DOWN, Neighbor Down: Interface down or detached

1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.9 on Tunnel1 from FULL to DOWN, Neighbor Down: Dead timer expired
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.9 on Tunnel1 from LOADING to FULL, Loading Done
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.8 on Tunnel0 from LOADING to FULL, Loading Done
1d07h: %TUN-5-RECURDOWN: Tunnel1 temporarily disabled due to recursive routing
1d07h: %TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to down
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.9 on Tunnel1 from FULL to DOWN, Neighbor Down: Interface down or detached
1d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
1d07h: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.8 on Tunnel0 from FULL to DOWN, Neighbor Down: Interface down or detached

Ohhhhh boy….we’ll see this on each of the routers. So we set the cost of our tunnels pretty damn low…what’s happening is that we are learning the route to the destination of our tunnels THROUGH THE TUNNELS…that would be bad. Therefore, we see the tunnel go up, down, up, down, forever. Bah!

To fix this issue, we’ll make sure all the tunnel endpoints are reachable by some other means…namely static routes.

R1(config)#ip route 200.0.0.8 255.255.255.255 150.100.12.2
R1(config)#ip route 200.0.0.9 255.255.255.255 150.100.12.2

R8(config)#ip route 200.0.0.1 255.255.255.255 150.100.78.7
R8(config)#ip route 200.0.0.9 255.255.255.255 150.100.78.7

R9(config)#ip route 200.0.0.1 255.255.255.255 150.100.69.6
R9(config)#ip route 200.0.0.8 255.255.255.255 150.100.69.6

OK how about NOW? Lets check out R9 again

R9(config)#
*Nov 21 03:10:30.481: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.1 on Tunnel0 from FULL to DOWN, Neighbor Down: Dead timer expired
R9(config)#
*Nov 21 03:10:37.528: %OSPF-5-ADJCHG: Process 1, Nbr 200.0.0.1 on Tunnel0 from LOADING to FULL, Loading Done
R9(config)#do sh ip route 200.0.0.1
Routing entry for 200.0.0.1/32
Known via “static”, distance 1, metric 0
Routing Descriptor Blocks:
* 150.100.69.6
Route metric is 0, traffic share count is 1

R9(config)#do ping 200.0.0.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 200.0.0.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)

Ugh…so we aren’t getting the recursive routing anymore, but our tunnels are still flapping and we can’t ping the tunnel end point. Well we kind of set that ospf cost of the tunnels too low…what is happening is that R9 is trying to get to the tunnel end point at 200.0.0.1…it has a static route to 200.0.0.1 pointing to R6 so it sends it there. R6 looks up 200.0.0.1 and because the cost of the tunnel was so low, the path back to R9 and over the tunnel is actually still LOWER than the regular path through the network..so what we end up with is a routing loop. We need to VERY carefully choose our tunnel ospf costs to make this work. So lets figure out the costs through the regular network, then set the tunnels just slightly lower.

Cost of R1 –> R8: R1 –> R2 via Ethernet = 10
R2 –> R6 via Frame = 64
R6 –> R7 via FastEthernet = 1
R7 –> R8 via Serial = 64
R8 Loopback = 1
——————————-
Total Cost = 140

Cost of R1 –> R9: R1 –> R2 Ethernet = 10
R2 –> R6 Frame = 64
R6 –> R9 PPP = 64
R9 Loopback = 1
————————
Total Cost = 139

Cost of R8 –> R9: R8 –> R7 via Serial = 64
R7 –> R6 via FastEthernet = 1
R6 –> R9 via Serial = 64
R9 Loopback = 1
————————-
Total Cost = 130

So lets set our tunnel costs just slightly lower:

R1(config)#int tun0
R1(config-if)#ip ospf cost 135
R1(config-if)#int tun1
R1(config-if)#ip ospf cost 135

R8(config)#int tun0
R8(config-if)#ip ospf cost 135
R8(config-if)#int tun1
R8(config-if)#ip ospf cost 120

R9(config)#int tun0
R9(config-if)#ip ospf cost 135
R9(config-if)#int tun1
R9(config-if)#ip ospf cost 120

Finally…let’s check it out on R9.

R9(config-router)#do sh ip route 100.100.100.0
Routing entry for 100.100.100.0/24
Known via “ospf 1”, distance 110, metric 145, type intra area
Redistributing via bgp 1001
Advertised by bgp 1001
Last update from 192.168.1.1 on Tunnel0, 00:06:41 ago
Routing Descriptor Blocks:
* 192.168.1.1, from 200.0.0.1, 00:06:41 ago, via Tunnel0
Route metric is 145, traffic share count is 1

R9(config-router)#do ping 64.58.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 64.58.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 120/123/125 ms
R9(config-router)#

BOOOOOOOOOYAH!

Now, lets just make it work from EVERY router in our AS. We’ll pump out a default route from our 3 edge routers. Now it won’t cause serious loop issues or blackholes, because no matter which one of the edge routers the packet gets to, each of them have all of the BGP routes via iBGP. Excellent

R1(config-router)#router ospf 1
R1(config-router)#default-information originate always

R8(config-router)#router ospf 1
R8(config-router)#default-information originate always

R9(config-router)#router ospf 1
R9(config-router)#default-information originate always

Moment of truth. Here are 3 routes we learned via our outside AS’ … one from each. We’ll go to a random internal router that is only speaking OSPF and try each one.

From BB1: 64.58.0.200
From BB2: 202.0.0.200
From BB3: 102.6.0.200

R4#ping 64.58.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 64.58.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 68/68/72 ms
R4#
R4#ping 202.0.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 202.0.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 176/177/184 ms

R4#ping 102.6.0.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 102.6.0.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 176/176/180 ms

Bickety Bam…full reachability…no BGP –> OSPF redistribution

– Joe A

Well my initial flight took off today on time, what I didn’t know was that one of the front tires blew out on take off. We had to turn around and land back at Buffalo because when the tire blew it loosened the landing gear cover. Landing on one wheel didn’t seem that bad until the second wheel blew and we started to come to a complete stop :) At least we came to a complete stop. The pilot and crew were excellent the whole time. Everything seemed normal….

The bad part was I missed my connecting flight and there was no way to get out until either really late tonight or sometime tomorrow. With everyone scrambling to get flights everywhere, I just decided I was better off rescheduling. I only had until 4:00 pm to cancel my hotel reservation and shuttle service or I was commited to them. If I got into California late tomorrow night Monday would be a complete waste at the camp.

I can’t believe I did all that packing and planning this week for nothing! Everything turned out well though so I can’t really complain, it could of been a lot worse :)

I have to admit, I’m pretty frustrated right now. It’s 11:30 at night…here I sit. I’ve spent the last 2 nights working on IPexpert Volume 1 Lab 11: BGP. I knew it was coming, dreaded it, feared it, and decided to just jump in, the water was warm!

I was in for the beating I expected. Still, after a rough first night, and spending most of today pondering a particular problem I just was not grasping, I finally understood it. I was about halfway through the lab and pretty much understood everything — Until tonight.

Is it just me, or is anybody else out there having occasional “issues” with the startup-configurations provided to you from IPexpert? I’ve noticed this before, but didn’t say anything because frankly hey it is a lot of information and there are bound to be tiny mistakes. But tonight seemed to present some fairly large issues. I mean here I am plowing through this material, I have enough things to worry about without having to worry about my configurations that I’m supposed to be learning from all jacked up. I’ll try to explain this.

grrrrrrrr…. a network diagram is necessary. Let me go make one quick…

So here is the deal… first off I should mention all routers have a loopback that is advertised into BGP of 200.0.0.x where x is the router number. simple enough. Also, R2,R5,R6 are a frame-relay cloud, whereby R2 is the Hub. Also, the diagram above shows BGP links, not physical links necessarily.

So I am about halfway through this lab, and I start pinging around for reachability to all my loopbacks. I am on R1 trying to ping 200.0.0.9 (R9’s loopback). It doesn’t work. I do a traceroute, and discover I have a routing loop. The packet goes from R1 to R2 to R5 to R2 to R5 to R2 …..OK what’s going on here?

The neighbor on R5 to R2 is set for next-hop-self. This was part of the lab solution, because without it we had no reachability to R7,R8, and anything injected via BB2. So, R9 advertises 200.0.0.9 via eBGP to R6. The neighbor relationship between R6 and R5 is also set for next-hop-self. So, now R6 sends the route to R5. R5 gets the route to 200.0.0.9, next hop R6. No problems. R5 advertises 200.0.0.9 to R2 with next-hop-self. Now R2 has a route to 200.0.0.9 next hop R5. Great.

So lets take this from the perspective of pinging from R2 to R9. On R2 you ping 200.0.0.9…OK it is in my routing table, next hop R5, send to R5. R5 gets the packet, looks at the destination IP address of 200.0.0.9…see’s that 200.0.0.9 is in it’s routing table, next hop R6. Here is the problem — R5 now does a recursive route lookup for the next hop address of R6. R5 and R6 are both spokes in a frame-relay cloud and R2 is the hub. For R5 to send to R6 is HAS to go to R2 (No spoke/spoke PVCs here boys!) R5 sends to R2, and boom we have a loop.

I beat my head into the wall for a good part of today going over the proctor guide, trying to see had I missed something? Have I gone insane?! Finally, I loaded the “final configurations” given by IPexpert and reloaded.

To my horror, after loading the “final configurations”, upon reboot I had a serious issue between R5 and R6. They were both complaining of “BGP identifier wrong” coming from the other side…as a 32 bit number (weird!) So I checked the ASNs configured on both ends, as well as the neighbor commands…all looked good. So the problem was that there was a task somewhere in this lab that told you to configure the same loopback addresses on both routers and advertise them into BGP (They were all /24 loopbacks). When the routers reboot, these loopbacks were the highest ones on the routers, and thus became the BGP RID. Both routers therefore had the same RID. None of this was mentioned in the lab solutions at all.

So after all that hoo-ha , the moment of truth… I am on R2 and attempt to ping R9 after loading the final configurations given to me…nothing. Same exact problem (routing loop). So, I had not missed something, it was just flat out wrong (if they expected full reachability). At this point I was pretty much too frustrated to continue for the night.

I did however play around a little more just for shits and giggles. So I did a little playing around on R5 just to see if I could make it work

R5
——
access-list 101 permit host 200.0.0.6
access-list 101 permit host 200.0.0.9

route-map wtf 10
match ip address 101
set ip next-hop 200.0.0.7

router bgp 64513
neighbor 150.100.100.6 route-map wtf in

clear ip bgp 150.100.100.6

Poof…reachability to R9 from R1 : )

*UPDATE* ….so how could I make this work for ANY networks advertised to R5 with a next hop of R6 ? The newest solution, quite a bit more scalable below! So this as-path access-list matches anything learned from subAS 64514. the \ characters are used in the regular expression to escape the parens…not sure if it is technically required or not, but this does work! So, now anything that R6 sends to R5 with next hop of R6 will be changed to a next hop of R7. So basically anything learned from R6 at R5 will now go back to R6 via the ethernet link to R7 instead of over the frame, thus defeating the evil loop issue.

ip as-path access-list 19 permit ^\(64514\)_

route-map wtf 10
match as-path 19
set ip next-hop 200.0.0.7

Wow, kind of a long post tonight. So what’s the point? I guess I just wanted to bitch a little about something :) This lesson has just been a bit frustrating so far. I knew BGP was going to be a struggle for me, because I have really only read about it before, but I was not expecting major problems from the study material out of the box. The other day during OSPF I had a similar issue with the startup configurations…as in major issues that I had to go in and dick with to get the practice scenarios given to work properly. For instance, sometimes there have been things missing in the startup configurations that are needed, yet those commands are not given in the solutions guide…but then are magically in the final configurations given.

Well, if nothing else, it is now 12:15 AM, I’ve plowed heavily into BGP for 2 nights now…although much of this fooling around has been due to what I perceive here as errors in the study material, at least it has been time spent on a router, and at least I have learned something and understand WHY it is broken, and was even able to make it work.

Regards,

Joe A

Well I have been pounding away on real equipment since I finished building my home lab! I have had my entire rack powered up 24/7 for about a week haha…allows for remote access whenever I need it. Should be an interesting power bill.Â

So, since getting my 28U rack in the mail and my other 2 3550 switches, I have been pounding on IPexpert volume 1 labs (technology specific)Â There are 22 labs in total, each one covering a specific technology.Â

Usually what happens is I read over the lab, and the first time really get my ass kicked hard … I typically have been using the proctor guide quite a bit the first time through to get the concepts down. After I go through and complete the entire lab, I wipe all my configs, and start over…I redo the lab 1 or 2 more times trying not to use the proctor guide at all. The good news is that I see to be doing OK on the 2nd and 3rd runs through. My memory has always been pretty decent…but as Scott Morris says, this is sort of bittersweet…one one hand you are doing well, but on the other hand, you are sort of memorizing the labs. I have made a specific point to really grasp and understand the solutions to each problem, so I am just not spewing out commands from memory.Â

I just finished my 3rd run through of the OSPF lab (lab # 10) and am quite honestly pretty much dreading lab 11 (BGP). I spent a lot of the last 3 or 4 days on this OSPF lab so it breaks my heart to have to write erase all those configs hahaha…

I have to say, there is absolutely NOTHING like having your own equipment and doing this stuff for real. I mean, I have read so much material, but until you actually go in and start doing it, it is not going to really sink in and stick (at least for me). For instance, I had read about route authentication in RIP, EIGRP, and OSPF before a hundred times, but I had never configured it for real. Now that I have actually typed the commands, the process has stuck in my head much more, and I know from experience what to look out for

speaking of startup-configs, I grew tired of trying to copy/paste text file configs into all my devices. Half the time, not all the configuration would get accepted…and I wouldn’t know until half way through the lab when something was broken. So, I’ve actually used some of this crazy CCIE knowledge to do something sort of neat.Â

Basically, I have my home network …. and 20 feet away is my rack. I wanted to connect my home network switch to the lab rack so I could have remote access and also the ability to TFTP configs from my home unix server to the lab, to avoid the copy/paste stuff. So, I have a long ethernet cable going from my home switch to the ethernet interface of the AS. The switch port on the home network is in it’s own little VLAN, which is routed by a 3725 router on a stick.Â

So the issue is that while I had connectivity to the AS, the AS was not in any way networked to the lab equipment from an IP perspective (only with the octal cables for console). All the other ethernet interfaces in the lab environment were used…OK I’ll go from a 3550 port back to the home switch instead…sorry I only have straight thru cables long enough, no crimper or that much inspiration to make my own crossover cable…solution?

Bridging! So now I have home switch –> AS ethernet –> R2 (over serial interface) … then I just bridge on the AS and on R2. Then I have a cat tools script that goes through all my lab devices, erases the configs, and gives them an IP address on the same vlan as the home switch and boom I have connectivity back to the home network to pull configs.