4 Label Stack in JunOS

Disclaimer - This 4 label stack has nothing to do with SR-MPLS or RSVP Pop&Forward

We are going to go full nerd with this one and exceed the “normal” 3 label stack we might have been used to with something like seamless MPLS. Today, we are shooting for VPN,BGP-LU,LDP,RSVP stack bottom to top. We are doing L3VPN Interprovider Option C, with LDP Tunneling over RSVP (LDPoRSVP).

What’s the point? Just to see how JunOS reacts to the label stack size larger than 3 by default for vMX routers. Also - for a core leveraging LDPoRSVP or doing some flavor of seamless MPLS and adding on some interprovider VPN this is a completely realistic scenario.


Topology:

What I’ve already configured

I’ve already configured RSVP LSP’s between PE1 and P3, and LDP tunneling over these LSPs. I’ve also configured the IP routing in 64510 via OSPF backbone area 0 and in 64511 as ISIS single level 2 area. P4 will advertise its loopback /32 into LDP, and P3 will pass this along as an advertisement as well as its own /32 via targeted LDP session to PE1. PE1 has LDP tunneling configured for the RSVP LSP from PE1->P3, so this will be viewed as the best next hop and the LDP routes for 192.168.1.4/32 and 192.168.1.5/32 will be tossed into inet.3.

Keep in mind LDP must always follow IGP next-hop, and you can see configuring LDP-tunneling in JunOS enables OSPF shortcuts in the background as hidden routes

[email protected]# run show ldp database
Input label database, 192.168.1.1:0--192.168.1.3:0
Labels received: 3
  Label     Prefix
 299792      192.168.1.1/32
      3      192.168.1.3/32
 299776      192.168.1.4/32

Output label database, 192.168.1.1:0--192.168.1.3:0
Labels advertised: 3
  Label     Prefix
      3      192.168.1.1/32
 299776      192.168.1.3/32
 299792      192.168.1.4/32

[email protected]# run show route table inet.3 hidden
inet.3: 3 destinations, 6 routes (2 active, 0 holddown, 3 hidden)
+ = Active Route, - = Last Active, * = Both

100.64.1.4/31       [OSPF] 00:10:34, metric 3
                    >  to 100.64.1.1 via ge-0/0/0.0, label-switched-path to-r3
192.168.1.3/32      [OSPF] 00:10:34, metric 2
                    >  to 100.64.1.1 via ge-0/0/0.0, label-switched-path to-r3
192.168.1.4/32      [OSPF] 00:10:34, metric 3
                    >  to 100.64.1.1 via ge-0/0/0.0, label-switched-path to-r3

[email protected]# run show route table inet.3 192.168.1.4
inet.3: 3 destinations, 6 routes (2 active, 0 holddown, 3 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.1.4/32     *[LDP/9] 00:11:08, metric 1
                    >  to 100.64.1.1 via ge-0/0/0.0, label-switched-path to-r3

What’s next?

We have an operational LDPoRSVP LSP between PE1 and P4. We also have LDP operating in AS64511. We need to configure eBGP-LU, as well as iBGP-LU sessions in both AS’s.

On P4:

[edit protocols]
+   bgp {
+       family inet {
+           labeled-unicast;
+           rib {
+           inet.3;
+           }
+       }
+       group ebgp {
+           type external;
+           peer-as 64511;
+           neighbor 100.64.2.1;
+       }
+       group ibgp {
+           type internal;
+           local-address 192.168.1.4;
+           neighbor 192.168.1.1;
+       }
+   }

On P5:

[edit protocols]
+   bgp {
+       family inet {
+           labeled-unicast;
+      		 rib {
+          		 inet.3;
+       	 } 
+       }
+       group ebgp {
+           type external;
+           peer-as 64510;
+           neighbor 100.64.2.0;
+       }
+       group ibgp {
+           type internal;
+           local-address 192.168.1.5;
+           neighbor 192.168.1.6;
+       }
+   }

On PE1:

[edit protocols]
+   bgp {
+       group ibgp {
+           type internal;
+           local-address 192.168.1.1;
+           family inet {
+               labeled-unicast;
+      		 	rib {
+           		inet.3;
+       		}
+           }
+           neighbor 192.168.1.4;
+       }
+   }

On PE6:

[edit protocols]
+   bgp {
+       group ibgp {
+           type internal;
+           local-address 192.168.1.6;
+           family inet {
+               labeled-unicast;
+       		rib {
+           		inet.3;
+       	}
+           }
+           neighbor 192.168.1.5;
+       }
+   }

Why did I choose “rib inet.3” under family inet labeled-unicast? Chris covers this really well in this post about BGP-LU. By keeping the BGP families inet unicast and inet labeled-unicast separate, we can learn and advertise prefixes from/to both tables without issue. Because of the likelihood of this happening in the real world, I’ve chosen to keep labeled advertisements in inet.3 RIB.

My iBGP sessions are established, but I need to get the PE1 loopback into inet.3 so I can actually advertise it. Because remember, that’s the RIB I’m using on every router for push/pull of labeled-unicast prefixes. I’m going to use a rib-group and interface-routes to do this.

[edit routing-options]
+   interface-routes {
+       rib-group inet copy-lo0;
+   }
+   rib-groups {
+       copy-lo0 {
+           import-rib [ inet.0 inet.3 ];
+       }
+   }

And then we apply export policy to ibgp groups on both PE1 and PE6 to get the loopbacks into iBGP

[edit]
+  policy-options {
+      policy-statement export-lo0 {
+          from interface lo0.0;
+          then accept;
+      }
+  }
[edit protocols bgp group ibgp]
+    export export-lo0;


[edit]
[email protected]# run show route advertising-protocol bgp 192.168.1.4 detail

inet.3: 6 destinations, 9 routes (5 active, 0 holddown, 3 hidden)
* 192.168.1.1/32 (1 entry, 1 announced)
 BGP group ibgp type Internal
     Route Label: 3
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [64510] I
     Entropy label capable

P5 is receiving the route for 192.168.1.1/32 with a label in the other AS64511

[email protected]# run show route receive-protocol bgp 100.64.2.0 detail
inet.3: 2 destinations, 2 routes (1 active, 0 holddown, 1 hidden)
* 192.168.1.1/32 (1 entry, 1 announced)
     Accepted
     Route Label: 299824
     Nexthop: 100.64.2.0
     AS path: 64510 I Unrecognized Attributes: 11 bytes
     AS path:  Attr flags e0 code 1c: 00 01 04 04 c0 a8 01 01

But P5 isn’t advertising this route to PE6, which is a problem

[email protected]# ...show route advertising-protocol bgp 192.168.1.6 detail
inet.3: 2 destinations, 2 routes (1 active, 0 holddown, 1 hidden)
* 192.168.1.1/32 (1 entry, 1 announced)
 BGP group ibgp type Internal
     BGP label allocation failure: family mpls not enabled on interface
     Nexthop: Not advertised
     Flags: Nexthop Change
     Localpref: 100
     AS path: [64511] 64510 I Unrecognized Attributes: 11 bytes
     AS path:  Attr flags e0 code 1c: 00 01 04 04 c0 a8 01 01

And the reason given is “nexthop: not advertised”.. A little cryptic. Well, the real reason is that MPLS is not enabled on the inter-AS link, and it needs to be to allow for MPLS traffic to forward on the link.

[edit interfaces ge-0/0/3 unit 0]
+      family mpls;
[email protected]# run show route advertising-protocol bgp 192.168.1.6 detail

inet.3: 2 destinations, 2 routes (1 active, 0 holddown, 1 hidden)
* 192.168.1.1/32 (1 entry, 1 announced)
 BGP group ibgp type Internal
     Route Label: 299776
     Nexthop: Self
     Flags: Nexthop Change
     Localpref: 100
     AS path: [64511] 64510 I Unrecognized Attributes: 11 bytes
     AS path:  Attr flags e0 code 1c: 00 01 04 04 c0 a8 01 01

PE1 has a route in inet.3 for PE6 and vice-versa

[email protected]# run show route protocol bgp

inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)

inet.3: 7 destinations, 10 routes (6 active, 0 holddown, 3 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.1.6/32     *[BGP/170] 00:00:06, localpref 100, from 192.168.1.4
                      AS path: 64511 I, validation-state: unverified
                    >  to 100.64.1.1 via ge-0/0/0.0, label-switched-path to-r3

The Problem is that we need this route in inet.0 to establish iBGP peering between the two. Rib-group to the rescue again for this simple operation.

[email protected]# show | compare
[edit routing-options rib-groups]
    copy-lo0 { ... }
+   copy-ebgplu-to-inet0 {
+       import-rib [ inet.3 inet.0 ];
+   }
[edit protocols bgp group ibgp family inet labeled-unicast]
+        rib-group copy-ebgplu-to-inet0;

[email protected]# run show route protocol bgp
inet.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.1.6/32     *[BGP/170] 00:00:08, localpref 100, from 192.168.1.4
                      AS path: 64511 I, validation-state: unverified
                    >  to 100.64.1.1 via ge-0/0/0.0, label-switched-path to-r3

inet.3: 7 destinations, 10 routes (6 active, 0 holddown, 3 hidden)
+ = Active Route, - = Last Active, * = Both

192.168.1.6/32     *[BGP/170] 00:02:33, localpref 100, from 192.168.1.4
                      AS path: 64511 I, validation-state: unverified
                    >  to 100.64.1.1 via ge-0/0/0.0, label-switched-path to-r3
[email protected]# run traceroute 192.168.1.6 source 192.168.1.1
traceroute to 192.168.1.6 (192.168.1.6) from 192.168.1.1, 30 hops max, 52 byte packets
 1  100.64.1.1 (100.64.1.1)  11.708 ms  8.978 ms  12.350 ms
     MPLS Label=299776 CoS=0 TTL=1 S=0
     MPLS Label=299776 CoS=0 TTL=1 S=0
     MPLS Label=299856 CoS=0 TTL=1 S=1
 2  100.64.1.3 (100.64.1.3)  14.698 ms  13.704 ms  15.115 ms
     MPLS Label=299776 CoS=0 TTL=1 S=0
     MPLS Label=299856 CoS=0 TTL=2 S=1
 3  100.64.1.5 (100.64.1.5)  22.004 ms  18.510 ms  19.497 ms
     MPLS Label=299856 CoS=0 TTL=1 S=1
 4  100.64.2.1 (100.64.2.1)  28.827 ms  28.709 ms  29.727 ms
     MPLS Label=299808 CoS=0 TTL=1 S=1
 5  192.168.1.6 (192.168.1.6)  22.977 ms  24.452 ms  24.528 ms

And connectivity between the two PE’s is looking fine.


As you saw in that last traceroute from PE1->PE6, that was already a 3-label-push operation. The bottom label was the BGP-LU label, the middle was the LDP label for the P4 router, and the RSVP label was the top label that gives us LSP reachability to P4. So what’s one more label? No problem, we are almost there.

For simplicity, I’m using a couple direct subnets and vrf-table-label for the L3VPN. We’ve done enough configuration, it’s time to be a little lazy to hurry and see this 4 label stack at work.

PE1:

 [edit interfaces lo0]
+    unit 1 {
+        family inet {
+            address 11.11.11.11/32;
+        }
+    }
[edit]
+  routing-instances {
+      test {
+          instance-type vrf;
+          interface lo0.1;
+          vrf-target target:1:1;
+          vrf-table-label;
+      }
+  }
[edit routing-options]
+  route-distinguisher-id 192.168.1.1;
[edit protocols bgp]
     group ibgp { ... }
+    group ebgp-vpn {
+        local-address 192.168.1.1;
+        multihop;
+        family inet-vpn {
+            unicast;
+        }
+        peer-as 64511;
+        neighbor 192.168.1.6;
+    }

PE6:

[email protected]# show | compare | no-more
[edit interfaces lo0]
+    unit 1 {
+        family inet {
+            address 66.66.66.66/32;
+        }
+    }
[edit]
+  routing-instances {
+      test {
+          instance-type vrf;
+          interface lo0.1;
+          vrf-target target:1:1;
+          vrf-table-label;
+      }
+  }
[edit routing-options]
+  route-distinguisher-id 192.168.1.6;
[edit protocols bgp]
     group ibgp { ... }
+    group ebgp-vpn {
+        local-address 192.168.1.6;
+        family inet-vpn {
+            unicast;
+        }
+        multihop;
+        peer-as 64510;
+        neighbor 192.168.1.1;
+    }

HOORAY! We did it, let’s see our amazing success and go out for pizza!

[email protected]# run show route table test hidden

test.inet.0: 2 destinations, 2 routes (1 active, 0 holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both

66.66.66.66/32      [BGP/170] 00:00:21, localpref 100, from 192.168.1.6
                      AS path: 64511 I, validation-state: unverified
                       Unusable

Ah man, that’s not what we expected, let’s get a little more detail.

[email protected]# run show route table test hidden detail

test.inet.0: 2 destinations, 2 routes (1 active, 0 holddown, 1 hidden)
66.66.66.66/32 (1 entry, 0 announced)
         BGP    Preference: 170/-101
                Route Distinguisher: 192.168.1.6:8
               Next hop type: Unusable, Next hop index: 0
                Address: 0xc3872d0
                Next-hop reference count: 3
                State: <Secondary Hidden Ext Changed ProtectionCand>
                Local AS: 64510 Peer AS: 64511
                Age: 1:16
                Validation State: unverified
                Task: BGP_64511.192.168.1.6
                AS path: 64511 I
                Communities: target:1:1
                Import Accepted
                VPN Label: 16
                Localpref: 100
                Router ID: 192.168.1.6
                Primary Routing Table bgp.l3vpn.0

Hmm, there’s that super informative next-hop unusable error. Maybe this is somehow related to the greater-than-3-label feat we have accomplished?

If you check out this link you will blatantly see that the default stack depth is 3. Let’s change this to 4 and see if we get this resolved.

[email protected]# show | compare
[edit interfaces ge-0/0/0 unit 0 family mpls]
+      maximum-labels 4;
[email protected]# run show route table test detail protocol bgp

test.inet.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
66.66.66.66/32 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Route Distinguisher: 192.168.1.6:8
                Next hop type: Indirect, Next hop index: 0
                Address: 0xc387dc0
                Next-hop reference count: 3
                Source: 192.168.1.6
                Next hop type: Router, Next hop index: 611
                Next hop: 100.64.1.1 via ge-0/0/0.0, selected
                Label-switched-path to-r3
                Label operation: Push 16, Push 299856, Push 299776, Push 299824(top)

Beautiful! The route is there.

[email protected]# run ping 66.66.66.66 routing-instance test
PING 66.66.66.66 (66.66.66.66): 56 data bytes
64 bytes from 66.66.66.66: icmp_seq=0 ttl=60 time=58.453 ms
64 bytes from 66.66.66.66: icmp_seq=1 ttl=60 time=24.834 ms
64 bytes from 66.66.66.66: icmp_seq=2 ttl=60 time=24.627 ms

And the pings are a great success!

Quickly, let’s follow the labels.

  1. PE1 pushes on VPN label 16, BGP-LU label 299856 for next-hop 192.168.1.4, LDP label 299776 for 192.168.1.4, and RSVP label 299824 for targeted LDP shortest next-hop 192.168.1.3.
  2. P2 receives label stack, sees top label 299824 and performs a pop operation as it is the PHP for the RSVP tunnel to P3
  3. P3 receives 3-label-stack with top label 299776 and does a pop operation as it is the PHP for the LDP LSP to 192.168.1.4
  4. P4 receives 2-label-stack with top label 299856, and swaps it for 299808 label that is received from P5 for prefix 192.168.1.6/32
  5. P5 receives 2-label-stack with top label 299808 and pops the top label, as it is the PHP for the iBGP-LU/LDP session from P6, and P5 sends the 1-label-stack toward PE6
  6. PE6 receives a 1-label-stack of just the VPN label 16, and because of vrf-table-label PE6 pops the 16 and sends bare IP packets to vrf test for lookup where it meets lo0.1 processing.

PCAP Image

Here is the beautiful 4 label stack in wireshark as it leaves PE1 toward P2, in all its glory.

If you made it this far, thanks for following along.

Written on January 23, 2021