Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded

To: Diederik de Haas <didi.debian@cknow.org>, 1071184@bugs.debian.org
Subject: Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
From: Development EasyNet <devel@easynet.dev>
Date: Fri, 17 May 2024 15:08:17 +0200
Message-id: <[🔎] 0453cea5-cfde-4b22-ae6f-e3102d882c80@easynet.dev>
Reply-to: Development EasyNet <devel@easynet.dev>, 1071184@bugs.debian.org
In-reply-to: <[🔎] 2839655.L3na9P5Ebz@bagend>
References: <[🔎] 2839655.L3na9P5Ebz@bagend> <1ccf1bda-ca69-4367-a625-f13b144b8e89@easynet.dev>

Hi Diederik,

I will try. Meanwhile I was troubleshooting this issue for some time and I notice a change in FRRouting between 9.1 and 10.0.
Before 10.0 FRRouting was installing the routes in kernel using the destination interface of the route. Starting from 10.0 FRRouting is installing all routes towards the VRF interface.

Here is my bug reported on FRRouting: https://github.com/FRRouting/frr/issues/15909

Example:
Working scenario with FRR 9.0.2 and 9.1:

root@FRR01:/opt/Kitts/frr/9.0.2# ip nexthop show
id 14 dev lo scope host proto zebra
id 15 dev ens33 scope host proto zebra
id 16 dev ens36 scope host proto zebra
id 17 dev ens37 scope host proto zebra
id 18 dev ens38 scope host proto zebra
id 19 dev ens33 scope link proto zebra
id 21 dev ens36 scope link proto zebra
id 23 dev ens37 scope link proto zebra
id 25 dev ens38 scope link proto zebra
id 26 dev lo3 scope link proto zebra
id 30 blackhole proto zebra
id 31 blackhole proto zebra
id 32 via 192.168.1.1 dev ens33 scope link proto zebra
id 36 dev ens37 scope host proto zebra
id 37 dev lo scope host proto zebra
id 38 dev ens38 scope host proto zebra

root@FRR01:/opt/Kitts/frr/9.0.2# ip nexthop show vrf red
id 18 dev ens38 scope host proto zebra
id 25 dev ens38 scope link proto zebra
id 38 dev ens38 scope host proto zebra

root@FRR01:/opt/Kitts/frr/9.0.2# ip route list
10.0.0.0/30 dev ens37 proto kernel scope link src 10.0.0.1
10.0.1.0/30 nhid 38 dev ens38 proto bgp metric 20

root@FRR01:/opt/Kitts/frr/9.0.2# ip route show table local
local 10.0.0.1 dev ens37 proto kernel scope host src 10.0.0.1
broadcast 10.0.0.3 dev ens37 proto kernel scope link src 10.0.0.1
local 10.100.0.1 dev lo proto kernel scope host src 10.100.0.1
broadcast 10.100.0.1 dev lo proto kernel scope link src 10.100.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1

root@FRR01:/opt/Kitts/frr/9.0.2# ip route show vrf red
blackhole default proto static metric 20
10.0.0.0/30 nhid 36 dev ens37 proto bgp metric 20
10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1
10.100.0.1 nhid 37 dev lo proto bgp metric 20

root@FRR01:/opt/Kitts/frr/9.0.2# ip route show table red
blackhole default proto static metric 20
10.0.0.0/30 nhid 36 dev ens37 proto bgp metric 20
10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1
local 10.0.1.1 dev ens38 proto kernel scope host src 10.0.1.1
broadcast 10.0.1.3 dev ens38 proto kernel scope link src 10.0.1.1
10.100.0.1 nhid 37 dev lo proto bgp metric 20

root@FRR01:/opt/Kitts/frr/9.0.2# ip route show vrf red
blackhole default proto static metric 20
10.0.0.0/30 nhid 36 dev ens37 proto bgp metric 20
10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1
10.100.0.1 nhid 37 dev lo proto bgp metric 20

root@FRR01:/opt/Kitts/frr/9.0.2# ip rule list
0:      from all lookup local
1000:   from all lookup [l3mdev-table]
32766:  from all lookup main
32767:  from all lookup default
root@FRR01:/opt/Kitts/frr/9.0.2#


Non-working scenario with FRR 10.0:

root@FRR01:/# ip nexthop show
id 2 dev lo0 scope link proto zebra
id 4 dev lo1 scope link proto zebra
id 6 dev lo2 scope link proto zebra
id 8 dev lo3 scope link proto zebra
id 10 dev ens36 scope host proto zebra
id 17 dev ens37 scope host proto zebra
id 18 dev ens38 scope host proto zebra
id 19 dev lo scope host proto zebra
id 20 dev ens33 scope host proto zebra
id 21 blackhole proto zebra
id 22 blackhole proto zebra
id 24 via 192.168.1.1 dev ens33 scope link proto zebra
id 32 dev ens33 scope link proto zebra
id 34 dev lo scope host proto zebra
id 36 dev red scope host proto zebra

root@FRR01:/# ip nexthop show vrf red
id 18 dev ens38 scope host proto zebra
id 25 dev ens38 scope link proto zebra

root@FRR01:/# ip route list

10.0.0.0/30 dev ens37 proto kernel scope link src 10.0.0.1
10.0.1.0/30 nhid 36 dev red proto bgp metric 20

root@FRR01:/# ip route show table local
local 10.0.0.1 dev ens37 proto kernel scope host src 10.0.0.1
broadcast 10.0.0.3 dev ens37 proto kernel scope link src 10.0.0.1
local 10.100.0.1 dev lo proto kernel scope host src 10.100.0.1
broadcast 10.100.0.1 dev lo proto kernel scope link src 10.100.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1

root@FRR01:/# ip route show vrf red
blackhole default proto static metric 20
10.0.0.0/30 nhid 34 dev lo proto bgp metric 20
10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1
10.100.0.1 nhid 34 dev lo proto bgp metric 20

root@FRR01:/# ip route show table red
blackhole default proto static metric 20
10.0.0.0/30 nhid 34 dev lo proto bgp metric 20
10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1
local 10.0.1.1 dev ens38 proto kernel scope host src 10.0.1.1
broadcast 10.0.1.3 dev ens38 proto kernel scope link src 10.0.1.1
10.100.0.1 nhid 34 dev lo proto bgp metric 20

root@FRR01:/# ip route show vrf red
blackhole default proto static metric 20
10.0.0.0/30 nhid 34 dev lo proto bgp metric 20
10.0.1.0/30 dev ens38 proto kernel scope link src 10.0.1.1
10.100.0.1 nhid 34 dev lo proto bgp metric 20

root@FRR01:/# ip ru l
0:      from all lookup local
1000:   from all lookup [l3mdev-table]
32766:  from all lookup main
32767:  from all lookup default

As you can see the difference is how FRRouting is installing the routes: 9.0.2 and 9.1 is pointing towards ensXX and 10.x is pointing to lo or red interfaces.
When is pointing to default VRF is pointing to interface lo. Then there there is a L3 loop until the packet is reaching TTL 0.
I have no clue how it suppose to work correctly: pointing the routes to VRF interface as 10.0 is doing or towards the route interface where is installed,like ensXX, as 9.1 is doing.

It suppose that lo interface should reinject the traffic in the kernel network stack or is this the normal behavior?

Kind regards,
Easynet

On 17.05.2024 14:52, Diederik de Haas wrote:

Control: tag -1 moreinfo

On 15 May 2024 16:08:27 +0200 Development EasyNet <devel@easynet.dev> wrote:

Package: linux-image
Version: 6.6.15-2 and 6.7.12-1

I'm facing for some time a strange behavior of the route-leak. It happen 
on both IPv4 and IPv6.
Configuration used: Debian Trixie, Kernel 6.7.12 with FRRouting 10.1 - git
VRF: internet
Default: just local management

Sid recently got a 6.8.9 kernel, can you test whether that fixes the issue?

Development @EasyNet

Web: www.easynet.dev

Reply to:

References:
- Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
  - From: Diederik de Haas <didi.debian@cknow.org>

Prev by Date: Processed: Re: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
Next by Date: Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
Previous by thread: Processed: Re: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
Next by thread: Bug#1071184: Kernel 6.6 and 6.7 route-leak between VRF and default leads to Time to live exceeded
Index(es):
- Date
- Thread