iBGP Sessions Flapping due to AIGP attribute after Upgrading 9.0.1/9.0.2 from 8.5.1 #15884

skyblueted · 2024-04-29T14:13:20Z

skyblueted
Apr 29, 2024

Description

Recently, we upgraded both the kernel and FRR, and noticed that after upgrading FRR from 8.5.1 to 9.0.2, upon reboot, the iBGP session starts flapping and fails to establish the BGP peer successfully.

Currently, we don't have any insights into the issue. However, we observed that reverting back to version 8.5.1 resolves the problem, with the same configuration resulting in normal iBGP session functionality

Version

FRR version:
8.5.1 -> 9.0.1/9.0.2

Kernel version:
5.15.69 -> 6.2

Ubuntu version:
20.04 -> 22.04

How to reproduce

Upgrade frr from 8.51. to 9.0.1/9.0.2
Restart frr service
vtysh -c 'show ip bgp summary'

Expected behavior

The iBGP sessions can be established successfully after upgrading

frr 8.5.1:

root@worker1:~# vtysh

Hello, this is FRRouting (version 8.5.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

worker1# show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 100.111.15.17, local AS number 64602 vrf-id 0
BGP table version 5
RIB entries 4, using 768 bytes of memory
Peers 2, using 1449 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
eno5            4      64602         7        11        0    0    0 00:00:22            1        3 N/A
ens3f0          4      64602         7        11        0    0    0 00:00:22            1        3 N/A

Total number of neighbors 2
worker1#

Actual behavior

frr 9.0.2:

root@worker1:/userap# vtysh

Hello, this is FRRouting (version 9.0.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

worker1# show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 100.111.15.17, local AS number 64602 vrf-id 0
BGP table version 3
RIB entries 4, using 768 bytes of memory
Peers 2, using 41 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
eno5            4      64602       240       193        0    0    0 00:00:03       Active        0 N/A
ens3f0          4      64602       245       196        0    0    0 00:00:02       Active        0 N/A

Additional context

frr 9.0.2
"We have observed that during the iBGP session flapping, numerous AIGP attribute logs frequently appear in the FRR log

frr logs

root@worker1:/userap# journalctl -u frr -e
Apr 29 19:47:46 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: ens3f0
Apr 29 19:47:48 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] ens3f0 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:47:56 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from eno5 in vrf default
Apr 29 19:47:56 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:47:56 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:47:56 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: eno5
Apr 29 19:47:58 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from ens3f0 in vrf default
Apr 29 19:47:58 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:47:58 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:47:58 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: ens3f0
Apr 29 19:47:58 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] eno5 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:00 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] ens3f0 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:08 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from eno5 in vrf default
Apr 29 19:48:08 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:08 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:08 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: eno5
Apr 29 19:48:09 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from ens3f0 in vrf default
Apr 29 19:48:09 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:09 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:09 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: ens3f0
Apr 29 19:48:10 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] eno5 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:11 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] ens3f0 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:20 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from eno5 in vrf default
Apr 29 19:48:20 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:20 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:20 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: eno5
Apr 29 19:48:21 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from ens3f0 in vrf default
Apr 29 19:48:21 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:21 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:21 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: ens3f0
Apr 29 19:48:22 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] eno5 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:23 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] ens3f0 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:32 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from eno5 in vrf default
Apr 29 19:48:32 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:32 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:32 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: eno5
Apr 29 19:48:34 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from ens3f0 in vrf default
Apr 29 19:48:34 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:34 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:34 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: ens3f0
Apr 29 19:48:34 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] eno5 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:36 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] ens3f0 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:46 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from eno5 in vrf default
Apr 29 19:48:46 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:46 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:46 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: eno5
Apr 29 19:48:47 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from ens3f0 in vrf default
Apr 29 19:48:47 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:47 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:47 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: ens3f0
Apr 29 19:48:48 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] eno5 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:49 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] ens3f0 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:48:59 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from eno5 in vrf default
Apr 29 19:48:59 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:48:59 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:48:59 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: eno5
Apr 29 19:49:01 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] eno5 [Error] bgp_read_packet error: Connection reset by peer
Apr 29 19:49:01 worker1 bgpd[40961]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from ens3f0 in vrf default
Apr 29 19:49:01 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Transitive"
Apr 29 19:49:01 worker1 bgpd[40961]: [NYD0B-A3BTB][EC 33554433] AIGP attribute must not be flagged as "Partial"
Apr 29 19:49:01 worker1 bgpd[40961]: [GW152-RVASS][EC 33554455] bgp_process_packet: BGP UPDATE receipt failed for peer: ens3f0
Apr 29 19:49:03 worker1 bgpd[40961]: [H4B4J-DCW2R][EC 33554455] ens3f0 [Error] bgp_read_packet error: Connection reset by peer

frr config:

frr defaults traditional
hostname worker1
log file /var/log/frr/bgpd.log informational
log syslog informational
service integrated-vtysh-config
!
interface eno5
 ipv6 nd ra-interval 4
 ipv6 nd ra-lifetime 10
 no ipv6 nd suppress-ra
exit
!
interface ens3f0
 ipv6 nd ra-interval 4
 ipv6 nd ra-lifetime 10
 no ipv6 nd suppress-ra
exit
!
router bgp 64602
 bgp router-id 100.111.15.17
 neighbor fabric peer-group
 neighbor fabric remote-as 64602
 neighbor fabric description Interal Fabric Network
 neighbor fabric bfd
 neighbor fabric bfd profile bfd_template
 neighbor fabric timers connect 10
 neighbor fabric capability extended-nexthop
 neighbor eno5 interface peer-group fabric
 neighbor ens3f0 interface peer-group fabric
 !
 address-family ipv4 unicast
  redistribute kernel route-map route_filter
  redistribute connected route-map route_filter
  neighbor fabric soft-reconfiguration inbound
  maximum-paths 64
  maximum-paths ibgp 64
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute kernel route-map v6_route_filter
  redistribute connected route-map v6_route_filter
  neighbor fabric activate
  neighbor fabric soft-reconfiguration inbound
  maximum-paths 64
  maximum-paths ibgp 64
 exit-address-family
exit
!
access-list block_default seq 5 permit [0.0.0.0/0](http://0.0.0.0/0) exact-match
access-list block_default seq 10 permit [192.168.0.0/16](http://192.168.0.0/16)
!
ipv6 access-list v6_block_default seq 5 permit ::/0 exact-match
!
route-map route_filter deny 10
 match ip address block_default
exit
!
route-map route_filter permit 20
exit
!
route-map v6_route_filter deny 10
 match ipv6 address v6_block_default
exit
!
route-map v6_route_filter permit 20
exit
!
bfd
 profile bfd_template
  detect-multiplier 6
  transmit-interval 500
  receive-interval 500
 exit
 !
exit
!

Checklist

I have searched the open issues for this bug.
I have not included sensitive information in this report.

ton31337 · 2024-04-29T14:35:22Z

ton31337
Apr 29, 2024
Collaborator

Quick question: what are the versions between FRR nodes? Is it like peering between 9.0.1 <--> 9.0.1, 9.0.1 <--> 9.0.2, or what actually?

0 replies

ton31337 · 2024-04-29T14:37:19Z

ton31337
Apr 29, 2024
Collaborator

Also, as a side note, 8.5.1 is broken in terms of AIGP, you shouldn't use it at all, please upgrade to 8.5.2 at least.

0 replies

skyblueted · 2024-04-29T14:52:01Z

skyblueted
Apr 29, 2024
Author

Quick question: what are the versions between FRR nodes? Is it like peering between 9.0.1 <--> 9.0.1, 9.0.1 <--> 9.0.2, or what actually?

Actually, our FRR daemon runs on bare metal, while the peer is Cisco equipment. The change that occurred is in the version on the server side.

It is like peering between 8.5.1 (server) <-> cisco (leaf), 9.0.1 (server) <-> cisco (leaf), 9.0.2 (server) <-> cisco (leaf)

0 replies

skyblueted · 2024-04-29T15:03:42Z

skyblueted
Apr 29, 2024
Author

Also, as a side note, 8.5.1 is broken in terms of AIGP, you shouldn't use it at all, please upgrade to 8.5.2 at least.

We're currently puzzled as the article suggests upgrading to 8.5.2, but when we upgraded to 9.0.1/9.0.2, we encountered iBGP flapping issues. We're uncertain how to address this.

0 replies

ton31337 · 2024-04-29T15:34:11Z

ton31337
Apr 29, 2024
Collaborator

Every node in the "cluster" should use a minimum of 8.5.2 (better the latest of 8.5.x).

0 replies

skyblueted · 2024-04-30T15:25:45Z

skyblueted
Apr 30, 2024
Author

Hi @ton31337 ,

Today, we made a new discovery from the recorded BGP packets.
We indeed observed malformed update messages with the Transitive/Partial bit set to 1 coming from the Leaf switch (Cisco). However, since we can't prevent them from sending malformed packets, we tried using "discard" but it was ineffective.

neighbor fabric path-attribute discard 26

We also attempted no neighbor fabric aigp without success.

We're seeking a solution that aligns with the description in RFC 7311, Section 3.3, which states that if an AIGP attribute is received on a BGP session where AIGP_SESSION is disabled, it should be treated as if it were an unrecognized non-transitive attribute. Specifically, it "must be quietly ignored and not passed along to other BGP peers."

Is there any other way to disable the AIGP attribute in iBGP?

1 reply

ton31337 May 1, 2024
Collaborator

So, here Cisco is sending the incorrect AIGP attribute, correct?

skyblueted · 2024-05-01T13:07:39Z

skyblueted
May 1, 2024
Author

So, here Cisco is sending the incorrect AIGP attribute, correct?

Yes, that's correct. We received an Update message from Cisco containing the AIGP attribute with both the transitive and partial bits set to 1.

However, we haven't succeeded in discarding the AIGP attribute using the path-attribute command, nor have we been able to directly disable AIGP in iBGP.

1. neighbor fabric path-attribute discard 26
2. no neighbor fabric aigp or ( no neighbor eno5 aigp & no neighbor ens3f0 aigp)

5 replies

ton31337 May 1, 2024
Collaborator

I have a fix for discard/withdraw to work, testing it a bit, and will push.

skyblueted May 1, 2024
Author

Thanks for your help!

ton31337 May 1, 2024
Collaborator

The fix is here: #15895.

ton31337 May 1, 2024
Collaborator

Would you be able to upgrade to 9.1 at least?

skyblueted May 1, 2024
Author

Is it also possible to backport to 9.0.x?

mdtancsa · 2024-05-01T17:55:03Z

mdtancsa
May 1, 2024

I am guessing this is the same issue I ran into here going from 8.5.1 to 8.5.4. Is there a similar work around that could be used in 8.x ?

https://lists.frrouting.org/pipermail/frog/2023-June/001278.html

---Mike

3 replies

ton31337 May 1, 2024
Collaborator

Not really. You were running FRR/FRR, while here it's FRR/Cisco.

mdtancsa May 1, 2024

But isnt it the same type of issue with FRR sending the wrong AIGP attribute value causing ibgp to flap ? I am still hoping to avoid having to take down my entire network to upgrade everything at once. Having the ability to tell the new ibgp peer to discard a broken AIGP attribute value would be helpful to allow me upgrade 8.5.1 routers one by one.

skyblueted May 3, 2024
Author

We've made a new discovery from packet analysis, which reveals a malformed AIGP attribute in the IPv6 default route. Upon tracing the source, we found it originating from another server of ours using FRR 8.5.1.
The illustration below depicts this:

After disabling IPv6 announce in our environment, upgrading FRR to version 9.0.2 still allows successful establishment of the iBGP session. We plan to utilize this approach to upgrade all server FRR versions to 8.5.2+ to rectify the wrong AIGP attribute issue.

(Adding to this, the current versions of Cisco devices here do not support AIGP.
Therefore, packets will be processed according to the behavior of the Transitive bit of path attribute)

iBGP Sessions Flapping due to AIGP attribute after Upgrading 9.0.1/9.0.2 from 8.5.1 #15884

Uh oh!

skyblueted Apr 29, 2024

Description

Version

How to reproduce

Expected behavior

Actual behavior

Additional context

Checklist

Replies: 8 comments · 10 replies

Uh oh!

ton31337 Apr 29, 2024 Collaborator

Uh oh!

ton31337 Apr 29, 2024 Collaborator

Uh oh!

Uh oh!

skyblueted Apr 29, 2024 Author

Uh oh!

skyblueted Apr 29, 2024 Author

Uh oh!

ton31337 Apr 29, 2024 Collaborator

Uh oh!

skyblueted Apr 30, 2024 Author

Uh oh!

ton31337 May 1, 2024 Collaborator

Uh oh!

skyblueted May 1, 2024 Author

Uh oh!

ton31337 May 1, 2024 Collaborator

Uh oh!

skyblueted May 1, 2024 Author

Uh oh!

ton31337 May 1, 2024 Collaborator

Uh oh!

ton31337 May 1, 2024 Collaborator

Uh oh!

Uh oh!

skyblueted May 1, 2024 Author

Uh oh!

mdtancsa May 1, 2024

Uh oh!

ton31337 May 1, 2024 Collaborator

Uh oh!

mdtancsa May 1, 2024

Uh oh!

Uh oh!

skyblueted May 3, 2024 Author

skyblueted
Apr 29, 2024

Replies: 8 comments 10 replies

ton31337
Apr 29, 2024
Collaborator

ton31337
Apr 29, 2024
Collaborator

skyblueted
Apr 29, 2024
Author

skyblueted
Apr 29, 2024
Author

ton31337
Apr 29, 2024
Collaborator

skyblueted
Apr 30, 2024
Author

ton31337 May 1, 2024
Collaborator

skyblueted
May 1, 2024
Author

ton31337 May 1, 2024
Collaborator

skyblueted May 1, 2024
Author

ton31337 May 1, 2024
Collaborator

ton31337 May 1, 2024
Collaborator

skyblueted May 1, 2024
Author

mdtancsa
May 1, 2024

ton31337 May 1, 2024
Collaborator

skyblueted May 3, 2024
Author