MC-LAG Configuration Guide

introduction introduction mc lag (multi chassis link aggregation group) is a mechanism for implementing cross device link aggregation by aggregating one device with two other devices across the switch, retaining all the benefits of normal link aggregation while providing device level redundancy mc lag provides a horizontal virtualization technique that virtualizes two physical devices into a single logical device, and this virtualized "single device" is "one to one" with its connected uplink or downlink device for link aggregation as shown in figure above, the mc lag group is externally represented as a logical device for link aggregation; the mc lag switch group deploys ethernet or lag type peer link for protocol information interaction between mc lag, as well as taking up the east west traffic in case of failure scenarios the mc lag group is externally presented as a single node, which does not pose a loop risk when implementing redundant backups, and the load balancing mode of link aggregation does not result in idle links, making link utilization more efficient as shown in figure below, the server side performs cross device link aggregation with the other two devices leaf1 and leaf2 via the mc lag mechanism basic concepts basic concepts mc lag peer mc lag peer refers to the two devices participating in the cross device link aggregation group the two mc lag peer devices assume the active and standby roles respectively according to the comparison of the size of the local ip and peer ip numbers in the configuration information, the smaller number will be the active side, the client side of the tcp connection; the larger number will be the standby side, the server side of the tcp connection, and the client side will initiate a connection request to the server note the active, standby role of mc lag peer is a control level concept only on the data forwarding side, mc lag peers each determine the traffic forwarding path and have the same status mc lag member interface mc lag member interface refers to a series of lags that participate in cross device link aggregation keep alive link keep alive link the keep alive link is a heartbeat detection link, typically implemented as a direct layer 3 connection between mc lag peer, periodically sends heartbeat packet it's used for transmitting iccp control protocol packets, synchronizing table information, establishing mc lag peer relationships, and implementing configuration consistency checks peer link peer link the peer link refers to a direct physical link between mc lag peers, used to forward traffic when there is a failure in downstream links generally, it is recommended to be shared with keep alive link for operational efficiency and resource optimization dad link dad link the dad (dual active detection) link is a layer 3 interoperable link used by mc lag peer to send dual active detection packets, applicable in scenarios when the peer link and keep alive link are shared when the keep alive link is detected as disconnected, the system will automatically shut down all interfaces on the standby node except logical interfaces, management ports, and peer link interfaces working principle working principle control protocols control protocols sonic uses a lightweight iccp protocol on the mc lag control plane, with only a small amount of consistency checking and packet synchronization while safeguarding functionality iccp (inter chassis communication protocol), a standard protocol defined in rfc 7275, uses tcp port 8888 to establish a connection between mc lag peers, and the lightweight iccp protocol performs configuration consistency checks, arp table entry and mac table entry synchronization neighbor establishment neighbor establishment the two mc lag peer devices establish iccp neighbor with local ip and peer ip as the source and destination addresses of the tcp connection when the iccp connection is established, the mc lag peer modifies the app port table or app lag table or app vxlan tunnel table table entry in app db and disables the mac learning function of the peer link, depending on the port type of the peer link information synchronization information synchronization the following information is synchronized between mc lag peers system information synchronize the mac addresses of mc lag member ports to ensure that the "system id" field in the lacp packets sent from mc lag peer to server is the same, so as to achieve cross device link aggregation to achieve cross device link aggregation, when the standby side receives the system mac information of the active side, it sends a netlink packet to modify the system id of the local mc lag member port to be the same as that of the active side mc lag member port configuration information records information such as the name of the local and peer mc lag member ports, which is used for consistency checking mc lag member port status information the local and peer lag member port status is recorded to ensure the isolation of peer link to mc lag member port in case of no fault, and the isolation of peer link to port of the same name in case of peer lag member port fault is released isolation is adding table entries to app db, and the table entry is deleted for isolation release arp information the arp table entries associated with mc lag member ports are synchronized between mc lag peers when an arp packet is received from the peer, the arp table entry is updated locally by sending a netlink packet fdb information the fdb table entries associated with mc lag member ports are synchronized between mc lag peers upon receipt of fdb packets from the peer, the local fdb table entries are updated by modifying the table entries in app db heartbeat detection heartbeat detection in sonic, a heartbeat packet is sent to the other end at 1s intervals if a heartbeat packet is not received for 15 consecutive cycles, it is judged to have timed out and the iccp connection is broken consistency check consistency check sonic uses the lightweight iccp protocol for consistency checking of the following attributes symmetrical configuration of mc lag peer in local ip, peer ip; same configuration of mc lag member ports name, number, joined vlan, routing port ip of vlan, routing port mac address of vlan; same configuration of mc lag peer in the port type of peer link the mc lag solution provides an automatic detection mechanism that supports the following three modes idle mode idle mode in this mode, consistency checks only display the relevant results without performing any processing default mode default mode in this mode, member ports failing detection will be shut down on two peers graceful mode graceful mode in this mode, member ports failing detection will be shut down on standby node the default mode for mc lag consistency checks is the idle mode mc lag configuration mc lag configuration table 1 overview of mc lag configuration tasks table 1 overview of mc lag configuration tasks configuration tasks description refer to mc lag configuration configure mc lag required docid\ rquz zbagfpylsr6qsyz4 configure peer link required docid\ rquz zbagfpylsr6qsyz4 configure keep alive required docid\ rquz zbagfpylsr6qsyz4 configure mc lag member required docid\ rquz zbagfpylsr6qsyz4 configure monitor link group required docid\ rquz zbagfpylsr6qsyz4 configure dual active gateway optional docid\ rquz zbagfpylsr6qsyz4 configure dad link optional docid\ rquz zbagfpylsr6qsyz4 configure consistency check mode optional docid\ rquz zbagfpylsr6qsyz4 configure mstp optional docid\ rquz zbagfpylsr6qsyz4 configure iccp backup channel optional docid\ rquz zbagfpylsr6qsyz4 configure loopback detect optional docid\ rquz zbagfpylsr6qsyz4 configure unique ip optional docid\ rquz zbagfpylsr6qsyz4 mc lag default setting mc lag default setting the default setting of the mc lag is shown in the table below table 2 mc lag default setting table 2 mc lag default setting parameters default value heartbeat interval 1s session timeout 15s consistency check mode idle mclag vrf default dad detection delay 60s dad recovery delay (mclag member) 60s dad recovery delay (not mclag member) 0s dad vrf default configure mc lag configure mc lag table 3 mc lag basic configuration table 3 mc lag basic configuration purpose commands description enter global configuration view configure terminal create mc lag domain mclag domain domain id only one domain is currently supported, with range of 1 4095 (optional) configure the mc lag session timeout session timeout timeout timeout time, in seconds, value range is 3 3600, default value is 15 seconds; the time interval of the heartbeat detection message should be less than 1/3 of the mc lag session timeout time; the session timeout time should be a multiple of the time interval of the heartbeat detection message configure peer link configure peer link peer link is recommended to use a high speed interface for interconnection, and it is recommended that a directly connected aggregation link located between two mc lag devices be used as a peer link link, and it is recommended that it be configured as a static aggregation in order to improve the reliability of mc lags after interface is configured as a peer link interface, no other services can be configured on this interface assign a dedicated vlan for the peer link and add the peer link to other mc lag service vlans configure interface delayed startup on the physical port where the peer link is located so that it is up before the up/downlink port up, so that the mc lag state is restored and the table entries are synchronized and then the traffic is forwarded, which reduces packet loss when the mc lag master and backup devices reboot table 4 mc lag member configuration table 4 mc lag member configuration purpose commands description enter global configuration view configure terminal create a peer link private vlan vlan vlan id vlan id range 1 4094 exit vlan configuration view exit enter lag configuration view and create a link aggregation interface link aggregation lag id lag id aggregation group id, range 1 9999 configure static aggregation mode mode static commit the configuration commit specify the private vlan and join other service vlans switchport trunk vlan vlan id exit lag interface configuration view exit enter ethernet interface view interface ethernet interface name add the corresponding lag link aggregation group lag id \[ port priority port priority ] port priority can be specified, in the range 0 to 65535, default 255 configure the delay to start the interface startup delay delay the default interface delay startup is 150 seconds, and it is recommended that 120 seconds be configured on the physical interface where peer link is located exit ethernet interface view exit enter mc lag domain configuration view mclag domain domain id domain id range 1 4095 specify peer link peer link link aggregation lag id commit configuration commit exit mc lag domain configuration view exit configure keep alive configure keep alive the heartbeat detection link is used to forward control messages of mc lag, which can be shared with peer link or use a separate physical link when the dual master detection function is enabled, the heartbeat detection link is required to be shared with the peer link, otherwise the function will be disabled the configuration using a separate physical link is as follow table 5 configure keep alive link table 5 configure keep alive link purpose commands description enter global configuration view configure terminal enter ethernet interface view interface { ethernet interface name | link aggregation lag id | vlan vlan id } when heartbeat detection link is shared with peer link, the interface uses vlan interface configure the ipv4 address ip address { a b c d/m | a b/m } exit ethernet interface view exit enter mc lag domain configuration view mclag domain domain id domain id range 1 4095 configure the peer ip of the keep alive link peer address { a b c d | a b } configure the local ip of the keep alive link local address { a b c d | a b } (optional) specify the vrf of the keep alive link vrf vrf name default is default vrf (optional) configure the time interval for sending mc lag heartbeat detection messages heartbeat interval interval the unit of interval time is second, the value range is 1 60, and the default value is 1 second; the time interval of heartbeat detection message should be less than 1/3 of the mc lag session timeout time; the session timeout time should be a multiple of the time interval of heartbeat detection message commit configuration commit exit mc lag domain configuration view exit configure mc lag member configure mc lag member it is recommended to use low speed interfaces as mc lag member interfaces to improve system reliability, it is recommended to use dynamic aggregation for cross device aggregation groups and enable lacp short timeout it is required that the port ids of the member physical interfaces of the two devices deployed with mc lag are the same, otherwise they cannot be aggregated properly table 6 mc lag member configuration table 6 mc lag member configuration purpose commands description enter global configuration view configure terminal enter vlan interface configuration view vlan vlan id vlan id range 1 4094 exit vlan configuration view exit enter lag interface configuration view and create an aggregation group interface link aggregation lag id lag id aggregation group id, range 1 9999 (optional) enable lacp short timeout lacp fast rate (optional) enable lacp fallback lacp fallback when not receiving lacp packets, activate one member port of a lag, which can only be configured on one of the mclag devices commit the configuration commit join the service vlan switchport trunk vlan vlan id exit lag interface configuration view exit enter mc lag configuration view mclag domain domain id domain id range 1 4095 add member lag port of mc lag member lag lag id lag id aggregation group id, range 1 9999 exit mc lag configuration view exit enter the lag interface view where peer link is located interface link aggregation lag id join the service vlan switchport trunk vlan vlan id configure monitor link group configure monitor link group it is recommended that you configure the monitor link group on the master and backup devices deployed with mc lag, with the uplink port configured as uplink and the downlink port configured as downlink after you enable this function, the downlink port will be linked down after the uplink port state is down, ensuring that it can converge quickly when a failure occurs in the topology table 7 configure monitor link group table 7 configure monitor link group purpose commands description enter global configuration view configure terminal create monitor link group monitor link group group name \[ delay time ] delay time is an optional parameter, which indicates the delay time of downlink port up after uplink port up, the unit is second, and the default value is 0 when not configured enter ethernet interface view interface ethernet interface name configure the uplink port as uplink monitor link group name uplink exit ethernet interface view exit enter downlink lag interface configuration view interface ethernet interface name configure the mc lag member interface as downlink monitor link group name downlink configure dual active gateway configure dual active gateway in scenarios where the mc lag accesses a layer 3 network or a vxlan network, the mc lag master and backup devices need to act as layer 3 gateways at the same time, and must ensure that the vlanif interfaces corresponding to the mc lag member interfaces have the same ip address and mac address; in scenarios with a vxlan network, it is also required that the vrfs corresponding to the same l3 vni on the mc lag master and backup devices have the mac is the same table 8 configure dual active gateway table 8 configure dual active gateway purpose commands description enter global configuration view configure terminal enter vrf configuration view vrf vrf name set the mac of vrf mac hh\ hh\ hh\ hh\ hh\ hh exit vrf configuration view exit enter the service vlanif configuration view interface vlan vlan id (optional)configure vrf for the interface vrf vrf name configure the vlanif interface ipv4 address ip address a b c d/m configure the mac address of the vlanif interface mac address hh\ hh\ hh\ hh\ hh\ hh configure dad link configure dad link dual active detection link (dad link) is recommended to be interconnected using low speed service ports and is recommended to be configured on a separate physical layer 3 interface the dad link and peer link cannot be shared if the peer link is configured before the dad link is not configured, configure the dad link when the peer link port is in the up state to prevent the standby device port from being mistakenly error down table 9 dad link configuration table 9 dad link configuration purpose commands description enter global configuration view configure terminal enter ethernet interface view interface ethernet interface name configure the interface ipv4 address to enable dad link interworking ip address a b c d/m exit ethernet interface view exit enter mc lag configuration view mclag domain domain id domain id range 1 4095 configure dad local ip dad local address a b c d configure dad peer ip dad peer address a b c d (optional) configure dad detection delay dad detection delay time time range 0 512; default value 15 (optional) configure the delay for recovering the error down status of mc lag member interfaces on the mc lag standby side dad recovery delay mlag time time recovery delay time , range 0 1000; default value 60 for mc lag member interfaces (optional) configure the delay of error down state recovery for non mc lag member interfaces on the mc lag standby side dad recovery delay non mlag time time recovery delay time , range 0 1000; default value 0 for non mc lag member interfaces (optional) configure the vrf where the dad link is located dad vrf { default | name } default default vrfname vrf name delete dad local ip configuration commit delete dad peer ip configuration exit configure consistency check mode configure consistency check mode table 10 configure consistency check mode table 10 configure consistency check mode purpose commands description enter global configuration view configure terminal enter mc lag domain configuration view mclag domain domain id domain id range 1 4095 configure the mode of mc lag consistency checking consistency check action { default | graceful | idle } the default consistency check is idle mode exit mc lag domain configuration view exit configure mstp configure mstp the mc lag scenario supports the configuration of mstp spanning tree protocol anti loop you should manually deploy the two devices of the mc lag as the root bridge of the layer 2 network when you turn on mstp, and configure the same bridge id to simulate the two devices as the same root bridge, and the root priority is configured to be the highest in both cases to ensure that the m lag device is the root node by default, the bridge mac on the switch that participates in the spanning tree calculation is the mac address of the switch when configuring manually, it is recommended that you additionally select a smaller mac address (not the mac address of the mc lag master or backup device) as the bridge mac since the switches at both ends of the mc lag need to be simulated as the same stp root bridge, it should be ensured that the switch's directly connected interface will not be blocked off, and it is necessary to disable the stp function of the peer link interface, that is, to turn on the bpdu filtering function the mc lag member interface needs to be manually configured as an edge interface does not support multi level mc lag scenarios table 11 configure mstp table 11 configure mstp purpose commands description enter global configuration view configure terminal enable mstp mstp enable enter the default mstp instance configuration view mstp instance 0 configure the mc lag master/standby device mstp instance priority to 0 priority 0 exit the mstp instance configuration view exit configure the mc lag root bridge mac on the mc lag master and standby device bridge mac hh\ hh\ hh\ hh\ hh\ hh disable the stp function for peer link links on mc lag master and backup devices mstp bpdu filter enable interface { ethernet interface name | link aggregation lag id } configure mc lag member interfaces as edge interfaces mstp edge port enable interface { ethernet interface name | link aggregation lag id } configure iccp backup channel configure iccp backup channel use a separate link, added to an independent vlan, as a backup path for iccp when the heartbeat detection link is disconnected, the system can automatically switch to this iccp backup path, ensuring mc lag reliability no manual ip address configuration is required on the iccp backup path; only a vlanif interface needs to be configured, and link local addresses will be used for iccp communication table 12 configure iccp backup channel table 12 configure iccp backup channel purpose commands description enter global configuration view configure terminal create an iccp backup channel dedicated vlan vlan vlan id vlan id range 1 4094 exit vlan configuration view exit create iccp backup channel vlanif configuration view interface vlan vlan id exit vlanif configuration view exit enter ethernet configuration view interface ethernet interface name add member for the iccp standby path dedicated vlan switchport trunk vlan vlan id enter mc lag domain configuration view mclag domain domain id domain id range 1 4095 configure the iccp backup channel backup channel vlan vlan id exit mc lag domain configuration view exit configure loopback detect configure loopback detect loop detection policies include two modes shutdown and block vlan in shutdown mode, the system administratively disables the interface where a loop is detected in block vlan mode, the system blocks the vlan in which a loop is detected the default mode is shutdown table 13 configure loopback detect table 13 configure loopback detect purpose commands description enter global configuration view configure terminal enter downlink lag view (mc lag member) interface link aggregation lag id lag id aggregation group id, range 1 9999 configure mc lag loopback detect loopback detect vlan vlan id \[ policy { shutdown | block vlan } interval interval \[ wait time wait time ]] vlan id vlan id, the range is \[1,4094] policy optionally configure the execution policy after a loopback has been detected the default is shutdown mode shutdown indicates that the switch will shut down interfaces with loopback detection configured block vlan indicates that the switch will block the vlan with loopback detection configured interval specify the interval time in seconds the default value is 5 s wait time specifies the wait time in seconds before restoring a blocked port or vlan the default value is 60 s configure unique ip configure unique ip this feature supports the mc lag peer to establish a routing protocol with the access side to transmit routes two options are currently available dual vlans, that is, the gateway vlan is separate from the routing protocol vlan the ip and the mac of the gateway vlan should be the same while the ip of the routing protocol vlan should be different from one another and the mac can just use the default value in addition, it is required to enable unique ip in diff mac mode and arp proxy in evpn mode for routing protocol vlan the gateway vlan and the routing protocol vlan are shared the primary ip of the svi are supposed to be different for protocol interactions, while the secondary ip be the same for external gateway the vlan mac should be set to the same value (not the same with system mac or interface mac of the peers) in addition, it is required to enable unique ip in same mac mode and arp proxy in evpn mode for this vlan the first one is configured as follows table 14 mc lag unique ip dual vlans configuration table 14 mc lag unique ip dual vlans configuration purpose commands description enter global configuration view configure terminal create two vlans, one for gateway and another for protocol communication vlan vlan id vlan id range 1 4094 exit vlan configuration view exit enter mc lag domain configuration view mclag domain domain id domain id range 1 4095 enable mc lag unique ip for protocol communication unique ip vlan vlan id \[ diff mac ] enable with mode diff mac (the default mode is diff mac ) exit mc lag configuration view exit enter protocol vlan interface view interface vlan vlan id set the ip address ip address a b c d/m assign with different ip on mc lag peers (optional)set the mac address mac address hh\ hh\ hh\ hh\ hh\ hh the mac of this vlanif needs to be different on mc lag peers it is recommended to use the default value configure arp proxy arp proxy mode evpn enter gateway vlan interface view interface vlan vlan id set the ip address ip address a b c d/m assign with the same ip on mc lag peers set the mac address mac address hh\ hh\ hh\ hh\ hh\ hh assign with the same mac on mc lag peers the second one is configured as follows table 15 mc lag unique ip single vlan with dual ip configuration table 15 mc lag unique ip single vlan with dual ip configuration purpose commands description enter global configuration view configure terminal create a vlan for both uses vlan vlan id vlan id range 1 4094 exit vlan configuration view exit enter mc lag domain configuration view mclag domain domain id domain id range 1 4095 enable mc lag unique ip unique ip vlan vlan id same mac enable with mode same mac exit mc lag configuration view exit enter vlan interface view interface vlan vlan id set the primary ip as the protocol ip ip address a b c d/m assign with different ip as protocol ip on mc lag peers set the secondary ip as the service gateway ip ip address a b c d/m secondary assign with the same ip as gateway ip on mc lag peers set the mac address mac address hh\ hh\ hh\ hh\ hh\ hh the mac of this vlanif needs to be consistent on mc lag peers (should not be the same with system mac or interface mac) configure arp proxy arp proxy mode evpn display and maintenance display and maintenance table 16 mc lag configuration display table 16 mc lag configuration display purpose commands description check mc lag status show mclag state \[ domain id ] domain id the created mc lag domain id display mc lag consistency check results show mclag consistency check result typical configuration example typical configuration example example of a single group mc lag configuration example of a single group mc lag configuration networking requirements in the network, leaf1 and leaf2 are two independently operating devices, connected to spine respectively due to the high service reliability requirements of the customer, mc lag is required to be configured between leaf1 and leaf2 to achieve cross device link aggregation topology configuration roadmap (1) check that the link status of the ports used on the switch is normal, and that all ports are stable in the up state (2) configure the switch interface ip addresses (3) configure vlan and configure ip address for vlan interface (4) configure link aggregation group and add it to vlan (5) configure mc lag (6) configure monitor link group (7) configure static routes to ensure reachability of the service network procedure spine configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 1/30 exit ! interface ethernet 0/52 ip address 10 93 0 9/30 exit ! configure service vlan with vlan id 20 ! vlan 20 exit ! interface vlan 20 ip address 10 20 0 1/24 exit add the interface to the vlan ! interface ethernet 0/0 switchport trunk vlan 20 configure static routes ip route 10 10 0 1/24 10 93 0 2 ip route 10 10 0 1/24 10 93 0 10 leaf1 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 2/30 exit configure service vlan with vlan id 10 and keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 1/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 1 peer address 10 245 0 2 peer link link aggregation 9999 commit member lag 1 configure monitor link ! monitor link group mc lag 90 ! interface ethernet 0/48 monitor link mc lag uplink exit ! interface ethernet 0/0 monitor link mc lag downlink exit configure static routes ip route 10 20 0 1/24 10 93 0 1 leaf2 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 10/30 exit configure service vlan with vlan id 10 and keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 2/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 2 peer address 10 245 0 1 peer link link aggregation 9999 commit member lag 1 configure monitor link ! monitor link group mc lag 90 ! interface ethernet 0/48 monitor link mc lag uplink exit ! interface ethernet 0/0 monitor link mc lag downlink exit configure static routes ip route 10 20 0 1/24 10 93 0 9 verify the configuration (1) check the mc lag status (leaf1) sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel primary channel vrf name default consistency check action idle local ip 10 245 0 1 peer ip 10 245 0 2 dad local ip dad peer ip peer link interface lag 9999 keepalive time 1 dad detection delay 15 dad recovery delay mlag intf 60 dad recovery delay non mlag intf 0 dad vrf name default dad status disable session timeout 15 peer link mac 60\ eb 5a 01 10\ b1 system mac 60\ eb 5a 01 10\ b1 peer mac 00 00 00 00 00 00 admin role none role active mclag interface lag 1 loglevel notice (2) verify vm connectivity vm a1 and vm b1 ping each other, can ping through, indicating successful configuration example of mc lag iccp backup channel configuration (keepalive and peer link are not the same link) example of mc lag iccp backup channel configuration (keepalive and peer link are not the same link) networking requirements in the network, leaf1 and leaf2 are two independently operating devices, connected to spine respectively due to the high service reliability requirements of the customer, mc lag is required to be configured between leaf1 and leaf2 to achieve cross device link aggregation also configure iccp backup channel to ensure keep alive link failure does not affect the work of mc lag topology steps (1)check that the link status of the ports used on the switch is normal, and that all ports are stable in the up state (2)configure the switch interface ip addresses (3)configure vlan and configure ip address for vlan interface (4)configure link aggregation group and add it to vlan (5)configure mc lag (6)configure monitor link group (7)configure static routes to ensure reachability of the service network procedure spine configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 1/30 exit ! interface ethernet 0/52 ip address 10 93 0 9/30 exit ! configure service vlan with vlan id 20 ! vlan 20 exit ! interface vlan 20 ip address 10 20 0 1/24 exit add the interface to the vlan ! interface ethernet 0/0 switchport trunk vlan 20 configure static routes ip route 10 10 0 1/24 10 93 0 2 ip route 10 10 0 1/24 10 93 0 10 leaf1 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 2/30 exit configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 and iccp backup vlan with vlan id 4093 vlan 10 exit ! vlan 4094 exit ! vlan 4093 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 1/30 exit ! interface vlan 4093 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/64 switchport trunk vlan 4094 ! interface ethernet 0/68 switchport trunk vlan 4093 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 1 peer address 10 245 0 2 peer link link aggregation 9999 backup channel vlan 4093 commit member lag 1 configure monitor link group ! monitor link group mc lag 90 ! interface ethernet 0/48 monitor link mc lag uplink exit ! interface ethernet 0/0 monitor link mc lag downlink exit configure static routes ip route 10 20 0 1/24 10 93 0 1 leaf2 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 10/30 exit configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 and iccp backup vlan with vlan id 4093 vlan 10 exit ! vlan 4094 exit ! vlan 4093 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 2/30 exit ! interface vlan 4093 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/64 switchport trunk vlan 4094 ! interface ethernet 0/68 switchport trunk vlan 4093 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 2 peer address 10 245 0 1 peer link link aggregation 9999 backup channel vlan 4093 commit member lag 1 configure monitor link group ! monitor link group mc lag 90 ! interface ethernet 0/48 monitor link mc lag uplink exit ! interface ethernet 0/0 monitor link mc lag downlink exit configure static routes ip route 10 20 0 1/24 10 93 0 9 verify the configuration (1) check the mc lag status (leaf1) sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel primary channel vrf name default consistency check action idle local ip 10 245 0 1 peer ip 10 245 0 2 dad local ip dad peer ip peer link interface lag 9999 keepalive time 1 dad detection delay 15 dad recovery delay mlag intf 60 dad recovery delay non mlag intf 0 dad vrf name default dad status disable session timeout 15 peer link mac 60\ eb 5a 01 10\ b1 system mac 60\ eb 5a 01 10\ b1 peer mac 00 00 00 00 00 00 admin role none role active mclag interface lag 1 loglevel notice (2) verify vm connectivity vm a1 and vm b1 ping each other, can ping through, indicating successful configuration (3) check if the iccp backup channel is active disconnect the keep alive link and then check the mc lag status sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel backup channel（vlan4093） vrf name default consistency check action idle local ip 10 245 0 1 peer ip 10 245 0 2 dad local ip dad peer ip peer link interface lag 9999 keepalive time 1 dad detection delay 15 dad recovery delay mlag intf 60 dad recovery delay non mlag intf 0 dad vrf name default dad status disable session timeout 15 peer link mac 60\ eb 5a 01 10\ b1 system mac 60\ eb 5a 01 10\ b1 peer mac 00 00 00 00 00 00 admin role none role active mclag interface lag 1 loglevel notice example of mc lag scenario loop detection example of mc lag scenario loop detection topology configuration roadmap (1) check that the link status of the ports used on the switch is normal, and that all ports are stable in the up state (2) configure the switch interface ip addresses (3) configure vlan and configure ip address for vlan interface (4) configure link aggregation group and add it to vlan (5) configure mc lag (6) configure loop detection on the interface (7) configure static routes to ensure reachability of the service network procedure spine configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 1/30 exit ! interface ethernet 0/52 ip address 10 93 0 9/30 exit ! configure service vlan with vlan id 20 ! vlan 20 exit ! interface vlan 20 ip address 10 20 0 1/24 exit add the interface to the vlan ! interface ethernet 0/0 switchport trunk vlan 20 configure static routes ip route 10 10 0 1/24 10 93 0 2 ip route 10 10 0 1/24 10 93 0 10 leaf1 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 2/30 exit configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 1/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/68 switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 1 peer address 10 245 0 2 peer link link aggregation 9999 commit member lag 1 configure loopback detection on the interface ! interface link aggregation 1 loopback detect vlan 10 40 policy block vlan interval 20 wait time 60 configure static routes ip route 10 20 0 1/24 10 93 0 1 leaf2 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 10/30 exit configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 2/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/68 switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 2 peer address 10 245 0 1 peer link link aggregation 9999 commit member lag 1 configure loopback detection on the interface ! interface link aggregation 1 loopback detect vlan 10 40 policy block vlan interval 20 wait time 60 configure static routes ip route 10 20 0 1/24 10 93 0 9 verify the configuration (1) check the mc lag status (leaf1) sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel primary channel vrf name default consistency check action idle local ip 10 245 0 1 peer ip 10 245 0 2 dad local ip dad peer ip peer link interface lag 9999 keepalive time 1 dad detection delay 15 dad recovery delay mlag intf 60 dad recovery delay non mlag intf 0 dad vrf name default dad status disable session timeout 15 peer link mac 60\ eb 5a 01 10\ b1 system mac 60\ eb 5a 01 10\ b1 peer mac 00 00 00 00 00 00 admin role none role active mclag interface lag 1 loglevel notice (2) check the loop detection results sonic# show loopback detect port policy vlan interval wait time vlans with loop \ lag1 block vlan 10 40 20 60 10 example of two level mc lag configuration example of two level mc lag configuration networking requirements the customer needs to build a large layer 2 data center network using a two level mc lag approach to expand the number of accessible hosts topology configuration roadmap (1) check that the link status of the ports used on the switch is normal, and that all ports are stable in the up state (2) configure vlan and configure ip address for vlan interface (3) configure link aggregation group and add it to vlan (4) configure mc lag procedure leaf1 configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 1/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 2 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit ! interface ethernet 0/1 link aggregation group 1 exit ! interface ethernet 0/2 link aggregation group 2 exit ! interface ethernet 0/3 link aggregation group 2 exit configure mc lag mclag domain 1 local address 10 245 0 1 peer address 10 245 0 2 peer link link aggregation 9999 commit member lag 1 member lag 2 leaf2 configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 2/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 2 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit ! interface ethernet 0/1 link aggregation group 1 exit ! interface ethernet 0/2 link aggregation group 2 exit ! interface ethernet 0/3 link aggregation group 2 exit configure mc lag mclag domain 1 local address 10 245 0 2 peer address 10 245 0 1 peer link link aggregation 9999 commit member lag 1 member lag 2 leaf3 configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 4094 ip address 10 245 0 5/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 100 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 101 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit ! interface ethernet 0/1 link aggregation group 1 exit ! interface ethernet 0/2 link aggregation group 100 exit ! interface ethernet 0/3 link aggregation group 101 exit configure mc lag mclag domain 1 local address 10 245 0 5 peer address 10 245 0 6 peer link link aggregation 9999 commit member lag 1 member lag 100 member lag 101 leaf4 configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 4094 ip address 10 245 0 6/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 100 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 101 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit ! interface ethernet 0/1 link aggregation group 1 exit ! interface ethernet 0/2 link aggregation group 100 exit ! interface ethernet 0/3 link aggregation group 101 exit configure mc lag mclag domain 1 local address 10 245 0 6 peer address 10 245 0 5 peer link link aggregation 9999 commit member lag 1 member lag 100 member lag 101 leaf5 configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 4094 ip address 10 245 0 9/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 2 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 100 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 101 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 2 exit ! interface ethernet 0/1 link aggregation group 2 exit ! interface ethernet 0/2 link aggregation group 100 exit ! interface ethernet 0/3 link aggregation group 101 exit configure mc lag mclag domain 1 local address 10 245 0 9 peer address 10 245 0 10 peer link link aggregation 9999 commit member lag 2 member lag 100 member lag 101 leaf6 configure service vlan with vlan id 10, keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 4094 ip address 10 245 0 10/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 2 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 100 lacp fast rate commit switchport trunk vlan 10 ! interface link aggregation 101 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 2 exit ! interface ethernet 0/1 link aggregation group 2 exit ! interface ethernet 0/2 link aggregation group 100 exit ! interface ethernet 0/3 link aggregation group 101 exit configure mc lag mclag domain 1 local address 10 245 0 10 peer address 10 245 0 9 peer link link aggregation 9999 commit member lag 2 member lag 100 member lag 101 verify the configuration (1) check the mc lag status (leaf1) sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel primary channel vrf name default consistency check action idle local ip 10 245 0 1 peer ip 10 245 0 2 dad local ip dad peer ip peer link interface lag 9999 keepalive time 1 dad detection delay 15 dad recovery delay mlag intf 60 dad recovery delay non mlag intf 0 dad vrf name default dad status disable session timeout 15 peer link mac 60\ eb 5a 01 10\ b1 system mac 60\ eb 5a 01 10\ b1 peer mac 18 45 85 98 43 9b admin role none role active mclag interface lag 1 loglevel notice (2) verify server connectivity configure ips for the servers within the subnet of vlan 10 assign 10 10 0 11 to server1, 10 10 0 12 to server2, 10 10 0 13 to server3, and 10 10 0 14 to server4 server1, server2, server3, server4, ping each other, can ping through, indicating successful configuration example of mc lag l3 backup link configuration example of mc lag l3 backup link configuration networking requirements in the network, leaf1 and leaf2 are two independently operating devices, connected to spine respectively due to the high service reliability requirements of the customer, mc lag is required to be configured between leaf1 and leaf2 to achieve cross device link aggregation an additional l3 backup link needs to be configured to ensure that traffic can be forwarded normally in the event of an uplink failure topology configuration roadmap (1) check that the link status of the ports used on the switch is normal, and that all ports are stable in the up state (2) configure the switch interface ip addresses (3) configure vlan and configure ip address for vlan interface (4) configure link aggregation group and add it to vlan (5) configure mc lag (6) configure l3 backup link (7) configure static routes to ensure reachability of the service network procedure spine configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 1/30 exit ! interface ethernet 0/52 ip address 10 93 0 9/30 exit ! configure service vlan with vlan id 20 ! vlan 20 exit ! interface vlan 20 ip address 10 20 0 1/24 exit add the interface to the vlan ! interface ethernet 0/0 switchport trunk vlan 20 configure static routes ip route 10 10 0 1/24 10 93 0 2 ip route 10 10 0 1/24 10 93 0 10 leaf1 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 2/30 exit configure service vlan with vlan id 10 and keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 1/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 1 peer address 10 245 0 2 peer link link aggregation 9999 commit member lag 1 configure static routes ip route 10 20 0 1/24 10 93 0 1 configure l3 backup link ！ interface vlan 4094 ip ospf bfd ip ospf network point to point exit ! ospf enable ! router ospf network 10 245 0 1/30 area 0 0 0 0 network 10 20 0 1/24 area 0 0 0 0 leaf2 configure ip addresses for interfaces interface ethernet 0/48 ip address 10 93 0 10/30 exit configure service vlan with vlan id 10 and keep alive vlan with vlan id 4094 vlan 10 exit ! vlan 4094 exit ! interface vlan 10 ip address 10 10 0 1/24 mac address 00 00 00 10 00 00 exit ! interface vlan 4094 ip address 10 245 0 2/30 exit configure lag interface link aggregation 9999 mode static commit switchport trunk vlan 4094 switchport trunk vlan 10 exit ! interface link aggregation 1 lacp fast rate commit switchport trunk vlan 10 ! interface ethernet 0/72 link aggregation group 9999 exit ! interface ethernet 0/76 link aggregation group 9999 exit ! interface ethernet 0/0 link aggregation group 1 exit configure mc lag mclag domain 1 local address 10 245 0 2 peer address 10 245 0 1 peer link link aggregation 9999 commit member lag 1 configure static routes ip route 10 20 0 1/24 10 93 0 9 configure l3 backup link ！ interface vlan 4094 ip ospf bfd ip ospf network point to point ! ospf enable ! router ospf network 10 245 0 2/30 area 0 0 0 0 network 10 20 0 1/24 area 0 0 0 0 verify the configuration (1) check the mc lag status (leaf1) sonic# show mclag state the mclag's keepalive is ok mclag info sync is completed domain id 1 mclag session channel primary channel vrf name default consistency check action idle local ip 10 245 0 1 peer ip 10 245 0 2 dad local ip dad peer ip peer link interface lag 9999 keepalive time 1 dad detection delay 15 dad recovery delay mlag intf 60 dad recovery delay non mlag intf 0 dad vrf name default dad status disable session timeout 15 peer link mac 60\ eb 5a 01 10\ b1 system mac 60\ eb 5a 01 10\ b1 peer mac 00 00 00 00 00 00 admin role none role active mclag interface lag 1 loglevel notice (2) verify vm connectivity vm a1 and vm b1 ping each other, can ping through, indicating successful configuration