存储服务器不想放在 OVH 了。所以自己来托管一台机器，顺便折腾下 2x1Gbps 组 LACP Bonding。

前提：服务器需要至少 2 个千兆物理网卡，上联交换机支持 802.3ad。

配置交换机

这里使用的是 Cisco Nexus 3064PQ-10GE 交换机，我们的接口在 Eth1/21-22，port-channel 的配置如下：

1
2
3
4
5
6
7
8
9
# show interface trunk

--------------------------------------------------------------------------------
Port          Native  Status        Port
              Vlan                  Channel
--------------------------------------------------------------------------------
Eth1/21       1       trnk-bndl     Po100
Eth1/22       1       trnk-bndl     Po100
Po100         1       trunking      --
1
2
3
4
5
6
7
8
9
10
show port-channel database
port-channel100
    Last membership update is successful
    2 ports in total, 2 ports up
    First operational port is Ethernet1/21
    Age of the port-channel is 0d:00h:02m:16s
    Time since last bundle is 0d:00h:02m:04s
    Last bundled member is Ethernet1/22
    Ports:   Ethernet1/21    [active ] [up] *
             Ethernet1/22    [active ] [up]

配置服务器

服务器操作系统是 Arch Linux，由于蜜汁问题 netctl 无法启动网卡，就只好用 systemd-networkd 啦。

麻烦一些，但是也还算顺利。与往常一样，折腾服务器网络的时候需要备着 IPMI 以防 connection lost。

内核模块

需要加载 bonding 模块。将模块名写入列表，文件 /etc/modules-load.d/bonding.conf，内容只需要一行：

1
bonding

先别急着加载模块，为了防止模块自动建立一个默认网卡影响后续配置，以及设置 LACP Mode=4 … 等等，先加入一行参数。文件 /etc/modprobe.d/bonding.conf

1
options bonding mode=4 miimon=100 max_bonds=0

然后安装 ifenslave 包，再 modprobe bonding 即可。

bonding 虚拟网卡

首先创建一个虚拟网卡的设备。文件 /etc/systemd/network/bond0.netdev 内容为

1
2
3
4
5
6
7
8
9
[NetDev]
Name=bond0
Kind=bond

[Bond]
Mode=802.3ad
TransmitHashPolicy=layer2+3
LACPTransmitRate=fast
AdSelect=bandwidth

然后在此虚拟网卡上创建网络。这里使用两个物理网卡 eth0 和 eth1 作为 bundle，交换机上的 VLAN id 是 113。文件 /etc/systemd/network/bond0.network 内容为

1
2
3
4
5
6
[Match]
Name=bond0

[Network]
VLAN=vlan113
BindCarrier=eth0 eth1

接下来分别为 eth0 和 eth1 建立网络设置。

  • /etc/systemd/network/eth0.network
1
2
3
4
5
[Match]
Name=eth0

[Network]
Bond=bond0
  • /etc/systemd/network/eth1.network
1
2
3
4
5
[Match]
Name=eth1

[Network]
Bond=bond0

最后是 VLAN 的设置。前面设置了上联 VLAN id 是 113，这里分别建立 VLAN 的虚拟网卡(based on bond0) 并设置网络(IP, etc)。

  • /etc/systemd/network/vlan113.netdev
1
2
3
4
5
6
[NetDev]
Name=vlan113
Kind=vlan

[VLAN]
Id=113
  • /etc/systemd/network/vlan113.network
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[Match]
Name=vlan113

[Network]
VLAN=vlan113

[Address]
Address=10.1.0.100/24

[Route]
Destination=0.0.0.0/0
Gateway=10.1.0.1
DNS=1.1.1.1

[Address]
Address=2600:x:x:x::2/64

[Route]
Gateway=2600:x:x:x::1

多个地址、IPv6 等可以写多个 [Address] 和 [Route]。

至此就完成啦。开启 systemd-networkd 的自启动：

1
systemctl enable systemd-networkd.service

然后重启网络：

1
systemctl restart systemd-networkd.service

如果配置都没有问题，网络会中断十几秒然后恢复。现在查看网卡列表已经可以看到组合的网卡了：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether <REDACTED> brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether <REDACTED> brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether <REDACTED> brd ff:ff:ff:ff:ff:ff
5: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether <REDACTED> brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether <REDACTED> brd ff:ff:ff:ff:ff:ff
7: vlan113@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether <REDACTED> brd ff:ff:ff:ff:ff:ff

ethtool 查看 bond0 的速率显示 2000Mb/s：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# ethtool bond0
Settings for bond0:
        Supported ports: [ ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 2000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Link detected: yes

搞定收工(‘・ω・’)

Reference: