Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings
Discussion options

bluven
Jul 4, 2023

根据这个文档Use OpenELB in Layer 2 Mode做实验，机器环境如下：

kubectl get nodes -o wide
NAME       STATUS   ROLES                            AGE     VERSION                    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
edge1      Ready    agent,edge                       5h27m   v1.22.6-kubeedge-v1.12.2   10.22.48.13   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3
edge2      Ready    agent,edge                       5h24m   v1.22.6-kubeedge-v1.12.2   10.22.48.27   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3
node1      Ready    <none>                           5h42m   v1.22.5                    10.22.48.12   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3
shanghai   Ready    connector,control-plane,master   5h56m   v1.22.5                    10.22.48.21   <none>        CentOS Linux 7 (Core)   6.2.10-1.el7.elrepo.x86_64   docker://18.6.3

其中edge1, edge2是边缘节点。

openELB的相关pod状况如下:

[root@shanghai openelb]# kubectl get po -o wide
NAME                                READY   STATUS             RESTARTS         AGE   IP             NODE       NOMINATED NODE   READINESS GATES
layer2-openelb-7b4fdf6f85-554ds     1/1     Running            0                44m   10.233.66.16   edge1      <none>           <none>
layer2-openelb-7b4fdf6f85-nnszz     1/1     Running            0                44m   10.233.67.18   edge2      <none>           <none>
openelb-admission-create--1-4xpb5   0/1     Completed          0                53m   10.233.64.21   shanghai   <none>           <none>
openelb-admission-patch--1-dflkw    0/1     Completed          2                53m   10.233.64.20   shanghai   <none>           <none>
openelb-keepalive-vip-b8gkb         1/1     Running            0                51m   10.22.48.12    node1      <none>           <none>
openelb-keepalive-vip-j8jqq         0/1     CrashLoopBackOff   14 (3m57s ago)   51m   10.22.48.13    edge1      <none>           <none>
openelb-keepalive-vip-v9xmf         0/1     CrashLoopBackOff   14 (4m19s ago)   51m   10.22.48.27    edge2      <none>           <none>
openelb-keepalive-vip-xg4kd         1/1     Running            0                51m   10.22.48.21    shanghai   <none>           <none>
openelb-manager-76f5d5cf65-897hq    1/1     Running            0                53m   10.22.48.12    node1      <none>           <none>

EIP的配置如下:

kubectl describe eip layer2-eip -n openelb-system
Name:         layer2-eip
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  network.kubesphere.io/v1alpha2
Kind:         Eip
Metadata:
  Creation Timestamp:  2023-07-04T07:41:35Z
  Finalizers:
    finalizer.ipam.kubesphere.io/v1alpha1
Spec:
  Address:    10.22.48.200-10.22.48.210
  Interface:  eth0
  Protocol:   layer2
Status:
  First IP:   10.22.48.200
  Last IP:    10.22.48.210
  Pool Size:  11
  Ready:      true
  Usage:      1
  Used:
    10.22.48.200:  openelb-system/layer2-svc
  v4:              true
Events:            <none>

layer-svc2的状态如下:

[root@shanghai ~]# kubectl get svc
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
layer2-svc          LoadBalancer   10.233.38.100   10.22.48.200   80:31221/TCP   40m

[root@shanghai ~]# kubectl describe svc layer2-svc
Name:                     layer2-svc
Namespace:                openelb-system
Labels:                   eip.openelb.kubesphere.io/v1alpha2=layer2-eip
Annotations:              eip.openelb.kubesphere.io/v1alpha2: layer2-eip
                          layer2.openelb.kubesphere.io/v1alpha1: shanghai
                          lb.kubesphere.io/v1alpha1: openelb
                          protocol.openelb.kubesphere.io/v1alpha1: layer2
Selector:                 app=layer2-openelb
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.233.38.100
IPs:                      10.233.38.100
LoadBalancer Ingress:     10.22.48.200
Port:                     http  80/TCP
TargetPort:               8080/TCP
NodePort:                 http  31221/TCP
Endpoints:                10.233.66.16:8080,10.233.67.18:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

在另一个客户端机器(10.22.48.20)ping 10.22.48.200时，发现ping 不通

[root@gateway ~]# ping 10.22.48.200
PING 10.22.48.200 (10.22.48.200) 56(84) bytes of data.
From 10.22.48.21: icmp_seq=2 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=3 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=4 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=5 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=6 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=8 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=11 Redirect Host(New nexthop: 10.22.48.200)
From 10.22.48.21: icmp_seq=17 Redirect Host(New nexthop: 10.22.48.200)

在shanghai(10.22.48.21)机器上抓包结果如下:

bash-5.1# tcpdump -nn -i any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:34:06.368538 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 1, length 64
08:34:06.368681 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 1, length 64
08:34:07.367277 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 2, length 64
08:34:07.367362 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:07.367396 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 2, length 64
08:34:08.367326 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 3, length 64
08:34:08.367371 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:08.367396 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 3, length 64
08:34:09.367295 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 4, length 64
08:34:09.367358 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:09.367384 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 4, length 64
08:34:10.367306 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 5, length 64
08:34:10.367371 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:10.367404 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 5, length 64
08:34:11.367324 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 6, length 64
08:34:11.367388 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:11.367416 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 6, length 64
08:34:12.367302 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 7, length 64
08:34:12.367358 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 7, length 64
08:34:13.367285 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 8, length 64
08:34:13.367350 eth0  Out IP 10.22.48.21 > 10.22.48.20: ICMP redirect 10.22.48.200 to host 10.22.48.200, length 92
08:34:13.367377 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 8, length 64
08:34:14.367347 eth0  In  IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 9, length 64
08:34:14.367406 eth0  Out IP 10.22.48.20 > 10.22.48.200: ICMP echo request, id 2022, seq 9, length 64

需要一提的是，虽然ping的结果不理想，但服务还是通了

[root@gateway ~]# curl 10.22.48.200
You've hit layer2-openelb-7b4fdf6f85-nnszz
[root@gateway ~]# curl 10.22.48.200
You've hit layer2-openelb-7b4fdf6f85-554ds

用 ip neigh查看发现10.22.48.200相关的记录是FAILED状态，

[root@gateway ~]# ip neigh
10.22.48.200 dev eth0  FAILED
10.22.48.21 dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
10.22.48.11 dev eth0 lladdr fa:16:3e:ba:d4:52 STALE
10.22.48.254 dev eth0 lladdr 00:1e:08:0f:3b:ba REACHABLE
fe80::ea30:60b3:32e3:36f4 dev eth0 lladdr fa:16:3e:a2:8e:b3 STALE
2003::66:d dev eth0 lladdr fa:16:3e:15:62:a9 STALE
2003::66:5 dev eth0 lladdr fa:16:3e:86:96:86 STALE
fe80::5880:9aff:fe6c:da29 dev eth0 lladdr 5a:80:9a:6c:da:29 STALE
fe80::cac:79d1:cfe7:f69c dev eth0 lladdr fa:16:3e:15:62:a9 STALE
fe80::24f2:4d6c:a01e:7fbe dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
fe80::a4e5:68ff:fe4c:6730 dev eth0 lladdr a6:e5:68:4c:67:30 STALE

但如果是在执行curl 10.22.48.200后再执行ip neigh，就会发现

[root@gateway ~]# ip neigh
10.22.48.200 dev eth0 lladdr fa:16:3e:70:d5:b3 REACHABLE
10.22.48.21 dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
10.22.48.11 dev eth0 lladdr fa:16:3e:ba:d4:52 STALE
10.22.48.254 dev eth0 lladdr 00:1e:08:0f:3b:ba REACHABLE
fe80::ea30:60b3:32e3:36f4 dev eth0 lladdr fa:16:3e:a2:8e:b3 STALE
2003::66:d dev eth0 lladdr fa:16:3e:15:62:a9 STALE
2003::66:5 dev eth0 lladdr fa:16:3e:86:96:86 STALE
fe80::5880:9aff:fe6c:da29 dev eth0 lladdr 5a:80:9a:6c:da:29 STALE
fe80::cac:79d1:cfe7:f69c dev eth0 lladdr fa:16:3e:15:62:a9 STALE
fe80::24f2:4d6c:a01e:7fbe dev eth0 lladdr fa:16:3e:70:d5:b3 STALE
fe80::a4e5:68ff:fe4c:6730 dev eth0 lladdr a6:e5:68:4c:67:30 STALE
You must be logged in to vote
Answered by renyunkang Jul 5, 2023

这个是符合预期的，openelb 对于分配的 loadbalancer ip 不会绑定到物理网卡中，而 svc 本身也应用于四层的转发，因此对于 icmp 协议无法做出回应

View full answer

Replies: 1 comment · 2 replies

Comment options

renyunkang
Jul 5, 2023
Maintainer

这个是符合预期的，openelb 对于分配的 loadbalancer ip 不会绑定到物理网卡中，而 svc 本身也应用于四层的转发，因此对于 icmp 协议无法做出回应
You must be logged in to vote
2 replies
@bluven
Comment options

bluven Jul 5, 2023
Author

那为什么你们的例子看起来很正常？
@renyunkang
Comment options

renyunkang Jul 5, 2023
Maintainer

感谢提醒，这块文档上确实有一些不完善。
我们默认的使用的是 ipvs 的 kube-proxy，对于 svc 的 ip，会写到 kube-ipvs0 网卡上，因此会响应 icmp 协议。
Answer selected by bluven
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
2 participants
@bluven @renyunkang
Morty Proxy This is a proxified and sanitized view of the page, visit original site.