POD 통신 암호화(네트워크 레벨)
WireGuard
IPSec, OpenVPN의 대항으로 만들어진 OSS VPN Project이며, Kernel 5.6에 버전 1.0.0이 포함되었다. 커널단에서 처리되기 때문에 간결하고 빠르다.
Calico /w WireGuard
WireGuard 설정 추가
calicoctl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true}}'
확인
calicoctl get felixconfiguration default -o yaml | grep wireguardEnabled
wireguardEnabled: true
public key 확인
calicoctl get node -o yaml | grep wireguardPublicKey
wireguardPublicKey: LIiPIm83EWSbDflfv32XXG6Uu8YuLp6WPvr0OV6mZDI=
wireguardPublicKey: lrgwZmoOVfmmpWYUN6r2k80iUtH9+LQggv/DFGO7Uys=
wireguardPublicKey: SiAGfn8R7+zHlt7JOD6pnkPfHG6gqnAH36x7YjkKmSU=
wireguardPublicKey: Cbx47lafvq8Wb0kBuEhmClAkVXQrOWiO3IIdARWJK0U=
Worker0 인터페이스 확인
ip -c -d addr show wireguard.cali
12: wireguard.cali: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000
link/none promiscuity 0 minmtu 0 maxmtu 2147483552
wireguard numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 172.16.34.4/32 scope global wireguard.cali
valid_lft forever preferred_lft forever
ifconfig wireguard.cali
wireguard.cali: flags=209<UP,POINTOPOINT,RUNNING,NOARP> mtu 1440
inet 172.16.34.4 netmask 255.255.255.255 destination 172.16.34.4
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 1000 (UNSPEC)
RX packets 1 bytes 148 (148.0 B)
RX errors 17 dropped 0 overruns 0 frame 17
TX packets 3 bytes 156 (156.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wireguard.cali 정보 확인
wg showconf wireguard.cali
[Interface]
ListenPort = 51820
FwMark = 0x100000
PrivateKey = 0HPGb/sB89CS/DF7kEpoxAS4bGum8WZ9vUMSYdOpO3E=
[Peer]
PublicKey = SiAGfn8R7+zHlt7JOD6pnkPfHG6gqnAH36x7YjkKmSU=
AllowedIPs = 172.16.158.0/24, 172.16.158.0/32, 172.16.158.1/32
Endpoint = 192.168.10.101:51820
[Peer]
PublicKey = Cbx47lafvq8Wb0kBuEhmClAkVXQrOWiO3IIdARWJK0U=
AllowedIPs = 172.16.184.0/24, 172.16.184.0/32, 172.16.184.1/32
Endpoint = 192.168.10.102:51820
[Peer]
PublicKey = LIiPIm83EWSbDflfv32XXG6Uu8YuLp6WPvr0OV6mZDI=
AllowedIPs = 172.16.116.0/24, 172.16.116.0/32, 172.16.116.2/32
Endpoint = 192.168.10.10:51820
peer 정보
wg show
interface: wireguard.cali
public key: lrgwZmoOVfmmpWYUN6r2k80iUtH9+LQggv/DFGO7Uys=
private key: (hidden)
listening port: 51820
fwmark: 0x100000
peer: LIiPIm83EWSbDflfv32XXG6Uu8YuLp6WPvr0OV6mZDI=
endpoint: 192.168.10.10:51820
allowed ips: 172.16.116.0/24, 172.16.116.0/32, 172.16.116.2/32
latest handshake: 6 minutes, 44 seconds ago
transfer: 148 B received, 156 B sent
peer: SiAGfn8R7+zHlt7JOD6pnkPfHG6gqnAH36x7YjkKmSU=
endpoint: 192.168.10.101:51820
allowed ips: 172.16.158.0/24, 172.16.158.0/32, 172.16.158.1/32
peer: Cbx47lafvq8Wb0kBuEhmClAkVXQrOWiO3IIdARWJK0U=
endpoint: 192.168.10.102:51820
allowed ips: 172.16.184.0/24, 172.16.184.0/32, 172.16.184.1/32
wireguard 적용 해제
calicoctl patch felixconfiguration default --patch '{"spec":{"wireguardEnabled": false}}'
worker0 내 인터페이스 정보 확인
ip -c link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 02:6d:da:b3:d4:d3 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:95:ec:cd brd ff:ff:ff:ff:ff:ff
4: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/ether 66:8d:da:b4:58:1a brd ff:ff:ff:ff:ff:ff
5: cali53ccf54a47b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-150e7a2d-7daa-bdbd-de1e-b7efb50d8580
6: calibf28d443af8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-bc4b5fdf-5856-0a78-742d-ee762e388ac4
Network Policy
K8s에서 POD간 통신시 트래픽 룰을 정하는 것으로, 미사용시 POD간 통신은 서로 통신이 가능하다. Network Policy 사용시 네임스페이스별로 트래픽 전송이 안되거나 특정한 파드 통신만 지원하도록 할수도 있다. CNI도 지원을 해야 이 기능을 사용할 수 있다. 기본적으로 이그레스/인그레스로 구성되며, 설정범위를 podSelector로 지정하며, 네임스페이스별로 구성해야한다.
실습
POD 생성
calicoctl get wep -n default;echo;calicoctl get wep -n nptest
NAMESPACE WORKLOAD NODE NETWORKS INTERFACE
default sample-pod-np1 k8s-w1 172.16.158.3/32 cali19b0bcd175f
default sample-pod-np2 k8s-w2 172.16.184.4/32 calicc8bba29d1b
NAMESPACE WORKLOAD NODE NETWORKS INTERFACE
nptest sample-pod-np3 k8s-w0 172.16.34.6/32 cali8e3546b3e16
nptest sample-pod-np4 k8s-w1 172.16.158.4/32 cali3fd7eae6008
네임스페이스별 레이블 지정
kubectl label ns default ns=default
kubectl label ns nptest ns=nptest
임시 변수 지정
DEFAULTPOD1=$(calicoctl get workloadEndpoint | grep np1 | awk '{print $3}' | cut -d "/" -f 1)
echo $DEFAULTPOD1
DEFAULTPOD2=$(calicoctl get workloadEndpoint | grep np2 | awk '{print $3}' | cut -d "/" -f 1)
echo $DEFAULTPOD2
NPTESTPOD3=$(calicoctl get workloadEndpoint -n nptest | grep np3 | awk '{print $4}' | cut -d "/" -f 1)
echo $NPTESTPOD3
NPTESTPOD4=$(calicoctl get workloadEndpoint -n nptest | grep np4 | awk '{print $4}' | cut -d "/" -f 1)
echo $NPTESTPOD4
파드간 통신 상태 확인 : 정상통신 확인
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 $DEFAULTPOD2
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 $NPTESTPOD3
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 $NPTESTPOD4
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 $DEFAULTPOD2 | grep nginx!
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 $NPTESTPOD3 | grep nginx!
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 $NPTESTPOD4 | grep nginx!
파드 -> 인터넷 통신 확인 : 정상 통신 확인
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 www.google.com
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 ipinfo.io/city
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 ipinfo.io/org
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 ipinfo.io/loc
인바운드 차단, 아웃 바운드 허용 적용
cat <<EOF> cloud-networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: cloud-networkpolicy
spec:
podSelector: {}
egress:
- {}
policyTypes:
- Ingress
- Egress
EOF
default 네임스페이스
k apply -f cloud-networkpolicy.yaml
networkpolicy.networking.k8s.io/cloud-networkpolicy created
nptest 네임스페이스
kubectl apply -n nptest -f cloud-networkpolicy.yaml
networkpolicy.networking.k8s.io/cloud-networkpolicy created
파드간 통신 상태 확인 : 통신 불가
PING 172.16.184.4 (172.16.184.4): 56 data bytes
--- 172.16.184.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1
PING 172.16.34.6 (172.16.34.6): 56 data bytes
--- 172.16.34.6 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1
PING 172.16.158.4 (172.16.158.4): 56 data bytes
--- 172.16.158.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1
command terminated with exit code 28
command terminated with exit code 28
command terminated with exit code 28
파드 -> 인터넷 통신 확인 : 정상 통신 확인
Calico 기능사용(GlobalNetworkPolicy:GNP)
cat <<EOF> gnp-allnamespaces-deny.yaml
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
name: default-app-policy
spec:
namespaceSelector: has(projectcalico.org/name) && projectcalico.org/name not in {"kube-system", "calico-system"}
types:
- Ingress
- Egress
EOF
적용
kubectl apply -f gnp-allnamespaces-deny.yaml
globalnetworkpolicy.crd.projectcalico.org/default-app-policy created
POD IP 확인
calicoctl get wep -A
NAMESPACE WORKLOAD NODE NETWORKS INTERFACE
default sample-pod-np1 k8s-w1 172.16.158.3/32 cali19b0bcd175f
default sample-pod-np2 k8s-w2 172.16.184.4/32 calicc8bba29d1b
kube-system calico-kube-controllers-77d59654f4-mhrj5 k8s-m 172.16.116.3/32 calidb0bb8d526a
kube-system coredns-55cb58b774-4zbmz k8s-w0 172.16.34.1/32 cali53ccf54a47b
kube-system coredns-55cb58b774-lnvp7 k8s-w0 172.16.34.2/32 calibf28d443af8
nptest sample-pod-np3 k8s-w0 172.16.34.6/32 cali8e3546b3e16
nptest sample-pod-np4 k8s-w1 172.16.158.4/32 cali3fd7eae6008
host -> POD 통신 : 정상
ping 172.16.34.1
PING 172.16.34.1 (172.16.34.1) 56(84) bytes of data.
64 bytes from 172.16.34.1: icmp_seq=1 ttl=63 time=1.55 ms
POD 간 통신 : 불가
--- 172.16.184.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1
PING 172.16.34.6 (172.16.34.6): 56 data bytes
POD - 인터넷 통신 : 불가
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 www.google.com
ping: bad address 'www.google.com'
특정 레이블이 설정된 POD에서 오는 통신만 허가하기
app=np2 레이블을 가진 POD는 app=np1 레이블을 가진 POD의 통신중 80 포트만 허가한다.
cat <<EOF> sample-podselector-ingress-networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: sample-podselector-ingress-networkpolicy
spec:
podSelector:
matchLabels:
app: np2
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: np1
ports:
- protocol: TCP
port: 80
EOF
적용
k apply -f sample-podselector-ingress-networkpolicy.yaml
networkpolicy.networking.k8s.io/sample-podselector-ingress-networkpolicy created
테스트
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 $DEFAULTPOD2 | grep nginx!
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 $DEFAULTPOD2
<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>
PING 172.16.184.4 (172.16.184.4): 56 data bytes
--- 172.16.184.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1
특정 네임스페이스에서 오는 통신만 허가하기
default 네임스페이스에서 오는 트래픽을 nptest 네임스페이스의 np3의 80 포트만 받도록 한다.
cat <<EOF> sample-namespaceselector-ingress-networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: sample-namespaceselector-ingress-networkpolicy
namespace: nptest
spec:
podSelector:
matchLabels:
app: np3
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
ns: default
ports:
- protocol: TCP
port: 80
EOF
테스트
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 $NPTESTPOD3 | grep nginx!
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 $NPTESTPOD3
<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>
PING 172.16.34.6 (172.16.34.6): 56 data bytes
--- 172.16.34.6 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1
특정 IP 블록에서 들어오는 통신 허가하기
특정 iP 블록에서 nptest 네임스페이스의 np4에서 80 포트만 받을수 있도록 한다.
cat <<EOF> sample-ipblock-ingress-networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: sample-ipblock-ingress-networkpolicy
namespace: nptest
spec:
podSelector:
matchLabels:
app: np4
policyTypes:
- Ingress
ingress:
- from:
- ipBlock:
cidr: $DEFAULTPOD1/32
ports:
- protocol: TCP
port: 80
EOF
적용 및 테스트
kubectl apply -f sample-ipblock-ingress-networkpolicy.yaml
kubectl exec -it sample-pod-np1 -- curl -s --connect-timeout 2 $NPTESTPOD4 | grep nginx!
kubectl exec -it sample-pod-np1 -- ping -i 1 -W 1 -c 1 $NPTESTPOD4
<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>
PING 172.16.158.4 (172.16.158.4): 56 data bytes
--- 172.16.158.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
command terminated with exit code 1
번외
POD에 static ip 지정 배포하기
클러스터내 pod 목록
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default sample-pod-np1 1/1 Running 0 23m 172.16.158.3 k8s-w1 <none> <none>
default sample-pod-np2 1/1 Running 0 23m 172.16.184.4 k8s-w2 <none> <none>
kube-system calico-kube-controllers-77d59654f4-mhrj5 1/1 Running 0 35m 172.16.116.3 k8s-m <none> <none>
kube-system calico-node-2b965 1/1 Running 0 27m 192.168.10.101 k8s-w1 <none> <none>
kube-system calico-node-7fmhw 1/1 Running 0 27m 192.168.10.102 k8s-w2 <none> <none>
kube-system calico-node-f9jws 1/1 Running 0 27m 192.168.20.100 k8s-w0 <none> <none>
kube-system calico-node-lw5wf 1/1 Running 0 27m 192.168.10.10 k8s-m <none> <none>
kube-system coredns-55cb58b774-4zbmz 1/1 Running 0 69m 172.16.34.1 k8s-w0 <none> <none>
kube-system coredns-55cb58b774-lnvp7 1/1 Running 0 69m 172.16.34.2 k8s-w0 <none> <none>
kube-system etcd-k8s-m 1/1 Running 0 69m 192.168.10.10 k8s-m <none> <none>
kube-system kube-apiserver-k8s-m 1/1 Running 0 69m 192.168.10.10 k8s-m <none> <none>
kube-system kube-controller-manager-k8s-m 1/1 Running 0 69m 192.168.10.10 k8s-m <none> <none>
kube-system kube-proxy-2r6h2 1/1 Running 0 66m 192.168.20.100 k8s-w0 <none> <none>
kube-system kube-proxy-99rf6 1/1 Running 0 69m 192.168.10.10 k8s-m <none> <none>
kube-system kube-proxy-nh8c9 1/1 Running 0 65m 192.168.10.101 k8s-w1 <none> <none>
kube-system kube-proxy-snxm4 1/1 Running 0 64m 192.168.10.102 k8s-w2 <none> <none>
kube-system kube-scheduler-k8s-m 1/1 Running 0 69m 192.168.10.10 k8s-m <none> <none>
nptest sample-pod-np3 1/1 Running 0 23m 172.16.34.6 k8s-w0 <none> <none>
nptest sample-pod-np4 1/1 Running 0 23m 172.16.158.4 k8s-w1 <none> <none>
pod 명세 및 추가
cat <<EOF| kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: static-ip-pod
annotations:
"cni.projectcalico.org/ipAddrs": "[\"172.16.100.100\"]"
spec:
containers:
- name: static-ip-pod
image: nicolaka/netshoot
command: ["tail"]
args: ["-f", "/dev/null"]
terminationGracePeriodSeconds: 0
EOF
생성 확인
k get pods -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default sample-pod-np1 1/1 Running 0 24m 172.16.158.3 k8s-w1 <none> <none>
default sample-pod-np2 1/1 Running 0 24m 172.16.184.4 k8s-w2 <none> <none>
default static-ip-pod 1/1 Running 0 19s 172.16.100.100 k8s-w2 <none> <none>
kube-system calico-kube-controllers-77d59654f4-mhrj5 1/1 Running 0 36m 172.16.116.3 k8s-m <none> <none>
Floating IP를 POD에 할당하기
calico config 수정
k edit configmap -n kube-system calico-config
plugins 항목에 추가
...
"kubernetes": {
"kubeconfig": "__KUBECONFIG_FILEPATH__"
},
"feature_control": {
"floating_ips": true
}
},
....
설정 적용을 위한 재시작
kubectl delete pod -n kube-system -l k8s-app=calico-node
라우팅 테이블 갱신
route -n && echo && ip link set enp0s8 down && sleep 1 && ip link set enp0s8 up && route -n
felixconfigurations.crd.projectcalico.org 항목 수정
k edit felixconfigurations.crd.projectcalico.org default
...
floatingIPs: Enabled => Disabled에서 변경한다.
...
POD 명세 및 추가
cat <<EOF| kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: nginx-sample
labels:
app: nginx
annotations:
"cni.projectcalico.org/floatingIPs": "[\"172.16.200.200\"]"
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
EOF
테스트
curl http://172.16.200.200
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
라우팅 테이블 확인
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
....
172.16.200.200 192.168.10.101 255.255.255.255 UGH 0 0 0 tunl0
...