Istio的sidecar注入

之前那篇文章太长,直接导致网站的search模块废掉,现单独将sidecar的注入这块内容拉出来讲,也方面后面再更新更多内容。

K8S的MutatingAdmissionWebhook支持两种类型:validatemutate对应ValidatingAdmissionWebhookMutatingAdmissionWebhook,其中validate用来配置验证,mutate用来修改对象,这里我们使用mutate来注入。

MutatingWebhookConfiguration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[root@k8s-master ~]# kc get mutatingwebhookconfiguration
NAME CREATED AT
istio-sidecar-injector 2019-03-31T13:55:50Z
[root@k8s-master ~]# kc get mutatingwebhookconfiguration -o yaml
apiVersion: v1
items:
- apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
...
labels:
app: istio-sidecar-injector
chart: sidecarInjectorWebhook-1.0.4
heritage: Tiller
release: istio
name: istio-sidecar-injector
...
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: ...
service:
name: istio-sidecar-injector
namespace: istio-system
path: /inject
failurePolicy: Fail
name: sidecar-injector.istio.io
namespaceSelector:
matchLabels:
istio-injection: enabled
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: Unknown
timeoutSeconds: 30
kind: List
metadata:
resourceVersion: ""
selfLink: ""

当webhook中配置了namespaceSelector,该namespace的对象被调用的时候,会去clientConfig中定义的服务去请求webhook(也就是在这个时候注入的)。在service中可以看到对应的服务,基于此,我们可以查询到对应的pod。

1
2
3
4
5
6
7
[root@k8s-master ~]# kci get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-sidecar-injector ClusterIP 10.110.47.206 <none> 443/TCP 8d

[root@k8s-master ~]# kci get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
istio-sidecar-injector-8477498cbf-mgjjb 1/1 Running 1 8d 10.244.0.132 k8s-master <none> <none>

配置文件

至于具体注入了些什么内容,需要来查看配置文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
[root@k8s-master ~]# kci describe deploy istio-sidecar-injector
Name: istio-sidecar-injector
Namespace: istio-system
CreationTimestamp: Sun, 31 Mar 2019 21:55:50 +0800
Labels: app=sidecarInjectorWebhook
chart=sidecarInjectorWebhook-1.0.4
heritage=Tiller
istio=sidecar-injector
release=istio
...
Pod Template:
...
Containers:
sidecar-injector-webhook:
Image: docker.io/istio/sidecar_injector:1.0.4
Port: <none>
Host Port: <none>
Args:
--caCertFile=/etc/istio/certs/root-cert.pem
--tlsCertFile=/etc/istio/certs/cert-chain.pem
--tlsKeyFile=/etc/istio/certs/key.pem
--injectConfig=/etc/istio/inject/config
--meshConfig=/etc/istio/config/mesh
--healthCheckInterval=2s
--healthCheckFile=/health
...
Mounts:
/etc/istio/certs from certs (ro)
/etc/istio/config from config-volume (ro)
/etc/istio/inject from inject-config (ro)
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio
Optional: false
certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.istio-sidecar-injector-service-account
Optional: false
inject-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-sidecar-injector
Optional: false
...

这里使用了几个配置文件,都是从configmap中获取到的,分别来看看。

/etc/istio/config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
[root@k8s-master ~]# kci get cm istio  -o=go-template='{{ .data.mesh}}'                
# Set the following variable to true to disable policy checks by the Mixer.
# Note that metrics will still be reported to the Mixer.
disablePolicyChecks: false

# Set enableTracing to false to disable request tracing.
enableTracing: true

# Set accessLogFile to empty string to disable access log.
accessLogFile: "/dev/stdout"
#
# Deprecated: mixer is using EDS
mixerCheckServer: istio-policy.istio-system.svc.cluster.local:9091
mixerReportServer: istio-telemetry.istio-system.svc.cluster.local:9091

# policyCheckFailOpen allows traffic in cases when the mixer policy service cannot be reached.
# Default is false which means the traffic is denied when the client is unable to connect to Mixer.
policyCheckFailOpen: false

# Unix Domain Socket through which envoy communicates with NodeAgent SDS to get
# key/cert for mTLS. Use secret-mount files instead of SDS if set to empty.
sdsUdsPath: ""

# How frequently should Envoy fetch key/cert from NodeAgent.
sdsRefreshDelay: 15s

#
defaultConfig:
#
# TCP connection timeout between Envoy & the application, and between Envoys.
connectTimeout: 10s
#
### ADVANCED SETTINGS #############
# Where should envoy's configuration be stored in the istio-proxy container
configPath: "/etc/istio/proxy"
binaryPath: "/usr/local/bin/envoy"
# The pseudo service name used for Envoy.
serviceCluster: istio-proxy
# These settings that determine how long an old Envoy
# process should be kept alive after an occasional reload.
drainDuration: 45s
parentShutdownDuration: 1m0s
#
# The mode used to redirect inbound connections to Envoy. This setting
# has no effect on outbound traffic: iptables REDIRECT is always used for
# outbound connections.
# If "REDIRECT", use iptables REDIRECT to NAT and redirect to Envoy.
# The "REDIRECT" mode loses source addresses during redirection.
# If "TPROXY", use iptables TPROXY to redirect to Envoy.
# The "TPROXY" mode preserves both the source and destination IP
# addresses and ports, so that they can be used for advanced filtering
# and manipulation.
# The "TPROXY" mode also configures the sidecar to run with the
# CAP_NET_ADMIN capability, which is required to use TPROXY.
#interceptionMode: REDIRECT
#
# Port where Envoy listens (on local host) for admin commands
# You can exec into the istio-proxy container in a pod and
# curl the admin port (curl http://localhost:15000/) to obtain
# diagnostic information from Envoy. See
# https://lyft.github.io/envoy/docs/operations/admin.html
# for more details
proxyAdminPort: 15000
#
# Set concurrency to a specific number to control the number of Proxy worker threads.
# If set to 0 (default), then start worker thread for each CPU thread/core.
concurrency: 0
#
# Zipkin trace collector
zipkinAddress: zipkin.istio-system:9411
#
# Mutual TLS authentication between sidecars and istio control plane.
controlPlaneAuthPolicy: NONE
#
# Address where istio Pilot service is running
discoveryAddress: istio-pilot.istio-system:15007

/etc/istio/inject

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
[root@k8s-master ~]# kci get cm istio-sidecar-injector -o=go-template={{.data.config}}
policy: enabled
template: |-
initContainers:
- name: istio-init
image: "docker.io/istio/proxy_init:1.0.4"
args:
- "-p"
- [[ .MeshConfig.ProxyListenPort ]]
- "-u"
- 1337
- "-m"
- [[ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode ]]
- "-i"
- "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges` "*" ]]"
- "-x"
- "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges` "" ]]"
- "-b"
- "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` (includeInboundPorts .Spec.Containers) ]]"
- "-d"
- "[[ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port` 0 ) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts` "" ) ]]"
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
add:
- NET_ADMIN
privileged: true
restartPolicy: Always
containers:
- name: istio-proxy
image: [[ annotation .ObjectMeta `sidecar.istio.io/proxyImage` "docker.io/istio/proxyv2:1.0.4" ]]

ports:
- containerPort: 15090
protocol: TCP
name: http-envoy-prom

args:
- proxy
- sidecar
- --configPath
- [[ .ProxyConfig.ConfigPath ]]
- --binaryPath
- [[ .ProxyConfig.BinaryPath ]]
- --serviceCluster
[[ if ne "" (index .ObjectMeta.Labels "app") -]]
- [[ index .ObjectMeta.Labels "app" ]]
[[ else -]]
- "istio-proxy"
[[ end -]]
- --drainDuration
- [[ formatDuration .ProxyConfig.DrainDuration ]]
- --parentShutdownDuration
- [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
- --discoveryAddress
- [[ annotation .ObjectMeta `sidecar.istio.io/discoveryAddress` .ProxyConfig.DiscoveryAddress ]]
- --discoveryRefreshDelay
- [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]
- --zipkinAddress
- [[ .ProxyConfig.ZipkinAddress ]]
- --connectTimeout
- [[ formatDuration .ProxyConfig.ConnectTimeout ]]
- --proxyAdminPort
- [[ .ProxyConfig.ProxyAdminPort ]]
[[ if gt .ProxyConfig.Concurrency 0 -]]
- --concurrency
- [[ .ProxyConfig.Concurrency ]]
[[ end -]]
- --controlPlaneAuthPolicy
- [[ annotation .ObjectMeta `sidecar.istio.io/controlPlaneAuthPolicy` .ProxyConfig.ControlPlaneAuthPolicy ]]
[[- if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port` 0 ) "0") ]]
- --statusPort
- [[ annotation .ObjectMeta `status.sidecar.istio.io/port` 0 ]]
- --applicationPorts
- "[[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/applicationPorts` (applicationPorts .Spec.Containers) ]]"
[[- end ]]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: ISTIO_META_INTERCEPTION_MODE
value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
[[ if .ObjectMeta.Annotations ]]
- name: ISTIO_METAJSON_ANNOTATIONS
value: |
[[ toJson .ObjectMeta.Annotations ]]
[[ end ]]
[[ if .ObjectMeta.Labels ]]
- name: ISTIO_METAJSON_LABELS
value: |
[[ toJson .ObjectMeta.Labels ]]
[[ end ]]
imagePullPolicy: IfNotPresent
[[ if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port` 0 ) "0") ]]
readinessProbe:
httpGet:
path: /healthz/ready
port: [[ annotation .ObjectMeta `status.sidecar.istio.io/port` 0 ]]
initialDelaySeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds` 1 ]]
periodSeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds` 2 ]]
failureThreshold: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold` 30 ]]
[[ end -]]securityContext:

readOnlyRootFilesystem: true
[[ if eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) "TPROXY" -]]
capabilities:
add:
- NET_ADMIN
runAsGroup: 1337
[[ else -]]
runAsUser: 1337
[[ end -]]
restartPolicy: Always
resources:
[[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -]]
requests:
cpu: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` ]]"
memory: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` ]]"
[[ else -]]
requests:
cpu: 10m

[[ end -]]
volumeMounts:
- mountPath: /etc/istio/proxy
name: istio-envoy
- mountPath: /etc/certs/
name: istio-certs
readOnly: true
volumes:
- emptyDir:
medium: Memory
name: istio-envoy
- name: istio-certs
secret:
optional: true
[[ if eq .Spec.ServiceAccountName "" -]]
secretName: istio.default
[[ else -]]
secretName: [[ printf "istio.%s" .Spec.ServiceAccountName ]]
[[ end -]]
0%