Due to business requirements, a single Service is needed to manage multiple deployments for a simple grayscale scheme. When I associated a deployment tag change to an existing Service, I found that the traffic was skewed. Traffic access path: nginx --> Service nodePort --> deployment1 |--> deployment2
It was found that when grayscale a new deployment pod, severe traffic skew occurred. Recurrence scheme first run with deployment-1, the traffic is load balanced, but when deployment-2 is enabled, the traffic will tilt. Simulation request
for i in seq 1 10000;do sleep 1 ; curl -s http://10.4.83.27:8081/ ;done | tee /tmp/1.log
First time cat /tmp/12.log | grep http-deployment-1 | wc -l 94 cat /tmp/12.log | grep http-deployment-2 | wc -l 6
Try again cat /tmp/13.log | grep http-deployment-2 | wc -l 378 cat /tmp/13.log | grep http-deployment-1 | wc -l 622
Upstream load balancing configuration
upstream backend {
server 10.4.87.33:33632;
server 10.4.87.34:33632;
server 10.4.87.35:33632;
keepalive 300;
}
server {
listen 8081;
server_name ljohn.com;
location / {
proxy_pass http://backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
deployment-1
Deployment 2
apiVersion: apps/v1
kind: Deployment
metadata:
annotations: {}
name: http-deployment-1
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: nginx
deployment: one
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: nginx
deployment: one
spec:
containers:
- image: nginx:1.20.1
imagePullPolicy: IfNotPresent
name: http
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/conf.d
- name: html-files
mountPath: /usr/share/nginx/html
volumes:
- name: nginx-config
configMap:
name: nginx-config
- name: html-files
configMap:
name: html-files-config-one
---
# ConfigMap for NGINX configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
data:
default.conf: |
server {
listen 80;
server_name localhost;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location / {
root /usr/share/nginx/html;
index index.html;
}
}
---
# ConfigMap for HTML file
apiVersion: v1
kind: ConfigMap
metadata:
name: html-files-config-one
data:
index.html: |
<!DOCTYPE html>
<html>
<head>
<title>Deployment Name</title>
</head>
<body>
<h1>Deployment Name: http-deployment-1</h1>
</body>
</html>
---
# Service
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: NodePort
deployment-2
# Deployment 2
apiVersion: apps/v1
kind: Deployment
metadata:
annotations: {}
name: http-deployment-2
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: nginx
deployment: two
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: nginx
deployment: two
spec:
containers:
- image: nginx:1.20.1
imagePullPolicy: IfNotPresent
name: http
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- name: nginx-config
mountPath: /etc/nginx/conf.d
- name: html-files
mountPath: /usr/share/nginx/html
volumes:
- name: nginx-config
configMap:
name: nginx-config
- name: html-files
configMap:
name: html-files-config-two
---
# ConfigMap for NGINX configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-config
data:
default.conf: |
server {
listen 80;
server_name localhost;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location / {
root /usr/share/nginx/html;
index index.html;
}
}
---
# ConfigMap for HTML file
apiVersion: v1
kind: ConfigMap
metadata:
name: html-files-config-two
data:
index.html: |
<!DOCTYPE html>
<html>
<head>
<title>Deployment Name</title>
</head>
<body>
<h1>Deployment Name: http-deployment-2</h1>
</body>
</html>
---
# Service
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
type: NodePort