记录 - k8s入门2 - helloredis (1.16.0)

环境

本篇改编自官方guestbook教程 (https://kubernetes.io/docs/tutorials/stateless-application/guestbook)：redis一主双从 + phpweb。web应用改用golang编写以减小镜像体积方便试验。

本篇基于前一篇 记录 - k8s 入门搭建 (1.16.0, helloweb) 搭建的环境。主节点k8s0=192.168.199.200，工作节点k8s1=192.168.199.201。

启动redis master

helloredis应用设置数据到redis主节点，从redis从节点读取数据。

创建deploy

##以下在k8s0

mkdir -p /test/k8s/helloredis
cd /test/k8s/helloredis
vi redis-master-deployment.yaml

apiVersion: apps/v1  # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: redis-master
  labels:
    app: redis
spec:
  selector:
    matchLabels:
      app: redis
      role: master
      tier: backend
  replicas: 1
  template:
    metadata:
      labels:
        app: redis
        role: master
        tier: backend
    spec:
      containers:
      - name: master
        # image: k8s.gcr.io/redis:e2e  # or just image: redis
        image: redis:3.2.9
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 6379

#注：redis的版本“3.2.9” 参考 [问题 - redis 不能主从复制]

#部署：
alias kc=kubectl
kc apply -f redis-master-deployment.yaml

kc get pod -o wide

NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE   NOMINATED NODE   READINESS GATES
redis-master-8556bd886d-nthc4   1/1     Running   0          95s   10.100.166.254   k8s1   <none>           <none>

#查询pod日志
kc logs -f redis-master-8556bd886d-nthc4

1:M 06 Nov 07:53:54.349 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 06 Nov 07:53:54.349 # Server started, Redis version 3.2.9
1:M 06 Nov 07:53:54.349 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 06 Nov 07:53:54.349 * The server is now ready to accept connections on port 6379

创建service

vi redis-master-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: redis-master
  labels:
    app: redis
    role: master
    tier: backend
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    app: redis
    role: master
    tier: backend

#“spec.selector” 指定目标pod 的标签，这些对应deploy 创建的pod：
kc describe pod/redis-master-8556bd886d-nthc4

Name:         redis-master-8556bd886d-nthc4
......
Labels:       app=redis
              pod-template-hash=8556bd886d
              role=master
              tier=backend

#部署
kc apply -f redis-master-service.yaml

kc get service -o wide

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE     SELECTOR
kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP    4d21h   <none>
redis-master   ClusterIP   10.96.151.116   <none>        6379/TCP   9s      app=redis,role=master,tier=backend

#注：service host 在集群内可以通过dns或env 获取。

启动redis slave

虽然redis主只有一个pod，但可以通过配置redis从来提升HA。

创建deploy

#下面yaml里配置了2个replica，当apply时如果当时已经有多于2个pod，会自动删除多余的；反之如果当时还不足2个，会自动补充创建。

vi redis-slave-deployment.yaml

apiVersion: apps/v1  # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: redis-slave
  labels:
    app: redis
spec:
  selector:
    matchLabels:
      app: redis
      role: slave
      tier: backend
  replicas: 2
  template:
    metadata:
      labels:
        app: redis
        role: slave
        tier: backend
    spec:
      containers:
      - name: slave
        image: gcr.io/google_samples/gb-redisslave:v3
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: GET_HOSTS_FROM
          value: dns
          # value: env
        ports:
        - containerPort: 6379

#注：k8s 1.16内置dns，可以通过dns连接redis-slave。如果需要改用环境变量方式获取redis-slave host，将上面的配置"value: dns" 改为"value: env"。

#部署
kc apply -f redis-slave-deployment.yaml

kc get pod -o wide

NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE   NOMINATED NODE   READINESS GATES
redis-master-8556bd886d-nthc4   1/1     Running   0          30m   10.100.166.254   k8s1   <none>           <none>
redis-slave-7664787fbc-r4plx    1/1     Running   0          19s   10.100.166.255   k8s1   <none>           <none>
redis-slave-7664787fbc-vfx6t    1/1     Running   0          19s   10.100.166.193   k8s1   <none>           <none>

#看到按照yaml的配置只创建了2个pod。

#可以直接curl redis pod：

# curl 10.100.166.255:6379
-ERR wrong number of arguments for 'get' command
-ERR unknown command 'User-Agent:'

关于redis slave镜像

参考 https://github.com/kubernetes/examples/blob/master/guestbook/redis-slave/run.sh

if [[ ${GET_HOSTS_FROM:-dns} == "env" ]]; then
  redis-server --slaveof ${REDIS_MASTER_SERVICE_HOST} 6379
else
  redis-server --slaveof redis-master 6379
fi

“REDIS_MASTER_SERVICE_HOST” 环境变量由k8s自动设置，参考 https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/

创建service

#在集群内不应直连某个pod，因为pod 可以动态创建/删除/移动。应该创建一个service，由service动态路由到pod。

vi redis-slave-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: redis-slave
  labels:
    app: redis
    role: slave
    tier: backend
spec:
  ports:
  - port: 6379
  selector:
    app: redis
    role: slave
    tier: backend

#部署
kc apply -f redis-slave-service.yaml

kc get service -o wide

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE     SELECTOR
kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP    4d22h   <none>
redis-master   ClusterIP   10.96.151.116   <none>        6379/TCP   32m     app=redis,role=master,tier=backend
redis-slave    ClusterIP   10.96.146.129   <none>        6379/TCP   22s     app=redis,role=slave,tier=backend

# curl 10.96.146.129:6379
-ERR wrong number of arguments for 'get' command
-ERR unknown command 'User-Agent:'

helloredis 编写、编译

hello_redis.go

参考 https://github.com/kubernetes/examples/blob/master/guestbook/php-redis/guestbook.php

封装调用redis get、set 两个方法的http api，在80端口监听：

package main

import (
    "bufio"
    "fmt"
    "io"
    "net"
    "net/http"
    "os"
    "strconv"
)

func main() {
    http.Handle("/get", http.HandlerFunc(handle_get))
    http.Handle("/set", http.HandlerFunc(handle_set))

    if err := http.ListenAndServe(":80", nil); err != nil {
        fmt.Println("ListenAndServe:", err)
    }
}

func handle_get(w http.ResponseWriter, req *http.Request) {
    str, err := get()
    if err != nil {
        fmt.Fprintln(w, "Error:", err)
    } else {
        fmt.Fprintln(w, "Got:", str)
    }
}

func handle_set(w http.ResponseWriter, req *http.Request) {
    value := req.FormValue("value")
    str, err := set(value)
    if err != nil {
        fmt.Fprintln(w, "Error:", err)
    } else {
        fmt.Fprintln(w, "Got:", str)
    }
}

redis的协议其实比较简单，可网上搜索“redis 通讯协议”。自行实现get方法：

func get() (string, error) {
    host := "redis-slave"
    if os.Getenv("GET_HOSTS_FROM") == "env" {
        host = os.Getenv("REDIS_SLAVE_SERVICE_HOST")
    }

    conn, err := net.Dial("tcp", host + ":6379")
    if err != nil {
        return "", err
    }

    defer conn.Close()

    // GET key
    fmt.Fprint(conn, "*2\r\n$3\r\nGET\r\n$3\r\nkey\r\n")

    // 响应：$-1\r\n 或 $1\r\n1\r\n
    reader := bufio.NewReader(conn)
    str, err := reader.ReadString('\n')
    if err != nil {
        return "", err
    }

    lenstr := str[1:len(str)-2]
    len, _ := strconv.Atoi(lenstr)
    if len == -1 {
        return "", nil  // 没有值
    }

    buf := make([]byte, len)
    _, err = io.ReadFull(reader, buf)
    if err != nil {
        return "", err
    }

    return string(buf), nil
}

上面使用dns或环境变量值来连接redis从。

set方法类似，但连接redis主：

func set(value string) (string, error) {
    host := "redis-master"
    if os.Getenv("GET_HOSTS_FROM") == "env" {
        host = os.Getenv("REDIS_MASTER_SERVICE_HOST")
    }

    conn, err := net.Dial("tcp", host + ":6379")
    if err != nil {
        return "", err
    }

    defer conn.Close()

    // SET key value
    fmt.Fprintf(conn, "*3\r\n$3\r\nSET\r\n$3\r\nkey\r\n$%d\r\n%s\r\n", len(value), value)

    // 响应：+OK\r\n
    reader := bufio.NewReader(conn)
    str, err := reader.ReadString('\n')
    if err != nil {
        return "", err
    }

    return str[1:len(str)-2], nil
}

编译为可执行文件 hello_redis (7.2M)。

编译、上传镜像

mkdir -p /test/k8s/docker_redis
cd /test/k8s/docker_redis
vi Dockerfile

FROM alpine
WORKDIR /helloredis
ADD . /helloredis
RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
EXPOSE 80
ENTRYPOINT ["./hello_redis"]

#将hello_redis 拷过来，目录下有2个文件：Dockerfile hello_redis
chmod +x hello_redis
#编译镜像。注意：“YYY” 需替换为你的hub.docker.com 账户名
docker build -t YYY/helloredis:v0.1 .

#再按照上一篇的做法将镜像上传到hub.docker.com

启动helloredis 应用

创建deploy

cd /test/k8s/helloredis
vi frontend-deployment.yaml

apiVersion: apps/v1  # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: frontend
  labels:
    app: helloredis
spec:
  selector:
    matchLabels:
      app: helloredis
      tier: frontend
  replicas: 3
  template:
    metadata:
      labels:
        app: helloredis
        tier: frontend
    spec:
      containers:
      - name: go-redis
        # image: gcr.io/google-samples/gb-frontend:v4
        image: YYY/helloredis:v0.1
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: GET_HOSTS_FROM
          value: dns
          # value: env
        ports:
        - containerPort: 80

#注：同样，上面“YYY”实际是docker hub的账户名

#部署
kc apply -f frontend-deployment.yaml

#观察deploy “frontend” 直到Ready=3/3
watch kubectl get deploy

#查询3个pod
kc get pods -l app=helloredis -l tier=frontend -o wide

NAME                        READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
frontend-868f999cb9-c2kb9   1/1     Running   0          6m51s   10.100.166.195   k8s1   <none>           <none>
frontend-868f999cb9-gmqzc   1/1     Running   0          6m51s   10.100.166.194   k8s1   <none>           <none>
frontend-868f999cb9-gzghz   1/1     Running   0          6m51s   10.100.166.197   k8s1   <none>           <none>

#直接选一个pod试验下get/set接口

[root@k8s0 helloredis]# curl 10.100.166.195/get
Got: 
[root@k8s0 helloredis]# curl 10.100.166.195/set?value=Holly
Got: OK
[root@k8s0 helloredis]# curl 10.100.166.195/get
Got: Holly
[root@k8s0 helloredis]# curl 10.100.166.197/get
Got: Holly

#可见主从读写正常

创建service

#默认"redis-master"和"redis-slave" service 都只能在集群内部访问 (类型"ClusterIP")。为了能从集群外部访问helloredis应用，将其类型设为"NodePort" (类型"LoadBalancer" 只在云厂商环境下可用)。

vi frontend-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: helloredis
    tier: frontend
spec:
  type: NodePort
  # type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: helloredis
    tier: frontend

#部署
kc apply -f frontend-service.yaml

kc get service -o wide

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE     SELECTOR
frontend       NodePort    10.96.214.32    <none>        80:30418/TCP   4s      app=helloredis,tier=frontend
kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP        5d1h    <none>
redis-master   ClusterIP   10.96.151.116   <none>        6379/TCP       3h50m   app=redis,role=master,tier=backend
redis-slave    ClusterIP   10.96.146.129   <none>        6379/TCP       3h18m   app=redis,role=slave,tier=backend

#请求一下

[root@k8s0 helloredis]# curl 10.96.214.32:80/get
Got: Holly
[root@k8s0 helloredis]# curl 192.168.199.200:30418/get
Got: Holly

#注：在k8s1机器也可

#顺便验证下 host

[root@k8s0 helloredis]# kc exec pod/frontend-868f999cb9-gzghz -it /bin/sh
/helloredis # env | grep "SERVICE_HOST"
REDIS_SLAVE_SERVICE_HOST=10.96.146.129
REDIS_MASTER_SERVICE_HOST=10.96.151.116
KUBERNETES_SERVICE_HOST=10.96.0.1
/helloredis # ping redis-master
PING redis-master (10.96.151.116): 56 data bytes
......
/helloredis # exit

伸缩部署

kc scale deploy frontend --replicas=5

kc scale deploy frontend --replicas=2

kc get pod

NAME                           READY   STATUS    RESTARTS   AGE
frontend-868f999cb9-c2kb9      1/1     Running   0          90m
frontend-868f999cb9-gzghz      1/1     Running   0          90m
redis-master-98f9cfc96-xhzr9   1/1     Running   0          50m
redis-slave-7664787fbc-r4plx   1/1     Running   0          3h52m
redis-slave-7664787fbc-vfx6t   1/1     Running   0          3h52m

清理

kc delete deploy -l app=redis
kc delete svc -l app=redis
kc delete deploy -l app=helloredis
kc delete svc -l app=helloredis

kc get pod

No resources found in default namespace.

补充

虚机重启后在kubernetes完成启动前“kc get node”会报错：
Unable to connect to the server: net/http: TLS handshake timeout
或
The connection to the server apiserver.demo:6443 was refused - did you specify the right host or port?

修改应用重新打包镜像时如果镜像名称和版本 都不变，容器拉取的可能还是老的镜像（待确认）。

问题

redis 不能主从复制

描述====

原文 redis-master-deployment.yaml 里“image: k8s.gcr.io/redis:e2e” - 这个镜像有419M，所以最初改成“image: redis”。
部署master、slave后，发现master写入后不能从slave读到。

诊断====

#查询slave日志
kc logs deploy/redis-slave

8:S 06 Nov 04:08:42.009 * Connecting to MASTER redis-master:6379
8:S 06 Nov 04:08:42.013 * MASTER <-> SLAVE sync started
8:S 06 Nov 04:08:42.013 * Non blocking connect for SYNC fired the event.
8:S 06 Nov 04:08:42.013 * Master replied to PING, replication can continue...
8:S 06 Nov 04:08:42.014 * Partial resynchronization not possible (no cached master)
8:S 06 Nov 04:08:42.015 * Full resync from master: 68398c71799a8917a8902e371de8654483978bd4:812
8:S 06 Nov 04:08:42.110 * MASTER <-> SLAVE sync: receiving 176 bytes from master
8:S 06 Nov 04:08:42.110 * MASTER <-> SLAVE sync: Flushing old data
8:S 06 Nov 04:08:42.110 * MASTER <-> SLAVE sync: Loading DB in memory
8:S 06 Nov 04:08:42.110 # Can't handle RDB format version 9
8:S 06 Nov 04:08:42.110 # Failed trying to load the MASTER synchronization DB from disk

日志表明从同步到主的数据后不能处理RDB 9格式，所以可能是主、从版本不兼容。

slave是“image: gcr.io/google_samples/gb-redisslave:v3”。
进入一个slave pod后查询版本：

[root@k8s0 helloredis]# kc exec pod/redis-slave-7664787fbc-r4plx -it /bin/sh
# redis-cli --version
redis-cli 3.2.9
# exit

同样进入master pod查询版本发现是redis-cli 5.0.6。主的版本过高。

修复====
改为"redis:3.2.9"，重新apply后，查询日志：

8:S 06 Nov 06:41:46.511 * MASTER <-> SLAVE sync: receiving 76 bytes from master
8:S 06 Nov 06:41:46.511 * MASTER <-> SLAVE sync: Flushing old data
8:S 06 Nov 06:41:46.511 * MASTER <-> SLAVE sync: Loading DB in memory
8:S 06 Nov 06:41:46.511 * MASTER <-> SLAVE sync: Finished with success