K8s的流量分析（生产实例）

kubesphere的应用路由(名为srm-nginx)-->对应kubeshpre的ingress-nginx

进入服务service的应用（名也为srm-nginx）界面，对应k8s的service，属于Deployment其中一个（其余为Job/CronJob,StatusSet，DaemonSet）

 

进入容器组中（POD）srm-nginx-5c844bf94b-srzjb，Ip 为10.234.116.132

可见流量为Ingress-->Service-->Pods net（Containers）,即：应用路由-》服务-》容器组

10.234.14.84 30341，31203   <-->   10.234.31.62 30486

                        |                |                            |

10.234.126.202 443       80                10.234.116.132 80

  

Ps,k8s集群里本身没有lb的，这里的ingress是k8s之外的，（是属于是安装kubesphere的组件之一，即kubesphere-route），正常Ingress外部还有一个所有服务器区域的LB，比如硬件的F5，或者软件的nginx等。

 

root@prd-master02:~# kubectl describe service kubesphere-router-srm -n kubesphere-controls-system Name:                     kubesphere-router-srm Namespace:                kubesphere-controls-system Labels:                   app.kubernetes.io/component=controller                           app.kubernetes.io/instance=kubesphere-router-srm-ingress                           app.kubernetes.io/managed-by=Helm                           app.kubernetes.io/name=ingress-nginx                           app.kubernetes.io/version=0.48.1                           helm.sh/chart=ingress-nginx-3.35.0 Annotations:              <none> Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=kubesphere-router-srm-ingress,app.kubernetes.io/name=ingress-nginx Type:                     NodePort IP Family Policy:         SingleStack IP Families:              IPv4 IP:                       10.234.14.84 IPs:                      10.234.14.84 Port:                     http  80/TCP TargetPort:               http/TCP NodePort:                 http  31203/TCP Endpoints:                10.234.126.202:80 Port:                     https  443/TCP TargetPort:               https/TCP NodePort:                 https  30341/TCP Endpoints:                10.234.126.202:443 Session Affinity:         None External Traffic Policy:  Cluster Events:                   <none> root@prd-master02:~# kubectl get svc -n kubesphere-controls-system NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE default-http-backend             ClusterIP   10.234.29.211   <none>        80/TCP                       339d kubesphere-router-itam           NodePort    10.234.40.132   <none>        80:32304/TCP,443:32322/TCP   234d kubesphere-router-itam-metrics   ClusterIP   10.234.49.73    <none>        10254/TCP                    234d kubesphere-router-srm            NodePort    10.234.14.84    <none>        80:31203/TCP,443:30341/TCP   304d kubesphere-router-srm-metrics    ClusterIP   10.234.40.30    <none>        10254/TCP                    304d root@prd-master02:~# kubectl get svc -n srm NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE srm-gateway           ClusterIP   10.234.6.135    <none>        8080/TCP                         303d srm-im                NodePort    10.234.32.153   <none>        11888:30141/TCP,9326:31650/TCP   303d srm-interface         NodePort    10.234.47.211   <none>        8080:31816/TCP                   309d srm-interface-nginx   NodePort    10.234.7.106    <none>        80:30610/TCP                     309d srm-kkfile            NodePort    10.234.12.81    <none>        8012:31437/TCP                   309d srm-mq-console        NodePort    10.234.27.142   <none>        8080:32401/TCP                   310d srm-mqnamesrv         ClusterIP   10.234.43.232   <none>        9876/TCP                         310d srm-nacos             NodePort    10.234.39.242   <none>        8848:30909/TCP                   310d srm-nginx             NodePort    10.234.31.62    <none>        80:30486/TCP                     303d srm-report            ClusterIP   10.234.6.24     <none>        22000/TCP                        303d srm-sentinel          NodePort    10.234.36.203   <none>        8080:30602/TCP                   310d root@prd-master02:~# kubectl describe service srm-nginx -n srm Name:                     srm-nginx Namespace:                srm Labels:                   app=srm-nginx                           ver=1.20.1 Annotations:              kubesphere.io/creator: srm Selector:                 app=srm-nginx,ver=1.20.1 Type:                     NodePort IP Family Policy:         SingleStack IP Families:              IPv4 IP:                       10.234.31.62 IPs:                      10.234.31.62 Port:                     srm-nginx  80/TCP TargetPort:               80/TCP NodePort:                 srm-nginx  30486/TCP Endpoints:                10.234.116.132:80 Session Affinity:         None External Traffic Policy:  Cluster Events:                   <none> root@prd-master02:~#

            

 

 

# ls srm-v5.conf # cat srm-v5.conf server {     listen       80;     server_name  srm.holitech.net;     charset utf-8;     add_header X-Frame-Options "SAMEORIGIN";     add_header X-XSS-Protection "1; mode=block";     add_header X-Content-Type-Options "nosniff";     access_log  /var/log/nginx/srm_nginx_access.log  main;     error_log  /var/log/nginx/srm_nginx_error.log;     #rewrite ^/$ https://$host permanent;   ####前端访问配置         location / {                 root   /opt/nfsshare/v5sit/dist/;                 index  index.html index.htm;                 try_files $uri $uri/ /index.html;         }   #####移动前端访问配置##         location ^~/mobile {                 alias /opt/nfsshare/v5sit/app/mobile/;                 index  index.html index.htm;                 #try_files $uri $uri/index.html /index.html;         }   ###客服前端配置###         location ^~/kefu {                 if (-d $request_filename){                 rewrite ^/(.*)([^/])$   https://$host/$1$2/ permanent;                 }                 alias /opt/nfsshare/v5sit/im/dist/;                 index  index.html index.htm;                 try_files $uri $uri/ /kefu/index.html;                 #try_files $uri $uri/ /kefu/index.html;         }   ###opt前端挂载共享###         location ~ ^/(images|img|javascript|js|im|css|tinymce|flash|media|static)/                            {             root /opt/nfsshare/v5sit/dist;             gzip_static on;              autoindex off;         }           ###srm-后端####         location ^~/els {             proxy_pass http://srm-gateway.srm.svc.cluster.local:8080;             proxy_set_header X-real-ip $remote_addr;             #proxy_set_header X-Forwarded-For $remote_addr;             proxy_set_header X-Forwarded-For $http_x_forwarded_for;             proxy_connect_timeout       3000;             proxy_send_timeout          3000;             proxy_read_timeout          3000;             send_timeout                3000;         }           ###srm-websocket后端#-集成##         location /els/websocket  {             proxy_pass http://srm-gateway.srm.svc.cluster.local:8080;             proxy_http_version 1.1;             proxy_set_header Upgrade $http_upgrade;             proxy_set_header Connection "Upgrade";             proxy_set_header X-real-ip $remote_addr;             proxy_set_header X-Forwarded-For $remote_addr;         }           ####im-websocket集成im###         location /els/imChat  {             proxy_pass http://srm-im.srm.svc.cluster.local:9326;             proxy_http_version 1.1;             proxy_set_header Upgrade $http_upgrade;             proxy_set_header Connection "Upgrade";             proxy_set_header X-real-ip $remote_addr;             proxy_set_header X-Forwarded-For $remote_addr;         }           ####报表--单独###         location /els/report  {             proxy_pass http://srm-report.srm.svc.cluster.local:22000;             proxy_set_header X-real-ip $remote_addr;             proxy_set_header X-Forwarded-For $remote_addr;         }           ###后端共享挂载js         location /opt/ {             alias /opt/nfsshare/v5sit/;             #alias /opt/;             gzip_static on;             autoindex off;         }         error_page 405 =200 http://$host$request_uri; }# pwd /usr/local/nginx/conf/domain # # ifconfig -a eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1440         inet 10.234.116.132  netmask 255.255.255.255  broadcast 10.234.116.132         ether f6:e0:66:fa:9a:e3  txqueuelen 0  (Ethernet)         RX packets 919505  bytes 344147263 (328.2 MiB)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 696268  bytes 1021955662 (974.6 MiB)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536         inet 127.0.0.1  netmask 255.0.0.0         loop  txqueuelen 1000  (Local Loopback)         RX packets 0  bytes 0 (0.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 tunl0: flags=128<NOARP>  mtu 1480         tunnel   txqueuelen 1000  (IPIP Tunnel)         RX packets 0  bytes 0 (0.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 0  bytes 0 (0.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0 #