k8s ingress部署教程

ingress原理概念

k8s暴露服务给外网有NodePort这个方法，但是一但服务过多就会导致端口混乱

一种实现方法是使用DaemonSet在每个节点上部署nginx，然后nginx配置好域名转发serviceIP的规则即可，但是这也会导致一但有新的服务加入就得修改nginx的配置，进行手动更新，一但服务过多和频繁，想想就很美

解决这个问题的办法是使用Ingress，将查询服务和对应pod IP的操作封装成Ingress，只要修改更新对应的单个Ingress即可，对其他的服务器没有影响，也就是对nginx的配置进行解耦

上面讲了nginx的配置问题，而nginx变成了Ingress Controoler
Ingress Controoler通过与 Kubernetes API 交互，动态的去感知集群中 Ingress 规则变化，然后读取他，按照他自己模板生成一段 Nginx 配置，再写到 Nginx Pod 里，最后 reload 一下

ingress下载安装

我的环境:k8s1.26.8

官方地址 https://github.com/kubernetes/ingress-nginx
官方文档 https://kubernetes.github.io/ingress-nginx/deploy/#quick-start
ingress-nginx和k8s的版本对应关系如下

下载不下来，打开复制保存即可
https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/cloud/deploy.yaml

或者使用gitee下载
wget https://gitee.com/mirrors/ingress-nginx/raw/controller-v1.10.1/deploy/static/provider/cloud/deploy.yaml

看看deploy配置文件都干了啥
1、创建ingress-nginx命名空间
2、创建ingress-nginx serviceaccount
3、创建ingress-nginx-admission serviceaccount
4、创建上面两个账号的集群role和非集群role分别进行绑定
5、创建ingress-nginx-controller和ingress-nginx-controller-admission的service
6、创建deployment ingress-nginx/controller
7、创建job和ingressClass（job中有镜像ingress-nginx/kube-webhook-certgen）

修改配置文件，将镜像改为阿里云的
vim deploy.yaml

:%s/registry.k8s.io\/ingress-nginx/registry.cn-hangzhou.aliyuncs.com\/google_containers/g

方式一：DaemonSet+HostNetwork+nodeSelector+ingress暴露服务

ingress-nginx/controller安装

DaemonSet+HostNetwork+nodeSelector
修改deploy.yml文件


该文件的内容我放在后面了

[root@k8s-master01 ingress]# kubectl apply -f deploy.yaml

ingress-nginx/controller拉不下来。各位各显神通吧

外网服务器下载保存成tar包

docker pull registry.k8s.io/ingress-nginx/controller:v1.10.1
docker save -o ingress-nginx-con-1.10.1.tar registry.k8s.io/ingress-nginx/controller:v1.10.1
docker rmi registry.k8s.io/ingress-nginx/controller:v1.10.1

下载tar包到本地导入

ctr -n=k8s.io images import ingress-nginx-con-1.10.1.tar 

查看导入情况
[root@k8s-worker02 software]# ctr -n=k8s.io images ls |grep ingress
registry.k8s.io/ingress-nginx/controller:v1.10.1                                                                                                 application/vnd.oci.image.manifest.v1+json                sha256:56756ff0528ddf4bfa4bec59a5455ae4b5d77890e334e8002f57e5a9081e6d3e 270.8 MiB linux/amd64                                                                                                                        io.cri-containerd.image=managed

[root@k8s-master01 ingress]# kubectl apply -f deploy.yaml

namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx unchanged
serviceaccount/ingress-nginx-admission unchanged
role.rbac.authorization.k8s.io/ingress-nginx unchanged
role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
configmap/ingress-nginx-controller unchanged
service/ingress-nginx-controller unchanged
service/ingress-nginx-controller-admission unchanged
daemonset.apps/ingress-nginx-controller configured
job.batch/ingress-nginx-admission-create unchanged
job.batch/ingress-nginx-admission-patch unchanged
ingressclass.networking.k8s.io/nginx unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured


[root@k8s-master01 ingress]# kubectl get pod -n ingress-nginx
NAME                             READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-jk8st   1/1     Running   0          4m24s

部署成功，宿主机监听三个端口443，80，8181，8181是nginx-controller默认配置的一个 default backend（Ingress 资源没有匹配的 rule 对象时，流量就会被导向这个 default backend）。

可用性测试

创建一个ingress绑定服务

cat > ingress-app.yaml<<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-app-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: test.ingress.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-app-svc
            port:
              number: 80
EOF
kubectl apply -f ingress-app.yaml -n ingress-nginx 

ingressClassName 指定 IngressClass，用来指定选择的 Ingress Controller

host 主机名可以是精确匹配，或者使用通配符来匹配，但通配符仅覆盖一个 DNS 标签（例如 *.foo.com 不匹配 baz.bar.foo.com）。

pathType 支持的路径类型有三种：

• Exact：精确匹配 URL 路径，且区分大小写。
• Prefix：基于以 / 分隔的 URL 路径前缀匹配。匹配区分大小写。如果路径的最后一个元素是请求路径中最后一个元素的子字符串，则不会匹配 （例如：/foo/bar 匹配 /foo/bar/baz, 但不匹配 /foo/barbaz）。
• ImplementationSpecific：对于这种路径类型，匹配方法取决于 IngressClass。具体实现可以将其作为单独的 pathType 处理或者与 Prefix 或 Exact 类型作相同处理。

创建一个测试用的pod和service

cat >ingress-deploy.yml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-app-svc
spec:
  type: ClusterIP
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  selector:
    app: ingress-nginx
EOF
kubectl apply -f ingress-deploy.yml

hosts中添加解析

192.168.126.23 test.ingress.com

可以看到根据域名访问到了k8s集群内部的服务，如果需要可以继续添加ingress绑定对应的服务

该方式实现了用域名来暴露内部服务，缺点是由于直接利用宿主机节点的网络和端口，安全性不高，一个node只能部署一个ingress-controller pod。

ingress controller pod内部情况

在ingress-nginx-controller的pod中对nginx的配置文件进行了修改
/etc/nginx/nginx.conf中添加了test.ingress.com虚拟主机配置

http {
        server {
                server_name _ ;
                http2 on;
                listen 80 default_server reuseport backlog=511 ;
        ## end server _
        ## start server test.ingress.com
        server {
                server_name test.ingress.com ;

                http2 on;
                listen 80  ;
                listen [::]:80  ;
                listen 443  ssl;
                listen [::]:443  ssl;

                set $proxy_upstream_name "-";

                ssl_certificate_by_lua_block {
                        certificate.call()
                }

                location / {

                        set $namespace      "default";
                        set $ingress_name   "nginx-app-ingress";
                        set $service_name   "nginx-app-svc";
                        set $service_port   "80";
                        set $location_path  "/";

                        proxy_pass http://upstream_balancer;
                        proxy_redirect                          off;
                }

        }
        ## end server test.ingress.com

方式二：Deployment+NodePort使用ingress暴露服务

本质是将ingress controller当作普通的pod使用，使用Node Port的service来将ingress暴露在集群节点ip的特定端口上。一般在前面搭建一套负载均衡器来转发请求。

方式一：pod固定在一个node上，适合ip固定的情况
方式二：pod在集群漂移，适合不固定ip，由于多了一层转发，在访问量大的时候可能有一定影响

将之前修改和注释的deploy.yml文件还原或者下载一个新文件
只需要改ingress-nginx-controller的service几个地方
– externalTrafficPolicy:Cluster#如果是local表示只能转发给当前node节点的pod
– type: NodePort
– nodePort: 30080
– nodePort：30043

首先删除之前的部署，重新部署

kubectl delete -f ingress-app.yaml  
kubectl delete -f ingress-deploy.yml.yaml
kubectl delete -f deploy.yaml
kubectl delete ns ingress-nginx --force --grace-period 0
kubectl apply -f deploy2.yaml

[root@k8s-master01 ingress]# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.105.105.1     <none>        80:30080/TCP,443:30043/TCP   2m3s
ingress-nginx-controller-admission   ClusterIP   10.100.199.248   <none>        443/TCP                      2m3s


[root@k8s-master01 ingress]# kubectl apply -f ingress-app.yaml
ingress.networking.k8s.io/nginx-app-ingress created
[root@k8s-master01 ingress]# kubectl apply -f ingress-deploy.yml
deployment.apps/nginx-app created

配置https+域名访问

安装mkcert

sudo yum install nss-tools
wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.4/mkcert-v1.4.4-linux-amd64

chmod +x mkcert-v1.4.4-linux-amd64
mv mkcert-v1.4.4-linux-amd64 /usr/local/bin/mkcert
source ~/.bashrc
mkcert -install
Created a new local CA 💥
The local CA is now installed in the system trust store! ⚡️
The local CA is now installed in the Firefox and/or Chrome/Chromium trust store (requires browser restart)! 🦊

[root@k8s-master01 ingress]# mkcert test.ingress.com
[root@k8s-master01 ingress]# kubectl create secret tls ingress-tls –cert test.ingress.com.pem –key test.ingress.com-key.pem
[root@k8s-master01 ingress]# kubectl get secrets
NAME TYPE DATA AGE
ingress-tls kubernetes.io/tls 2 34s

配置SSL

kubectl delete -f ingress-app.yaml
修改配置文件

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-app-ingress
spec:
  tls:
   - hosts:
      - test.ingress.com
     secretName: ingress-tls
  ingressClassName: nginx
  rules:
  - host: test.ingress.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-app-svc
            port:
              number: 80

kubectl apply -f ingress-app.yaml

[root@k8s-master01 ingress]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx-app-ingress nginx test.ingress.com 80, 443 20s

由于是虚拟主机中配置的，证书没起作用,但已经可以通过https访问了

方式一deploy.yml内容

镜像可能需要替换

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx
  namespace: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx
  namespace: ingress-nginx
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - configmaps
  - pods
  - secrets
  - endpoints
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - coordination.k8s.io
  resourceNames:
  - ingress-nginx-leader
  resources:
  - leases
  verbs:
  - get
  - update
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission
  namespace: ingress-nginx
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - nodes
  - pods
  - secrets
  - namespaces
  verbs:
  - list
  - watch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses/status
  verbs:
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingressclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission
rules:
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - validatingwebhookconfigurations
  verbs:
  - get
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
- kind: ServiceAccount
  name: ingress-nginx
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
- kind: ServiceAccount
  name: ingress-nginx
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
  name: ingress-nginx-admission
  namespace: ingress-nginx
---
apiVersion: v1
data:
  allow-snippet-annotations: "false"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  ports:
  - appProtocol: https
    name: https-webhook
    port: 443
    targetPort: webhook
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: ClusterIP
---
apiVersion: apps/v1
#kind: Deployment
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  minReadySeconds: 0
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  #strategy:
  #  rollingUpdate:
  #    maxUnavailable: 1
  #  type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.10.1
    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --enable-metrics=false
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: LD_PRELOAD
          value: /usr/local/lib/libmimalloc.so
        image: registry.k8s.io/ingress-nginx/controller:v1.10.1 
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /wait-shutdown
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: controller
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          name: https
          protocol: TCP
        - containerPort: 8443
          name: webhook
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 90Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsNonRoot: true
          runAsUser: 101
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
        - mountPath: /usr/local/certificates/
          name: webhook-cert
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true #使用d宿主机网络
      nodeSelector:
        #kubernetes.io/os: linux
        kubernetes.io/hostname: k8s-worker02 #指定节点运行
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
      - name: webhook-cert
        secret:
          secretName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission-create
  namespace: ingress-nginx
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.10.1
      name: ingress-nginx-admission-create
    spec:
      containers:
      - args:
        - create
        - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
        - --namespace=$(POD_NAMESPACE)
        - --secret-name=ingress-nginx-admission
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.4.1
        imagePullPolicy: IfNotPresent
        name: create
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 65532
          seccompProfile:
            type: RuntimeDefault
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission-patch
  namespace: ingress-nginx
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: admission-webhook
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        app.kubernetes.io/version: 1.10.1
      name: ingress-nginx-admission-patch
    spec:
      containers:
      - args:
        - patch
        - --webhook-name=ingress-nginx-admission
        - --namespace=$(POD_NAMESPACE)
        - --patch-mutating=false
        - --secret-name=ingress-nginx-admission
        - --patch-failure-policy=Fail
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.4.1
        imagePullPolicy: IfNotPresent
        name: patch
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 65532
          seccompProfile:
            type: RuntimeDefault
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: nginx
spec:
  controller: k8s.io/ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/component: admission-webhook
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.10.1
  name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: ingress-nginx-controller-admission
      namespace: ingress-nginx
      path: /networking/v1/ingresses
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: validate.nginx.ingress.kubernetes.io
  rules:
  - apiGroups:
    - networking.k8s.io
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - ingresses
  sideEffects: None

问题及解决办法

ingress无法被contoller发现

原因是ingress中需要添加注解

metadata:
  name: nginx-app-ingress
  annotations:
    kubernetes.io/ingress.class: nginx #重点
spec:

kubernetes.io/ingress.class” is deprecated

[root@k8s-master01 ingress]# kubectl apply -f ingress-app.yaml
Warning: annotation “kubernetes.io/ingress.class” is deprecated, please use ‘spec.ingressClassName’ instead

  annotations:
    kubernetes.io/ingress.class: nginx #重点

上面是旧版本的配置
新版本

spec:
  ingressClassName: nginx
  rules:

cannot unmarshal string into Go struct field IngressTLS.spec.tls.hosts

Error from server (BadRequest): error when creating “ingress-app.yaml”: Ingress in version “v1” cannot be handled as a Ingress: json: cannot unmarshal string into Go struct field IngressTLS.spec.tls.hosts of type []string
[root@k8s-master01 ingress]# vim ingress-app.yaml
原因是域名前少了一个-
错误写法

spec:
  tls:
   - hosts:
      test.ingress.com
     secretName: ingress-tls
  ingressClassName: nginx

正确写法

spec:
  tls:
   - hosts:
      - test.ingress.com
     secretName: ingress-tls
  ingressClassName: nginx

ingress-nginx/controller镜像拉不下来

外网服务器下载保存成tar包

docker pull registry.k8s.io/ingress-nginx/controller:v1.10.1
docker save -o ingress-nginx-con-1.10.1.tar registry.k8s.io/ingress-nginx/controller:v1.10.1
docker rmi registry.k8s.io/ingress-nginx/controller:v1.10.1

下载tar包到本地导入

ctr -n=k8s.io images import ingress-nginx-con-1.10.1.tar 

参考

【kubernetes】关于k8s集群中的ingress规则案例