Kubernetes Ingress — Kong - HelloWorld开发者社区

Kubernetes Ingress — Kong Kong 是由 Mashape 公司开源的一个高性能、高可用、易扩展的 API Gateway 项目，基于OpenResty（Nginx + Lua模块），并提供了插件实现 API 的 AOP 功能。可以通过负载均衡、插件扩展等方式，来处理网络请求。

Kong 主要的概念

Service 服务，也称服务对象，是 Kong 管理的上游 API 和微服务，通过 Kong 转发后根据请求的协议，host，method，path 匹配到实际的服务地址。

Route 路由，作为客户端的入口，可以通过定义一些规则来匹配客户端的请求，每个路由都会关联一个 Service , 并且 Service 可以关联多个 Route，当匹配到客户端的请求时，每个请求都会被代理到其配置的 Service 中。

Upstream 相当于Kong提供了一个负载均衡的功能，基于 Nginx 的虚拟主机的方式做的负载功能，在 Service 中指定 host 的时候，也可以指定 upstream 。

Target 是 upstream 进行负载均衡的终端，当我们的服务是多节点时，那每个节点都是作为一个 target 。可以负载的权重，也可以通过 upstream 对 target 进行健康检查。

Consumer 可以代表一个 Service 的请求消费者，也可以代表对应到实际应用中的一个用户。

Plugin 在请求被代理到上游 API 之前或之后执行相关动作的插件，如限流插件。

Kong-ingress-controller 是对 Kubernetes Ingress API 的实现，通过和 Kubernetes API 交互，动态的去感知 Kubernetes 中 Ingress 规则变化。

整体的链路是 Route > Service > Upstream > Target 。

Kong 安装

Kong 在 1.1 版本后引入了一个新特性 DB-less 模式，即无 DB 运行 Kong，官方推荐在 Kubernetes 采用该模式运行，所有配置都存储在 Kubernetes 控制面板中。简化了 Kong 的操作，而无需再维护一个关系型数据库。

当然也可以根据自己的需求，采用数据库的模式运行 Kong，官方推荐 Postgres。需要注意的是，如果采用了无 DB 模式，那么在 Konga 的管理页面是无法添加插件和证书等资源。

在 Kubernetes 环境下，Admin API 方式不是很适应 Kubernetes 声明式管理方式。所以 Kong Ingress Controller 定义了五个 CRDs，基本上涵盖了原 Admin API 的各个方面。

[root@k8s-test01 ~]# kubectl get crds | grep konghq.com 
kongclusterplugins.configuration.konghq.com           2021-02-04T16:02:43Z
kongconsumers.configuration.konghq.com                2021-02-04T16:02:43Z
kongingresses.configuration.konghq.com                2021-02-04T16:02:43Z
kongplugins.configuration.konghq.com                  2021-02-04T16:02:43Z
tcpingresses.configuration.konghq.com                 2021-02-04T16:02:43Z

kongconsumers：给不同的API用户提供不同的消费者身份 kongcredentials：用户的认证凭证 kongplugins：插件的配置 kongingresses、tcpingresses：定义代理行为规则，是对 Ingress 的补充配置

安装普罗米修斯监控 Kong 网络请求，如果已经安装则跳过。

# 创建 namespace
kubectl create ns monitoring
# 安装 namespace
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm search repo prometheus-community
helm install  my-release prometheus-community/prometheus --namespace monitoring --values https://bit.ly/2RgzDtg --version 13.2.1 \
--set alertmanager.persistentVolume.enabled=false \
--set server.persistentVolume.enabled=false

# 安装 grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm search repo grafana
helm install my-grafana  grafana/grafana --namespace monitoring --values http://bit.ly/2FuFVfV --version 6.2.1 --set persistence.enabled=false --set service.type=NodePort --set service.nodePort=3000

# 查看 grafana admin用户的密码
kubectl get secret --namespace monitoring my-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Kong 可以使用环境变量的方式调整配置，但是 helm 的安装方式文档有点糟糕，这里使用 yaml 的方式安装，无 DB 模式运行。

wget https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/master/deploy/single/all-in-one-dbless.yaml
kubectl create -f all-in-one-dbless.yaml
# service 改成 nodePort 模式
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: kong-proxy
  namespace: kong
spec:
  ports:
  - name: proxy
    port: 80
    protocol: TCP
    targetPort: 8000
    nodePort: 8000
  - name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: 8443
    nodePort: 8443
  - name: admin
    port: 8444
    protocol: TCP
    targetPort: 8444
    nodePort: 8444
  selector:
    app: ingress-kong
  type: NodePort
  externalTrafficPolicy: Local

Kong 的一些配置

        # Kong 管理地址，这里使用的http，所以 kong-ingress-controller、Konga 只能用 http 连接
        - name: KONG_ADMIN_LISTEN
          value: 0.0.0.0:8444
        # 无 DB 模式
        - name: KONG_DATABASE
          value: "off"
        # nginx 工作进程数
        - name: KONG_NGINX_WORKER_PROCESSES
          value: "2"
        # 开启压缩
        - name: KONG_NGINX_HTTP_GZIP
          value: "on"
        - name: KONG_NGINX_HTTP_GZIP_TYPES
          value: "*"
        - name: KONG_NGINX_HTTP_GZIP_PROXIED
          value: "any"
        - name: KONG_NGINX_HTTP_GZIP_MIN_LENGTH
          value: "1000"
        # 日志级别
        - name: KONG_LOG_LEVEL
          value: debug
        # 获取真实IP
        - name: KONG_TRUSTED_IPS
          value: "0.0.0.0/0,::/0"
        - name: KONG_REAL_IP_HEADER
          value: "X-Forwarded-For"

Konga 依赖 Postgres 数据库，如果已经有 Postgres 可以不用安装，创建一个用户和数据库即可。

[root@k8s-test01 kong]# cat pg.yaml 
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  selector:
    matchLabels:
      app: postgres
  serviceName: postgres
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: postgres
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: postgres
        imagePullPolicy: IfNotPresent
        image: docker.io/library/postgres:10-alpine
        env:
        - name: POSTGRES_PASSWORD
          value: "PGadmin"
        - name : PGTZ
          value: "Asia/Shanghai"
        resources:
          requests:
            cpu: 500m
            memory: 500Mi
          limits:
            cpu: 3000m
            memory: 3096Mi
        ports:
        - containerPort: 5432
          name: tcp
        livenessProbe:
          tcpSocket:
            port: 5432
          initialDelaySeconds: 60
          periodSeconds: 60
        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          readOnly: false
          name: data
        - mountPath: /etc/localtime
          readOnly: false 
          name: time-data
        - mountPath: /etc/timezone
          readOnly: false
          name: timezone
      volumes:
      - name: time-data 
        hostPath: 
          path: /usr/share/zoneinfo/Asia/Shanghai
      - name: timezone
        hostPath:
          path: /etc/timezone
      - name: data
        emptyDir: {} # 注意！！！ 数据没有持久化！
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: postgres
  name: postgres
spec:
  type: ClusterIP
  ports:
  - port: 5432
    targetPort: 5432
    name: tcp
  selector:
    app: postgres

进入 Postgres 创建数据库和用户

[root@k8s-test01 kong]# kubectl -n kong exec -it postgres-0 -- bash
bash-5.1# psql -h 127.0.0.1 -p 5432 -U postgres 
psql (10.15)
Type "help" for help.

postgres=# CREATE USER konga WITH PASSWORD 'Konga2021'; 
CREATE ROLE
postgres=# CREATE DATABASE konga OWNER konga ENCODING UTF8; 
CREATE DATABASE
postgres=# GRANT ALL PRIVILEGES ON DATABASE konga TO konga; 
GRANT
postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
 konga     | konga    | UTF8     | en_US.utf8 | en_US.utf8 | =Tc/konga            +
           |          |          |            |            | konga=CTc/konga
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(4 rows)

postgres=# \q
bash-5.1#

安装 Konga

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: konga
  name: konga
  namespace: kong
spec:
  replicas: 1
  selector:
    matchLabels:
      app: konga
  template:
    metadata:
      labels:
        app: konga
    spec:
      restartPolicy: Always
      containers:
      - env:
        - name: PORT
          value: "1337"
        - name: DB_ADAPTER
          value: postgres
        - name: DB_URI
          value: "postgresql://konga:Konga2021@postgres:5432/konga"
        image: pantsel/konga:0.14.9
        name: konga
        ports:
        - containerPort: 1337
          protocol: TCP
          name: http
---
apiVersion: v1
kind: Service
metadata:
  name: konga
  namespace: kong
spec:
  type: NodePort
  ports:
  - name: http
    port: 1337
    targetPort: 1337
    protocol: TCP
    nodePort: 1337
  selector:
    app: konga

浏览器打开 nodeport 端口打开 Konga，配置 Kong 的管理地址即可。

至此，kong 环境已经部署完成，下面我们来简单测试下 Kong 在 Kubernetes 下的应用，使用 Kubernetes 声明式的方式配置。也可以在Konga 页面上配置，但笔者嫌麻烦，在页面配置的时间，yaml 文件早就编排好了....

Prometheus插件

创建测试的 pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-svc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-svc
  template:
    metadata:
      labels:
        app: http-svc
    spec:
      containers:
      - name: http-svc
        image: docker.io/kennethreitz/httpbin
        ports:
        - containerPort: 80
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

在服务上创建普罗米修斯插件配置，global 为 true 时，全局网络请求都使用普罗米修斯插件。

[root@k8s-test01 kong]# cat prometheus-plugin.yaml
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: prometheus
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "false"
plugin: prometheus
[root@k8s-test01 kong]# kubectl get KongClusterPlugin
NAME         PLUGIN-TYPE   AGE
prometheus   prometheus    38h
[root@k8s-test01 kong]#

创建测试 pod 的 service 和 ingress，通过 annotations 加载普罗米修斯插件。

---
apiVersion: v1
kind: Service
metadata:
  name: billing
  labels:
    app: billing
  annotations:
    konghq.com/plugins: prometheus 
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: http-svc
---
apiVersion: v1
kind: Service
metadata:
  name: invoice
  labels:
    app: invoice
  annotations:
    konghq.com/plugins: prometheus
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: http-svc
---
apiVersion: v1
kind: Service
metadata:
  name: comments
  labels:
    app: comments
  annotations:
    konghq.com/plugins: prometheus
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: http-svc
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo
  annotations:
    konghq.com/strip-path: "true"
    kubernetes.io/ingress.class: kong
spec:
  rules:
  - host: "foo.bar.com"
    http:
      paths:
      - pathType: Prefix
        path: /billing
        backend:
          service:
            name: billing
            port:
              number: 80
      - pathType: Prefix
        path: /comments
        backend:
          service:
            name: comments
            port:
              number: 80
      - pathType: Prefix
        path: /invoice
        backend:
          service:
            name: invoice
            port:
              number: 80

创建完成后，可以在 Konga 查看详情 Kubernetes Ingress — Kong

我们制造一些流量，查看效果。

while true; do \
curl http://foo.bar.com:8000/billing/status/200; \
curl http://foo.bar.com:8000/billing/status/501; \
curl http://foo.bar.com:8000/invoice/status/201; \
curl http://foo.bar.com:8000/invoice/status/404; \
curl http://foo.bar.com:8000/comments/status/200; \
curl http://foo.bar.com:8000/comments/status/200; \
sleep 0.01; \
done

打开grafana Kubernetes Ingress — Kong 官方模板:https://grafana.com/grafana/dashboards/7424

我们可以根据这些网络指标配置普罗米修斯的告警规则。

key-auth 认证登录

key-auth 类似 nginx 的 auth_basic 认证登录。

开启前

[root@k8s-test01 kong]# curl -i  http://foo.bar.com:8000/billing/status/200
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:00:13 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 0
Via: kong/2.2.1

开启key-auth 验证

# 在路由上创建 key-auth 插件，配置身份验证
echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: httpbin-auth
plugin: key-auth
" | kubectl apply -f -

# 创建一个消费者
echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: harry
  annotations:
    kubernetes.io/ingress.class: kong
username: harry" | kubectl apply -f -

# 创建带有 API 密钥的 Secret 资源
kubectl create secret generic harry-apikey  \
  --from-literal=kongCredType=key-auth  \
  --from-literal=key=my-sooper-secret-key

# API密钥和消费者关联
echo "apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: harry
  annotations:
    kubernetes.io/ingress.class: kong
username: harry
credentials:
- harry-apikey" | kubectl apply -f -
更新 Ingress 

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo
  annotations:
    konghq.com/strip-path: "true"
    kubernetes.io/ingress.class: kong
    konghq.com/plugins: httpbin-auth
spec:
  rules:
  - host: "foo.bar.com"
    http:
      paths:
      - pathType: Prefix
        path: /billing
        backend:
          service:
            name: billing
            port:
              number: 80
      - pathType: Prefix
        path: /comments
        backend:
          service:
            name: comments
            port:
              number: 80
      - pathType: Prefix
        path: /invoice
        backend:
          service:
            name: invoice
            port:
              number: 80

测试

# 不带 key 访问，401
[root@k8s-test01 kong]# curl -i  http://foo.bar.com:8000/billing/status/200
HTTP/1.1 401 Unauthorized
Date: Sat, 06 Feb 2021 09:16:48 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
WWW-Authenticate: Key realm="kong"
Content-Length: 45
X-Kong-Response-Latency: 0
Server: kong/2.2.1

{
  "message":"No API key found in request"
}
# 带 key 访问，200
[root@k8s-test01 kong]# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:17:02 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 1
Via: kong/2.2.1

rate-limiting 速率限制

没有限制之前

[root@k8s-test01 kong]# while true; do curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200; done 
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:27:18 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 0
Via: kong/2.2.1

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:27:18 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 0
Via: kong/2.2.1

在服务上创建 rate-limiting 插件限制配置，

echo "
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: http-ratelimit
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: \"false\"
config:
  policy: local
  second: 1
plugin: rate-limiting
" | kubectl apply -f -
更新 Service

---
apiVersion: v1
kind: Service
metadata:
  name: billing
  labels:
    app: billing
  annotations:
    konghq.com/plugins: http-ratelimit, prometheus 
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: http-svc

测试

[root@k8s-test01 kong]#  curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200;
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 09:55:02 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-RateLimit-Limit-Second: 1
X-RateLimit-Remaining-Second: 0
X-RateLimit-Remaining-Minute: 0
RateLimit-Limit: 1
RateLimit-Remaining: 0
X-RateLimit-Limit-Minute: 1
RateLimit-Reset: 58
X-Kong-Upstream-Latency: 4
X-Kong-Proxy-Latency: 1
Via: kong/2.2.1

[root@k8s-test01 kong]#  curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/billing/status/200;
HTTP/1.1 429 Too Many Requests
Date: Sat, 06 Feb 2021 09:55:04 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Retry-After: 56
Content-Length: 41
X-RateLimit-Limit-Second: 1
X-RateLimit-Remaining-Second: 1
X-RateLimit-Remaining-Minute: 0
RateLimit-Limit: 1
RateLimit-Remaining: 0
X-RateLimit-Limit-Minute: 1
RateLimit-Reset: 56
X-Kong-Response-Latency: 0
Server: kong/2.2.1

{
  "message":"API rate limit exceeded"
}

ip-restriction IP限制

在路由上创建 ip-restriction 插件配置，允许8.210.1.244 访问，拒绝 47.242.91.20 访问。

echo "
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: ip-restriction-test
config: 
  allow:
  - 8.210.1.244
  deny:
  - 47.242.91.20
plugin: ip-restriction
" | kubectl apply -f -

更新 Ingress

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo
  annotations:
    konghq.com/strip-path: "true"
    kubernetes.io/ingress.class: kong
    konghq.com/plugins: httpbin-auth, ip-restriction-test
spec:
  rules:
  - host: "foo.bar.com"
    http:
      paths:
      - pathType: Prefix
        path: /billing
        backend:
          service:
            name: billing
            port:
              number: 80
      - pathType: Prefix
        path: /comments
        backend:
          service:
            name: comments
            port:
              number: 80
      - pathType: Prefix
        path: /invoice
        backend:
          service:
            name: invoice
            port:
              number: 80

在 47.242.91.20 上访问拒绝

[root@k8s-test01 kong]# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/invoice/status/200
HTTP/1.1 403 Forbidden
Date: Sat, 06 Feb 2021 10:18:54 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
X-Kong-Response-Latency: 1
Server: kong/2.2.1

{
  "message":"Your IP address is not allowed"
}

在 8.210.1.244 上正常访问

---
root@k8s-test02:~# curl -i -H 'apikey: my-sooper-secret-key' http://foo.bar.com:8000/invoice/status/200
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Date: Sat, 06 Feb 2021 10:30:20 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 25
X-Kong-Proxy-Latency: 0
Via: kong/2.2.1

总结

通过上面的示例，我们大概了解了在 Kubernetes 环境下 Kong 与 Ingress 的结合使用，目前对 Consumer 没有太多的使用，后续有机会再跟大家介绍下。官方也有很多的示例，感兴趣的朋友可以去官网查看。

Kong 拥有很强的扩展能力，如果自带的插件不能满足我们的需求，我们可以写插件来实现。笔者的同事也写了一些插件在生产环境中运行。 Kubernetes Ingress — Kong

写好的插件，重新构建镜像，通过下面环境变量启用。

  - name: KONG_LUA_PACKAGE_PATH
  value: "/go/kong-lua/kong-plugin/?.lua;;"
- name: KONG_PLUGINS
  value: "bundled,myplugin,jwt-plus,oauth2-plus,oss-simple"

感兴趣的读者可以关注下微信号 Kubernetes Ingress — Kong