参考：https://www.rabbitmq.com/kubernetes/operator/install-topology-operator.html

参考：https://github.com/rabbitmq/messaging-topology-operator

Operator说明

  • 官方的描述：该operator允许开发人员使用声明式kubernetes API去创建和管理RabbitMQ集群的消息传递拓扑(messaging topologies)。所谓消息传递拓扑，指的是交换机(exchanges)，消息队列(queues)，绑定(Binding)，策略(Policy)等对象的集合。通俗一点来说就是该Operator将RabbitMQ集群里面的多种资源对象变成了kubernetes资源对象，通过创建和删除kubernetes资源对象就可以完成RabbitMQ集群资源对象的操作。
  • 该Operator是配合RabbitMQ Cluster Kubernetes Operator来使用的。
    • RabbitMQ Cluster Kubernetes Operator用来在kubernetes集群上自动配置，创建，管理RabbitMQ集群；
    • RabbitMQ Messaging Topology Operator用来管理上面RabbitMQ集群的消息传递拓扑

该Operator包含多个admission webhooks，启用了TLS通信，因此依赖一组TLS证书。该TLS证书可以自己手动生成，也可以通过官方推荐的cert-manager工具去生成。

下面使用的是cert-manager生成TLS证书和部署RabbitMQ Messaging Topology Operator的过程

如果需要手动生成证书来部署Operator，请参考这里。

一、安装RabbitMQ Messaging Topology Operator

部署cert-manager

该步骤会创建一个cert-manager的namespace，然后在该namespace下面创建所需要的相关资源

root@ub62286:~/rabbitmq-messaging# kubectl apply -f cert-manager.yaml 
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created

安装Operator

cert-manager自签的证书有效期是3个月，到期了是会自动续签的。或者可以在Certificate资源的.spec字段里面添加duration: 99999h

root@ub62286:~/rabbitmq-messaging# kubectl apply -f messaging-topology-operator-with-certmanager.yaml 
namespace/rabbitmq-system unchanged
customresourcedefinition.apiextensions.k8s.io/bindings.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/exchanges.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/federations.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/permissions.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/policies.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/queues.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/schemareplications.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/shovels.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/superstreams.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/topicpermissions.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/users.rabbitmq.com created
customresourcedefinition.apiextensions.k8s.io/vhosts.rabbitmq.com created
serviceaccount/messaging-topology-operator created
role.rbac.authorization.k8s.io/messaging-topology-leader-election-role created
clusterrole.rbac.authorization.k8s.io/messaging-topology-manager-role created
rolebinding.rbac.authorization.k8s.io/messaging-topology-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/messaging-topology-manager-rolebinding created
service/webhook-service created
deployment.apps/messaging-topology-operator created
certificate.cert-manager.io/serving-cert created
issuer.cert-manager.io/selfsigned-issuer created
validatingwebhookconfiguration.admissionregistration.k8s.io/topology.rabbitmq.com created

二、使用Operator

参考：https://www.rabbitmq.com/kubernetes/operator/using-topology-operator.html#queues-policies

通过RabbitMQ Cluster Kubernetes Operator部署的RabbitMQ集群，会把存放账号密码的secret记录在rabbitmqclusters资源的status字段里

root@ub62286:~# kubectl get rabbitmqclusters.rabbitmq.com hello-world -ojsonpath="{.status.binding.name}"
hello-world-default-user

RabbitMQ Messaging Topology Operator会读取这个配置信息连接RabbitMQ集群。

如果RabbitMQ集群不是以RabbitMQ Cluster Kubernetes Operator部署的，或者在部署的时候另外指定了账号密码，那么RabbitMQ Messaging Topology Operator就会读不到正确的RabbitMQ集群信息，后面创建资源会失败。

这时候就需要自己将RabbitMQ集群的连接信息放到secret里面，在创建RabbitMQ集群资源的时候使用rabbitmqClusterReference指定存放RabbitMQ集群连接信息的secret。参考这里。

创建RabbitMQ集群

这里使用的是RabbitMQ Cluster Kubernetes Operator去创建的。

root@ub62286:~/rabbitmq-cluster1# cat rabbitmq.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: rabbitmq-cluster1
---
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-cluster1
  namespace: rabbitmq-cluster1
spec:
  image: vestack-cr.starbucks.net/release/rabbitmq:3.10.2-management
  replicas: 1
  service:
    type: NodePort
root@ub62286:~/rabbitmq-cluster1# kubectl apply -f rabbitmq.yaml 
namespace/rabbitmq-cluster1 created
rabbitmqcluster.rabbitmq.com/rabbitmq-cluster1 created

root@ub62286:~/rabbitmq-cluster1# kubectl get all -n rabbitmq-cluster1 
NAME                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster1-server-0   1/1     Running   0          58s

NAME                              TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                          AGE
    service/rabbitmq-cluster1         NodePort    10.254.24.8   <none>        5672:30295/TCP,15672:30327/TCP,15692:31804/TCP   58s
service/rabbitmq-cluster1-nodes   ClusterIP   None          <none>        4369/TCP,25672/TCP                               58s

NAME                                        READY   AGE
statefulset.apps/rabbitmq-cluster1-server   1/1     58s

NAME                                             ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmqcluster.rabbitmq.com/rabbitmq-cluster1   True               True               58s

创建随机用户名密码账号

root@ub62286:~/rabbitmq-cluster1# cat user.yaml 
apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
  name: zhangsan
  namespace: rabbitmq-cluster1
spec:
  tags:
  - policymaker
  rabbitmqClusterReference:
    name: rabbitmq-cluster1

root@ub62286:~/rabbitmq-cluster1# kubectl apply -f user.yaml 
user.rabbitmq.com/zhangsan created

账号和密码保存在secret里面

root@ub62286:~/rabbitmq-cluster1# kubectl -n rabbitmq-cluster1 get secret zhangsan-user-credentials  -o jsonpath="{.data.username}" | base64 --decode
PYz6AwliiOE3Ti6EEnUi4ZYSeFplRMlY
root@ub62286:~/rabbitmq-cluster1# kubectl -n rabbitmq-cluster1 get secret zhangsan-user-credentials  -o jsonpath="{.data.password}" | base64 --decode
xG91pzXDzLUJ6D4ICpS2z37zaagkt5dP

登陆RabbitMQ管理页面可以看到这个用户

image-20230307170926648

创建指定用户名密码账号

root@ub62286:~/rabbitmq-cluster1# kubectl -n rabbitmq-cluster1 create secret generic lisi-user-credentials --from-env-file=./user-lisi.txt
secret/lisi-user-credentials created
root@ub62286:~/rabbitmq-cluster1# cat user-lisi.txt 
username=lisi
password=423dc10ee9f1
root@ub62286:~/rabbitmq-cluster1# cat user-lisi.yaml 
apiVersion: rabbitmq.com/v1beta1
kind: User
metadata:
  name: lisi
  namespace: rabbitmq-cluster1
spec:
  tags:
  - policymaker
  - monitoring
  rabbitmqClusterReference:
    name: rabbitmq-cluster1
  importCredentialsSecret:
    name: lisi-user-credentials
#user资源可能会报没有update secret的权限，修改clusterrole messaging-topology-manager-role，secret资源里面加上update权限，然后重新执行
root@ub62286:~/rabbitmq-cluster1# kubectl apply -f user-lisi.yaml 
user.rabbitmq.com/lisi created
image-20230307174532156

给现有用户lisi设置权限

操作前

image-20230307174649932
root@ub62286:~/rabbitmq-cluster1# cat user-lisi-permission.yaml
apiVersion: rabbitmq.com/v1beta1
kind: Permission
metadata:
  name: user-lisi-permission
  namespace: rabbitmq-cluster1
spec:
  vhost: "/"
  user: "lisi"
  permissions:
    write: ".*"
    configure: ".*"
    read: ".*"
  rabbitmqClusterReference:
    name: rabbitmq-cluster1
root@ub62286:~/rabbitmq-cluster1# kubectl apply -f user-lisi-permission.yaml
permission.rabbitmq.com/user-lisi-permission created

操作后

image-20230307175025183

创建虚拟主机(Virtual Host)

root@ub62286:~/rabbitmq-cluster1# cat vhost-test.yaml 
apiVersion: rabbitmq.com/v1beta1
kind: Vhost
metadata:
  name: test-vhost # name of this custom resource
  namespace: rabbitmq-cluster1
spec:
  name: test-vhost # name of the vhost
  rabbitmqClusterReference:
    name: rabbitmq-cluster1
root@ub62286:~/rabbitmq-cluster1# kubectl apply -f vhost-test.yaml
vhost.rabbitmq.com/test-vhost created
image-20230307175348526

创建消息队列(Queue)

root@ub62286:~/rabbitmq-cluster1# cat queue-test.yaml 
apiVersion: rabbitmq.com/v1beta1
kind: Queue
metadata:
  name: test # name of this custom resource; does not have to the same as the actual queue name
  namespace: rabbitmq-cluster1
spec:
  name: test # name of the queue
  autoDelete: false
  durable: true
  rabbitmqClusterReference:
    name: rabbitmq-cluster1
root@ub62286:~/rabbitmq-cluster1# kubectl apply -f queue-test.yaml
queue.rabbitmq.com/test created
image-20230307175713918

创建交换机(Exchange)

root@ub62286:~/rabbitmq-cluster1# cat exchange-fanout.yaml 
apiVersion: rabbitmq.com/v1beta1
kind: Exchange
metadata:
  name: fanout
  namespace: rabbitmq-cluster1
spec:
  name: fanout-exchange # name of the exchange
  type: fanout # default to 'direct' if not provided; can be set to 'direct', 'fanout', 'headers', and 'topic'
  autoDelete: false
  durable: true
  rabbitmqClusterReference:
    name: rabbitmq-cluster1
root@ub62286:~/rabbitmq-cluster1# kubectl apply -f exchange-fanout.yaml 
exchange.rabbitmq.com/fanout created
image-20230307180249415

创建绑定(Binding)

将fanout-exchange交换机和test消息队列绑定

root@ub62286:~/rabbitmq-cluster1# cat binding.yaml 
apiVersion: rabbitmq.com/v1beta1
kind: Binding
metadata:
  name: binding-test
  namespace: rabbitmq-cluster1
spec:
  source: fanout-exchange # an existing exchange
  destination: test # an existing queue
  destinationType: queue # can be 'queue' or 'exchange'
  rabbitmqClusterReference:
    name: rabbitmq-cluster1
root@ub62286:~/rabbitmq-cluster1# kubectl apply -f binding.yaml 
binding.rabbitmq.com/binding-test created
image-20230307180625276

其他更多功能参考官网文档。

更新资源

修改yaml文件，然后apply即可。但是有些字段是不可更新的，会报错，具体参考官网API文档

删除资源

使用kubectl delete删除资源，会同步删除RabbitMQ集群里面的资源

三、其他内容

指定集群认证信息

适用于不是以RabbitMQ Cluster Kubernetes Operator部署的RabbitMQ集群，或者在部署rabbitmq集群的时候另外指定了账号密码

创建secret存放rabbitmq集群信息

---
apiVersion: v1
kind: Secret
metadata:
  name: my-rabbit-creds
type: Opaque
stringData:
  username: a-user # has to be an existing user
  password: a-secure-password
  uri: https://my.rabbit:15672 # uri for the management api; when scheme is not provided in uri, operator defaults to 'http'

创建资源时候指定该secret

apiVersion: rabbitmq.com/v1beta1
kind: Queue
metadata:
  name: qq-example
spec:
  name: my-queue
  rabbitmqClusterReference:
    connectionSecret:
      name: my-rabbit-creds # has to an existing secret in the same namespace as this Queue object

TLS配置

如果RabbitmqClusters配置了HTTPS，那么需要在Messaging Topology Operator配置TLS，也就是挂载CA证书文件。参考这里。

将CA证书文件放入secret里面。这里使用的是generic类型的secret

kubectl -n rabbitmq-system create secret generic rabbitmq-ca --from-file=ca.crt=$CA_PATH

在messaging-topology-operator控制器里面挂载CA证书

$ kubectl -n rabbitmq-system patch deployment messaging-topology-operator --patch "spec:
  template:
    spec:
      containers:
      - name: manager
        volumeMounts:
        - mountPath: /etc/ssl/certs/rabbitmq-ca.crt
          name: rabbitmq-ca
          subPath: ca.crt
      volumes:
      - name: rabbitmq-ca
        secret:
          defaultMode: 420
          secretName: rabbitmq-ca"