写点什么

TiDB Operator 实践

  • 2019-10-30
  • 本文字数:6851 字

    阅读完需:约 22 分钟

TiDB Operator实践

K8s 和 TiDB 都是目前开源社区中活跃的开源产品,TiDB Operator 项目是一个在 K8s 上编排管理 TiDB 集群的项目。本文详细记录了部署 K8s 及 install TiDB Operator 的详细实施过程,希望能对刚"入坑"的同学有所帮助。

一、环境

Ubuntu 16.04


K8s 1.14.1

二、Kubespray 安装 K8s

配置免密登录

1yum -y install expect
复制代码


  • vi /tmp/autocopy.exp


 1#!/usr/bin/expect 2 3set timeout 4set user_hostname [lindex $argv ] 5set password [lindex $argv ] 6spawn ssh-copy-id $user_hostname 7    expect { 8        "(yes/no)?" 9        {10            send "yes\n"11            expect "*assword:" { send "$password\n"}12        }13        "*assword:"14        {15            send "$password\n"16        }17    }18expect eof
复制代码


 1ssh-keyscan addedip  >> ~/.ssh/known_hosts 2 3ssh-keygen -t rsa -P '' 4 5for i in 10.0.0.{31,32,33,40,10,20,50}; do  ssh-keyscan $i  >> ~/.ssh/known_hosts ; done 6 7/tmp/autocopy.exp root@addeip 8ssh-copy-id addedip 910/tmp/autocopy.exp root@10.0.0.3111/tmp/autocopy.exp root@10.0.0.3212/tmp/autocopy.exp root@10.0.0.3313/tmp/autocopy.exp root@10.0.0.4014/tmp/autocopy.exp root@10.0.0.1015/tmp/autocopy.exp root@10.0.0.2016/tmp/autocopy.exp root@10.0.0.50
复制代码

配置 Kubespray

1pip install -r requirements.txt2cp -rfp inventory/sample inventory/mycluster
复制代码


  • inventory/mycluster/inventory.ini

  • inventory/mycluster/inventory.ini


1# ## Configure 'ip' variable to bind kubernetes services on a 2# ## different ip than the default iface 3# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value. 4[all] 5# node1 ansible_host=95.54.0.12  # ip=10.3.0.1 etcd_member_name=etcd1 6# node2 ansible_host=95.54.0.13  # ip=10.3.0.2 etcd_member_name=etcd2 7# node3 ansible_host=95.54.0.14  # ip=10.3.0.3 etcd_member_name=etcd3 8# node4 ansible_host=95.54.0.15  # ip=10.3.0.4 etcd_member_name=etcd4 9# node5 ansible_host=95.54.0.16  # ip=10.3.0.5 etcd_member_name=etcd510# node6 ansible_host=95.54.0.17  # ip=10.3.0.6 etcd_member_name=etcd611etcd1 ansible_host=10.0.0.31 etcd_member_name=etcd112etcd2 ansible_host=10.0.0.32 etcd_member_name=etcd213etcd3 ansible_host=10.0.0.33 etcd_member_name=etcd314master1 ansible_host=10.0.0.4015node1 ansible_host=10.0.0.1016node2 ansible_host=10.0.0.2017node3 ansible_host=10.0.0.501819# ## configure a bastion host if your nodes are not directly reachable20# bastion ansible_host=x.x.x.x ansible_user=some_user2122[kube-master]23# node124# node225master126[etcd]27# node128# node229# node330etcd131etcd232etcd33334[kube-node]35# node236# node337# node438# node539# node640node141node242node34344[k8s-cluster:children]45kube-master46kube-node
复制代码

节点所需镜像的文件

由于某些镜像国内无法访问需要现将镜像通过代理下载到本地然后上传到本地镜像仓库或 DockerHub,同时修改配置文件,个别组件存放位置https://storage.googleapis.com,需要新建 Nginx 服务器分发文件。


建立 Nginx 服务器


  • ~/distribution/docker-compose.yml

  • 创建文件目录及 Nginx 配置文件目录

  • ~/distribution/conf.d/open_distribute.conf

  • 启动

  • 下载并上传所需文件 具体版本号参考 roles/download/defaults/main.yml 文件中 kubeadm_version、kube_version、image_arch 参数

  • 安装 Docker 及 Docker-Compose


 1apt-get install \ 2apt-transport-https \ 3ca-certificates \ 4curl \ 5gnupg-agent \ 6software-properties-common 7 8curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - 910add-apt-repository \11"deb [arch=amd64] https://download.docker.com/linux/ubuntu \12$(lsb_release -cs) \13stable"1415apt-get update1617apt-get install docker-ce docker-ce-cli containerd.io1819chmod +x /usr/local/bin/docker-compose20sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
复制代码


  • 新建 Nginx docker-compose.yml


1mkdir ~/distribution2vi ~/distribution/docker-compose.yml
复制代码


 1#  distribute 2version: '2' 3services:     4    distribute: 5        image: nginx:1.15.12 6        volumes: 7            - ./conf.d:/etc/nginx/conf.d 8            - ./distributedfiles:/usr/share/nginx/html 9        network_mode: "host"10        container_name: nginx_distribute 
复制代码


1mkdir ~/distribution/distributedfiles2mkdir ~/distribution/3mkdir ~/distribution/conf.d4vi ~/distribution/conf.d/open_distribute.conf
复制代码


 1#open_distribute.conf 2 3server { 4    #server_name distribute.search.leju.com; 5        listen 8888; 6 7    root /usr/share/nginx/html; 8 9    add_header Access-Control-Allow-Origin *;  10    add_header Access-Control-Allow-Headers X-Requested-With;  11    add_header Access-Control-Allow-Methods GET,POST,OPTIONS;  1213    location / {14    #    index index.html;15                autoindex on;        16    }17    expires off;18    location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|eot|ttf|woff|woff2|svg)$ {19        expires -1;20    }2122    location ~ .*\.(js|css)?$ {23        expires -1 ;24    }25} # end of public static files domain : [ distribute.search.leju.com ]
复制代码


1docker-compose up -d
复制代码


1wget https://storage.googleapis.com/kubernetes-release/release/v1.14.1/bin/linux/amd64/kubeadm23scp /tmp/kubeadm  10.0.0.60:/root/distribution/distributedfiles45wget https://storage.googleapis.com/kubernetes-release/release/v1.14.1/bin/linux/amd64/hyperkube
复制代码


  • 需要下载并上传到私有仓库的镜像


 1docker pull k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.4.0 2docker tag k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.4.0 jiashiwen/cluster-proportional-autoscaler-amd64:1.4.0 3docker push jiashiwen/cluster-proportional-autoscaler-amd64:1.4.0 4 5docker pull k8s.gcr.io/k8s-dns-node-cache:1.15.1 6docker tag k8s.gcr.io/k8s-dns-node-cache:1.15.1 jiashiwen/k8s-dns-node-cache:1.15.1 7docker push jiashiwen/k8s-dns-node-cache:1.15.1 8 9docker pull gcr.io/google_containers/pause-amd64:3.110docker tag gcr.io/google_containers/pause-amd64:3.1 jiashiwen/pause-amd64:3.111docker push jiashiwen/pause-amd64:3.11213docker pull gcr.io/google_containers/kubernetes-dashboard-amd64:v1.10.114docker tag gcr.io/google_containers/kubernetes-dashboard-amd64:v1.10.1 jiashiwen/kubernetes-dashboard-amd64:v1.10.115docker push jiashiwen/kubernetes-dashboard-amd64:v1.10.11617docker pull gcr.io/google_containers/kube-apiserver:v1.14.118docker tag gcr.io/google_containers/kube-apiserver:v1.14.1 jiashiwen/kube-apiserver:v1.14.119docker push jiashiwen/kube-apiserver:v1.14.12021docker pull gcr.io/google_containers/kube-controller-manager:v1.14.122docker tag gcr.io/google_containers/kube-controller-manager:v1.14.1 jiashiwen/kube-controller-manager:v1.14.123docker push jiashiwen/kube-controller-manager:v1.14.12425docker pull gcr.io/google_containers/kube-scheduler:v1.14.126docker tag gcr.io/google_containers/kube-scheduler:v1.14.1 jiashiwen/kube-scheduler:v1.14.127docker push jiashiwen/kube-scheduler:v1.14.12829docker pull gcr.io/google_containers/kube-proxy:v1.14.130docker tag gcr.io/google_containers/kube-proxy:v1.14.1 jiashiwen/kube-proxy:v1.14.131docker push jiashiwen/kube-proxy:v1.14.13233docker pull gcr.io/google_containers/pause:3.134docker tag gcr.io/google_containers/pause:3.1 jiashiwen/pause:3.135docker push jiashiwen/pause:3.13637docker pull gcr.io/google_containers/coredns:1.3.138docker tag gcr.io/google_containers/coredns:1.3.1 jiashiwen/coredns:1.3.139docker push  jiashiwen/coredns:1.3.1
复制代码


  • 用于下载上传镜像的脚本


 1#!/bin/bash 2 3privaterepo=jiashiwen 4 5k8sgcrimages=( 6cluster-proportional-autoscaler-amd64:1.4.0 7k8s-dns-node-cache:1.15.1 8) 910gcrimages=(11pause-amd64:3.112kubernetes-dashboard-amd64:v1.10.113kube-apiserver:v1.14.114kube-controller-manager:v1.14.115kube-scheduler:v1.14.116kube-proxy:v1.14.117pause:3.118coredns:1.3.119)202122for k8sgcrimageName in ${k8sgcrimages[@]} ; do23echo $k8sgcrimageName24docker pull k8s.gcr.io/$k8sgcrimageName25docker tag k8s.gcr.io/$k8sgcrimageName $privaterepo/$k8sgcrimageName26docker push $privaterepo/$k8sgcrimageName27done282930for gcrimageName in ${gcrimages[@]} ; do31echo $gcrimageName32docker pull gcr.io/google_containers/$gcrimageName33docker tag gcr.io/google_containers/$gcrimageName $privaterepo/$gcrimageName34docker push $privaterepo/$gcrimageName35done
复制代码


  • 修改文件 inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml,修改 K8s 镜像仓库


1# kube_image_repo: "gcr.io/google-containers"2kube_image_repo: "jiashiwen"
复制代码


  • 修改 roles/download/defaults/main.yml


 1#dnsautoscaler_image_repo: "k8s.gcr.io/cluster-proportional-autoscaler-{{   image_arch }}" 2dnsautoscaler_image_repo: "jiashiwen/cluster-proportional-autoscaler-{{   image_arch }}" 3 4#kube_image_repo: "gcr.io/google-containers" 5kube_image_repo: "jiashiwen" 6 7#pod_infra_image_repo: "gcr.io/google_containers/pause-{{ image_arch }}" 8pod_infra_image_repo: "jiashiwen/pause-{{ image_arch }}" 910#dashboard_image_repo: "gcr.io/google_containers/kubernetes-dashboard-{{   image_arch }}"11dashboard_image_repo: "jiashiwen/kubernetes-dashboard-{{ image_arch }}"1213#nodelocaldns_image_repo: "k8s.gcr.io/k8s-dns-node-cache"14nodelocaldns_image_repo: "jiashiwen/k8s-dns-node-cache"1516#kubeadm_download_url: "https://storage.googleapis.com/kubernetes-release/  release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"17kubeadm_download_url: "http://10.0.0.60:8888/kubeadm"1819#hyperkube_download_url: "https://storage.googleapis.com/  kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/  hyperkube"20hyperkube_download_url: "http://10.0.0.60:8888/hyperkube"
复制代码

三、执行安装

  • 安装命令


1ansible-playbook -i inventory/mycluster/inventory.ini cluster.yml
复制代码


  • 重置命令


1ansible-playbook -i inventory/mycluster/inventory.ini reset.yml
复制代码

四、验证 K8s 集群

安装 Kubectl


  • 本地浏览器打开得到最新版本为 v1.14.1

  • 用上一步得到的最新版本号 v1.7.1 替换下载地址中的 $(curl -s 下载地址)得到真正的下载地址

  • 上传下载好的 kubectl


1scp /tmp/kubectl root@xxx:/root
复制代码


  • 修改属性


1chmod +x ./kubectl2mv ./kubectl /usr/local/bin/kubectl
复制代码


  • Ubuntu


1sudo snap install kubectl --classic
复制代码


  • CentOS


将 master 节点上的~/.kube/config 文件复制到你需要访问集群的客户端上即可


1scp 10.0.0.40:/root/.kube/config ~/.kube/config
复制代码


执行命令验证集群


1kubectl get nodes2kubectl cluster-info
复制代码

五、TiDB-Operaor 部署

安装helm

  • 安装 helm


1curl https://raw.githubusercontent.com/helm/helm/master/scripts/get > get_helm.sh2chmod 700 get_helm.sh3./get_helm.sh
复制代码


  • 查看 helm 版本


1helm version
复制代码


  • 初始化


1helm init --upgrade -i registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.13.1 --stable-repo-url https://kubernetes.os
复制代码

为 K8s 提供 local volumes

  • 参考文档

  • tidb-operator 启动会为 pd 和 tikv 绑定 pv,需要在 discovery directory 下创建多个目录

  • 格式化并挂载磁盘


1mkfs.ext4 /dev/vdb2DISK_UUID=$(blkid -s UUID -o value /dev/vdb) 3mkdir /mnt/$DISK_UUID4mount -t ext4 /dev/vdb /mnt/$DISK_UUID
复制代码


  • /etc/fstab 持久化 mount


1echo UUID=`sudo blkid -s UUID -o value /dev/vdb` /mnt/$DISK_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab
复制代码


  • 创建多个目录并 mount 到 discovery directory


1for i in $(seq 1 10); do2sudo mkdir -p /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i}3sudo mount --bind /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i}4done
复制代码


  • /etc/fstab 持久化 mount


1for i in $(seq 1 10); do2echo /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i} none bind 0 0 | sudo tee -a /etc/fstab3done
复制代码


  • 为 tidb-operator 创建 local-volume-provisioner


1$ kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/local-dind/local-volume-provisioner.yaml2$ kubectl get po -n kube-system -l app=local-volume-provisioner3$ kubectl get pv --all-namespaces | grep local-storage 
复制代码

六、Install TiDB Operator

项目中使用了 gcr. io/google-containers/hyperkube,国内访问不了,简单的办法是把镜像重新 push 到 dockerhub 然后修改 charts/tidb-operator/values.yaml


1scheduler: 2  # With rbac.create=false, the user is responsible for creating this   account 3  # With rbac.create=true, this service account will be created 4  # Also see rbac.create and clusterScoped 5  serviceAccount: tidb-scheduler 6  logLevel: 2 7  replicas: 1 8  schedulerName: tidb-scheduler 9  resources:10    limits:11      cpu: 250m12      memory: 150Mi13    requests:14      cpu: 80m15      memory: 50Mi16  # kubeSchedulerImageName: gcr.io/google-containers/hyperkube17  kubeSchedulerImageName: yourrepo/hyperkube18  # This will default to matching your kubernetes version19  # kubeSchedulerImageTag: latest
复制代码


  • TiDB Operator 使用 CRD 扩展 Kubernetes,因此要使用 TiDB Operator,首先应该创建 TidbCluster 自定义资源类型。


1kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/crd.yaml2kubectl get crd tidbclusters.pingcap.com
复制代码


  • 安装 TiDB-Operator


1$ git clone https://github.com/pingcap/tidb-operator.git2$ cd tidb-operator3$ helm install charts/tidb-operator --name=tidb-operator   --namespace=tidb-admin4$ kubectl get pods --namespace tidb-admin -l app.kubernetes.io/  instance=tidb-operator
复制代码

七、部署 TiDB

1helm install charts/tidb-cluster --name=demo --namespace=tidb2watch kubectl get pods --namespace tidb -l app.kubernetes.io/instance=demo -o wide
复制代码

八、验证

安装 MySQL 客户端


1wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm2yum localinstall mysql80-community-release-el7-3.noarch.rpm -y3yum repolist all | grep mysql4yum-config-manager --disable mysql80-community5yum-config-manager --enable mysql57-community6yum install mysql-community-client
复制代码


  • Ubuntu 安装


1wget https://dev.mysql.com/get/mysql-apt-config_0.8.13-1_all.deb2dpkg -i mysql-apt-config_0.8.13-1_all.deb3apt update45# 选择MySQL版本6dpkg-reconfigure mysql-apt-config7apt install mysql-client -y
复制代码

九、映射 TiDB 端口

  • 查看 TiDB Service


1kubectl get svc --all-namespaces
复制代码


  • 映射 TiDB 端口


1# 仅本地访问2kubectl port-forward svc/demo-tidb 4000:4000 --namespace=tidb34# 其他主机访问5kubectl port-forward --address 0.0.0.0 svc/demo-tidb 4000:4000 --namespace=tidb
复制代码


  • 首次登录 MySQL


1mysql -h 127.0.0.1 -P 4000 -u root -D test
复制代码


  • 修改 TiDB 密码


1SET PASSWORD FOR 'root'@'%' = 'wD3cLpyO5M'; FLUSH PRIVILEGES;
复制代码


2019-10-30 17:591069

评论

发布
暂无评论
发现更多内容

业务连续性专题:DBless

agnostic

高可用架构

我的数据清洗能手小浣熊

轻口味

征文活动 办公小浣熊 小浣熊家族

小浣熊家族:自媒体新手的快速成长指南

程序员海军

AI 办公小浣熊 小浣熊家族 小浣熊

供配电学习笔记 day4

万里无云万里天

自动化 电力 工厂运维

从0-100:钓鱼场小程序开发笔记(上)

CC同学

java+uniapp实现微信JSSDK扫码功能

源字节1号

开源 软件开发 前端开发 后端开发 小程序开发

客户在哪儿AI用数据解决ToB企业市场和销售脱节问题

客户在哪儿AI

ToB营销 ToB获客 ToB增长 ToB销售

【开源鸿蒙】编译OpenHarmony轻量系统QEMU RISC-V版本

码匠许师傅

qemu OpenHarmony risc-v

加速数字化转型,信创自主可控:TapData 为银行业数据管理能力建设提供新思路

tapdata

我的数据分析大师-办公小浣熊

黑白相间

数据分析 办公小浣熊

ToB活动营销先要区分战略级活动、常规级活动和与我无关的活动

客户在哪儿AI

ToB营销 活动营销 ToB获客 ToB销售

开源创新引领未来|酷克数据亮相PostgreSQL中国技术大会,荣获数据库杰出贡献奖

酷克数据HashData

小浣熊,本领强,助我轻松把活忙

知日

AI #人工智能 小浣熊家族

想知道海外技术面试都考些什么吗?

王中阳Go

Go 数据库 面试 算法 面经

实测小浣熊AI办公神器(效率翻倍不是梦)

攻城先森

人工智能 智能助手 大模型 AIGC 办公小浣熊

XMind for Mac:专业思维导图软件,提升思维与工作效率

Mac相关知识分享

Dash for Mac(好用的API文档工具) v7.2.4版

Mac相关知识分享

ToB市场部如何选择最优的合作平台

客户在哪儿AI

ToB营销 ToB获客 ToB增长 ToB销售

如何彻底解决ToB企业市场和销售脱节的问题

客户在哪儿AI

ToB营销 ToB获客 ToB销售

破解缺陷管理:挑选顶尖系统的秘籍

爱吃小舅的鱼

缺陷跟踪

如有神威,办公小浣熊助我轻松拿捏数据分析

战场小包

小浣熊 AI办公助手

无损音乐播放器推荐:Audirvana for Mac 中文激活版

你的猪会飞吗

Mac 软件

Mac应用程序清理卸载工具:App Cleaner & Uninstaller for Mac 中文版

你的猪会飞吗

Mac软件下载站 mac破解软件下载

深入了解项目跟踪软件的关键优势

爱吃小舅的鱼

项目进度管理

Oracle,MySQL,SQLServer三种关系型数据库的特点介绍

源字节1号

小程序 开源 软件开发 前端开发 后端开发

多功能文件同步对比工具Beyond Compare 4 for Mac

Mac相关知识分享

TiDB Operator实践_架构_京东云_InfoQ精选文章