k8s集成gitlab-runner、maven、docker、kubelet等全自动发布全流程实践

Aug 5, 2026 // 大蚊子
虚拟化
k8s 虚拟化


基于gitlab runner 的did(docker in docker ) ci/cd k8s方案 首先，jenkins很强大，尤其是各种插件的支持，但实际个人工作中，用到的并不多，早期大型项目布署负载各种脚本和 远程调用，目前所有项目和k8s深耦合，已经拆解为各种云服务，jenkins的大部分功能用不到 其次，这只是一种可行的方案，并不是最优的方案，不同阶段也都有再调整和优化的空间 最后，对个人的需求，jenkins过于复杂，gitlab-runner足可胜任，对研发人员更透明友好，学习应用成本更低。 CI/CD集成k8s服务,对各种文件抽像的只有三步（部分语言生态只需要一步） 1 编译环境加载依赖编译可执行文件 2 执行环境+可执行文件打包为docker image并上传至docker hub 3 执行k8s布署，启动服务 k8s布署k8s gitlab-runner 服务 apiVersion: apps/v1 kind: StatefulSet metadata: name: gitlab-runner-java namespace: common spec: replicas:1 revisionHistoryLimit:10 selector: matchLabels: app: gitlab-runner-java serviceName: gitlab-runner-java template: metadata: creationTimestamp:null labels: app: gitlab-runner-java name: gitlab-runner-java spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms:

  • matchExpressions:

  • key: kubernetes.io/hostname operator: In values:

  • server-svc-2 containers:

  • image: gitlab/gitlab-runner:v12.4.1 imagePullPolicy: IfNotPresent name: gitlab-runner-java resources: limits: cpu:“4” memory: 4Gi requests: cpu:“2” memory: 1Gi securityContext: runAsUser:0 volumeMounts:

  • mountPath:/var/run/docker.sock name: docker-sock readOnly:true

  • mountPath:/etc/gitlab-runner name: config

  • mountPath:/cache name: cache

  • mountPath:/usr/share/maven/ref/repository name: maven dnsPolicy: ClusterFirst restartPolicy: Always schedulerName:default-scheduler securityContext:{} terminationGracePeriodSeconds:30 volumes:

  • hostPath: path:/var/run/docker.sock type:"" name: docker-sock

  • hostPath: path:/data1/pv/gitlab-runner-config/gitlab-runner-java type:"" name: config

  • hostPath: path:/data1/pv/gitlab-runner-cache/gitlab-runner-java type:"" name: cache

  • hostPath: path:/data2/pv/gitlab-runner-java-maven type:"" name: maven 首先布署gitlab-runner节点，使用did方案,did要直接访问因此需要较高的权限 个人比较熟悉k8s,k8s权限提升使用 securityContext: runAsUser: 0 另外需挂载几项目录， /var/run/docker.sock:/var/run/docker.sock 自不必说,did 必备 :/etc/gitlab-runner 默认容器内配置文件路径为/etc/gitlab-runner/config.toml，挂载方便维护 /cache 自不必说 :/usr/share/maven/ref/repository maven的默认的本地缓存repository路径 需要补充的是，只有/var/run/docker.sock是did必须其他三项是为了可以从k8s访问本地挂载目录提供的，方便在没有宿 主机权限下，通过k8s pod访问目录 docker ps |grep java 759e94f0dafd db5c21d7a3e4 “/usr/bin/dumb-init …” 6 weeks ago Up 6 weeks k8s_gitlab- runner-java_gitlab-runner-java-0_common_9112c414-cef3-4801-a234-4c8edb965e82_0 6bda34787976 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 “/pause” 6 weeks ago Up 6 weeks k8s_POD_gitlab- runner-java-0_common_9112c414-cef3-4801-a234-4c8edb965e82_0 目前为不同语言构建了不同的pod,以java的为例(java的本身较为复杂，maven打包为jar,再将jar以java包装为镜像) gitlab-runner gitlab-runner-java-0 1/1 Running 0 47d gitlab-runner gitlab-runner-node-0 1/1 Running 0 47d

gitlab-runner gitlab-runner-python-0 1/1 Running 0 47d gitlab-runner gitlab-runner-golang-0 1/1 Running 0 47d 进入gitlab后台，获取项目的token

Screen Shot 2021-01-11 at 3.49.31 PM

进入容器以token 注册gitlab-runner

root@gitlab-runner-java-0:/# gitlab-runner register -n \

–url https://[abc.gitlab.com]
–registration-token [your_project_token]
–tag-list java,prod,preview,test
–executor docker
–description “java”
–cache-dir /data1/pv/gitlab-runner-cache/gitlab-runner-java/cache-dir
–docker-cache-dir /data1/pv/gitlab-runner-cache/gitlab-runner-java/docker-cache- dir
–docker-image “docker:18.06.3”
–docker-pull-policy “if-not-present”
–docker-volumes /var/run/docker.sock:/var/run/docker.sock
–docker-volumes /root/.docker/config.json:/root/.docker/config.json
–docker-volumes /data1/pv/gitlab-runner-cache/gitlab-runner-java:/cache
–docker-volumes /data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository Runtime platform arch=amd64 os=linux pid=94 revision=05161b14 version=12.4.1 Running in system-mode. Registering runner… succeeded runner=joTBdSTv 重点注意–docker-volumes的几项参数，因为是did的执行方式，外部目录尤其是(/root/.m2/repository)需要在maven容器 持久化，k8s yaml mount的路径只是查看需要，gitlab-runner注册时的docker-volumes才是真正在不同任务间共享数据 的必须 例如–docker-volumes /root/.docker/config.json:/root/.docker/config.json 宿主机docker login的授权信息，可以共享使用 ​ –docker-volumes /data2/pv/gitlab-runner-java-maven/repository:/root/.m2/repository 对maven容器，不同项目共同挂 载宿主机目录，减少jar包的重复下载 注册完后的内容如 root@gitlab-runner-java-0:/# cat /etc/gitlab-runner/config.toml concurrent = 8 check_interval = 0 [session_server]

session_timeout = 1800 [[runners]] name = “java” url = “https://[abc.123.com]” token = “[your token]” executor = “docker” cache_dir = “/data1/pv/gitlab-runner-cache/gitlab-runner-java/cache-dir” [runners.custom_build_dir] [runners.docker] tls_verify = false image = “docker:18.06.3” privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/var/run/docker.sock:/var/run/docker.sock", “/root/.docker/config.json:/root/.docker/config.json”, “/data1/pv/gitlab-runner- cache/gitlab-runner-java:/cache”, “/data2/pv/gitlab-runner-java- maven/repository:/root/.m2/repository”] cache_dir = “/data1/pv/gitlab-runner-cache/gitlab-runner-java/docker-cache-dir” pull_policy = “if-not-present” shm_size = 0 [runners.cache] [runners.cache.s3] [runners.cache.gcs] 重点的项目配置文件，具体项目结构见https://github.com/cclient/hanlp-remote-dict . ├── Dockerfile –打包docker镜像配置 ├── deploy –镜像打包成功，更新上线至k8s配置 │   ├── deployment_preview.yaml │   ├── deployment_prod.yaml │   └── deployment_test.yaml ├── .gitlab-ci.yml –gitlab-runner ci/cd配置 ├── pom.xml –java项目依赖配置

.gitlab-ci.yml variables: DOCKER_API_VERSION: ‘1.38’ DOCKER_HUB: “[docker hub/registry]” DOCKER_USERNAME: “[your docker user]” DOCKER_PASSWORD: “[your docker passwd]” PROJECT_NAME: $CI_PROJECT_NAME PROJECT_VERSION: $CI_COMMIT_TAG K8S_API_URL: “https://[k8s api host]:6443” K8S_ACCESS_TOKEN: “[your k8s token]” K8S_DEPLOY_BASE_NAME: “hanlp-remote-dict” CONTAINER_IMAGE: $DOCKER_HUB/$CI_PROJECT_NAME image: docker:18.06.3 stages:

  • prepare-env
  • maven-package
  • docker-build
  • k8s-deploy

根据git branch/tag匹配 cicd上线环境，添加相关环境变量

prepare-env-develop: stage: prepare-env tags:

  • java only:
  • develop before_script:
  • mkdir -p ./sartifacts/ script:
  • echo BUILD_ENV=“test” » ./sartifacts/version
  • echo BUILD_IMAGE_VERSION=“test” » ./sartifacts/version artifacts: name: “${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}” paths:
  • ./sartifacts/* expire_in: 2 day prepare-env-preview:

stage: prepare-env tags:

  • java only:
  • /^release/.*$/
  • master before_script:
  • echo docker login hub.intra.github.com -u “’$DOCKER_USERNAME’” -p $DOCKER_PASSWORD
  • mkdir -p ./sartifacts/ script:
  • echo BUILD_ENV=“preview” » ./sartifacts/version
  • echo BUILD_IMAGE_VERSION=“preview” » ./sartifacts/version artifacts: name: “${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}” paths:
  • ./sartifacts/* expire_in: 2 day prepare-env-production: stage: prepare-env tags:
  • java only:
  • tags before_script:
  • mkdir -p ./sartifacts/ script:
  • echo BUILD_ENV=“prod” » ./sartifacts/version
  • echo BUILD_IMAGE_VERSION=$CI_COMMIT_TAG » ./sartifacts/version artifacts: name: “${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}” paths:
  • ./sartifacts/* expire_in: 2 day

以下为通用部分

mvn 打包java项目，生成jar包，并将jar包添加至./sartifacts/ 传至下一级

maven-package: image: maven:3.6.3-adoptopenjdk-8 stage: maven-package tags:

  • java only:
  • tags
  • /^release/.*$/
  • master
  • develop before_script:

source加载./sartifacts/version的环境变量

  • source ./sartifacts/version

echo只是调试用，打印gitlab-runner结合git的变量

  • echo ‘CI_CONFIG_PATH’ $CI_CONFIG_PATH
  • echo ‘CI_COMMIT_REF_NAME’ $CI_COMMIT_REF_NAME
  • echo ‘CI_COMMIT_BRANCH’ $CI_COMMIT_BRANCH
  • echo ‘CI_COMMIT_TAG’ $CI_COMMIT_TAG
  • echo ‘CI_COMMIT_TITLE’ $CI_COMMIT_TITLE
  • echo ‘CI_ENVIRONMENT_NAME’ $CI_ENVIRONMENT_NAME
  • echo ‘CONTAINER_IMAGE’ $CONTAINER_IMAGE
  • echo ‘ENV’ $ENV
  • echo ‘MAVEN_OPTS’ “$MAVEN_OPTS”
  • echo ‘BUILD_ENV’ “$BUILD_ENV”
  • echo ‘BUILD_IMAGE_VERSION’ “$BUILD_IMAGE_VERSION” script:
  • mvn clean package -Dmaven.test.skip=true -Dmaven.repo.local=/root/.m2/repository
  • mv target ./sartifacts/ artifacts: name: “${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}” paths:
  • ./sartifacts/* expire_in: 2 day

获取上一级的sartifacts，提取target目录，打包image并上传

docker-build: image: docker:18.06.3 stage: docker-build tags:

  • java only:

  • tags

  • /^release/.*$/

  • master

  • develop before_script:

  • pwd

  • ls -alh

  • source ./sartifacts/version

  • mv ./sartifacts/target ./target script:

  • docker build -t $CONTAINER_IMAGE:$BUILD_IMAGE_VERSION ./

- docker login hub.intra.github.com -u $DOCKER_USERNAME -p $DOCKER_PASSWORD

  • docker push $CONTAINER_IMAGE:$BUILD_IMAGE_VERSION

执行k8s yaml 上线pod/deploy/sts等服务

k8s-deploy: image: telefonica/kubectl:1.15.10 stage: k8s-deploy tags:

  • java only:
  • tags
  • /^release/.*$/
  • master
  • develop before_script:
  • source ./sartifacts/version
  • echo K8S_DEPLOY_BASE_NAME-BUILD_ENV “$K8S_DEPLOY_BASE_NAME-$BUILD_ENV”
  • echo CONTAINER_IMAGE “$CONTAINER_IMAGE” script:
  • kubectl –server="${K8S_API_URL}" –token="$K8S_ACCESS_TOKEN" –insecure-skip-tls- verify apply -f ./deploy/deployment_$BUILD_ENV.yaml -n default
  • kubectl –server="${K8S_API_URL}" –token="$K8S_ACCESS_TOKEN" –insecure-skip-tls- verify set image deployment/$K8S_DEPLOY_BASE_NAME-$BUILD_ENV $K8S_DEPLOY_BASE_NAME-$BUILD_ENV=$CONTAINER_IMAGE:$BUILD_IMAGE_VERSION -n default – record 基本流程为 1 第一阶段 prepare-env 根据gitlab 提交和分枝信息，构造version文件(保存env),供后续阶段使用，目前实现了三项 prepare-env-develop，prepare-env-preview，prepare-env-production，内容一样是写入version,为了保证这里不够 精简，看了下gitlab-runner的rule，应该可以有更优雅的方案，但个人没有精力深入，有知道更好办法的，烦请告 知。 2 第二阶段 maven-package did以maven:3.6.3-adoptopenjdk-8为底包打包项目，生成target目录(下有jar包)，并将 targer整个打包为sartifacts传给下一阶段，大包的话，只用jar即可 3 第三阶段 docker-build dit以docker:18.06.3为底包，自动下载解压sartifacts，拷备出target至项目目标地址，根据 Dockerfile(对该示例而言底包为openjdk:8u162-jre-slim-stretch) 生成image 并上传,

docker login hub.intra.github.com -u $DOCKER_USERNAME -p $DOCKER_PASSWORD 这一步如果挂载了宿主机的docker 授权则不必要 4 第四阶段 k8s-deploy did 以telefonica/kubectl:1.15.10为底包,因需选择版本，个人用该镜像从k8s v1.15.0用到 v1.18.5，目前使用正常，通过api 提交相应deploy.yaml至k8s api，这一阶段执行两部，kubectl apply -f deploy.yaml 提交上线，再kubectl set image 更新deploy内服务image的真实版本 docker file FROM openjdk:8u162-jre-slim-stretch WORKDIR /home/spring ADD target/*.jar /home/spring/app.jar deploy.yaml 以测试deploy为例 apiVersion: apps/v1 kind: Deployment metadata: labels: app: hanlp-remote-dict-test name: hanlp-remote-dict-test namespace: default spec: replicas: 1 selector: matchLabels: app: hanlp-remote-dict-test strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 template: metadata: labels: app: hanlp-remote-dict-test spec: containers:

  • image: cclient/arm64v8-kafka:test command:
  • java args:
  • -Xms1g
  • -Xmx2g
  • -XX:+UseG1GC
  • -XX:G1ReservePercent=10
  • -XX:+UseStringDeduplication
  • -jar
  • app.jar imagePullPolicy: Always name: hanlp-remote-dict-test resources: limits: cpu: 1 memory: 2Gi requests: cpu: 1 memory: 1Gi restartPolicy: Always

apiVersion: v1 kind: Service metadata: name: hanlp-remote-dict-test namespace: default spec: externalTrafficPolicy: Local ports:

  • port: 8080 protocol: TCP targetPort: 8080 selector: app: hanlp-remote-dict-test sessionAffinity: None type: NodePort

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hanlp-remote-dict-test

namespace: default spec: rules:

  • host: hanlp-remote-dict-test.github.com http: paths:
  • path: / backend: serviceName: hanlp-remote-dict-test servicePort: 8080 执行成功样例

Screen Shot 2021-01-11 at 5.58.36 PM

[session_server].listen_address not defined, session endpoints disabled builds=0 Checking for jobs… received job=196962 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN Job succeeded duration=21.427262629s job=196962 project=11190 runner=sdWbANrN Checking for jobs… received job=196963 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN Job succeeded duration=2m3.362519176s job=196963 project=11190 runner=sdWbANrN Checking for jobs… received job=196964 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN Job succeeded duration=4m8.502822318s job=196964 project=11190 runner=sdWbANrN Checking for jobs… received job=196965 repo_url=https://github.com/cclient/hanlp-remote-dict.git runner=sdWbANrN 其他 1 目前spring boot 构造为一个大包，docker image上传下载io开销较大，可调整为先下载准备了依赖，再打小包 2 个人没有使用https://github.com/fabric8io/docker-maven-plugin等mvn 组件，主要原因是要支持多语言，方案不同， 但共用通用的ci/cd抽像，对其他语言而言，可能不存在相同定位的包管理插件，docker cmd 更通用，对环境的依赖也 小 3 test,preview,prod 不同上线分枝可以提交不同的配置 限于篇幅太长，最后test,preview,prod对应git 分枝管理，以后再弹独写一篇