Job?

Job개념

• 일회성으로 실행되고 나서 종료되어야 하는 성격의 작업을 실행할 때 사용되는 controller
• helm chart 에서는 주로 hook 의 개념으로 helm의 life cycle 중 (helm install, helm upgrade 등) 특정 단계에서 사전에 정의해둔 Job이 수행되도록 trigger 처럼 동작하게 많이 사용함
• linux의 crontab 처럼 정해진 스케쥴 마다 Job이 수행되게 하는 cronjob 도 있음

Job 사용현황

$ kubectl get job

NAME                                                  COMPLETIONS   DURATION   AGE
job.batch/kimdubi-test-redis-cluster-cluster-create   1/1           37s        4m27s
job.batch/kimdubi-test-redis-cluster-cluster-update   1/1           24s        3m50s

• kimdubi-test-redis-cluster-cluster-create Job은 helm install로 redis cluster를 구성할 때 위 Job이 install 후에 수행되어 redis cluster를 구성하는 용도로 사용되고 있음
• kimdubi-test-redis-cluster-cluster-update Job은 helm upgrade 커맨드로 redis cluster의 node를 추가할 때 동작하는 Job

Job template을 살펴보자

job.batch/kimdubi-test-redis-cluster-cluster-create

$ kubectl get -o yaml job.batch/kimdubi-test-redis-cluster-cluster-create

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    helm.sh/hook: post-install,post-upgrade
  creationTimestamp: "2021-02-21T07:33:55Z"
  labels:
    app.kubernetes.io/instance: kimdubi-test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: redis-cluster
    helm.sh/chart: redis-cluster-4.3.0
  name: kimdubi-test-redis-cluster-cluster-create
  namespace: default
  resourceVersion: "25079"
  selfLink: /apis/batch/v1/namespaces/default/jobs/kimdubi-test-redis-cluster-cluster-create
  uid: 91756ad1-444b-4593-90d1-8703a170cbc8
spec:
  activeDeadlineSeconds: 600
  backoffLimit: 6
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: 91756ad1-444b-4593-90d1-8703a170cbc8
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: kimdubi-test
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: redis-cluster
        controller-uid: 91756ad1-444b-4593-90d1-8703a170cbc8
        helm.sh/chart: redis-cluster-4.3.0
        job-name: kimdubi-test-redis-cluster-cluster-create
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: kimdubi-test
                  app.kubernetes.io/name: redis-cluster
              namespaces:
              - default
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - args:
        - |
          # Backwards compatibility change
          if ! [[ -f /opt/bitnami/redis/etc/redis.conf ]]; then
              cp /opt/bitnami/redis/etc/redis-default.conf /opt/bitnami/redis/etc/redis.conf
          fi
          /opt/bitnami/scripts/redis-cluster/entrypoint.sh /opt/bitnami/scripts/redis-cluster/run.sh
        command:
        - /bin/bash
        - -c
        env:
        - name: REDIS_TLS_ENABLED
          value: "no"
        - name: REDIS_PORT
          value: "6379"
        - name: REDIS_NODES
          value: 'kimdubi-test-redis-cluster-0.kimdubi-test-redis-cluster-headless
            kimdubi-test-redis-cluster-1.kimdubi-test-redis-cluster-headless kimdubi-test-redis-cluster-2.kimdubi-test-redis-cluster-headless
            kimdubi-test-redis-cluster-3.kimdubi-test-redis-cluster-headless kimdubi-test-redis-cluster-4.kimdubi-test-redis-cluster-headless
            kimdubi-test-redis-cluster-5.kimdubi-test-redis-cluster-headless '
        - name: REDISCLI_AUTH
          valueFrom:
            secretKeyRef:
              key: redis-password
              name: kimdubi-test-redis-cluster
        - name: REDIS_CLUSTER_CREATOR
          value: "yes"
        - name: REDIS_CLUSTER_REPLICAS
          value: "1"
        image: docker.io/bitnami/redis-cluster:6.0.10-debian-10-r5
        imagePullPolicy: IfNotPresent
        name: trigger
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 50Mi
        securityContext:
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: OnFailure
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
status:
  completionTime: "2021-02-21T07:34:55Z"
  conditions:
  - lastProbeTime: "2021-02-21T07:34:55Z"
    lastTransitionTime: "2021-02-21T07:34:55Z"
    status: "True"
    type: Complete
  startTime: "2021-02-21T07:33:56Z"
  succeeded: 1

.metatdata.annotations

• 여지껏 본 다른 오브젝트들과는 다르게 helm.sh/hook: post-install,post-upgrade 이라는 설정이 있음
• Hook은 생성한 chart를 생성하거나 update 수행 등 chart 의 cycle 중에 trigger 처럼 어떤 action을 추가할 수 있게 해주는 기능
• 위처럼 annotation 으로 helm hook 임을 명시하고 post-install,post-upgrade 처럼 어느 시점에 이 Job이 수행될 것인지를 정의 할 수 있음
• 이 Job은 helm chart 설치 후, helm chart upgrade 후 수행되는 Job임

.spec.completion && .spec.parallelism

• parallelism 은 해당 Job을 수행하는 Pod가 병렬로 생성되어 실행되는 개 수, parallelistm 3이면 Pod는 3개가 떠서 동작함
• completion 은 Job이 작업을 완료해야하는 Pod의 개 수
• parallelism 5 & completion 2 일 때 Pod를 동시에 5개를 띄울 수 있으나 Pod 2개만 작업 성공하면 되기 때문에 Pod 2개만 뜨게됨
• parallelistm 2 & completion 5 일 때 Pod는 동시에 2개를 띄울 수 있고 작업이 완료되려면 completion 5를 채워야 하기 때문에 Pod 2대, 2대, 1대 씩 떠서 작업을 처리하게됨

.spec.template.spec

• Pod 생성 template과 동일하지만 .spec.template.spec.containers.env 부분에 추가되는 내용이 있음

.spec.template.spec.containers.env

        - name: REDIS_CLUSTER_CREATOR
          value: "yes"
        - name: REDIS_CLUSTER_REPLICAS
          value: "1"

• 전체적으로 Pod 생성구문과 비슷하나 Job에서 생성하는 Pod은 위 환경 변수가 추가되었음
• 이 환경 변수는 container 생성 시 수행되는 스크립트에서 redis cluster를 구성하는 key로써 동작하게 됨

containers:
- args:
  - |
    # Backwards compatibility change
    if ! [[ -f /opt/bitnami/redis/etc/redis.conf ]]; then
        cp /opt/bitnami/redis/etc/redis-default.conf /opt/bitnami/redis/etc/redis.conf
    fi
    /opt/bitnami/scripts/redis-cluster/entrypoint.sh /opt/bitnami/scripts/redis-cluster/run.sh

• Pod 내 redis container 가 생성되고 /opt/bitnami/scripts/redis-cluster/run.sh 이 스크립트를 수행할 때 REDIS_CLUSTER_CREATOR : yes 인 값을 가지고 생성된 container와 아닌 container가 분기되어 다른 역할을 수행함
• 이 Job을 통해 생성된 Pod는 helm install 후에 hook 호출 되어 REDIS_CLUSTER_CREATOR 값을 가지고 앞서 생성된 Pod들을 cluster구성하는 역할을 수행하고 종료됨

job.batch/kimdubi-test-redis-cluster-cluster-update

$ kubectl get -o yaml job.batch/kimdubi-test-redis-cluster-cluster-update

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    helm.sh/hook: post-upgrade
  creationTimestamp: "2021-03-06T08:32:53Z"
  labels:
    app.kubernetes.io/instance: kimdubi-test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: redis-cluster
    helm.sh/chart: redis-cluster-4.4.0
  name: kimdubi-test-redis-cluster-cluster-update
  namespace: default
  resourceVersion: "4073462"
  selfLink: /apis/batch/v1/namespaces/default/jobs/kimdubi-test-redis-cluster-cluster-update
  uid: 1f2e3f6f-ba6e-4150-88e8-e5c692dbc801
spec:
  activeDeadlineSeconds: 600
  backoffLimit: 6
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: 1f2e3f6f-ba6e-4150-88e8-e5c692dbc801
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: kimdubi-test
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: redis-cluster
        controller-uid: 1f2e3f6f-ba6e-4150-88e8-e5c692dbc801
        helm.sh/chart: redis-cluster-4.4.0
        job-name: kimdubi-test-redis-cluster-cluster-update
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: kimdubi-test
                  app.kubernetes.io/name: redis-cluster
              namespaces:
              - default
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - args:
        - |
          . /opt/bitnami/scripts/libnet.sh
          . /opt/bitnami/scripts/libos.sh
          # Backwards compatibility change
          if ! [[ -f /opt/bitnami/redis/etc/redis.conf ]]; then
              cp /opt/bitnami/redis/etc/redis-default.conf /opt/bitnami/redis/etc/redis.conf
          fi
          previousNodeIp=""
          firstNodeIp=$(wait_for_dns_lookup kimdubi-test-redis-cluster-0.kimdubi-test-redis-cluster-headless 120 5)
          for node in $(seq $((1+6)) 7); do
            new_node_index="$(($node - 1))"
            new_node_ip=$(wait_for_dns_lookup kimdubi-test-redis-cluster-"$new_node_index".kimdubi-test-redis-cluster-headless 120 5)
            while [[ $(redis-cli -h "$new_node_ip" -p "$REDIS_PORT" ping) != 'PONG' ]]; do
              echo "Node $new_node_ip not ready, waiting for all the nodes to be ready..."
              sleep 5
            done
            slave=()
            if (( $REDIS_CLUSTER_REPLICAS >= 1 )) && (( new_node_index % (( $REDIS_CLUSTER_REPLICAS + 1 )) )); then
              slave+=("--cluster-slave")
              [ -z previousNodeIp ] && previousNodeIp=$firstNodeIp
              if [ -z "$REDISCLI_AUTH" ]; then
                masterID=$(redis-cli -c -h $previousNodeIp cluster myid)
              else
                masterID=$(redis-cli -c -h $previousNodeIp -a $REDISCLI_AUTH cluster myid)
              fi
              slave+=("--cluster-master-id $masterID")
              echo "Adding Node $new_node_index ${new_node_ip} as slave of ${masterID}"
            else
              previousNodeIp=$new_node_ip
              echo "Adding Node $new_node_index ${new_node_ip} as master"
            fi
            while ! redis-cli --cluster add-node "${new_node_ip}:${REDIS_PORT}" "${firstNodeIp}:${REDIS_PORT}" ${slave[@]}; do
              echo "Add-node ${new_node_index} ${new_node_ip} failed, retrying"
              sleep 5
              firstNodeIp=$(wait_for_dns_lookup kimdubi-test-redis-cluster-0.kimdubi-test-redis-cluster-headless 120 5)
            done
          done
        command:
        - /bin/bash
        - -c
        env:
        - name: REDIS_PORT
          value: "6379"
        - name: REDIS_CLUSTER_REPLICAS
          value: "1"
        - name: REDISCLI_AUTH
          valueFrom:
            secretKeyRef:
              key: redis-password
              name: kimdubi-test-redis-cluster
        image: docker.io/bitnami/redis-cluster:6.0.12-debian-10-r0
        imagePullPolicy: IfNotPresent
        name: trigger
        resources: {}
        securityContext:
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: OnFailure
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
status:
  completionTime: "2021-03-06T08:33:17Z"
  conditions:
  - lastProbeTime: "2021-03-06T08:33:17Z"
    lastTransitionTime: "2021-03-06T08:33:17Z"
    status: "True"
    type: Complete
  startTime: "2021-03-06T08:32:53Z"
  succeeded: 1

.metatdata.annotations

• helm.sh/hook: post-upgrade , upgrade 수행 시 생성되는 Job으로 아래 커맨드 수행 시 동작함

helm upgrade --timeout 600s kimdubi-test --set "password=${REDIS_PASSWORD},cluster.nodes=7,cluster.update.addNodes=true,cluster.update.currentNumberOfNodes=6,global.storageClass=kimdubi-test"  bitnami/redis-cluster

.spec.template.spec.container.args

• 새로운 Pod 내 redis container 생성하면서 redis cluster에 신규 node를 추가하기 위한 스크립트를 전달함
• 자세한 내용은 autoscale 편에서 다룸