Pushデバッグってなんぞ？

〜1年前〜　CircleCIド素人ぼく

「CircleCI configに記述したコードがうまく動かないな……でもGithubにPushして、CircleCIが動いてるところ見ないとわからん……」

などど言い出し、何度もPushしまくってcommitを荒らしたのは私です。

そんな一か八か、動くかわからないコードをPushしまくるのはやめていきたいものです。

CircleCI CLIをインストール
- Linux, macOS
Homebrew
ローカルで確認する
- configtest
- Job実行
  - ちょっとした注意点
  - You attempted to run a local build with version '2.1' of configuration.
SSHする
参考

CircleCI CLIをインストール

インストール方法はそこまで大変でないので、公式に書いてあることをそのまま書きます。

Linux, macOS

curl -fLSs https://circle.ci/cli | bash

Homebrew

brew install circleci

ローカルで確認する

configtest

CircleCIはローカルで検証することができます。

 % circleci config validate .circleci/config.yml
Config file at config.yml is valid.

まずはこのコマンドでTypoや構文エラーがないか確認できます。

Job実行

ローカルでJobをテストすることができます。

とりあえず挙動を試してみたい、という場合はデモデータがあるのでそれを引っ張ってきましょう。

git clone https://github.com/CircleCI-Public/circleci-demo-go.git
cd circleci-demo-go
circleci local execute --job build

デモの実行結果

 % circleci local execute --job build
Docker image digest: sha256:e889c8ffff4fb4054229a4f9e32d04871be150472ab7940b02cf165df2d14384
====>> Spin up Environment
Build-agent version 1.0.11912-4fd916c5 (2019-06-20T18:22:57+0000)
Docker Engine Version: 18.09.2
Kernel Version: Linux 5308afaba763 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 Linux
Starting container circleci/golang:1.12
  using image circleci/golang@sha256:15e1d40d63836cafb864b6e6e3dc74a3426e5bd208d020f5fbaa4878aee33d21
Starting container circleci/postgres:9.6-alpine
  using image circleci/postgres@sha256:8d3433352c13f9a29c50b01dba06d606e2992621c6c691d790fd3d6bb5c55fea
====>> Container circleci/postgres:9.6-alpine
Service containers logs streaming is disabled for local builds
You can manually monitor container 2e4ce7223c1da344d5720f0d10430806105cd624886209db6df6ea178858a4ee
====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /home/circleci/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /home/circleci/project && cp -a /tmp/_circleci_local_build_repo/.git /home/circleci/project
====>> mkdir -p $TEST_RESULTS
  #!/bin/bash -eo pipefail
mkdir -p $TEST_RESULTS
====>> Restoring Cache
Error:
Skipping cache - error checking storage: not supported

Step failed
====>> Waiting for Postgres to be ready
  #!/bin/bash -eo pipefail
dockerize -wait tcp://localhost:5432 -timeout 1m
2019/06/30 09:41:13 Waiting for: tcp://localhost:5432
2019/06/30 09:41:13 Problem with dial: dial tcp 127.0.0.1:5432: connect: connection refused. Sleeping 1s
2019/06/30 09:41:14 Problem with dial: dial tcp 127.0.0.1:5432: connect: connection refused. Sleeping 1s
2019/06/30 09:41:15 Problem with dial: dial tcp 127.0.0.1:5432: connect: connection refused. Sleeping 1s
2019/06/30 09:41:16 Problem with dial: dial tcp 127.0.0.1:5432: connect: connection refused. Sleeping 1s
2019/06/30 09:41:17 Problem with dial: dial tcp 127.0.0.1:5432: connect: connection refused. Sleeping 1s
2019/06/30 09:41:18 Connected to tcp://localhost:5432
====>> Run unit tests
  #!/bin/bash -eo pipefail
PACKAGE_NAMES=$(go list ./... | circleci tests split --split-by=timings --timings-type=classname)
gotestsum --junitfile ${TEST_RESULTS}/gotestsum-report.xml -- $PACKAGE_NAMES

go: finding github.com/stretchr/testify v0.0.0-20170130113145-4d4bfba8f1d1
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/mattes/migrate v0.0.0-20170302213955-305a63a4472b
go: finding github.com/julienschmidt/httprouter v0.0.0-20170104185816-8a45e95fc75c
go: finding github.com/mattn/go-sqlite3 v1.10.0
go: finding github.com/cznic/ql v1.2.0
go: finding github.com/lib/pq v0.0.0-20170309164912-e4af84aab01e
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/edsrzf/mmap-go v0.0.0-20170320065105-0bce6a688712
go: finding github.com/cznic/internal v0.0.0-20180608152220-f44710a21d00
go: finding github.com/cznic/strutil v0.0.0-20171016134553-529a34b1c186
go: finding github.com/cznic/lldb v1.1.0
go: finding github.com/cznic/sortutil v0.0.0-20150617083342-4c7342852e65
go: finding github.com/cznic/mathutil v0.0.0-20180504122225-ca4c9f2c1369
go: finding github.com/cznic/golex v0.0.0-20170803123110-4ab7c5e190e4
go: finding github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
go: finding github.com/cznic/zappy v0.0.0-20160723133515-2533cb5b45cc
go: finding github.com/cznic/fileutil v0.0.0-20180108211300-6a051e75936f
go: finding github.com/cznic/b v0.0.0-20180115125044-35e9bbe41f07
go: downloading github.com/stretchr/testify v0.0.0-20170130113145-4d4bfba8f1d1
go: downloading github.com/julienschmidt/httprouter v0.0.0-20170104185816-8a45e95fc75c
go: downloading github.com/mattes/migrate v0.0.0-20170302213955-305a63a4472b
go: extracting github.com/julienschmidt/httprouter v0.0.0-20170104185816-8a45e95fc75c
go: extracting github.com/stretchr/testify v0.0.0-20170130113145-4d4bfba8f1d1
go: extracting github.com/mattes/migrate v0.0.0-20170302213955-305a63a4472b
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/lib/pq v0.0.0-20170309164912-e4af84aab01e
go: extracting github.com/pmezard/go-difflib v1.0.0
go: extracting github.com/davecgh/go-spew v1.1.1
go: extracting github.com/lib/pq v0.0.0-20170309164912-e4af84aab01e
Requested historical based timing, but they are not present.  Falling back to name based sorting
∅  . (3ms)
✓  formatter (16ms)
✓  math (5.019s)
✓  service (111ms)
✓  validator (11ms)
✓  bigdata (15.02s)
∅  test

DONE 9 tests in 30.732s
====>> make
  #!/bin/bash -eo pipefail
make
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o workdir/contacts .
====>> Saving Cache
Error:
Skipping cache - error checking storage: not supported

Step failed
====>> Start service
  #!/bin/bash -eo pipefail
./workdir/contacts
====>> Validate service is working
  #!/bin/bash -eo pipefail
sleep 5
curl --retry 10 --retry-delay 1 -X POST --header "Content-Type: application/json" -d '{"email":"test@example.com","name":"Test User"}' http://localhost:8080/contacts

{"contact":{"id":1,"email":"test@example.com","name":"Test User"}}
====>> Uploading artifacts
Uploading /tmp/test-results to raw-test-output
Uploading /tmp/test-results/gotestsum-report.xml (2.8 kB): Error: FAILED with error not supported

====>> Uploading test results
Archiving the following test results
  * /tmp/test-results

Error: Failed uploading test results directory
Error &errors.errorString{s:"not supported"}

{"Runner":true,"level":"error","msg":"Can't add file //tmp/test-results to tar: io: read/write on closed pipe","task-id":"localbuild-1561887664","time":"2019-06-30T09:42:21Z"}
{"Runner":true,"level":"error","msg":"Can't close tar writer: io: read/write on closed pipe","task-id":"localbuild-1561887664","time":"2019-06-30T09:42:21Z"}
Step canceled
Success!

ローカル実行は、WorkflowやExecutor、restore_cacheなどには対応してないです。

実行結果を見ると、save_cacheやrestore_cacheがSkipされています。

configをみれば何が実行されて何が実行できないかわかるので暇だったら読んでみましょう。

 % cat .circleci/config.yml

version: 2 # use CircleCI 2.0
jobs: # basic units of work in a run
  build: # runs not using Workflows must have a `build` job as entry point
    docker: # run the steps with Docker
      # CircleCI Go images available at: https://hub.docker.com/r/circleci/golang/
      - image: circleci/golang:1.12
      # CircleCI PostgreSQL images available at: https://hub.docker.com/r/circleci/postgres/
      - image: circleci/postgres:9.6-alpine
        environment: # environment variables for primary container
          POSTGRES_USER: circleci-demo-go
          POSTGRES_DB: circle_test

    parallelism: 2

    environment: # environment variables for the build itself
      TEST_RESULTS: /tmp/test-results # path to where test results will be saved

    steps: # steps that comprise the `build` job
      - checkout # check out source code to working directory
      - run: mkdir -p $TEST_RESULTS # create the test results directory

      - restore_cache: # restores saved cache if no changes are detected since last run
          keys:
            - go-mod-v4-{{ checksum "go.sum" }}

      #  Wait for Postgres to be ready before proceeding
      - run:
          name: Waiting for Postgres to be ready
          command: dockerize -wait tcp://localhost:5432 -timeout 1m

      - run:
          name: Run unit tests
          environment: # environment variables for the database url and path to migration files
            CONTACTS_DB_URL: "postgres://circleci-demo-go@localhost:5432/circle_test?sslmode=disable"
            CONTACTS_DB_MIGRATIONS: /home/circleci/project/db/migrations

          # store the results of our tests in the $TEST_RESULTS directory
          command: |
            PACKAGE_NAMES=$(go list ./... | circleci tests split --split-by=timings --timings-type=classname)
            gotestsum --junitfile ${TEST_RESULTS}/gotestsum-report.xml -- $PACKAGE_NAMES

      - run: make # pull and build dependencies for the project

      - save_cache:
          key: go-mod-v4-{{ checksum "go.sum" }}
          paths:
            - "/go/pkg/mod"

      - run:
          name: Start service
          environment:
            CONTACTS_DB_URL: "postgres://circleci-demo-go@localhost:5432/circle_test?sslmode=disable"
            CONTACTS_DB_MIGRATIONS: /home/circleci/project/db/migrations
          command: ./workdir/contacts
          background: true # keep service running and proceed to next step

      - run:
          name: Validate service is working
          command: |
            sleep 5
            curl --retry 10 --retry-delay 1 -X POST --header "Content-Type: application/json" -d '{"email":"test@example.com","name":"Test User"}' http://localhost:8080/contacts

      - store_artifacts: # upload test summary for display in Artifacts
          path: /tmp/test-results
          destination: raw-test-output

      - store_test_results: # upload test results for display in Test Summary
          path: /tmp/test-results
workflows:
  version: 2
  build-workflow:
    jobs:
      - build

ちょっとした注意点

ただしAWS Credentialなどがローカルに存在しない場合は実行できないので注意です。

% circleci local execute -c .circleci/process.yml  --job build
Docker image digest: sha256:xxxxxxxxxxxxxxxxx
====>> Spin up Environment
Build-agent version 1.0.11151-deadd97a (2019-05-09T22:12:43+0000)
Docker Engine Version: 18.09.2
Kernel Version: Linux d95f9bb8f051 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 Linux
Error:
error authentication with ECR: AWS credentials not found

Step failed
Task failed
Error: error authentication with ECR: AWS credentials not found

You attempted to run a local build with version '2.1' of configuration.

configのバージョンが2.1だとこのJob実行はできないようです。

以下のコマンドを実行して実行できるバージョンを一時的に作成すればよいでしょう。

circleci config process .circleci/config.yml > .circleci/process.yml

なおコマンドの詳細はhelpで見られます。

 % circleci config --help                                                                                                      
Operate on build config files


Usage:
  circleci config [command]

Available Commands:
  pack        Pack up your CircleCI configuration into a single file.
  process     Process the config.
  validate    Check that the config file is well formed.

Flags:
  -h, --help   help for config

Global Flags:
      --host string    URL to your CircleCI host, also CIRCLECI_CLI_HOST (default "https://circleci.com")
      --token string   your token for using CircleCI, also CIRCLECI_CLI_TOKEN
Use "circleci config [command] --help" for more information about a command.

SSHする

ここだけローカルではなく、コードをPushしたあとにデバッグする方法です。実際にCircleCIで動いているコンテナやVMにSSHして挙動を確認する方法です。

私は主にCircleCI上で実行させるシェルスクリプトをテストしたりします。変数が設定されるかみたいなことは実際のコンテナ上で確認するのが速いですよね。

f:id:vtryo:20190630212247p:plain

実行すると、そのWorkflowのJobがすべて実行されたあと、成功失敗に関わらずSSH可能になります。

f:id:vtryo:20190630212606p:plain

コピペしてSSHします。

 % ssh -p xxxxxx xx.xxx.xxx.xxx
The authenticity of host '[xx.xxx.xxx.xxx]:xxxxx (xx.xxx.xxx.xxx]:xxxxx)' can't be established.
RSA key fingerprint is SHA256:xxxxxxxxxxxxxxxxxxxxxx.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'xx.xxx.xxx.xxx]:xxxxx' (RSA) to the list of known hosts.
circleci@5bc02f45a8dd:~$

CircleCIに設定してある環境変数がセットされているかなども確認できますね。（CircleCIの環境変数については別でブログを書きます）

circleci@5bc02f45a8dd:~$ env

/home/circleci配下にあるprojectディレクトリの中に、Git checkoutしたソースが配置されるようです。

circleci@5bc02f45a8dd:~$ ls project/
xxxx-xxxx

参考

circleci.com

qiita.com

VTRyo Blog

一歩ずつ前に進むブログ

まさかPushデバッグしてないよね？よく使うCircleCIのデバッグ方法