ko  ^beta 

ko enables fast, standardized, reproducible, configuration-less, Docker-less, and multi-platform container image builds for Go apps.

Skaffold embeds ko as a library, so you do not need to download ko to use the ko builder.

Benefits of the ko builder

Compared to …

• the Cloud Native buildpacks builder, the ko builder is fast, doesn’t require Docker, and uses a default base image that has a small attack surface.

• the Docker builder, the ko builder standardizes builds, avoiding artisanal snowflake Dockerfiles. It also doesn’t require the Docker daemon, so builds can run in environments where Docker isn’t available for security reasons.

• the Kaniko builder, the ko builder doesn’t need a Kubernetes cluster, and it avoids the previously-mentioned artisanal Dockerfiles.

• the Bazel builder, the ko builder doesn’t require users to adopt Bazel. However, we recommend the Bazel builder for users who already use Bazel for their Go apps.

• the custom builder, the ko builder standardizes builds, as it doesn’t require running ko using custom shell scripts.

Configuring the ko builder

The ko builder default configuration is sufficient for many Go apps. To use the ko builder with its default configuration, provide an empty map in the ko field, e.g.:

build:
  artifacts:
  - image: my-simple-go-app
    ko: {}

Base image

ko uses the cgr.dev/chainguard/static image as the default base image. This is a small image that provides a minimal environment for Go binaries. The default base image does not provide a shell, and it does not include glibc.

You can specify a different base image using the ko builder fromImage config field. For instance, if you want to use a base image that contains glibc, you can use this configuration:

    ko:
      fromImage: cgr.dev/chainguard/glibc-dynamic

If you run Skaffold in a Google Cloud VPC network with limited or no internet access, you can use a distroless base image from Google Container Registry, accessible via Private Google Access:

    ko:
      fromImage: gcr.io/distroless/static-debian11:nonroot

Multi-platform images

The ko builder supports building multi-platform images. The default platform is linux/amd64, but you can configure a list of platforms using the artifact’s platforms configuration field, e.g.:

build:
  platforms: ["linux/amd64", "linux/arm64"]
  artifacts:
  - image: foo
    ko: {}

You can also supply ["all"] as the value of platforms. all means that the ko builder builds images for all platforms supported by the base image.

Image labels

Use the labels configuration field to add image labels (a.k.a. Dockerfile LABELs),

For example, you can add labels based on the pre-defined annotations keys from the Open Container Initiative (OCI) Image Format Specification:

    ko:
      labels:
        org.opencontainers.image.licenses: Apache-2.0
        org.opencontainers.image.source: https://github.com/GoogleContainerTools/skaffold

The labels section supports templating of values based on environment variables, e.g.:

    ko:
      labels:
        org.opencontainers.image.revision: "{{.GITHUB_SHA}}"
        org.opencontainers.image.source: "{{.GITHUB_SERVER_URL}}/{{.GITHUB_REPOSITORY}}"

Build time environment variables

Use the env configuration field to specify build-time environment variables.

Example:

    ko:
      env:
      - GOCACHE=/workspace/.gocache
      - GOPRIVATE=git.internal.example.com,source.developers.google.com

The env field supports templating of values using environment variables, for example:

    ko:
      env:
      - GOPROXY={{.GOPROXY}}

Dependencies

The dependencies section configures what files Skaffold should watch for changes when in dev mode.

paths and ignore are arrays that list file patterns to include and ignore. Any patterns in ignore will be ignored by the Skaffold file watcher, even if they are also specified in paths. ignore is only used when paths is not empty.

Example:

    ko:
      dependencies:
        paths:
        - cmd
        - go.mod
        - pkg
        ignore:
        - vendor

If no dependencies are specified, the default values are as follows:

    ko:
      dependencies:
        paths: ["**/*.go"]
        ignore: []

Build flags

Use the flags configuration field to provide flag arguments to go build, e.g.:

    ko:
      flags:
      - -mod=vendor
      - -v

Use the ldflags configuration field to provide linker flag arguments, e.g.:

    ko:
      ldflags:
      - -s
      - -w

The flags and ldflags fields support templating using environment variables, e.g.:

    ko:
      ldflags:
      - -X main.version={{.VERSION}}

These templates are evaluated by Skaffold. Note that the syntax is slightly different to ko’s template expansion, specifically, there’s no .Env prefix.

Source file locations

If your Go source files and go.mod are not in the context directory, use the dir configuration field to specify the path, relative to the context directory, e.g.:

    ko:
      dir: ./compat-go114

If your package main is not in the context directory (or in dir if specified), use the main configuration field to specify the path or target, e.g.:

    ko:
      main: ./cmd/foo

If your context directory only contains one package main directory, you can use the ... wildcard in the main field value, e.g., ./....

Both dir and main default to ..

Existing ko users

Useful tips for existing ko users:

• Specify your destination image registry using Skaffold’s default-repo functionality. The ko builder does not read the KO_DOCKER_REPO environment variable.

• Image naming follows the Skaffold image naming strategy. Skaffold removes the ko:// prefix, if present, before determining the image name.

• If your image references use the ko:// prefix and you are pushing the images to a registry, you must set the default repo.

• The ko builder supports reading base image configuration from the .ko.yaml file. If you already configure your base images using this file, you do not need to specify the fromImage field for the artifact in skaffold.yaml.

• The ko builder supports reading build configs from the .ko.yaml file if skaffold.yaml does not specify any of the build config fields (dir, main, env, flags, and ldflags). If you already specify these fields in .ko.yaml, you do not need to repeat them in skaffold.yaml.

• You can generate skaffold.yaml files by examining an existing code base, using the command skaffold init --XXenableKoInit=true. Select the Ko builder for your images when prompted.

ko commands and workflows in Skaffold

Here are some examples of Skaffold equivalents of ko commands and worflows.

Using vendored dependencies

If vendor your dependencies and your go.mod specifies a Go version < 1.14, you can pass -mod=vendor to ko using the GOFLAGS environment variable:

GOFLAGS="-mod=vendor" ko publish .

To achieve the same using Skaffold’s ko builder, use the flags field in skaffold.yaml:

    ko:
      flags:
      - -mod=vendor

skaffold build

Capturing image name from stdout

If you want Skaffold to print out the full image name and digest (and nothing else) to stdout, similar to what ko build does, use the --quiet and --output flags. These flags enable you to capture the full image references in an environment variable or redirect to a file, e.g.:

skaffold build --quiet --output='{{range .Builds}}{{.Tag}}{{end}}' > out.txt

Note that Skaffold produces a JSON file with the image names if you run skaffold build with the --file-output flag. You can then use this flag as input to skaffold render to render Kubernetes manifests. For details on how to do this, see the next section.

Rendering Kubernetes manifests

When you use the Skaffold ko builder, Skaffold takes care of replacing the image placeholder name in your Kubernetes manifest files using its render functionality.

The ko builder supports image name placeholders that consist of the ko:// prefix, followed by the Go import path to the main package. This means that Skaffold works with existing Kubernetes manifest files that use this image name placeholder format. Note that Skaffold only replaces image references in fields that have the name image.

If you previously built images and rendered Kubernetes manifests using ko, e.g.:

ko resolve --filename k8s/*.yaml > out.yaml

You can instead use Skaffold’s render subcommand with the --digest-source local flag to build and render manifests:

skaffold render --digest-source local --offline --output out.yaml

Or you can perform the action as two steps: first build the images, then render the manifests using the output file from the build step:

skaffold build --file-output artifacts.json --push
skaffold render --build-artifacts artifacts.json --digest-source none --offline --output out.yaml

Specify the location of your Kubernetes manifests in skaffold.yaml:

manifests:
  rawYaml:
  - k8s/*.yaml

To build images in parallel, consider setting the SKAFFOLD_BUILD_CONCURRENCY environment variable value to 0:

SKAFFOLD_BUILD_CONCURRENCY=0 skaffold [...]

You can also set the concurrency value in your skaffold.yaml:

build:
  local:
    concurrency: 0

Advanced usage

Debugging

Cloud Code and skaffold debug can debug images built using ko.

Images built using ko are automatically identified as Go apps by the presence of the KO_DATA_PATH environment variable.

Skaffold configures ko to build with compiler optimizations and inlining disabled (-gcflags='all=-N -l') when you run skaffold debug or use Cloud Code to debug a Kubernetes application.

If you debug using VS Code and need to configure a “remote path” or “path on remote container”, then this value should match your local path, typically ${workspaceFolder}. The reason is that “remote path” in this case means the path to your Go source code where it was compiled. The ko builder currently only supports local builds, so the remote path will be same as the local path.

To learn more about how Skaffold debugs Go applications, read the Go section in the Debugging guide.

File sync

The ko builder can sync files to a running container when you run skaffold dev.

The sync feature for the ko builder only works for static assets bundled with the container image.

Use infer mode to specify patterns for the files you want to sync. The infer patterns are relative to the context directory.

For instance, if your main package is in the context directory, you can use this configuration to sync all the static files bundled with the container image:

    sync:
      infer:
      - kodata/**/*

Note that the file sync feature requires the tar command to be available in the container. The default ko builder base image does not include the tar command. Use the fromImage field in the ko builder configuration in your skaffold.yaml file to specify a base image that contains the tar command, such as gcr.io/distroless/base-debian11:debug.

You can use profiles with activation by command to override the fromImage value only when running skaffold dev, such as in this example:

profiles:
- name: sync
  activation:
  - command: dev
  patches:
  - op: add
    path: /build/artifacts/0/ko/fromImage
    value: gcr.io/distroless/base-debian11:debug

Remote builds

The ko builder supports remote builds on Google Cloud Build. See the example.

Using the custom builder

If the ko builder doesn’t support your use of ko, you can instead use the custom builder.

See the custom builder example.

build:
  artifacts:
  - image: ko://github.com/googlecontainertools/skaffold/examples/custom
    custom:
      buildCommand: ./build.sh
      dependencies:
        paths:
        - "**/*.go"
        - go.*
        - .ko.yaml

If you need to use ko via the custom builder rather than the ko builder, please consider filing an issue that describes your use case.

Collecting coverage profiles from integration tests

Go 1.20 introduced support for collecting coverage profile data from running Go application when running integration or end-to-end tests. To see how you can use the ko builder to configure this, see the tutorial Go integration test coverage profiles.

SBOM synthesis and upload

The ko CLI by default generates a software bill of materials (SBOM) and uploads it to the image registry. The Skaffold ko builder does not generate or upload SBOMs. If you need this feature, please raise an issue.