How to invest two hours in building an image

Why it's essential to understand the tools we use.

This post is part of a series that documents my journey of rewriting my Callmebot integration service into an extensible notification service.

Building the Container Image

When building my container image, I followed Kelsey Hightower’s guide and got to the part where the image was built. The Dockerfile in the guide includes many things that are not needed in 2025, so I used an image that made sense to me: docker.io/golang:1.23.4.
The line that compiles the code is the most interesting one:

RUN CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags ‘-w’ .

My general practice is not to run code I don't understand, so I included only what I thought I needed to compile the app:

RUN go build .

The build process was completed without issues, and when running the image, it started correctly.

Light Optimizations Amirite

The baseline

I started the nerd sniping with a baseline Dockerfile

FROM docker.io/golang:1.23.4 AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o main .

The output:

REPOSITORY                TAG       SIZE
localhost/noti-service    base     1.26 GB

That's a bit too big…

Using the Alpine image

Alpine Linux images are often smaller than their Debian counterparts, so why not try that?

A simple line change from:

FROM docker.io/golang:1.23.4

To:

FROM docker.io/golang:1.23.4-alpine

And I got a significant reduction

REPOSITORY                TAG       SIZE
localhost/noti-service    base     1.26 GB
localhost/noti-service    alpine   641 MB

How about…

Multi-Stage Builds

What if we split the build image from the runtime image? Since I'm compiling everything related to Go, I don’t need the packages I downloaded.

FROM docker.io/golang:1.23.4-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o main .

FROM docker.io/golang:1.23.4-alpine
COPY --from=builder /app/main /main
CMD ["/main"]

Not too shabby!

REPOSITORY                TAG           SIZE
localhost/noti-service    base         1.26 GB
localhost/noti-service    alpine       641 MB
localhost/noti-service    multi-stage  267 MB

267 megabytes is manageable; perhaps squashing all layers will give me better gains

REPOSITORY                TAG              SIZE
localhost/noti-service    base            1.26 GB
localhost/noti-service    alpine          641 MB
localhost/noti-service    multi-stage     267 MB
localhost/noti-service    multi-stage-sq  267 MB

Nope… and I lose the benefit of caching layers.

If Kelsey says it can be done

I should have been happy at this point, but I got greedy. If Kelsey said it could be done, I should be able to do it. We’re going for the scratch image.

Let's do a simple change.

FROM docker.io/golang:1.23.4-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go build -o main .

#FROM docker.io/golang:1.23.4-alpine
FROM scratch
COPY --from=builder /app/main /main
CMD ["/main"]

Wow, that's cool!

REPOSITORY                TAG              SIZE
localhost/noti-service    base            1.26 GB
localhost/noti-service    alpine          641 MB
localhost/noti-service    multi-stage     267 MB
localhost/noti-service    multi-stage-sq  267 MB
localhost/noti-service    scratch         11.8 MB

Does it run? Yes, but with some errors. That’s when Kelsey's guide helped a lot, and I expanded upon it in my article about handling TLS in this series.

Back to the build command?

The command that Kelsey’s guide uses to compile the application includes some interesting additions that I removed

RUN CGO_ENABLED=0 GOOS=linux go build -a -tags netgo -ldflags ‘-w’ .

I’d like to focus our attention on the CGO_ENABLED=0 line since it took me a couple of tries to understand how it works.

CGO_ENABLED is an environment variable that’s used by the Golang compiler to determine what should be done with C code that is referenced by any Go code.

This is not something all applications will do explicitly, but it might be done by libraries imported and used by our code. It certainly broke my notification service image when building it with CGO_ENABLED=1.

When building with CGO_ENABLED=1, the compiler will reference the operating system’s libraries whenever needed. The result is usually a smaller binary that relies on the underlying operating system to fulfill its purpose.

According to the official docs:

The cgo tool is enabled by default for native builds on systems where it is expected to work

Depending on your source image, it may be enabled. I tested golang:1.23.4-bookworm and golang:1.23.4-alpine, and only bookworm works as expected, while alpine throws an error related to the C compiler not being found.

It’s essential to remember that compiling with CGO_ENABLED=1 requires the runtime image to include the necessary libraries that the compiler assumes will be present during runtime.

If you build this Dockerfile:

FROM golang:1.23.4-bookworm AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ENV CGO_ENABLED=1
RUN go build -o main .

FROM scratch
COPY --from=builder /app/main /main
CMD ["/main"]

It will be successful, but upon running the image, you’ll get the following error:

{"msg":"exec container process (missing dynamic library?) `/main`: No such file or directory","level":"error","time":"2025–01–16T18:30:48.384359Z"}

The default CGO_ENABLED=0 in the Alpine image should be good enough in most scenarios, and enabling the feature should only be considered when necessary. Be mindful of older guides that include explicit declarations of the variable; in most cases, it's unnecessary and might cause confusion.

Sources

Here’s a compilation of links I used to understand what was going on:

• https://stackoverflow.com/questions/61515186/when-using-cgo-enabled-is-must-and-what-happens
• https://www.reddit.com/r/golang/comments/pi97sp/what_is_the_consequence_of_using_cgo_enabled0/

Other thoughts on understanding our tools

This experience with Go reminded me of a project I worked on some time ago. We shipped around 50 megabytes of Javascript code to load the login portion of a web application.
We added a CDN in front of our web servers to load things faster, but since we deployed daily, those caching layers were frequently invalidated.

The underlying problem we never invested the time in was understanding what part of that Javascript code was needed to load a login page and utilize Webpack to split the bundle into chunks, and load only was necessary.

As developers, we focus a lot on delivery: getting features out the door as fast as possible.
Companies want their products shipped yesterday, project managers commit to dates without understanding the underlying complexities of what's being done, and developers are terrible at estimating the level of effort of a task.

Who should pay for the time invested in understanding the tools we work with? Employers don’t want to, Project Managers will bury those tasks in the backlog, and Employees don't want to work for free.
Everyone wants the benefits, and no one wants to pay the cost.

I don’t see a correct answer here; it's a push-pull dynamic we will always see in our industry. The one thing I know (and can act on) is who is responsible for being employable and estimating how long a task will take.