I recently revived this blog, which is now running as a pod in my Kubernetes cluster. In order to deploy a new version, I had to (manualy) perform the following steps:

Build static website from the Jekyll source
Build container image using the static website
Push container image to my container registry
Update Kubernetes deployment manifest to use the new container image
Apply the new manifest to the running blog deployment

This quickly gets annoying and I started looking for a Continuous Integration (CI) system to automate this process. I am using Forgejo as my git server, which recently had its own CI system, Forgejo Actions, graduated to alpha status. Even though this is still alpha software, I decided to try it out for myself.

Setting up a Forgejo Actions runner

Forgejo Actions is forked from Gitea Actions which in turn is forked from act. Act is project to run GitHub Actions locally, which is the reason why Forgejo Actions are quite similar to Github Actions. In fact, there are quite a few references to GitHub still.

Forgejo Actions work roughly as follows:

You install a runner, which is a server that accepts workloads from Forgejo Actions.
You register this runner with Forgejo.
You define workflows on a git repository that execute steps.
These workflows are submitted to a runner.
Either a Docker container or an LXC container is spinned up to run the workflow to completion.
The result is communicated back to Forgejo.

All my workloads run on Kubernetes, and wanted the Forgejo runner to run on Kubernetes as well. Unfortunately, there is no Kubernetes-native way of running this runner, like for example GitLab does. However, there is an example deployment using a Docker-in-Docker setup. This does however require to run the image as a privileged container, and (spoiler) this setup seems to be very unreliable.

Using Forgejo Actions to build Nix derivations

Both the static website and container image for my blog are built using Nix (which I wrote about here). Therefore, it made sense to me to use the official Docker image. However, it seems Forgeo Actions has an undocumented dependency on /bin/sleep being present inside the container image. It seems sleep is being used to check whether the image is working correctly. Nix being very unconventional, does not have /bin/sleep by default in its Docker image. Therefore, I extended the offical Docker image using Nix below. The coreutils package contains the sleep binary which is linked inside the image using pathsToLink.

let
  nixFromDockerHub = pkgs.dockerTools.pullImage {
    imageName = "nixos/nix";
    imageDigest = "sha256:b3dc72ab3216606d52357ee46f0830a0cc32f3e50e00bd490efa1a8304e9f99d";
    sha256 = "sha256-FvDlbSnCmPtWTn4eG3hu8WVK1Wm3RSi2T+CdmIDLkG4=";
    finalImageTag = "2.22.0";
    finalImageName = "nix";
  };
  };
in
{
  packages.forgejo-nix-action = pkgs.dockerTools.buildImage {
    name = "forgejo-nix-action";
    tag = "latest";
    fromImage = nixFromDockerHub;

    copyToRoot = pkgs.buildEnv {
      name = "image-root";
      paths = with pkgs; [ coreutils ];
      pathsToLink = [ "/bin" ];
    };
  };
}

This is sufficient to be able to run nix build commands inside Forgejo Action, for example:

on: [ push ]
jobs:
  blog-pim:
    runs-on: docker
    container:
      image: git.kun.is/home/forgejo-nix-action:687d16c49ea7936068bac64ec68c480a9d681962
    steps:
      - name: Clone repository
        run: git clone ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git src
      - name: Build image
        run: nix build --out-link image ./src#packages.x86_64-linux.container-image

This workflows clones the source code of my blog, then builds a container image for it using Nix.

Pushing the image to a container registry

Being able to automatically build container images is useless if we don’t publish them somewhere. Skopeo is a real life-saver here, and I would strongly recommend this tool when interacting with container images. Before settling with Skopeo, I first tried using just Docker. However, for many operations, Docker needs a running Docker daemon and this is annoying to setup inside an already annoying Docker-in-Docker setup. Another option I explored was Podman but it was not even able to read the container images built by Nix.

These are the steps I ended up with to push a container image to a registry using Skopeo:

- name: Log into container registry
  run: /bin/skopeo login --tls-verify --username $ --password $ ${GITHUB_SERVER_URL}
- name: Push image to container registry
  run: |
    /bin/skopeo --insecure-policy copy docker-archive:image docker://${GITHUB_SERVER_URL#https://}/${GITHUB_REPOSITORY_OWNER}/blog-pim:latest

Failing to deploy to Kubernetes

Having pushed a new container image to a registry, we just have to updated the Kubernetes deployment to use the new image. This is unfortunately something I have not been able to get working. For some reason, anytime I run nix run inside a step, the k3s systemd service crashes. It seems during the execution of nix run, etcd is continuously timing out. After a few seconds of this, k3s deems it unhealthy and restarts itself.

I have a hunch running nix run puts so much strain on the disk that etcd is not able to quickly write to disk. It is known that etcd needs very quick disk times to function correctly. I have however not been able to reproduce this issue outside of the container. This leads me to believe that perhaps the Docker-in-Docker setup is the cause of this issue.

Conclusions

I think Forgejo Actions really needs a first-class Kubernetes treatment. It seems to me having Action runners on Kubernetes gives free improvements in terms of scalability and management. Going further, I will not be using Forgejo Actions for CI, but will explore some systems that run on Kubernetes natively. In particular, I’m interested in Argo Workflows, and perhaps I will give GitLab a try.

However in general, and this is not a critique on Forgejo Actions in particular, I don’t fully agree with the way these “Actions” systems work. It seems to leading way to use these actions is to use a runs-on directive and specify some container image, perhaps the latest Ubuntu version (who thought it was a good idea to turn OSes into images?). Then, people install some tools using the image’s package manager after which their actual jobs runs. Unless they are very well maintained, these actions will probably stop working after a while when the underlying container image is discontinued from support. Or perhaps a package is updated, which is incompatable with the action. Or there is some hidden dependency that changes in the underlying container image. Or …

I believe Nix could be an amazing tool to improve these “Actions” systems. Using Nix, you wouldn’t need a bloated container image with lots of tools you won’t need. You could take advantage of Nixpkgs and install only the tools you need to run your job. Everything inside the image would be version-pinned and reproducible and will continue to work in 50 years. I’m not entirely sure how such a system would even look, but my gut tells me it should be possible. I’m unfortunately not aware whether such a system currently exists though.