<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-US"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://pim.kun.is/feed.xml" rel="self" type="application/atom+xml" /><link href="https://pim.kun.is/" rel="alternate" type="text/html" hreflang="en-US" /><updated>2025-03-02T14:38:00+0000</updated><id>https://pim.kun.is/feed.xml</id><title type="html">Pim Kunis</title><subtitle>Automation enjoyer</subtitle><author><name>Pim Kunis</name></author><entry><title type="html">Automating a Parking Expenses Report</title><link href="https://pim.kun.is/parking-expenses-automation/" rel="alternate" type="text/html" title="Automating a Parking Expenses Report" /><published>2025-03-02T14:38:00+00:00</published><updated>2025-03-02T14:38:00+00:00</updated><id>https://pim.kun.is/parking-expenses-automation</id><content type="html" xml:base="https://pim.kun.is/parking-expenses-automation/"><![CDATA[<p>For the past year, I was consulting at a place where I could request
reimbursement of parking expenses. This had to be claimed using a standard Excel
template.</p>

<p>Doing this manually every month seemed boring, so I automated a significant part
of that:</p>

<ul>
  <li>Fetch parking costs for a certain month from my Paperless-ngx instance</li>
  <li>Calculate costs after taxes and a grand total</li>
  <li>Fill in an Excel sheet with the details</li>
  <li>Convert the Excel sheet to PDF and append photos of parking costs</li>
</ul>

<p>Note: I don’t expect anybody to have the same requirements as me here, but
hopefully pieces can be useful to some. The open-sourced project can be found
<a href="https://git.kun.is/pim/parking-expenses">here</a>.</p>

<h1 id="organizing-the-parking-costs">Organizing the Parking Costs</h1>

<p>Each time I would park, I would accumulate a parking ticket. This ticket
contains the following information:</p>

<ul>
  <li>Date and time</li>
  <li>Car license plate</li>
  <li>Price</li>
  <li>VAT</li>
</ul>

<p>In order to automate the report, I had to have this information in a structured
way. Therefore, I turned to <a href="https://docs.paperless-ngx.com/">Paperless-ngx</a>.
Paperless-ngx is an self-hostable open-source service that helps digitalizing
paper documents. Third-party mobile apps help quickly scanning documents and
uploading them to the server. This is exactly what I did with the parking
tickets as well: each time I received a ticket, I would scan it with the
Paperless app.</p>

<p>Paperless-ngx uses Optical Character Recognition (OCR) to cleverly extract the
date of the ticket automatically. I also created a custom label to indicate a
parking ticket, and I added custom fields to indicate a ticket’s cost and VAT
amount. You can see this information in the screenshot below.</p>

<p><img src="ticket.png" alt="foo" /> <em>Translated from Dutch: aanmaakdatum means creation date,
parkeerkaart means parking ticket, BTW means VAT and bedrag means cost</em></p>

<p>Paperless-ngx will also automatically learn to label the tickets with the
“parking ticket” label, based on the document’s content. That just leaves the
VAT and price data points. Unfortunately I found the OCR to be too unreliable to
extract that text from the documents. Therefore, I had to manually set those two
data points for each ticket.</p>

<p>Neat, we now have all parking tickets organized on Paperless-ngx! In order to
create a report, we can just query the Paperless-ngx API!</p>

<h1 id="using-the-paperless-ngx-api">Using the Paperless-ngx API</h1>

<p>The remaining part of this post will be code explanations. You can find the full
open-sourced Python code
<a href="https://git.kun.is/pim/parking-expenses/src/commit/645fd8b4e4e46bc8f3d8ba831ca28261cd3edb39/main.py">here</a>.</p>

<p>Paperless-ngx has <a href="https://docs.paperless-ngx.com/api/">an API</a> we can now use
to query our parking tickets. I found the filtering logic pretty hard to
understand. Therefore I simply looked at what API calls the web UI makes with
the query I needed. This turned out to be something like this:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span>
    <span class="sa">f</span><span class="sh">"</span><span class="si">{</span><span class="n">paperless_ngx_url</span><span class="si">}</span><span class="s">/api/documents/?page_size=50&amp;query=created:[</span><span class="si">{</span><span class="n">start_date</span><span class="si">}</span><span class="s"> TO </span><span class="si">{</span><span class="n">end_date</span><span class="si">}</span><span class="s">]&amp;tags__id__all=</span><span class="si">{</span><span class="n">FILTER_TAG_ID</span><span class="si">}</span><span class="s">&amp;correspondent__id__in=</span><span class="si">{</span><span class="n">FILTER_CORRESPONDENT_ID</span><span class="si">}</span><span class="sh">"</span><span class="p">,</span>
    <span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="sh">"</span><span class="s">Authorization</span><span class="sh">"</span><span class="p">:</span> <span class="sa">f</span><span class="sh">"</span><span class="s">Token </span><span class="si">{</span><span class="n">token</span><span class="si">}</span><span class="sh">"</span><span class="p">},</span>
<span class="p">)</span>
</code></pre></div></div>

<p>In the above code snippet, <code class="language-plaintext highlighter-rouge">FILTER_TAG_ID</code> is the ID of the “Parking ticket”
label and <code class="language-plaintext highlighter-rouge">FILTER_CORRESPONDENT_ID</code> is the ID of the correspondent (I included
this to filter out potential unrelated parking tickets).</p>

<p>Perfect, this returns a JSON list of parking tickets we can use to create the
report!</p>

<h1 id="creating-the-report">Creating the Report</h1>

<p>All that’s left is to fill in an Excel template now. This actually proved to be
the most annoying part but also the most boring part. I used the
<a href="https://openpyxl.readthedocs.io/en/stable/">OpenPyXL</a> Python library to
manipulate the Excel sheets and basically just add a row for each parking
ticket. For code, check
<a href="https://git.kun.is/pim/parking-expenses/src/commit/645fd8b4e4e46bc8f3d8ba831ca28261cd3edb39/main.py#L104">here</a>.</p>

<p>To convert the Excel sheet to a PDF, I used the (apparently now deprecated)
<a href="https://github.com/unoconv/unoconv">unoconv</a> utility. It uses LibreOffice under
the hood for the conversion, which adds a 2GiB dependency for the project 🫠.</p>

<p>Finally, to merge the report PDF with photos of the parking tickets, I used the
<a href="https://pypdf2.readthedocs.io/en/3.x/">PyPDF2</a> Python library.</p>

<p>I would now show some example output, but it contains quite some sensitive
information so unfortunately I can’t… But I never had any complaints from the
finance department, so it worked great!</p>]]></content><author><name>Pim Kunis</name></author><category term="paperless" /><category term="paperless-ngx" /><category term="nix" /><category term="python" /><category term="python3" /><summary type="html"><![CDATA[For the past year, I was consulting at a place where I could request reimbursement of parking expenses. This had to be claimed using a standard Excel template.]]></summary></entry><entry><title type="html">Notes on Forgejo Actions</title><link href="https://pim.kun.is/i-tried-forgejo-actions/" rel="alternate" type="text/html" title="Notes on Forgejo Actions" /><published>2024-05-03T19:12:00+00:00</published><updated>2024-05-03T19:12:00+00:00</updated><id>https://pim.kun.is/i-tried-forgejo-actions</id><content type="html" xml:base="https://pim.kun.is/i-tried-forgejo-actions/"><![CDATA[<p>I recently revived this blog, which is now running as a pod in my Kubernetes cluster.
In order to deploy a new version, I had to (manualy) perform the following steps:</p>

<ol>
  <li>Build static website from the Jekyll source</li>
  <li>Build container image using the static website</li>
  <li>Push container image to my container registry</li>
  <li>Update Kubernetes deployment manifest to use the new container image</li>
  <li>Apply the new manifest to the running blog deployment</li>
</ol>

<p>This quickly gets annoying and I started looking for a Continuous Integration (CI) system to automate this process.
I am using <a href="https://forgejo.org/">Forgejo</a> as my git server, which recently had its own CI system, Forgejo Actions, <a href="https://forgejo.org/2023-07-release-v1201-0/">graduated</a> to alpha status.
Even though this is still alpha software, I decided to try it out for myself.</p>

<h1 id="setting-up-a-forgejo-actions-runner">Setting up a Forgejo Actions runner</h1>

<p><a href="https://forgejo.org/docs/v1.20/user/actions/">Forgejo Actions</a> is forked from <a href="https://docs.gitea.com/usage/actions/overview">Gitea Actions</a> which in turn is forked from <a href="https://github.com/nektos/act">act</a>.
Act is project to run GitHub Actions locally, which is the reason why Forgejo Actions are quite similar to Github Actions.
In fact, there are quite a few references to GitHub still.</p>

<p>Forgejo Actions work roughly as follows:</p>
<ol>
  <li>You install a runner, which is a server that accepts workloads from Forgejo Actions.</li>
  <li>You register this runner with Forgejo.</li>
  <li>You define workflows on a git repository that execute steps.</li>
  <li>These workflows are submitted to a runner.</li>
  <li>Either a Docker container or an LXC container is spinned up to run the workflow to completion.</li>
  <li>The result is communicated back to Forgejo.</li>
</ol>

<p>All my workloads run on Kubernetes, and wanted the Forgejo runner to run on Kubernetes as well.
Unfortunately, there is no Kubernetes-native way of running this runner, like for example <a href="https://docs.gitlab.com/runner/install/operator.html">GitLab</a> does.
However, there is <a href="https://code.forgejo.org/forgejo/runner/src/branch/main/examples/kubernetes">an example deployment</a> using a Docker-in-Docker setup.
This does however require to run the image as a privileged container, and (spoiler) this setup seems to be very unreliable.</p>

<h1 id="using-forgejo-actions-to-build-nix-derivations">Using Forgejo Actions to build Nix derivations</h1>

<p>Both the static website and container image for my blog are built using Nix (which I wrote about <a href="/nix-jekyll-derivation/">here</a>).
Therefore, it made sense to me to use the <a href="https://hub.docker.com/r/nixos/nix">official Docker image</a>.
However, it seems Forgeo Actions has an undocumented dependency on <code class="language-plaintext highlighter-rouge">/bin/sleep</code> being present inside the container image.
It seems sleep is being used to check whether the image is working correctly.
Nix being very unconventional, does not have <code class="language-plaintext highlighter-rouge">/bin/sleep</code> by default in its Docker image.
Therefore, I extended the offical Docker image using Nix below.
The <code class="language-plaintext highlighter-rouge">coreutils</code> package contains the <code class="language-plaintext highlighter-rouge">sleep</code> binary which is linked inside the image using <code class="language-plaintext highlighter-rouge">pathsToLink</code>.</p>

<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span>
  <span class="nv">nixFromDockerHub</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">dockerTools</span><span class="o">.</span><span class="nv">pullImage</span> <span class="p">{</span>
    <span class="nv">imageName</span> <span class="o">=</span> <span class="s2">"nixos/nix"</span><span class="p">;</span>
    <span class="nv">imageDigest</span> <span class="o">=</span> <span class="s2">"sha256:b3dc72ab3216606d52357ee46f0830a0cc32f3e50e00bd490efa1a8304e9f99d"</span><span class="p">;</span>
    <span class="nv">sha256</span> <span class="o">=</span> <span class="s2">"sha256-FvDlbSnCmPtWTn4eG3hu8WVK1Wm3RSi2T+CdmIDLkG4="</span><span class="p">;</span>
    <span class="nv">finalImageTag</span> <span class="o">=</span> <span class="s2">"2.22.0"</span><span class="p">;</span>
    <span class="nv">finalImageName</span> <span class="o">=</span> <span class="s2">"nix"</span><span class="p">;</span>
  <span class="p">};</span>
  <span class="err">}</span><span class="p">;</span>
<span class="kn">in</span>
<span class="p">{</span>
  <span class="nv">packages</span><span class="o">.</span><span class="nv">forgejo-nix-action</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">dockerTools</span><span class="o">.</span><span class="nv">buildImage</span> <span class="p">{</span>
    <span class="nv">name</span> <span class="o">=</span> <span class="s2">"forgejo-nix-action"</span><span class="p">;</span>
    <span class="nv">tag</span> <span class="o">=</span> <span class="s2">"latest"</span><span class="p">;</span>
    <span class="nv">fromImage</span> <span class="o">=</span> <span class="nv">nixFromDockerHub</span><span class="p">;</span>

    <span class="nv">copyToRoot</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">buildEnv</span> <span class="p">{</span>
      <span class="nv">name</span> <span class="o">=</span> <span class="s2">"image-root"</span><span class="p">;</span>
      <span class="nv">paths</span> <span class="o">=</span> <span class="kn">with</span> <span class="nv">pkgs</span><span class="p">;</span> <span class="p">[</span> <span class="nv">coreutils</span> <span class="p">];</span>
      <span class="nv">pathsToLink</span> <span class="o">=</span> <span class="p">[</span> <span class="s2">"/bin"</span> <span class="p">];</span>
    <span class="p">};</span>
  <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This is sufficient to be able to run <code class="language-plaintext highlighter-rouge">nix build</code> commands inside Forgejo Action, for example:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on</span><span class="pi">:</span> <span class="pi">[</span> <span class="nv">push</span> <span class="pi">]</span>
<span class="na">jobs</span><span class="pi">:</span>
  <span class="na">blog-pim</span><span class="pi">:</span>
    <span class="na">runs-on</span><span class="pi">:</span> <span class="s">docker</span>
    <span class="na">container</span><span class="pi">:</span>
      <span class="na">image</span><span class="pi">:</span> <span class="s">git.kun.is/home/forgejo-nix-action:687d16c49ea7936068bac64ec68c480a9d681962</span>
    <span class="na">steps</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Clone repository</span>
        <span class="na">run</span><span class="pi">:</span> <span class="s">git clone ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git src</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Build image</span>
        <span class="na">run</span><span class="pi">:</span> <span class="s">nix build --out-link image ./src#packages.x86_64-linux.container-image</span>
</code></pre></div></div>

<p>This workflows clones the source code of my blog, then builds a container image for it using Nix.</p>

<h1 id="pushing-the-image-to-a-container-registry">Pushing the image to a container registry</h1>

<p>Being able to automatically build container images is useless if we don’t publish them somewhere.
<a href="https://github.com/containers/skopeo">Skopeo</a> is a real life-saver here, and I would strongly recommend this tool when interacting with container images.
Before settling with Skopeo, I first tried using just Docker.
However, for many operations, Docker needs a running Docker daemon and this is annoying to setup inside an already annoying Docker-in-Docker setup.
Another option I explored was Podman but it was not even able to read the container images built by Nix.</p>

<p>These are the steps I ended up with to push a container image to a registry using Skopeo:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Log into container registry</span>
  <span class="na">run</span><span class="pi">:</span> <span class="s">/bin/skopeo login --tls-verify --username $ --password $ ${GITHUB_SERVER_URL}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Push image to container registry</span>
  <span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
    <span class="s">/bin/skopeo --insecure-policy copy docker-archive:image docker://${GITHUB_SERVER_URL#https://}/${GITHUB_REPOSITORY_OWNER}/blog-pim:latest</span>
</code></pre></div></div>

<h1 id="failing-to-deploy-to-kubernetes">Failing to deploy to Kubernetes</h1>

<p>Having pushed a new container image to a registry, we just have to updated the Kubernetes deployment to use the new image.
This is unfortunately something I have not been able to get working.
For some reason, anytime I run <code class="language-plaintext highlighter-rouge">nix run</code> inside a step, the k3s systemd service crashes.
It seems during the execution of <code class="language-plaintext highlighter-rouge">nix run</code>, <code class="language-plaintext highlighter-rouge">etcd</code> is continuously timing out.
After a few seconds of this, k3s deems it unhealthy and restarts itself.</p>

<p>I have a hunch running <code class="language-plaintext highlighter-rouge">nix run</code> puts so much strain on the disk that <code class="language-plaintext highlighter-rouge">etcd</code> is not able to quickly write to disk.
It is known that <code class="language-plaintext highlighter-rouge">etcd</code> needs very quick disk times to function correctly.
I have however not been able to reproduce this issue outside of the container.
This leads me to believe that perhaps the Docker-in-Docker setup is the cause of this issue.</p>

<h1 id="conclusions">Conclusions</h1>

<p>I think Forgejo Actions really needs a first-class Kubernetes treatment.
It seems to me having Action runners on Kubernetes gives free improvements in terms of scalability and management.
Going further, I will not be using Forgejo Actions for CI, but will explore some systems that run on Kubernetes natively.
In particular, I’m interested in <a href="https://argo-workflows.readthedocs.io/en/latest/">Argo Workflows</a>, and perhaps I will give <a href="https://docs.gitlab.com/runner/install/operator.html">GitLab</a> a try.</p>

<p>However in general, and this is not a critique on Forgejo Actions in particular, I don’t fully agree with the way these “Actions” systems work.
It seems to leading way to use these actions is to use a <code class="language-plaintext highlighter-rouge">runs-on</code> directive and specify some container image, perhaps the latest Ubuntu version (who thought it was a good idea to turn OSes into images?).
Then, people install some tools using the image’s package manager after which their actual jobs runs.
Unless they are very well maintained, these actions will probably stop working after a while when the underlying container image is discontinued from support.
Or perhaps a package is updated, which is incompatable with the action.
Or there is some hidden dependency that changes in the underlying container image.
Or …</p>

<p>I believe Nix could be an amazing tool to improve these “Actions” systems.
Using Nix, you wouldn’t need a bloated container image with lots of tools you won’t need.
You could take advantage of Nixpkgs and install only the tools you need to run your job.
Everything inside the image would be version-pinned and reproducible and will continue to work in 50 years.
I’m not entirely sure how such a system would even look, but my gut tells me it should be possible.
I’m unfortunately not aware whether such a system currently exists though.</p>]]></content><author><name>Pim Kunis</name></author><category term="forgejo" /><category term="actions" /><category term="ci" /><category term="act" /><category term="nix" /><category term="nixos" /><category term="kubernetes" /><summary type="html"><![CDATA[I recently revived this blog, which is now running as a pod in my Kubernetes cluster. In order to deploy a new version, I had to (manualy) perform the following steps:]]></summary></entry><entry><title type="html">Building a Jekyll Static Website using Nix</title><link href="https://pim.kun.is/nix-jekyll-derivation/" rel="alternate" type="text/html" title="Building a Jekyll Static Website using Nix" /><published>2024-05-03T15:44:00+00:00</published><updated>2024-05-03T15:44:00+00:00</updated><id>https://pim.kun.is/nix-jekyll-derivation</id><content type="html" xml:base="https://pim.kun.is/nix-jekyll-derivation/"><![CDATA[<p>This blog has been down for several months, and I recently at last decided to revive it.
The blog is written in <a href="https://jekyllrb.com/">Jekyll</a> and building it generally means:</p>

<ol>
  <li>Installing the right version of the Ruby programming language.</li>
  <li>Installing the correct version of Ruby libraries.</li>
  <li>Building the website using <code class="language-plaintext highlighter-rouge">jekyll build</code>.</li>
</ol>

<p>In order to improve reproducability, I’ll show how to create a Nix derivation for a Jekyll static website.
Much of this post also applies to other Ruby projects.</p>

<h1 id="generate-a-gemsetnix">Generate a <code class="language-plaintext highlighter-rouge">gemset.nix</code></h1>

<p>Ruby uses <a href="https://bundler.io/">Bundler</a> to manage dependencies (called gems).
Each Ruby project generally has two files related to Bundler: <code class="language-plaintext highlighter-rouge">Gemfile</code> and <code class="language-plaintext highlighter-rouge">Gemfile.lock</code>.
These are similar to <code class="language-plaintext highlighter-rouge">flake.nix</code> and <code class="language-plaintext highlighter-rouge">flake.lock</code> in the Nix realm: the <code class="language-plaintext highlighter-rouge">Gemfile</code> specifies a project’s high-level dependencies, which is converted to a <code class="language-plaintext highlighter-rouge">Gemfile.lock</code> file with a snapshot of exact versions these dependencies revolve to.</p>

<p>Nix, however, doesn’t understand this <code class="language-plaintext highlighter-rouge">Gemfile.lock</code>.
Therefore we will first have to convert this to a Nix definition.
We can do this using <a href="https://github.com/nix-community/bundix">Bundix</a>; simply execute <code class="language-plaintext highlighter-rouge">nix run nixpkgs#bundix</code> in the directory containing your <code class="language-plaintext highlighter-rouge">Gemfile.lock</code>.
If you don’t have a <code class="language-plaintext highlighter-rouge">Gemfile.lock</code> yet, you can also have Bundix generate it using <code class="language-plaintext highlighter-rouge">nix run nixpkgs#bundix --lock</code>
This creates a <code class="language-plaintext highlighter-rouge">gemset.nix</code> file containing an attrset like:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
  <span class="nv">jekyll</span> <span class="o">=</span> <span class="p">{</span>
    <span class="nv">dependencies</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"addressable"</span> <span class="s2">"colorator"</span> <span class="s2">"em-websocket"</span> <span class="s2">"i18n"</span> <span class="s2">"jekyll-sass-converter"</span> <span class="s2">"jekyll-watch"</span> <span class="s2">"kramdown"</span> <span class="s2">"kramdown-parser-gfm"</span> <span class="s2">"liquid"</span> <span class="s2">"mercenary"</span> <span class="s2">"pathutil"</span> <span class="s2">"rouge"</span> <span class="s2">"safe_yaml"</span> <span class="s2">"terminal-table"</span><span class="p">];</span>
    <span class="nv">groups</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"default"</span> <span class="s2">"jekyll_plugins"</span><span class="p">];</span>
    <span class="nv">platforms</span> <span class="o">=</span> <span class="p">[];</span>
    <span class="nv">source</span> <span class="o">=</span> <span class="p">{</span>
      <span class="nv">remotes</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"https://rubygems.org"</span><span class="p">];</span>
      <span class="nv">sha256</span> <span class="o">=</span> <span class="s2">"192k1ggw99slpqpxb4xamcvcm2pdahgnmygl746hmkrar0i3xa5r"</span><span class="p">;</span>
      <span class="nv">type</span> <span class="o">=</span> <span class="s2">"gem"</span><span class="p">;</span>
    <span class="p">};</span>
    <span class="nv">version</span> <span class="o">=</span> <span class="s2">"4.1.1"</span><span class="p">;</span>
  <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>

<p>As you can see, apart from pinning the version (<code class="language-plaintext highlighter-rouge">4.1.1</code>), Bundix also pins the SHA256 hash of the Ruby gem guaranteeing reproducability.</p>

<h1 id="packaging-the-ruby-gems">Packaging the Ruby Gems</h1>

<p>Now that have all Ruby dependencies neatly version-pinned in Nix, we can have Nix build them for us.
Here, <a href="https://github.com/NixOS/nixpkgs/blob/123ed385f88545e6037b20a6cda113b936614605/doc/languages-frameworks/ruby.section.md#developing-with-ruby-developing-with-ruby"><code class="language-plaintext highlighter-rouge">bundlerEnv</code></a> does the heavy lifting for us:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
    <span class="nv">gems</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">bundlerEnv</span> <span class="p">{</span>
        <span class="nv">name</span> <span class="o">=</span> <span class="s2">"blog-pim"</span><span class="p">;</span>
        <span class="nv">gemdir</span> <span class="o">=</span> <span class="sx">./src</span><span class="p">;</span>
    <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>

<p>The attribute <code class="language-plaintext highlighter-rouge">gemdir</code> specifies the directory containing our <code class="language-plaintext highlighter-rouge">gemset.nix</code>.
Using <code class="language-plaintext highlighter-rouge">bundlerEnv</code> like this, <code class="language-plaintext highlighter-rouge">gems</code> contains <code class="language-plaintext highlighter-rouge">/lib</code> and <code class="language-plaintext highlighter-rouge">/bin</code> of all direct and indirect dependencies and puts all executables from these gems on PATH.
Also, <code class="language-plaintext highlighter-rouge">gems.wrappedRuby</code> contains a Ruby package with access to each of these Ruby gems.</p>

<h1 id="the-nix-derivation">The Nix Derivation</h1>

<p>Having access to all Ruby dependencies as Nix packages, we can now build static website using Nix:</p>

<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
    <span class="nv">static-website</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">stdenv</span><span class="o">.</span><span class="nv">mkDerivation</span> <span class="p">{</span>
      <span class="nv">name</span> <span class="o">=</span> <span class="s2">"blog-pim"</span><span class="p">;</span>
      <span class="nv">src</span> <span class="o">=</span> <span class="sx">./src</span><span class="p">;</span>
      <span class="nv">sourceRoot</span> <span class="o">=</span> <span class="s2">"src"</span><span class="p">;</span>

      <span class="nv">buildInputs</span> <span class="o">=</span> <span class="p">[</span>
        <span class="nv">gems</span>
        <span class="nv">gems</span><span class="o">.</span><span class="nv">wrappedRuby</span>
      <span class="p">];</span>

      <span class="nv">buildPhase</span> <span class="o">=</span> <span class="s2">''</span><span class="err">
</span><span class="s2">        bundle exec jekyll build</span><span class="err">
</span><span class="s2">      ''</span><span class="p">;</span>

      <span class="nv">installPhase</span> <span class="o">=</span> <span class="s2">''</span><span class="err">
</span><span class="s2">        mkdir -p $out</span><span class="err">
</span><span class="s2">        cp -r _site/* $out/</span><span class="err">
</span><span class="s2">      ''</span><span class="p">;</span>
    <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Set the <code class="language-plaintext highlighter-rouge">src</code> attribute to the directory containing your Jekyll source code.
Also set <code class="language-plaintext highlighter-rouge">sourceRoot</code> to this directory, which <code class="language-plaintext highlighter-rouge">jekyll build</code> expects.
As inputs to our build process, we have the Ruby gems and the wrapped Ruby program from before.
The build phase should be obvious: with our setup we can just build Jekyll as normal.
The compiled static website is put in the <code class="language-plaintext highlighter-rouge">_site</code> directory, which we copy to our build result in the install phase.</p>

<p>And that’s it for the most part!
A fully working flake can be found on my <a href="https://git.kun.is/home/blog-pim/src/commit/824b03b4763487efd39f0cc475f45fa9e3195b43/flake.nix#L53">Git website</a>.
Let’s build the derivation and check its result:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix build .#packages.x86_64-linux.static-website
<span class="nv">$ </span>ll result/
total 180
<span class="nt">-r--r--r--</span> 1 root root  7391 Jan  1  1970 404.html
dr-xr-xr-x 2 root root  4096 Jan  1  1970 about
dr-xr-xr-x 2 root root  4096 Jan  1  1970 ansible-edit-grub
dr-xr-xr-x 2 root root  4096 Jan  1  1970 archive
dr-xr-xr-x 7 root root  4096 Jan  1  1970 assets
dr-xr-xr-x 2 root root  4096 Jan  1  1970 backup-failure
<span class="nt">-r--r--r--</span> 1 root root   262 Jan  1  1970 browserconfig.xml
dr-xr-xr-x 2 root root  4096 Jan  1  1970 concourse-apprise-notifier
...
</code></pre></div></div>

<h1 id="jekyll-feeds-monkey-wrench"><code class="language-plaintext highlighter-rouge">jekyll-feed</code>’s Monkey Wrench</h1>

<p>The above works fine for the most part, but if you use the <a href="https://github.com/jekyll/jekyll-feed">jekyll-feed</a> Ruby gem, you will unfortunately get a non-deterministic derivation!
<code class="language-plaintext highlighter-rouge">jekyll-feed</code> creates an Atom feed for your blog posts when compiling the static website.
Let’s see why it creates non-deterministic builds:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix build <span class="nt">--rebuild</span> <span class="nt">--keep-failed</span> .#packages.x86_64-linux.static-website 
note: keeping build directory <span class="s1">'/tmp/nix-build-blog-pim.drv-3'</span>
error: derivation <span class="s1">'/nix/store/sdphangdgj348ps9x93qwly7kzqalwqf-blog-pim.drv'</span> may not be deterministic: output <span class="s1">'/nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim'</span> differs from <span class="s1">'/nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim.check'</span>
</code></pre></div></div>

<p>Nix complains our derivation differs between different runs.
Let’s see what is different between these two executions:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ nix run nixpkgs#diffoscope /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim.check
--- /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim
+++ /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim.check
│   --- /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim/feed.xml
├── +++ /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim.check/feed.xml
│ │   --- /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim/feed.xml
│ ├── +++ /nix/store/193gdfza7d98l573b53sjqswhwfpc30g-blog-pim.check/feed.xml
│ │ @@ -1,13 +1,13 @@
│ │  &lt;?xml version="1.0" encoding="utf-8"?&gt;
│ │  &lt;feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en-US"&gt;
│ │    &lt;generator uri="https://jekyllrb.com/" version="4.1.1"&gt;Jekyll&lt;/generator&gt;
│ │    &lt;link href="https://pim.kun.is/feed.xml" rel="self" type="application/atom+xml"/&gt;
│ │    &lt;link href="https://pim.kun.is/" rel="alternate" type="text/html" hreflang="en-US"/&gt;
│ │ -  &lt;updated&gt;2024-05-03T11:12:54+00:00&lt;/updated&gt;
│ │ +  &lt;updated&gt;2024-05-03T11:12:58+00:00&lt;/updated&gt;
│ │    &lt;id&gt;https://pim.kun.is/feed.xml&lt;/id&gt;
│ │    &lt;title type="html"&gt;Pim Kunis&lt;/title&gt;
│ │    &lt;subtitle&gt;A pig's gotta fly&lt;/subtitle&gt;
│ │    &lt;author&gt;
│ │      &lt;name&gt;Pim Kunis&lt;/name&gt;
│ │    &lt;/author&gt;
│ │    &lt;entry&gt;
</code></pre></div></div>

<p>Aha, it seems <code class="language-plaintext highlighter-rouge">jekyll-feed</code> uses the current datetime inside the feed to indicate the last change.
Patching this was quite easy: I simply replaced this date with the date of the last post.
I’ll spare you my terrible Python code, but this can be read <a href="https://git.kun.is/home/blog-pim/src/commit/824b03b4763487efd39f0cc475f45fa9e3195b43/patch-feed-date.py">here</a>.</p>

<p>I packaged this script like so:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
  <span class="nv">patch-feed-date</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">stdenv</span><span class="o">.</span><span class="nv">mkDerivation</span> <span class="p">{</span>
    <span class="nv">name</span> <span class="o">=</span> <span class="s2">"path-feed-date"</span><span class="p">;</span>
    <span class="nv">propagatedBuildInputs</span> <span class="o">=</span> <span class="p">[</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">python3</span> <span class="p">];</span>
    <span class="nv">dontUnpack</span> <span class="o">=</span> <span class="kc">true</span><span class="p">;</span>
    <span class="nv">installPhase</span> <span class="o">=</span> <span class="s2">"install -Dm755 </span><span class="si">${</span><span class="sx">./patch-feed-date.py</span><span class="si">}</span><span class="s2"> $out/bin/patch-feed-date"</span><span class="p">;</span>
  <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>

<p>And I updated the derivation as follows:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
    <span class="nv">static-website</span> <span class="o">=</span> <span class="nv">pkgs</span><span class="o">.</span><span class="nv">stdenv</span><span class="o">.</span><span class="nv">mkDerivation</span> <span class="p">{</span>
      <span class="nv">name</span> <span class="o">=</span> <span class="s2">"blog-pim"</span><span class="p">;</span>
      <span class="nv">src</span> <span class="o">=</span> <span class="sx">./src</span><span class="p">;</span>
      <span class="nv">sourceRoot</span> <span class="o">=</span> <span class="s2">"src"</span><span class="p">;</span>

      <span class="nv">buildInputs</span> <span class="o">=</span> <span class="p">[</span>
        <span class="nv">gems</span>
        <span class="nv">gems</span><span class="o">.</span><span class="nv">wrappedRuby</span>
        <span class="nv">patch-feed-date</span> <span class="c"># Updated</span>
      <span class="p">];</span>

      <span class="nv">buildPhase</span> <span class="o">=</span> <span class="s2">''</span><span class="err">
</span><span class="s2">        bundle exec jekyll build</span><span class="err">
</span><span class="s2">      ''</span><span class="p">;</span>

      <span class="nv">installPhase</span> <span class="o">=</span> <span class="s2">''</span><span class="err">
</span><span class="s2">        mkdir -p $out</span><span class="err">
</span><span class="s2">        cp -r _site/* $out/</span><span class="err">
</span><span class="s2">        patch-feed-date --file _site/feed.xml &gt; $out/feed.xml # Updated</span><span class="err">
</span><span class="s2">      ''</span><span class="p">;</span>
    <span class="p">};</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Let’s check if this fixed our non-deterministic build:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>nix build <span class="nt">--rebuild</span> <span class="nt">--keep-failed</span>  .#packages.x86_64-linux.static-website 
<span class="nv">$ </span><span class="nb">echo</span> <span class="nv">$?</span>
0
</code></pre></div></div>]]></content><author><name>Pim Kunis</name></author><category term="jekyll" /><category term="nix" /><category term="blog" /><category term="ruby" /><summary type="html"><![CDATA[This blog has been down for several months, and I recently at last decided to revive it. The blog is written in Jekyll and building it generally means:]]></summary></entry><entry><title type="html">It’s alive!</title><link href="https://pim.kun.is/its-alive/" rel="alternate" type="text/html" title="It’s alive!" /><published>2024-04-21T10:02:00+00:00</published><updated>2024-04-21T10:02:00+00:00</updated><id>https://pim.kun.is/its-alive</id><content type="html" xml:base="https://pim.kun.is/its-alive/"><![CDATA[<p>Finally, after several months this website is up and running again!</p>

<p>My homelab has completely changed, but the reason why it initially went offline is because of my failing CI installation.
I was using <a href="https://concourse-ci.org/">Concourse CI</a> which I was initially interested in due to the reproducible nature of its builds using containers.
However, for some reason pipelines were sporadically getting stuck when I reboot the virtual machine it was running on.
The fix was very annoying: I had to re-create the pipelines manually (which feels very backwards for a CI/CD system!)
Additionally, my virtual machine setup back then was also quite fragile and I decided to get rid of that as well.</p>

<p>I have learned that having an escape hatch to deploy something is probably a good idea 😅
Expect a new overview of my homelab soon, in the same vein as <a href="/infrastructure-snapshot/">this post from last year</a>!</p>]]></content><author><name>Pim Kunis</name></author><category term="jekyll" /><category term="blog" /><summary type="html"><![CDATA[Finally, after several months this website is up and running again!]]></summary></entry><entry><title type="html">Home Lab Infrastructure Snapshot August 2023</title><link href="https://pim.kun.is/infrastructure-snapshot/" rel="alternate" type="text/html" title="Home Lab Infrastructure Snapshot August 2023" /><published>2023-08-27T22:23:00+00:00</published><updated>2023-08-27T22:23:00+00:00</updated><id>https://pim.kun.is/infrastructure-snapshot</id><content type="html" xml:base="https://pim.kun.is/infrastructure-snapshot/"><![CDATA[<p>I have been meaning to write about the current state of my home lab infrastructure for a while now.
Now that the most important parts are quite stable, I think the opportunity is ripe.
I expect this post to get quite long, so I might have to leave out some details along the way.</p>

<p>This post will be a starting point for future infrastructure snapshots which I can hopefully put out periodically.
That is, if there is enough worth talking about.</p>

<p>Keep an eye out for the <i class="fa-solid fa-code-branch"></i> icon, which links to the source code and configuration of anything mentioned.
Oh yeah, did I mention everything I do is open source?</p>

<h1 id="networking-and-infrastructure-overview">Networking and Infrastructure Overview</h1>

<h2 id="hardware-and-operating-systems">Hardware and Operating Systems</h2>

<p>Let’s start with the basics: what kind of hardware do I use for my home lab?
The most important servers are my three <a href="https://www.gigabyte.com/Mini-PcBarebone/GB-BLCE-4105-rev-10">Gigabyte Brix GB-BLCE-4105</a>.
Two of them have 16 GB of memory, and one 8 GB.
I named these servers as follows:</p>
<ul>
  <li><strong>Atlas</strong>: because this server was going to “lift” a lot of virtual machines.</li>
  <li><strong>Lewis</strong>: we started out with a “Max” server named after the Formula 1 driver Max Verstappen, but it kind of became an unmanagable behemoth without infrastructure-as-code. Our second server we subsequently named Lewis after his colleague Lewis Hamilton. Note: people around me vetoed these names and I am no F1 fan!</li>
  <li><strong>Jefke</strong>: it’s a funny Belgian name. That’s all.</li>
</ul>

<p>Here is a picture of them sitting in their cosy closet:</p>

<p><img src="servers.jpeg" alt="A picture of my servers." /></p>

<p>If you look look to the left, you will also see a Raspberry pi 4B.
I use this Pi to do some rudimentary monitoring whether servers and services are running.
More on this in the relevant section below.
The Pi is called <strong>Iris</strong> because it’s a messenger for the other servers.</p>

<p>I used to run Ubuntu on these systems, but I have since migrated away to Debian.
The main reasons were Canonical <a href="https://askubuntu.com/questions/1434512/how-to-get-rid-of-ubuntu-pro-advertisement-when-updating-apt">putting advertisements in my terminal</a> and pushing Snap which has a <a href="https://hackaday.com/2020/06/24/whats-the-deal-with-snap-packages/">proprietry backend</a>.
Two of my servers run the newly released Debian Bookworm, while one still runs Debian Bullseye.</p>

<h2 id="networking">Networking</h2>

<p>For networking, I wanted hypervisors and virtual machines separated by VLANs for security reasons.
The following picture shows a simplified view of the VLANs present in my home lab:</p>

<p><img src="vlans.png" alt="Picture showing the VLANS in my home lab." /></p>

<p>All virtual machines are connected to a virtual bridge which tags network traffic with the DMZ VLAN.
The hypervisors VLAN is used for traffic to and from the hypervisors.
Devices from the hypervisors VLAN are allowed to connect to devices in the DMZ, but not vice versa.
The hypervisors are connected to a switch using a trunk link, allows both DMZ and hypervisors traffic.</p>

<p>I realised the above design using ifupdown.
Below is the configuration for each hypervisor, which creates a new <code class="language-plaintext highlighter-rouge">enp3s0.30</code> interface with all DMZ traffic from the <code class="language-plaintext highlighter-rouge">enp3s0</code> interface <a href="https://git.kun.is/home/hypervisors/src/commit/71b96d462116e4160b6467533fc476f3deb9c306/ansible/dmz.conf.j2"><i class="fa-solid fa-code-branch"></i></a>.</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto enp3s0.30
iface enp3s0.30 inet manual
iface enp3s0.30 inet6 auto
	accept_ra 0
	dhcp 0
	request_prefix 0
	privext 0
	pre-up sysctl -w net/ipv6/conf/enp3s0.30/disable_ipv6=1
</code></pre></div></div>

<p>This configuration seems more complex than it actually is.
Most of it is to make sure the interface is not assigned an IPv4/6 address on the hypervisor host.
The magic <code class="language-plaintext highlighter-rouge">.30</code> at the end of the interface name makes this interface tagged with VLAN ID 30 (DMZ for me).</p>

<p>Now that we have an interface tagged for the DMZ VLAN, we can create a bridge where future virtual machines can connect to:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>auto dmzbr
iface dmzbr inet manual
	bridge_ports enp3s0.30
	bridge_stp off
iface dmzbr inet6 auto
	accept_ra 0
	dhcp 0
	request_prefix 0
	privext 0
	pre-up sysctl -w net/ipv6/conf/dmzbr/disable_ipv6=1
</code></pre></div></div>

<p>Just like the previous config, this is quite bloated because I don’t want the interface to be assigned an IP address on the host.
Most importantly, the <code class="language-plaintext highlighter-rouge">bridge_ports enp3s0.30</code> line here makes this interface a virtual bridge for the <code class="language-plaintext highlighter-rouge">enp3s0.30</code> interface.</p>

<p>And voilà, we now have a virtual bridge on each machine, where only DMZ traffic will flow.
Here I verify whether this configuration works:</p>
<details>
  <summary>Show</summary>

  <p>We can see that the two virtual interfaces are created, and are only assigned a MAC address and not a IP address:</p>
  <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@atlas:~# ip a show enp3s0.30
4: enp3s0.30@enp3s0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue master dmzbr state UP group default qlen 1000
    link/ether d8:5e:d3:4c:70:38 brd ff:ff:ff:ff:ff:ff
5: dmzbr: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4e:f7:1f:0f:ad:17 brd ff:ff:ff:ff:ff:ff
</code></pre></div>  </div>

  <p>Pinging a VM from a hypervisor works:</p>
  <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@atlas:~# ping -c1 maestro.dmz
PING maestro.dmz (192.168.30.8) 56(84) bytes of data.
64 bytes from 192.168.30.8 (192.168.30.8): icmp_seq=1 ttl=63 time=0.457 ms
</code></pre></div>  </div>

  <p>Pinging a hypervisor from a VM does not work:</p>
  <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@maestro:~# ping  -c1 atlas.hyp
PING atlas.hyp (192.168.40.2) 56(84) bytes of data.

--- atlas.hyp ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
</code></pre></div>  </div>
</details>

<h2 id="dns-and-dhcp">DNS and DHCP</h2>

<p>Now that we have a working DMZ network, let’s build on it to get DNS and DHCP working.
This will enable new virtual machines to obtain a static or dynamic IP address and register their host in DNS.
This has actually been incredibly annoying due to our friend <a href="https://en.wikipedia.org/wiki/Network_address_translation?useskin=vector">Network address translation (NAT)</a>.</p>
<details>
  <summary>NAT recap</summary>

  <p>Network address translation (NAT) is a function of a router which allows multiple hosts to share a single IP address.
This is needed for IPv4, because IPv4 addresses are scarce and usually one household is only assigned a single IPv4 address.
This is one of the problems IPv6 attempts to solve (mainly by having so many IP addresses that they should never run out).
To solve the problem for IPv4, each host in a network is assigned a private IPv4 address, which can be reused for every network.</p>

  <p>Then, the router must perform address translation.
It does this by keeping track of ports opened by hosts in its private network.
If a packet from the internet arrives at the router for such a port, it forwards this packet to the correct host.</p>
</details>

<p>I would like to host my own DNS on a virtual machine (called <strong>hermes</strong>, more on VMs later) in the DMZ network.
This basically gives two problems:</p>

<ol>
  <li>The upstream DNS server will refer to the public internet-accessible IP address of our DNS server.
This IP-address has no meaning inside the private network due to NAT and the router will reject the packet.</li>
  <li>Our DNS resolves hosts to their public internet-accessible IP address.
This is similar to the previous problem as the public IP address has no meaning.</li>
</ol>

<p>The first problem can be remediated by overriding the location of the DNS server for hosts inside the DMZ network.
This can be achieved on my router, which uses Unbound as its recursive DNS server:</p>

<p><img src="unbound_overrides.png" alt="Unbound overides for kun.is and dmz domains." /></p>

<p>Any DNS requests to Unbound to domains in either <code class="language-plaintext highlighter-rouge">dmz</code> or <code class="language-plaintext highlighter-rouge">kun.is</code> will now be forwarded <code class="language-plaintext highlighter-rouge">192.168.30.7</code> (port 5353).
This is the virtual machine hosting my DNS.</p>

<p>The second problem can be solved at the DNS server.
We need to do some magic overriding, which <a href="https://dnsmasq.org/docs/dnsmasq-man.html">dnsmasq</a> is perfect for <a href="https://git.kun.is/home/hermes/src/commit/488024a7725f2325b8992e7a386b4630023f1b52/ansible/roles/dnsmasq/files/dnsmasq.conf"><i class="fa-solid fa-code-branch"></i></a>:</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">alias</span>=<span class="m">84</span>.<span class="m">245</span>.<span class="m">14</span>.<span class="m">149</span>,<span class="m">192</span>.<span class="m">168</span>.<span class="m">30</span>.<span class="m">8</span>
<span class="n">server</span>=/<span class="n">kun</span>.<span class="n">is</span>/<span class="m">192</span>.<span class="m">168</span>.<span class="m">30</span>.<span class="m">7</span>
</code></pre></div></div>

<p>This always overrides the public IPv4 address to the private one.
It also overrides the DNS server for <code class="language-plaintext highlighter-rouge">kun.is</code> to <code class="language-plaintext highlighter-rouge">192.168.30.7</code>.</p>

<p>Finally, behind the dnsmasq server, I run <a href="https://www.powerdns.com/">Powerdns</a> as authoritative DNS server <a href="https://git.kun.is/home/hermes/src/branch/master/ansible/roles/powerdns"><i class="fa-solid fa-code-branch"></i></a>.
I like this DNS server because I can manage it with Terraform <a href="https://git.kun.is/home/hermes/src/commit/488024a7725f2325b8992e7a386b4630023f1b52/terraform/dns/kun_is.tf"><i class="fa-solid fa-code-branch"></i></a>.</p>

<p>Here is a small diagram showing my setup (my networking teacher would probably kill me for this):
<img src="nat.png" alt="Shitty diagram showing my DNS setup." /></p>

<h1 id="virtualization">Virtualization</h1>
<p>https://github.com/containrrr/shepherd
Now that we have laid out the basic networking, let’s talk virtualization.
Each of my servers are configured to run KVM virtual machines, orchestrated using Libvirt.
Configuration of the physical hypervisor servers, including KVM/Libvirt is done using Ansible.
The VMs are spun up using Terraform and the <a href="https://registry.terraform.io/providers/dmacvicar/libvirt/latest/docs">dmacvicar/libvirt</a> Terraform provider.</p>

<p>This all isn’t too exciting, except that I created a Terraform module that abstracts the Terraform Libvirt provider for my specific scenario <a href="https://git.kun.is/home/tf-modules/src/commit/e77d62f4a2a0c3847ffef4434c50a0f40f1fa794/debian/main.tf"><i class="fa-solid fa-code-branch"></i></a>:</p>
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="s2">"maestro"</span> <span class="p">{</span>
  <span class="nx">source</span>          <span class="o">=</span> <span class="s2">"git::https://git.kun.is/home/tf-modules.git//debian"</span>
  <span class="nx">name</span>            <span class="o">=</span> <span class="s2">"maestro"</span>
  <span class="nx">domain_name</span>     <span class="o">=</span> <span class="s2">"tf-maestro"</span>
  <span class="nx">memory</span>          <span class="o">=</span> <span class="mi">10240</span>
  <span class="nx">mac</span>             <span class="o">=</span> <span class="s2">"CA:FE:C0:FF:EE:08"</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This automatically creates a Debian virtual machines with the properties specified.
It also sets up certificate-based SSH authentication which I talked about <a href="/homebrew-ssh-ca/">before</a>.</p>

<h1 id="clustering">Clustering</h1>

<p>With virtualization explained, let’s move up one level further.
Each of my three physical servers hosts a virtual machine running Docker, which together form a Docker Swarm.
I use Traefik as a reverse proxy which routes requests to the correct container.</p>

<p>All data is hosted on a single machine and made available to containers using NFS.
This might not be very secure (as NFS is not encrypted and no proper authentication), it is quite fast.</p>

<p>As of today, I host the following services on my Docker Swarm <a href="https://git.kun.is/home/shoarma"><i class="fa-solid fa-code-branch"></i></a>:</p>
<ul>
  <li><a href="https://forgejo.org/">Forgejo</a> as Git server</li>
  <li><a href="https://www.freshrss.org/">FreshRSS</a> as RSS aggregator</li>
  <li><a href="https://hedgedoc.org/">Hedgedoc</a> as Markdown note-taking</li>
  <li><a href="https://hedgedoc.org/">Inbucket</a> for disposable email</li>
  <li><a href="https://cyberchef.org/">Cyberchef</a> for the lulz</li>
  <li><a href="https://kitchenowl.org/">Kitchenowl</a> for grocery lists</li>
  <li><a href="https://joinmastodon.org/">Mastodon</a> for microblogging</li>
  <li>A monitoring stack (read more below)</li>
  <li><a href="https://nextcloud.com/">Nextcloud</a> for cloud storage</li>
  <li><a href="https://pi-hole.net/">Pihole</a> to block advertisements</li>
  <li><a href="https://radicale.org/v3.html">Radicale</a> for calendar and contacts sync</li>
  <li><a href="https://www.seafile.com/en/home/">Seafile</a> for cloud storage and sync</li>
  <li><a href="https://github.com/containrrr/shepherd">Shephard</a> for automatic container updates</li>
  <li><a href="https://nginx.org/en/">Nginx</a> hosting static content (like this page!)</li>
  <li><a href="https://hub.docker.com/r/charypar/swarm-dashboard/#!">Docker Swarm dashboard</a></li>
  <li><a href="https://syncthing.net/">Syncthing</a> for file sync</li>
</ul>

<h1 id="ci--cd">CI / CD</h1>

<p>For CI / CD, I run <a href="https://concourse-ci.org/">Concourse CI</a> in a separate VM.
This is needed, because Concourse heavily uses containers to create reproducible builds.</p>

<p>Although I should probably use it for more, I currently use my Concourse for three pipelines:</p>

<ul>
  <li>A pipeline to build this static website and create a container image of it.
The image is then uploaded to the image registry of my Forgejo instance.
I love it when I can use stuff I previously built :)
The pipeline finally deploys this new image to the Docker Swarm <a href="https://git.kun.is/pim/static/src/commit/eee4f0c70af6f2a49fabb730df761baa6475db22/pipeline.yml"><i class="fa-solid fa-code-branch"></i></a>.</li>
  <li>A pipeline to create a Concourse resource that sends Apprise alerts (Concourse-ception?) <a href="https://git.kun.is/pim/concourse-apprise-notifier/src/commit/b5d4413c1cd432bc856c45ec497a358aca1b8b21/pipeline.yml"><i class="fa-solid fa-code-branch"></i></a></li>
  <li>A pipeline to build a custom Fluentd image with plugins installed <a href="https://git.kun.is/pim/fluentd"><i class="fa-solid fa-code-branch"></i></a></li>
</ul>

<h1 id="backups">Backups</h1>

<p>To create backups, I use <a href="https://www.borgbackup.org/">Borg</a>.
As I keep all data on one machine, this backup process is quite simple.
In fact, all this data is stored in a single Libvirt volume.
To configure Borg with a simple declarative script, I use <a href="https://torsion.org/borgmatic/">Borgmatic</a>.</p>

<p>In order to back up the data inside the Libvirt volume, I create a snapshot to a file.
Then I can mount this snapshot in my file system.
The files can then be backed up while the system is still running.
It is also possible to simply back up the Libvirt image, but this takes more time and storage <a href="https://git.kun.is/home/hypervisors/src/commit/71b96d462116e4160b6467533fc476f3deb9c306/ansible/roles/borg/backup.yml.j2"><i class="fa-solid fa-code-branch"></i></a>.</p>

<h1 id="monitoring-and-alerting">Monitoring and Alerting</h1>

<p>The last topic I would like to talk about is monitoring and alerting.
This is something I’m still actively improving and only just set up properly.</p>

<h2 id="alerting">Alerting</h2>

<p>For alerting, I wanted something that runs entirely on my own infrastructure.
I settled for Apprise + Ntfy.</p>

<p><a href="https://github.com/caronc/apprise">Apprise</a> is a server that is able to send notifications to dozens of services.
For application developers, it is thus only necessary to implement the Apprise API to gain access to all these services.
The Apprise API itself is also very simple.
By using Apprise, I can also easily switch to another notification service later.
<a href="https://ntfy.sh/">Ntfy</a> is free software made for mobile push notifications.</p>

<p>I use this alerting system in quite a lot of places in my infrastructure, for example when creating backups.</p>

<h2 id="uptime-monitoring">Uptime Monitoring</h2>

<p>The first monitoring setup I created, was using <a href="https://github.com/louislam/uptime-kuma">Uptime Kuma</a>.
Uptime Kuma periodically pings a service to see whether it is still running.
You can do a literal ping, test HTTP response codes, check database connectivity and much more.
I use it to check whether my services and VMs are online.
And the best part is, Uptime Kuma supports Apprise so I get push notifications on my phone whenever something goes down!</p>

<h2 id="metrics-and-log-monitoring">Metrics and Log Monitoring</h2>

<p>A new monitoring system I am still in the process of deploying is focused on metrics and logs.
I plan on creating a separate blog post about this, so keep an eye out on that (for example using RSS :)).
Safe to say, it is no basic ELK stack!</p>

<h1 id="conclusion">Conclusion</h1>

<p>That’s it for now!
Hopefully I inspired someone to build something… or how not to :)</p>]]></content><author><name>Pim Kunis</name></author><category term="infrastructure" /><category term="homelab" /><summary type="html"><![CDATA[I have been meaning to write about the current state of my home lab infrastructure for a while now. Now that the most important parts are quite stable, I think the opportunity is ripe. I expect this post to get quite long, so I might have to leave out some details along the way.]]></summary></entry><entry><title type="html">Hashicorp’s License Change and my Home Lab - Update</title><link href="https://pim.kun.is/hashicorp-license-change/" rel="alternate" type="text/html" title="Hashicorp’s License Change and my Home Lab - Update" /><published>2023-08-17T18:15:00+00:00</published><updated>2023-08-17T18:15:00+00:00</updated><id>https://pim.kun.is/hashicorp-license-change</id><content type="html" xml:base="https://pim.kun.is/hashicorp-license-change/"><![CDATA[<p><em>See the <a href="#update">Update</a> at the end of the article.</em></p>

<p>Already a week ago, Hashicorp <a href="https://www.hashicorp.com/blog/hashicorp-adopts-business-source-license">announced</a> it would change the license on almost all its projects.
Unlike <a href="https://github.com/hashicorp/terraform/commit/ab411a1952f5b28e6c4bd73071194761da36a83f">their previous license</a>, which was the Mozilla Public License 2.0, their new license is no longer truly open source.
It is called the Business Source License™ and restricts use of their software for competitors.
In their own words:</p>
<blockquote>
  <p>Vendors who provide competitive services built on our community products will no longer be able to incorporate future releases, bug fixes, or security patches contributed to our products.</p>
</blockquote>

<p>I found <a href="https://meshedinsights.com/2021/02/02/rights-ratchet/">a great article</a> by MeshedInsights that names this behaviour the “rights ratchet model”.
They define a script start-ups use to garner the interest of open source enthusiasts but eventually turn their back on them for profit.
The reason why Hashicorp can do this, is because contributors signed a copyright license agreement (CLA).
This agreement transfers the copyright of contributors’ code to Hashicorp, allowing them to change the license if they want to.</p>

<p>I find this action really regrettable because I like their products.
This sort of action was also why I wanted to avoid using an Elastic stack, which also had their <a href="https://www.elastic.co/pricing/faq/licensing">license changed</a>.<sup id="fnref:elastic"><a href="#fn:elastic" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>
These companies do not respect their contributors and the software stack beneath they built their product on, which is actually open source (Golang, Linux, etc.).</p>

<h1 id="impact-on-my-home-lab">Impact on my Home Lab</h1>

<p>I am using Terraform in my home lab to manage several important things:</p>
<ul>
  <li>Libvirt virtual machines</li>
  <li>PowerDNS records</li>
  <li>Elasticsearch configuration</li>
</ul>

<p>With Hashicorp’s anti open source move, I intend to move away from Terraform in the future.
While I will not use Hashicorp’s products for new personal projects, I will leave my current setup as-is for some time because there is no real need to quickly migrate.</p>

<p>I might also investigate some of Terraform’s competitors, like Pulumi.
Hopefully there is a project that respects open source which I can use in the future.</p>

<h1 id="update">Update</h1>

<p>A promising fork of Terraform has been announced called <a href="https://opentf.org/announcement">OpenTF</a>.
They intend to take part of the Cloud Native Computing Foundation, which I think is a good effort because Terraform is so important for modern cloud infrastructures.</p>

<h1 id="footnotes">Footnotes</h1>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:elastic">
      <p>While I am still using Elasticsearch, I don’t use the rest of the Elastic stack in order to prevent a vendor lock-in. <a href="#fnref:elastic" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Pim Kunis</name></author><category term="hashicorp" /><category term="terraform" /><category term="vault" /><category term="nomad" /><summary type="html"><![CDATA[See the Update at the end of the article.]]></summary></entry><entry><title type="html">Monitoring Correct Memory Usage in Fluent Bit</title><link href="https://pim.kun.is/fluent-bit-memory/" rel="alternate" type="text/html" title="Monitoring Correct Memory Usage in Fluent Bit" /><published>2023-08-09T16:19:00+00:00</published><updated>2023-08-09T16:19:00+00:00</updated><id>https://pim.kun.is/fluent-bit-memory</id><content type="html" xml:base="https://pim.kun.is/fluent-bit-memory/"><![CDATA[<p>Previously, I have used <a href="https://github.com/prometheus/node_exporter">Prometheus’ node_exporter</a> to monitor the memory usage of my servers.
However, I am currently in the process of moving away from Prometheus to a new Monioring stack.
While I understand the advantages, I felt like Prometheus’ pull architecture does not scale nicely.
Everytime I spin up a new machine, I would have to centrally change Prometheus’ configuration in order for it to query the new server.</p>

<p>In order to collect metrics from my servers, I am now using <a href="https://fluentbit.io/">Fluent Bit</a>.
I love Fluent Bit’s way of configuration which I can easily express as code and automate, its focus on effiency and being vendor agnostic.
However, I have stumbled upon one, in my opinion, big issue with Fluent Bit: its <code class="language-plaintext highlighter-rouge">mem</code> plugin to monitor memory usage is <em>completely</em> useless.
In this post I will go over the problem and my temporary solution.</p>

<h1 id="the-problem-with-fluent-bits-mem-plugin">The Problem with Fluent Bit’s <code class="language-plaintext highlighter-rouge">mem</code> Plugin</h1>

<p>As can be seen in <a href="https://docs.fluentbit.io/manual/pipeline/inputs/memory-metrics">the documentation</a>, Fluent Bit’s <code class="language-plaintext highlighter-rouge">mem</code> input plugin exposes a few metrics regarding memory usage which should be self-explaining: <code class="language-plaintext highlighter-rouge">Mem.total</code>, <code class="language-plaintext highlighter-rouge">Mem.used</code>, <code class="language-plaintext highlighter-rouge">Mem.free</code>, <code class="language-plaintext highlighter-rouge">Swap.total</code>, <code class="language-plaintext highlighter-rouge">Swap.used</code> and <code class="language-plaintext highlighter-rouge">Swap.free</code>.
The problem is that <code class="language-plaintext highlighter-rouge">Mem.used</code> and <code class="language-plaintext highlighter-rouge">Mem.free</code> do not accurately reflect the machine’s actual memory usage.
This is because these metrics include caches and buffers, which can be reclaimed by other processes if needed.
Most tools reporting memory usage therefore include an additional metric that specifices the memory <em>available</em> on the system.
For example, the command <code class="language-plaintext highlighter-rouge">free -m</code> reports the following data on my laptop:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>               total        used        free      shared  buff/cache   available
Mem:           15864        3728        7334         518        5647       12136
Swap:           2383         663        1720
</code></pre></div></div>

<p>Notice that the <code class="language-plaintext highlighter-rouge">available</code> memory is more than <code class="language-plaintext highlighter-rouge">free</code> memory.</p>

<p>While the issue is known (see <a href="https://github.com/fluent/fluent-bit/pull/3092">this</a> and <a href="https://github.com/fluent/fluent-bit/pull/5237">this</a> link), it is unfortunately not yet fixed.</p>

<h1 id="a-temporary-solution">A Temporary Solution</h1>

<p>The issues I linked previously provide stand-alone plugins that fix the problem, which will hopefully be merged in the official project at some point.
However, I didn’t want to install another plugin so I used Fluent Bit’s <code class="language-plaintext highlighter-rouge">exec</code> input plugin and the <code class="language-plaintext highlighter-rouge">free</code> Linux command to query memory usage like so:</p>
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[<span class="n">INPUT</span>]
    <span class="n">Name</span> <span class="n">exec</span>
    <span class="n">Tag</span> <span class="n">memory</span>
    <span class="n">Command</span> <span class="n">free</span> -<span class="n">m</span> | <span class="n">tail</span> -<span class="m">2</span> | <span class="n">tr</span> <span class="s1">'\n'</span> <span class="s1">' '</span>
    <span class="n">Interval_Sec</span> <span class="m">1</span>
</code></pre></div></div>

<p>To interpret the command’s output, I created the following filter:</p>
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[<span class="n">FILTER</span>]
    <span class="n">Name</span> <span class="n">parser</span>
    <span class="n">Match</span> <span class="n">memory</span>
    <span class="n">Key_Name</span> <span class="n">exec</span>
    <span class="n">Parser</span> <span class="n">free</span>
</code></pre></div></div>

<p>Lastly, I created the following parser (warning: regex shitcode incoming):</p>
<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[<span class="n">PARSER</span>]
    <span class="n">Name</span> <span class="n">free</span>
    <span class="n">Format</span> <span class="n">regex</span>
    <span class="n">Regex</span> ^<span class="n">Mem</span>:\<span class="n">s</span>+(?&lt;<span class="n">mem_total</span>&gt;\<span class="n">d</span>+)\<span class="n">s</span>+(?&lt;<span class="n">mem_used</span>&gt;\<span class="n">d</span>+)\<span class="n">s</span>+(?&lt;<span class="n">mem_free</span>&gt;\<span class="n">d</span>+)\<span class="n">s</span>+(?&lt;<span class="n">mem_shared</span>&gt;\<span class="n">d</span>+)\<span class="n">s</span>+(?&lt;<span class="n">mem_buff_cache</span>&gt;\<span class="n">d</span>+)\<span class="n">s</span>+(?&lt;<span class="n">mem_available</span>&gt;\<span class="n">d</span>+) <span class="n">Swap</span>:\<span class="n">s</span>+(?&lt;<span class="n">swap_total</span>&gt;\<span class="n">d</span>+)\<span class="n">s</span>+(?&lt;<span class="n">swap_used</span>&gt;\<span class="n">d</span>+)\<span class="n">s</span>+(?&lt;<span class="n">swap_free</span>&gt;\<span class="n">d</span>+)
    <span class="n">Types</span> <span class="n">mem_total</span>:<span class="n">integer</span> <span class="n">mem_used</span>:<span class="n">integer</span> <span class="n">mem_free</span>:<span class="n">integer</span> <span class="n">mem_shared</span>:<span class="n">integer</span> <span class="n">mem_buff_cache</span>:<span class="n">integer</span> <span class="n">mem_available</span>:<span class="n">integer</span> <span class="n">swap_total</span>:<span class="n">integer</span> <span class="n">swap_used</span>:<span class="n">integer</span>
</code></pre></div></div>

<p>With this configuration, you can use the <code class="language-plaintext highlighter-rouge">mem_available</code> metric to get accurate memory usage in Fluent Bit.</p>

<h1 id="conclusion">Conclusion</h1>

<p>Let’s hope Fluent Bit’s <code class="language-plaintext highlighter-rouge">mem</code> input plugin is improved upon soon so this hacky solution is not needed.
I also intend to document my new monitoring pipeline, which at the moment consists of:</p>
<ul>
  <li>Fluent Bit</li>
  <li>Fluentd</li>
  <li>Elasticsearch</li>
  <li>Grafana</li>
</ul>]]></content><author><name>Pim Kunis</name></author><category term="fluentd" /><category term="fluentbit" /><category term="memory" /><summary type="html"><![CDATA[Previously, I have used Prometheus’ node_exporter to monitor the memory usage of my servers. However, I am currently in the process of moving away from Prometheus to a new Monioring stack. While I understand the advantages, I felt like Prometheus’ pull architecture does not scale nicely. Everytime I spin up a new machine, I would have to centrally change Prometheus’ configuration in order for it to query the new server.]]></summary></entry><entry><title type="html">Error Handling in Borgmatic</title><link href="https://pim.kun.is/backup-failure/" rel="alternate" type="text/html" title="Error Handling in Borgmatic" /><published>2023-08-08T11:51:00+00:00</published><updated>2023-08-08T11:51:00+00:00</updated><id>https://pim.kun.is/backup-failure</id><content type="html" xml:base="https://pim.kun.is/backup-failure/"><![CDATA[<p><a href="https://borgbackup.readthedocs.io/en/stable/">BorgBackup</a> and <a href="https://torsion.org/borgmatic/">Borgmatic</a> have been my go-to tools to create backups for my home lab since I started creating backups.
Using <a href="https://wiki.archlinux.org/title/systemd/Timers">Systemd Timers</a>, I regularly create a backup every night.
I also monitor successful execution of the backup process, in case some error occurs.
However, the way I set this up resulted in not receiving notifications.
Even though it boils down to RTFM, I’d like to explain my error and how to handle errors correctly.</p>

<p>I was using the <code class="language-plaintext highlighter-rouge">on_error</code> option to handle errors, like so:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on_error</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="s1">'</span><span class="s">apprise</span><span class="nv"> </span><span class="s">--body="Error</span><span class="nv"> </span><span class="s">while</span><span class="nv"> </span><span class="s">performing</span><span class="nv"> </span><span class="s">backup"</span><span class="nv"> </span><span class="s">&lt;URL&gt;</span><span class="nv"> </span><span class="s">||</span><span class="nv"> </span><span class="s">true'</span>
</code></pre></div></div>

<p>However, <code class="language-plaintext highlighter-rouge">on_error</code> does not handle errors from the execution of <code class="language-plaintext highlighter-rouge">before_everything</code> and <code class="language-plaintext highlighter-rouge">after_everything</code> hooks.
My solution to this was moving the error handling up to the Systemd service that calls Borgmatic.
This results in the following Systemd service:</p>

<div class="language-systemd highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">[Unit]</span>
<span class="nt">Description</span><span class="p">=</span>Backup data using Borgmatic
<span class="c"># Added</span>
<span class="nt">OnFailure</span><span class="p">=</span>backup-failure.service

<span class="k">[Service]</span>
<span class="nt">ExecStart</span><span class="p">=</span>/usr/bin/borgmatic --config /root/backup.yml
<span class="nt">Type</span><span class="p">=</span>oneshot
</code></pre></div></div>

<p>This handles any error, be it from Borgmatic’s hooks or itself.
The <code class="language-plaintext highlighter-rouge">backup-failure</code> service is very simple, and just calls Apprise to send a notification:</p>

<div class="language-systemd highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">[Unit]</span>
<span class="nt">Description</span><span class="p">=</span>Send backup failure notification

<span class="k">[Service]</span>
<span class="nt">Type</span><span class="p">=</span>oneshot
<span class="nt">ExecStart</span><span class="p">=</span>apprise --body="Failed to create backup!" &lt;URL&gt;

<span class="k">[Install]</span>
<span class="nt">WantedBy</span><span class="p">=</span>multi-user.target
</code></pre></div></div>

<h1 id="the-aftermath-or-what-i-learned">The Aftermath (or what I learned)</h1>

<p>Because the error handling and alerting weren’t working propertly, my backups didn’t succeed for two weeks straight.
And, of course, you only notice your backups aren’t working when you actually need them.
This is exactly what happened: my disk was full and a MariaDB database crashed as a result of that.
Actually, the whole database seemed to be corrupt and I find it worrying MariaDB does not seem to be very resilient to failures (in comparison a PostgreSQL database was able to recover automatically).
I then tried to recover the data using last night’s backup, only to find out there was no such backup.
Fortunately, I had other means to recover the data so I incurred no data loss.</p>

<p>I already knew it is important to test backups, but I learned it is also important to test failures during backups!</p>]]></content><author><name>Pim Kunis</name></author><category term="backup" /><category term="borg" /><category term="borgmatic" /><summary type="html"><![CDATA[BorgBackup and Borgmatic have been my go-to tools to create backups for my home lab since I started creating backups. Using Systemd Timers, I regularly create a backup every night. I also monitor successful execution of the backup process, in case some error occurs. However, the way I set this up resulted in not receiving notifications. Even though it boils down to RTFM, I’d like to explain my error and how to handle errors correctly.]]></summary></entry><entry><title type="html">Using Ansible to alter Kernel Parameters</title><link href="https://pim.kun.is/ansible-edit-grub/" rel="alternate" type="text/html" title="Using Ansible to alter Kernel Parameters" /><published>2023-06-19T09:31:00+00:00</published><updated>2023-06-19T09:31:00+00:00</updated><id>https://pim.kun.is/ansible-edit-grub</id><content type="html" xml:base="https://pim.kun.is/ansible-edit-grub/"><![CDATA[<p>For months, I’ve had a peculiar problem with my laptop: once in a while, seemingly without reason, my laptop screen would freeze.
This only happened on my laptop screen, and not on an external monitor.
I had kind of learned to live with it as I couldn’t find a solution online.
The only remedy I had was reloading my window manager, which would often unfreeze the screen.</p>

<p>Yesterday I tried Googling once more and I actually found <a href="https://bbs.archlinux.org/viewtopic.php?id=246841">a thread</a> about it on the Arch Linux forums!
They talk about the same laptop model, the Lenovo ThinkPad x260, having the problem.
Fortunately, they also propose <a href="https://bbs.archlinux.org/viewtopic.php?pid=1888932#p1888932">a temporary fix</a>.</p>

<h1 id="trying-the-fix">Trying the Fix</h1>

<p>Apparently, a problem with the Panel Self Refresh (PSR) feature of Intel iGPUs is the culprit.
According to the <a href="https://github.com/torvalds/linux/blob/45a3e24f65e90a047bef86f927ebdc4c710edaa1/drivers/gpu/drm/i915/display/intel_psr.c#L42">Linux source code</a>, PSR enables the display to go into a lower standby mode when the sytem is idle but the screen is in use.
These lower standby modes can reduce power usage of your device when idling.</p>

<p>This all seems useful, except when it makes your screen freeze!
The proposed fix disables the PSR feature entirely.
To do this, we need to change a parameter to the Intel Graphics Linux Kernel Module (LKM).
The LKM for Intel Graphics is called <code class="language-plaintext highlighter-rouge">i915</code>.
There are <a href="https://wiki.archlinux.org/title/Kernel_parameters">multiple ways</a> to change kernel parameters, but I chose to edit my Grub configuration.</p>

<p>First, I wanted to test whether it actually works.
When booting into my Linux partition via Grub, you can press <code class="language-plaintext highlighter-rouge">e</code> to edit the Grub definition.
Somewhere there, you can find the <code class="language-plaintext highlighter-rouge">linux</code> command which specifies to boot Linux and how to do that.
I simply appended the option <code class="language-plaintext highlighter-rouge">i915.enable_psr=0</code> to this line.
After rebooting, I noticed my screen no longer freezes!
Success!</p>

<h1 id="persisting-the-fix">Persisting the Fix</h1>

<p>To make the change permanent, we need to permanently change Grub’s configuration.
One way to do this, is by changing Grub’s defaults in <code class="language-plaintext highlighter-rouge">/etc/default/grub</code>.
Namely, the <code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT</code> option specifies what options Grub should pass to the Linux kernel by default.
For me, this is a nice solution as the problem exists for both Linux OSes I have installed.
I changed this option to:</p>
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="py">GRUB_CMDLINE_LINUX_DEFAULT</span><span class="p">=</span><span class="s">"quiet splash i915.enable_psr=0"</span>
</code></pre></div></div>

<p>Next, I wanted to automate this solution using Ansible.
This turned out to be quite easy, as the Grub configuration looks a bit like an ini file (maybe it is?):</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Edit grub to disable Panel Self Refresh</span>
  <span class="na">become</span><span class="pi">:</span> <span class="kc">true</span>
  <span class="na">ini_file</span><span class="pi">:</span>
    <span class="na">path</span><span class="pi">:</span> <span class="s">/etc/default/grub</span>
    <span class="na">section</span><span class="pi">:</span> <span class="kc">null</span>
    <span class="na">option</span><span class="pi">:</span> <span class="s2">"</span><span class="s">GRUB_CMDLINE_LINUX_DEFAULT"</span>
    <span class="na">value</span><span class="pi">:</span> <span class="s1">'</span><span class="s">"quiet</span><span class="nv"> </span><span class="s">splash</span><span class="nv"> </span><span class="s">i915.enable_psr=0"'</span>
    <span class="na">no_extra_spaces</span><span class="pi">:</span> <span class="kc">true</span>
  <span class="na">notify</span><span class="pi">:</span> <span class="s">update grub</span>
</code></pre></div></div>

<p>Lastly, I created the <code class="language-plaintext highlighter-rouge">notify</code> hook to update the Grub configuration:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">update grub</span>
  <span class="na">become</span><span class="pi">:</span> <span class="kc">true</span>
  <span class="na">command</span><span class="pi">:</span>
    <span class="na">cmd</span><span class="pi">:</span> <span class="s">update-grub</span>
</code></pre></div></div>

<h1 id="update-just-use-nix">Update: Just use Nix</h1>

<p>Lately, I have been learning a bit of NixOS with the intention of replacing my current setup.
Compared to Ansible, applying this fix is a breeze on NixOS:</p>
<div class="language-nix highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
  <span class="nv">boot</span><span class="o">.</span><span class="nv">kernelParams</span> <span class="o">=</span> <span class="p">[</span> <span class="s2">"i915.enable_psr=0"</span> <span class="p">];</span>
<span class="p">}</span>
</code></pre></div></div>

<p>That’s it, yep.</p>

<h1 id="conclusion">Conclusion</h1>

<p>It turned out to be quite easy to change Linux kernel parameters using Ansible.
Maybe some kernel gurus have better ways to change parameters, but this works for me for now.</p>

<p>As a sidenote, I started reading a bit more about NixOS and realised that it can solve issues like these much more nicely than Ansible does.
I might replace my OS with NixOS some day, if I manage to rewrite my Ansible for it.</p>]]></content><author><name>Pim Kunis</name></author><category term="ansible" /><category term="grub" /><category term="linux" /><summary type="html"><![CDATA[For months, I’ve had a peculiar problem with my laptop: once in a while, seemingly without reason, my laptop screen would freeze. This only happened on my laptop screen, and not on an external monitor. I had kind of learned to live with it as I couldn’t find a solution online. The only remedy I had was reloading my window manager, which would often unfreeze the screen.]]></summary></entry><entry><title type="html">Sending Apprise Notifications from Concourse CI</title><link href="https://pim.kun.is/concourse-apprise-notifier/" rel="alternate" type="text/html" title="Sending Apprise Notifications from Concourse CI" /><published>2023-06-14T23:39:00+00:00</published><updated>2023-06-14T23:39:00+00:00</updated><id>https://pim.kun.is/concourse-apprise-notifier</id><content type="html" xml:base="https://pim.kun.is/concourse-apprise-notifier/"><![CDATA[<p>Recently, I deployed <a href="https://concourse-ci.org/">Concourse CI</a> because I wanted to get my feet wet with a CI/CD pipeline.
However, I had a practical use case lying around for a long time: automatically compiling my static website and deploying it to my docker Swarm.
This took some time getting right, but the result works like a charm (<a href="https://git.kun.is/pim/static">source code</a>).</p>

<p>It’s comforting to know I don’t have move a finger and my website is automatically deployed.
However, I would still like to receive some indication of what’s happening.
And what’s a better way to do that, than using my <a href="https://github.com/caronc/apprise">Apprise</a> service to keep me up to date.
There’s a little snag though: I could not find any Concourse resource that does this.
That’s when I decided to just create it myself.</p>

<h1 id="the-plagiarism-hunt">The Plagiarism Hunt</h1>

<p>As any good computer person, I am lazy.
I’d rather just copy someone’s work, so that’s what I did.
I found <a href="https://github.com/mockersf/concourse-slack-notifier">this</a> GitHub repository that does the same thing but for Slack notifications.
For some reason it’s archived, but it seemed like it should work.
I actually noticed lots of repositories for Concourse resource types are archived, so not sure what’s going on there.</p>

<h1 id="getting-to-know-concourse">Getting to know Concourse</h1>

<p>Let’s first understand what we need to do reach our end goal of sending Apprise notifications from Concourse.</p>

<p>A Concourse pipeline takes some inputs, performs some operations on them which result in some outputs.
These inputs and outputs are called <em>resources</em> in Concourse.
For example, a Git repository could be a resource.
Each resource is an instance of a <em>resource type</em>.
A resource type therefore is simply a blueprint that can create multiple resources.
To continue the example, a resource type could be “Git repository”.</p>

<p>We therefore need to create our own resource type that can send Apprise notifications.
A resource type is simply a container that includes three scripts:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">check</code>: check for a new version of a resource</li>
  <li><code class="language-plaintext highlighter-rouge">in</code>: retrieve a version of the resource</li>
  <li><code class="language-plaintext highlighter-rouge">out</code>: create a version of the resource</li>
</ul>

<p>As Apprise notifications are basically fire-and-forget, we will only implement the <code class="language-plaintext highlighter-rouge">out</code> script.</p>

<h1 id="writing-the-out-script">Writing the <code class="language-plaintext highlighter-rouge">out</code> script</h1>

<p>The whole script can be found <a href="https://git.kun.is/pim/concourse-apprise-notifier/src/branch/master/out">here</a>, but I will explain the most important bits of it.
Note that I only use Apprise’s persistent storage solution, and not its stateless solution.</p>

<p>Concourse provides us with the working directory, which we <code class="language-plaintext highlighter-rouge">cd</code> to:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> <span class="s2">"</span><span class="k">${</span><span class="nv">1</span><span class="k">}</span><span class="s2">"</span>
</code></pre></div></div>

<p>We create a timestamp, formatted in JSON, which we will use for the resource’s new version later.
Concourse requires us to set a version for the resource, but since Apprise notifications don’t have that, we use the timestamp:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">timestamp</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-n</span> <span class="s2">"{version:{timestamp:</span><span class="se">\"</span><span class="si">$(</span><span class="nb">date</span> +%s<span class="si">)</span><span class="se">\"</span><span class="s2">}}"</span><span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<p>First some black magic Bash to redirect file descriptors.
Not sure why this is needed, but I copied it anyways.
After that, we create a temporary file holding resource’s parameters.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">exec </span>3&gt;&amp;1
<span class="nb">exec </span>1&gt;&amp;2

<span class="nv">payload</span><span class="o">=</span><span class="si">$(</span><span class="nb">mktemp</span> /tmp/resource-in.XXXXXX<span class="si">)</span>
<span class="nb">cat</span> <span class="o">&gt;</span> <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span> &lt;&amp;0
</code></pre></div></div>

<p>We then extract the individual parameters.
The <code class="language-plaintext highlighter-rouge">source</code> key contains values how the resource type was specified, while the <code class="language-plaintext highlighter-rouge">params</code> key specifies parameters for this specific resource.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">apprise_host</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-r</span> <span class="s1">'.source.host'</span> &lt; <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>
<span class="nv">apprise_key</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-r</span> <span class="s1">'.source.key'</span> &lt; <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>

<span class="nv">alert_body</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-r</span> <span class="s1">'.params.body'</span> &lt; <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>
<span class="nv">alert_title</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-r</span> <span class="s1">'.params.title // null'</span> &lt; <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>
<span class="nv">alert_type</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-r</span> <span class="s1">'.params.type // null'</span> &lt; <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>
<span class="nv">alert_tag</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-r</span> <span class="s1">'.params.tag // null'</span> &lt; <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>
<span class="nv">alert_format</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span>jq <span class="nt">-r</span> <span class="s1">'.params.format // null'</span> &lt; <span class="s2">"</span><span class="k">${</span><span class="nv">payload</span><span class="k">}</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<p>We then format the different parameters using JSON:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">alert_body</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">eval</span> <span class="s2">"printf </span><span class="se">\"</span><span class="k">${</span><span class="nv">alert_body</span><span class="k">}</span><span class="se">\"</span><span class="s2">"</span> | jq <span class="nt">-R</span> <span class="nt">-s</span> .<span class="si">)</span><span class="s2">"</span>
<span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">alert_title</span><span class="k">}</span><span class="s2">"</span> <span class="o">!=</span> <span class="s2">"null"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nv">alert_title</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">eval</span> <span class="s2">"printf </span><span class="se">\"</span><span class="k">${</span><span class="nv">alert_title</span><span class="k">}</span><span class="se">\"</span><span class="s2">"</span> | jq <span class="nt">-R</span> <span class="nt">-s</span> .<span class="si">)</span><span class="s2">"</span>
<span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">alert_type</span><span class="k">}</span><span class="s2">"</span> <span class="o">!=</span> <span class="s2">"null"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nv">alert_type</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">eval</span> <span class="s2">"printf </span><span class="se">\"</span><span class="k">${</span><span class="nv">alert_type</span><span class="k">}</span><span class="se">\"</span><span class="s2">"</span> | jq <span class="nt">-R</span> <span class="nt">-s</span> .<span class="si">)</span><span class="s2">"</span>
<span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">alert_tag</span><span class="k">}</span><span class="s2">"</span> <span class="o">!=</span> <span class="s2">"null"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nv">alert_tag</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">eval</span> <span class="s2">"printf </span><span class="se">\"</span><span class="k">${</span><span class="nv">alert_tag</span><span class="k">}</span><span class="se">\"</span><span class="s2">"</span> | jq <span class="nt">-R</span> <span class="nt">-s</span> .<span class="si">)</span><span class="s2">"</span>
<span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">alert_format</span><span class="k">}</span><span class="s2">"</span> <span class="o">!=</span> <span class="s2">"null"</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="nv">alert_format</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">eval</span> <span class="s2">"printf </span><span class="se">\"</span><span class="k">${</span><span class="nv">alert_format</span><span class="k">}</span><span class="se">\"</span><span class="s2">"</span> | jq <span class="nt">-R</span> <span class="nt">-s</span> .<span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<p>Next, from the individual parameters we construct the final JSON message body we send to the Apprise endpoint.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">body</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">cat</span> <span class="o">&lt;&lt;</span><span class="no">EOF</span><span class="sh">
{
  "body": </span><span class="k">${</span><span class="nv">alert_body</span><span class="k">}</span><span class="sh">,
  "title": </span><span class="k">${</span><span class="nv">alert_title</span><span class="k">}</span><span class="sh">,
  "type": </span><span class="k">${</span><span class="nv">alert_type</span><span class="k">}</span><span class="sh">,
  "tag": </span><span class="k">${</span><span class="nv">alert_tag</span><span class="k">}</span><span class="sh">,
  "format": </span><span class="k">${</span><span class="nv">alert_format</span><span class="k">}</span><span class="sh">
}
</span><span class="no">EOF
</span><span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<p>Before sending it just yet, we compact the JSON and remove any values that are <code class="language-plaintext highlighter-rouge">null</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">compact_body</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">body</span><span class="k">}</span><span class="s2">"</span> | jq <span class="nt">-c</span> <span class="s1">'.'</span><span class="si">)</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$compact_body</span><span class="s2">"</span> | jq <span class="s1">'del(..|nulls)'</span> <span class="o">&gt;</span> /tmp/compact_body.json
</code></pre></div></div>

<p>Here is the most important line, where we send the payload to the Apprise endpoint.
It’s quite straight-forward.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-v</span> <span class="nt">-X</span> POST <span class="nt">-T</span> /tmp/compact_body.json <span class="nt">-H</span> <span class="s2">"Content-Type: application/json"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">apprise_host</span><span class="k">}</span><span class="s2">/notify/</span><span class="k">${</span><span class="nv">apprise_key</span><span class="k">}</span><span class="s2">"</span>
</code></pre></div></div>

<p>Finally, we print the timestamp (fake version) in order to appease the Concourse gods.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"</span><span class="k">${</span><span class="nv">timestamp</span><span class="k">}</span><span class="s2">"</span> <span class="o">&gt;</span>&amp;3
</code></pre></div></div>

<h1 id="building-the-container">Building the Container</h1>

<p>As said earlier, to actually use this script, we need to add it to a image.
I won’t be explaining this whole process, but the source can be found <a href="https://git.kun.is/pim/concourse-apprise-notifier/src/branch/master/pipeline.yml">here</a>.
The most important take-aways are these:</p>
<ul>
  <li>Use <code class="language-plaintext highlighter-rouge">concourse/oci-build-task</code> to build a image from a Dockerfile.</li>
  <li>Use <code class="language-plaintext highlighter-rouge">registry-image</code> to push the image to an image registry.</li>
</ul>

<h1 id="using-the-resource-type">Using the Resource Type</h1>

<p>Using our newly created resource type is surprisingly simple.
I use it for the blog you are reading right now and the pipeline definition can be found <a href="https://git.kun.is/pim/static/src/branch/main/pipeline.yml">here</a>.
Here we specify the resource type in a Concourse pipeline:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">resource_types</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">apprise</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">registry-image</span>
  <span class="na">source</span><span class="pi">:</span>
    <span class="na">repository</span><span class="pi">:</span> <span class="s">git.kun.is/pim/concourse-apprise-notifier</span>
    <span class="na">tag</span><span class="pi">:</span> <span class="s2">"</span><span class="s">1.1.1"</span>
</code></pre></div></div>

<p>We simply have to tell Concourse where to find the image, and which tag we want.
Next, we instantiate the resource type to create a resource:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">resources</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">apprise-notification</span>
  <span class="na">type</span><span class="pi">:</span> <span class="s">apprise</span>
  <span class="na">source</span><span class="pi">:</span>
    <span class="na">host</span><span class="pi">:</span> <span class="s">https://apprise.kun.is:444</span>
    <span class="na">key</span><span class="pi">:</span> <span class="s">concourse</span>
  <span class="na">icon</span><span class="pi">:</span> <span class="s">bell</span>
</code></pre></div></div>

<p>We simply specify the host to send Apprise notifications to.
Yeah, I even gave it a little bell because it’s cute.</p>

<p>All that’s left to do, is actually send the notification.
Let’s see how that is done:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">deploy-static-website</span>
  <span class="na">plan</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">task</span><span class="pi">:</span> <span class="s">deploy-site</span>
      <span class="na">config</span><span class="pi">:</span> <span class="s">...</span>

      <span class="na">on_success</span><span class="pi">:</span>
<span class="err">	</span><span class="na">put</span><span class="pi">:</span> <span class="s">apprise-notification</span>
<span class="na">	params</span><span class="pi">:</span>
<span class="err">	</span>  <span class="na">title</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Static</span><span class="nv"> </span><span class="s">website</span><span class="nv"> </span><span class="s">deployed!"</span>
<span class="err">	</span>  <span class="na">body</span><span class="pi">:</span> <span class="s2">"</span><span class="s">New</span><span class="nv"> </span><span class="s">version:</span><span class="nv"> </span><span class="s">$(cat</span><span class="nv"> </span><span class="s">version/version)"</span>
<span class="err">	</span><span class="na">no_get</span><span class="pi">:</span> <span class="kc">true</span>
</code></pre></div></div>

<p>As can be seen, the Apprise notification can be triggered when a task is executed successfully.
We do this using the <code class="language-plaintext highlighter-rouge">put</code> command, which execute the <code class="language-plaintext highlighter-rouge">out</code> script underwater.
We set the notification’s title and body, and send it!
The result is seen below in my Ntfy app, which Apprise forwards the message to:
<img src="ntfy.png" alt="picture showing my Ntfy app with the Apprise notification" /></p>

<p>And to finish this off, here is what it looks like in the Concourse web UI:
<img src="pipeline.png" alt="the concourse web gui showing the pipeline of my static website including the the apprise notification resources" /></p>

<h1 id="conclusion">Conclusion</h1>

<p>Concourse’s way of representing everything as an image/container is really interesting in my opinion.
A resource type is quite easily implemented as well, although Bash might not be the optimal way to do this.
I’ve seen some people implement it in Rust, which might be a good excuse to finally learn that language :)</p>

<p>Apart from Apprise notifications, I’m planning on creating a resource type to deploy to a Docker swarm eventually.
This seems like a lot harder than simply sending notifications though.</p>]]></content><author><name>Pim Kunis</name></author><category term="concourse" /><category term="apprise" /><summary type="html"><![CDATA[Recently, I deployed Concourse CI because I wanted to get my feet wet with a CI/CD pipeline. However, I had a practical use case lying around for a long time: automatically compiling my static website and deploying it to my docker Swarm. This took some time getting right, but the result works like a charm (source code).]]></summary></entry></feed>