Shane Howearth

Dead Letter Queues

Shane — Sun, 08 Mar 2026 23:38:46 GMT

A Dead Letter Queue (DLQ) is an architectural pattern for handling messages or events that a system cannot process after multiple attempts.

Why is it named a Dead Letter Queue (DLQ)?

The physical mail in the snail mail/Postal service would occasionally encounter mail that was undeliverable - usually because the address was incorrect, illegible, or for some other reason.

What should happen to that mail? It got put into a bin/container where it could be dealt with later.

Why are DLQs used?

Standard resilience patterns, like retries, are perfect for handling transient failures - the quick network blips or short-term service degradations that resolve themselves. However, when a failure becomes permanent (exceeding your SLA or retry limits), you need a robust fallback strategy. This is where the DLQ comes in: it captures the message for later action while allowing you to surface a clean error to the user.

In the past, these failures were often just logged. The problem is that logs are easily buried in a flood of other data; if no one is manually auditing them, a failure can go unnoticed indefinitely. Moving these events to a DLQ ensures they aren’t just recorded, but stored in a dedicated location for resolution.

While it’s tempting to make every failure a pageable event, context matters. Nobody wants to be woken up at 2:00 AM because of a user’s malformed input. A DLQ allows for a more nuanced approach to on-call rotations:

Internal Failures: If your own upstream service fails, it may warrant a page.

Third-Party Failures: If an external vendor returns a 500 error or a “payment required” status, your support team often can’t fix it immediately.

Instead of triggering unnecessary alarms, the best strategy is to route these “un-processable” messages to the DLQ. This provides a central “to-do list” that your team can review and manage at the start of the next business day.

Common mistakes

A DLQ is great at capturing the fact that events have become unprocessable, but that’s of little use if the DLQ is never checked (it’s easy to “fire and forget”, messages being passed to the DLQ but never handled).

Keeping the failed messages forever - there should be a policy to remove the events after some given amount of time.

Overusing the DLQ - The DLQ is for messages that cannot be processed, it’s not a replacement for log messages that might be detailing poor use of the service.

The rise of "AI" (LLMs)

Shane — Sun, 17 Aug 2025 01:56:49 GMT

For anyone living under a rock, there has been major advances in AI technology over the past few years.

The initial release of ChatGPT (November 2022), led to wild claims about the value of AI, and the demise of various industries. As time has progressed, a lot of that hype has so far failed to be realised. Having said that, some of the anti-AI hysteria has also proved to be overdone.

As I write this, know that there is currently a “race” going on where a lot of money is being spent on finding ways to integrate these LLMs into products that serve us better. This post isn’t going to address that aspect (I’m not much of a technology forecaster, what’s going to happen is going to happen, but how it will look is beyond me).

What this post will focus on is how I am finding using AI in my day to day activity as a Software Engineer.

Let me further preface this by saying that I have been writing software (professionally) for the better part of the last 15 years, longer if you count before I went to University.

Because of the length of time that I have been in “the game” I have seen a few things come and go, and I will use that experience to evaluate the AI tools.

I have been using two LLMs over the past couple of months that are free to use, and “do the job” - they are Gemini and Claude - I have not been using ChatGPT, even though they have been”leading the race” and have released a new version (5.0).

The first thing to say, I use AI as a “pairing partner” and to help me “search” for answers - the LLMs have a better Natural Language Processing ability that “understands” my query than a query submitted to Google.

This means that I will get answers that I would get from sites like StackOverflow, or blog posts, or whatever else Google would find for me. The knowledge isn’t “knew”, it’s just (IMO) easier to access and/or find.

Subjectively I can say that I have found Claude to marginally “better” than Gemini, it has been producing less hallucinated code examples for me. I have had Gemini provide me example code that used incorrect types in function calls.

To give a better understanding, I am using TView in my version of Solitaire that runs in a terminal.

At one point Gemini suggested that the function it was proposing I use to solve an issue required a tview.Key, which doesn’t exist within the library. What was actually required was a tcell.Key (defined in TCell).

In general I have been able to do things like “paste all of my project code ino a Claude prompt” and ask Claude “Why is this broken”, and got a lis of issues to look at.

I have also had Claude help me immensely by telling me how to rearchitect my code (after I asked it to), which produced this commit (Note: I edited the code as suggested by Claude, and I found that asking Claude to present it’s suggestions as a diff made my life a LOT easier)

However there have been a couple of notable issues with both Claude and Gemini that I feel need raising. I had the experience of an issue with an incorrect cell being highlighted when clicked. The LLMs both suggested three possible issues, that proved to be (at least partially) wrong.

Further, I would tell the LLMs that they were wrong, and provide evidence, they would claim to accept that they had erred, but, some moments later… give me the same “solution” to the problem again.

Quite infuriating.

I eventually found a fix for the problem I was having, but that was through my own creativity, not due to any help from the LLMs (or Google for that matter).

I have found that I treat the LLMs as “pairing partners” - I can ask them for their input, and they give me their /opinions/ freely. I still have to use my experience as a developer, and my intuition to help me realise whether to use the advice, or not, but I have the same filter working for me when I interact with humans.

One thing that I do find the LLMs are superior than me at is rewording comments, and creating names. I had a linter complain that the comment I was using in a //nolint directive wasn’t good enough - but it gave no indication of why or how to fix it.

The following is the transcript from Claude

can you improve the comment on this for gocritic

// nolint: mnd // The magic numbers in here are too magic.

Edit
Here are some better comments that actually explain why you're disabling the magic number detector:

Option 1: Explain the context

go
//nolint:mnd // Card game constants (suits, ranks, deck size) are self-documenting
Option 2: Explain why constants aren't better

go
//nolint:mnd // Small numeric literals for UI positioning are clearer inline than as named constants
Option 3: Domain-specific explanation

go
//nolint:mnd // Standard card values (13 ranks, 4 suits, 52 cards) are universally understood
Option 4: Performance/readability trade-off

go
//nolint:mnd // Simple numeric values used once don't warrant separate constant declarations
Option 5: Configuration/layout explanation

go
//nolint:mnd // Layout dimensions and offsets are more readable as literal values
The key is to explain why magic numbers are acceptable in this specific context, not just acknowledge that they exist. Choose the option that best matches what your code is actually doing.

What kind of magic numbers are you dealing with? That would help me suggest the most appropriate comment.

I chose the final suggestion, and the linter was happy. I honestly wouldn’t have thought to come up with that.

Hopefully it’s clear to you that the LLMs are usable, and helpful, but only to a point. As a user I still need to understand, and sometimes verify, what the LLM is telling me. There’s no “blind acceptance”. It’s easy, if you don’t pay attention, for the LLMs to create problems.

The wonders of git

Shane — Wed, 25 Sep 2024 09:16:42 GMT

I’ve been wanting to write about git for a very long time. It’s a tool used for managing versions of software (but it can be used for other things).

One of the reasons that I haven’t written much about git is that, quite frankly, there’s a lot to learn.

I use git from the command line, because it’s the most flexible way I know, and I am about to show an example of what I had to do today, which will demonstrate git’s strength (I hope).

Before I start, if you’re unfamiliar with the terminology used here, I recommend reading the first 3 or 4 chapters of the Git book (it’s well worth the effort).

Right, some context. Today I was working on creating a topic branch because I had a change to write for the code at work. As I worked on the change I realised that I really needed to break the branch up into two branches, one with a large amount of change that prepared the way for the actual change that I was working on. Reviewing the two changes togather would be tiresome, and might lead to things being missed.

I created the preparation branch, with a commit inside it, and pushed it to our shared repository.

I then pushed the remaining change in the topic branch to the same repository.

Our CI runs on every branch, triggered by the push, and I duly got an email saying that tests had failed on the preparation branch, and I realised that I had included some of the change meant for the preparation branch in the topic branch. And this is what I did to get myself into a position where the preparation branch had the code it needed, and the topic branch no longer did.

git checkout <topic branch>
git reset HEAD~1
git add -p
<add the stuff meant for the preparation branch>
git commit -m "move me"
git commit -m "the rest" .
git log -n2
<grab the sha1>
git checkout <preparation branch>
git cherry-pick <saved sha1>
git rebase -i HEAD~2
<squash the two commits>
git push --force
git checkout <topic branch>
git rebase <preparation branch>
git push --force

Let’s break that down into (hopefully) manageable bites.

1) git checkout <topic branch>

This tells git which branch it is working on. Work on one branch at a time. Git will apply the commands to this branch, until told otherwise.

2) git reset HEAD~1

This “undoes” the last commit. But it’s a soft reset, meaning that the changes in that commit are now unstaged.

3) git add -p

(add the stuff meant for the preparation branch)

Git allows you to add parts of a change, hunk by hunk, and you can edit what is being added using this command.

4) git commit -m "move me"; git commit -m "the rest" .

These two commands commit the changes, the first is a commit comprised of what I had staged in the add -p step above, the second commits the rest. The -m tells git to use the message supplied, instead of going into the editor and mucking about.

5) git log -n2

I used this to show me a synopsis of the two commits that I had just created, and allowed me to copy the sha1 hash of the commit that I had named “move me” which contained the changes that I wanted to move.

6) git checkout <preparation branch>

Git is now applying the commands to the preparation branch.

7) git cherry-pick <saved sha1>

This moves the commit from the topic branch to the preparation branch.

8) git rebase -i HEAD~2

Moving into interactive mode, git willapply some rebase commands, in this case I squashed the two commits together (the existing commit in the preparation branch, and the commit that I just moved to this branch).

9) git push --force

Updating the remote repository with the new state of the preparation branch. The push must be forced because of the change to the commit made in the previous step.

10) git checkout <topic branch>

Git is now applying the commands to the topic branch.

11) git rebase <preparation branch>

This tells git to move the topic branch such that it is branched from the new tip of the preparation branch.

12) git push --force

The previous command had changed the branch in such a way that the remote repository needed to be forced to accept the new understanding of this branch.

That was a LOT of fun to actually do :)

Static site hosting on S3

Shane — Wed, 01 May 2024 04:21:31 GMT

A static site is one where there is no server side computation required, no database lookups, just ask for a resource by name, and get it back.

You can do client side computations with javascript, but not server side.

There are three parts to hosting a static site on AWS, the storage of the objects, the policy to access the objects, and the name used to access the objects.

S3 itself does not support https, if you want your static site to be accessed using https then use CloudFront.

With that out of the way let’s create a simple static site that is hosted on an S3 bucket on AWS.

From the AWS Console.

Search for S3 and click on the S3 link.

Click the “Create bucket” button

And give the new bucket a name (My example uses “random-name-12345678”, the rule for bucket names is that they must be unique across an AWS partition, be between 3 and 63 characters long, consist only of lowercase letters, numbers, dots, and hyphens, but dot usage isn’t recommended, and begin and end with a letter)

Select the new bucket from the list of buckets

Upload your files to the bucket. An example might be an index.html

<html>
  <head><title>Hello AWS</title></head>
  <body>
    <h1>Rock and/or Roll!</h1>
  </body>
</html>

Click the “Upload” and “Close” buttons to upload the files, then close the dialog once complete.

The task now is to make the objects within the bucket available for all to access. For the record, there is no way to have some objects in a bucket private and others public, so do NOT store any objects intended to be private in this bucket, public only.

Select the “Permissions” tab

Select the “edit” button in the “Block Public Access” box, and clear all of the checkboxes, then “Save Changes”

A policy is required to allow the objects to be accessed as well. -

Click the “Add New Statement” button so that we can add the following policy (replacing the defaulted in policy):

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Sid": "Only allow writes to my bucket with bucket owner full control",
         "Effect": "Allow",
         "Principal": {
            "AWS": [
               "arn:aws:iam::111122223333:user/ExampleUser"
            ]
         },
         "Action": [
            "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::random-name-12345678/*",
         "Condition": {
            "StringEquals": {
               "s3:x-amz-acl": "bucket-owner-full-control"
            }
         }
      }
   ]
}

And now you can access the site from the following URL - note that this example has long since been deleted, so you will need to determine the URL for your bucket, it’s a simple case of bucket name.AWS region.amazonaws.com

https://random-name-12345678.s3.us-east-1.amazonaws.com/index.html

Of course, if you want to have a friendlier name you will need to work with Route53, but that is the subject of another blog post :-)

The power of the index

Shane — Mon, 08 Apr 2024 10:03:14 GMT

A discussion came up on IRC today, someone was asking why a query was being done the way it was, by Postgres. The person had looked at the explain function of Postgres for a given query, and the planner showed how the query was going to be actioned.

The query was such

SELECT COUNT(*) FROM one, two
WHERE one.id = two.id AND d >= 56 AND d <= 69 AND a > 110;

The planner explained the query plan thus

Aggregate (cost=1365.77..1365.77 rows=1 width=0) (actual time=5.250..5.250 rows=1 loops=1)
  -> Nested Loop (cost=0.00..1365.12 rows=250 width=0) (actual time=0.034..5.181 rows=230 loops=1)
    -> Seq Scan on two (cost=0.00..180.00 rows=391 width=4) (actual time=0.014..3.737 rows=388 loops=1)
      Filter: (a > 110)
    -> Index Scan using one_id_idx on one (cost=0.00..3.02 rows=1 width=4) (actual time=0.003..0.003 rows=1 loops=388)
      Index Cond: (one.id = "outer".id)
      Filter: ((d >= 56) AND (d <= 69))
Total runtime: 5.339 ms

So, what’s happening? The query planner has done one loop on table two and 388 loops on the index of table one.

The question is, why did the planner choose that as the best way to implement the query?

This is where understanding an index is so important.

First I will explain why the query planner chose this route, then I will explain a bit more about indexes.

In the first place, the query planner had two create subsets of each table, rows from table one had to have field d between 56 and 69, and rows from table two had to have a field a greater than 110.

That created two sets with the following number of rows each.

SELECT COUNT(*) FROM one WHERE d >=56 AND d <=69;
 count
----------
   6544
(1 row)

SELECT COUNT(*) FROM two WHERE a > 119;
 count
----------
    388
(1 row)

Next the join - the part of the query that says

WHERE one.id = two.id

There are two choices to be made, for the same algorithm. That is, one set must have every one of its items searched for in the second set.

The cost of the query will be

O(n)*O(log2(m))

The reason for the cost - every item in one set = O(n) (Where n is the number of items in the first set)

Searching for the item in the other set O(log2(m)) (where m is the number of items in the second set)

A search is O(log2(n)) (where n is the number of rows in the set being searched) because the fastest way that we know how to search is binary search of a sorted list, that is, we look at the midpoint of a sorted list, and if our item is greater than the value held at the midpoint, we then search the top half of the list, and if our item is less than the value held at the midpoint we search the bottom half. And so on until we either find the item, or there is nothing left to search. Because the number of items being searched each time is reduced by half, we end up with a total number of times that we searched (worst case) of log2(n), the logarithm with a base of 2 of n.

An index is a sorted list (and that’s all an index is), so searching an index is going to cost O(log(n)) (where n is the number of rows in the set being searched).

The planner has chosen to iterate over every item in the smaller set two which has 388 items, and will search the larger set one which has 6544 items.

The reason why can be seen in the following math.

Search the larger set for the keys in the smaller set

log2(6544) = 12.67596
multiplied by the number of searches
388 * 13 = 5044 (we cannot do a partial operation)

Whereas searching the smaller set for the items in the larger set:

log2(388) = 8.5999
multiplied by the number of searches
6544 * 9 = 58896 (we cannot do a partial operation)

If the planner had iterated over every item in the larger set, searching the smaller set, there would have (worst case) been 58896 operations.

However, by iterating over every item in the smaller set, there are only (worst case) 5044 operations.

Summary

By having an index to search, what might have been a full table scan (O(n)), becomes a binary search (O(log2(n))).

How I design the layout of a Go program's code on disk

Shane — Sat, 23 Mar 2024 03:47:07 GMT

The layout of a Go application on disk is a question that almost everyone asks when starting out.

It’s a fair question - there’s no scaffolding that’s widely used, there was no guidelines for people for the first decade.

One of the biggest strengths of Go is that you can lay your project out on disk pretty much how you want to.

Unfortunately, though, that strength is its weakness, people lay their projects out on disk pretty much how they want to, leaving people new to the ecosystem scratching their heads.

The official Go blog did finally create a blog post that put some ideas out there for people to use.

Here’s howI think about the layout.

In the first place, the project, for example, has a root directory, that will have the README.md and go.mod and go.sum files in it.

myproject/
├── go.mod
├── go.sum
└── README.md

The project itself has a “core” library, which is typically the business logic for the project, so I put that at the root of the project.

The package that these files are in is going to be myproject as well.

myproject/
├── go.mod
├── go.sum
├── myproject.go
├── my_project_test.go
└── README.md

To build the project into a binary there needs to be a main package. There is a de facto standard (meaning everyone uses this standard, but it’s not an actual standard), and that is to put the main package in a directory named cmd.

myproject/
├── cmd
│   └── main.go
├── go.mod
├── go.sum
├── myproject.go
├── my_project_test.go
└── README.md

The next step in my thought process is “supporting libraries”, that is, libraries that my business logic will need in order to do its tasks.

I put these supporting libraries beneath the core library, so that all developers can see that the library at the top of the tree is supported by the libraries below.

There’s a little bit extra happening here, as a firm believer in the SOLID design principles - I create an interface that is owned by the core library that defines the contract that my datastores need to abide by to be used.

For this example the interface will be in feed.go, and the feeds directory holds the code that creates types that satisfy that interface and allow the core library to interact with the upstream data feeds (each in their own directory).

myproject/
├── cmd
│   └── main.go
├── feeds
│   ├── twitter
│   │   ├── twitter.go
│   │   └── twitter_test.go
│   └── reddit
│       ├── reddit.go
│       └── reddit_test.go
├── feed.go
├── go.mod
├── go.sum
├── myproject.go
├── my_project_test.go
└── README.md

I used to put the feed.go file, with interface definition, within the feeds directory, to show that that wasthe interface that the feeds need to satisfy in order to be used by the core library, but having the definition in the core library demonstrates that the core library owns the interface (which it should).

This should be enough to demonstrate the layout of a project, with the only other thing to mention being that if the supporting libraries themselves need supporting libraries, then those secondary support libraries should go below the libraries that they support.

Kubernetes initContainers vs Jobs

Shane — Sat, 06 Jan 2024 00:05:19 GMT

There are tasks that need to take place within your cluster on an ad hoc basis. These tasks tend to have short lifetimes, an example might be “updating the schema in a database”.

Kubernetes offers two pathways to achieve this goal, both very similar, but differ in very subtle ways.

Jobs, and init containers.

Both of these pathways execute the same tasks, and, as will be seen further down, they both have similar definitions in manifests.

Jobs

A Kubernetes Job will execute a task as soon as it is loaded into the cluster.

This is the manifest I have for a container that I have defined elsewhere to have a certain set of commands be applied to it (basically, I have a container with Goose and netcat in it, an image of which is sent to my container repository, and this job ‘runs’ it with the commands).

The key things to note are that it’s a job, it has a name, a container image that it will interact with, some environment variables to be set on that container, and a set of commands that will be run on that container.

apiVersion: batch/v1
kind: Job
metadata:
  name: accounts-schema-init
  namespace: "$NAMESPACE"
spec:
  template:
    metadata:
      name: accounts-schema
    spec:
      containers:
      - name: accounts-schema
        image: "onepage/accounts-schema:$TAG"
        imagePullPolicy: "$IMAGEPULLPOLICY"
        env:
        - name: PGUSER
          valueFrom:
            secretKeyRef:
              name: account-secret
              key: username
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: account-secret
              key: password
        - name: PGDBNAME
          value: "$DBNAME"
        - name: PGHOST
          value: "$HOST"
        command:
        - /bin/sh
        - -c
        - |
          while ! nc -z "$PGHOST" "5432"; do
            echo "Waiting for $PGHOST:5432 to become available..."
            sleep 5
          done
          goose -v -dir=migrations postgres "user=$PGUSER dbname=$PGDBNAME password=$PGPASSWORD host=$PGHOST sslmode=disable" up;
      restartPolicy: Never

Init containers

A bit of background first. A Pod will have one or more containers in it that will run. However before those containers are loaded and run on that pod an init container can be run on that pod (and will exit before the main container(s) will run).

This means that when the deployment of the pod takes place then the init container will run in each pod before the main container(s) in the pod do, and we can run something very similar to the above job on the init container.

Note the manifest below, the key things are: It’s a deployment, with a name, a container image that it will work with (for the init container) as well as the main container, and a sidecar container (I know that it muddies things a bit to have it in the manifest, but it demonstrates that the containers in the pod each have different purposes).

The initContainer specification is almost identical to the job above.

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: "$NAMESPACE"
  name: accountserver
  labels:
    app: account
spec: # this is the specification of the deployment
  replicas: $REPLICAS
  selector:
    matchLabels:
      app: account
  template:
    metadata:
      annotations: # inject linkerd into this pod.
        linkerd.io/inject: enabled
      labels:
        app: account
    spec: # this is the specification of the pods.
      initContainers: # container that will run when this pod is initialised.
      - name: accounts-schema
        image: "onepage/accounts-schema:$TAG"
        imagePullPolicy: "$IMAGEPULLPOLICY"
        env:
        - name: PGUSER
          valueFrom:
            secretKeyRef:
              name: account-secret
              key: username
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: account-secret
              key: password
        - name: PGDBNAME
          value: "$DBNAME"
        - name: PGHOST
          value: "$HOST"
        command:
        - /bin/sh
        - -c
        - |
          while ! nc -z "$PGHOST" "5432"; do
            echo "Waiting for $PGHOST:5432 to become available..."
            sleep 5
          done
          goose -v -dir=migrations postgres "user=$PGUSER dbname=$PGDBNAME password=$PGPASSWORD host=$PGHOST sslmode=disable" up;
      containers:
      - name: account
        image: onepage/accounts
        imagePullPolicy: "$IMAGEPULLPOLICY"
        ports:
        - containerPort: $TARGETPORT
        env:
        - name: PORT_NUM
          value: "$TARGETPORT"

When to use which.

If they’re doing the same thing, why use one or the other? Or, When should one approach be used over the other?

In this case, migrating the schema of a database, we would use the initContainers because it means that the coupling of the schema to a version of the application container is explicit, and, that when a new version of a container is rolled out that the schema will be in a proscribed state before that new container will begin its work.

However, there are times when out of band updates are required (eg. “Oh god, I rolled out a change to the cluster that needs to be undone without rolling out a new deployment” - nobody has ever encountered that… have they :). At that point a Job is appropriate, pushing it into the cluster has an immediate effect.

envsubstr

Shane — Thu, 04 Jan 2024 21:41:49 GMT

I’m writing a lot of yaml files at the moment, mostly manifests for kubernetes. Part of that task involves substituting variables into the yaml, because they change depending on the environment.

As an example, this manifest that kubernetes will use to create a namespace has a variable in it that allows me to use the name that I think is appropriate for the environment.

apiVersion: v1
kind: Namespace
metadata:
    name: "$NAMESPACE"

To set that $NAMESPACE variable to something useful I use a command called envsubstr

From the shell I would do something like

NAMESPACE=production envsubst < k8s/namespace.yaml | kubectl apply -f -

Which means that the information that gets passed to kubernetes would look something like

apiVersion: v1
kind: Namespace
metadata:
    name: "production"

If that’s all there was to know, this would be a very short blog post, but wait, there’s more!

envsubst will substitute all the things it sees that it thinks are variables. That is, if the example yaml looked like this

apiVersion: v1
kind: Namespace
metadata:
    name: "$NAMESPACE"
    labels:
        name: "$NAME"

And we ran our previous example command

NAMESPACE=production envsubst < k8s/namespace.yaml | kubectl apply -f -

Then envsubst will substitute both ‘variables’, but only one will have a value - creating

apiVersion: v1
kind: Namespace
metadata:
    name: "production"
    labels:
        name: ""

Which may not be what we want. We might only want the $NAMESPACE variable to be changed, and any other variables to be left alone (I will show a full example of a manifest that makes use of this at the bottom, but continue with this small example to keep focus on what’s happening).

To achieve this we explicitly tell envsubstr which variables to change - putting them in a string after the call, and surrounding the name of each with {}

NAMESPACE=production envsubst '${NAMESPACE}' < k8s/namespace.yaml | kubectl apply -f -

If there are multiple it would look something like

NAMESPACE=production envsubst '${NAMESPACE} ${FOO} ${BAR}' < k8s/namespace.yaml | kubectl apply -f -

And people can make bash expansions that look for prefixes and so on (but that’s well above my paygrade ;)

what gets produced by envsubstr is:

apiVersion: v1
kind: Namespace
metadata:
    name: "production"
    labels:
        name: "$NAME"

One more thing, gnu make has to use a slightly different syntax in the list, it needs a double $$ .

So, in a Makefile use

NAMESPACE=production envsubst '$${NAMESPACE}' < k8s/namespace.yaml | kubectl apply -f -

Finally, the example where I want some substitution to take place, and some variables to be left behind.

The reason is that I am building a command that I want to use the environment variables that are set inside the docker container’s shell.

You can see that I prefix the variables to be set in the env with PG, that’s purely so that I can differentiate them, there’s no other rhyme or reason.

So what I need to happen is my Makefile sets some of the variables, but I don’t want it to set some others to “” because they will be set by the shell when the command is run (and if I leave envsubst to its own devices, it deletes the variables, meaning the shell doesn’t know they’re even there(.

apiVersion: batch/v1
kind: Job
metadata:
  name: accounts-schema-init
  namespace: "$NAMESPACE"
spec:
  template:
    metadata:
      name: accounts-schema
    spec:
      containers:
      - name: accounts-schema
        image: "onepage/accounts-schema:$TAG"
        imagePullPolicy: "$IMAGEPULLPOLICY"
        env:
        - name: PGUSER
          valueFrom:
            secretKeyRef:
              name: account-secret
              key: username
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: account-secret
              key: password
        - name: PGDBNAME
          value: "$DBNAME"
        - name: PGHOST
          value: "$HOST"
        command:
        - /bin/sh
        - -c
        - goose -v -dir=migrations postgres "user=$PGUSER dbname=$PGDBNAME password=$PGPASSWORD host=$PGHOST sslmode=disable" up;
      restartPolicy: Never

Summary

Hopefully this gives you (almost) everything that you need to use envsubstr for substituting variables in files, and only the ones you want substituted.

kubectl wait

Shane — Mon, 18 Dec 2023 08:10:18 GMT

After writing a few million scripts over the years,I’ve learnt a thing or two (I hope). One of those things is how painful it is to wait for some condition before continuing with the next step in a script.

I’ve used expect scripts, I’ve added sleep calls.

I’ve tried a few other things too. It’s never fun :(

kubectl comes with a wait option.

You can ask kubectl to wait for any number of conditions eg:

Wait for a namespace named shane to become Active

$ kubectl wait --for jsonpath='{.status.phase}=Active' --timeout=5s namespace/shane

Wait for a deployment to become Available

$ kubectl wait deployment -n cnpg-system cnpg-controller-manager --for condition=Available=True --timeout=210s

Note that the condition can be specified using the jsonpath, or, in some cases, by name.

The timeout is a maximum time to wait for the condition to become true.

Finding the condition can be difficult. Typically it’s a matter of digging through .yaml files looking for selectors, or, getting whatever you want running, then inspecting it. eg

$ kubectl get  pods -n shane
NAME                            READY   STATUS    RESTARTS   AGE
accountserver-c8897d469-js9qs   1/1     Running   0          17m
accountserver-c8897d469-mr7jn   1/1     Running   0          17m
accountserver-c8897d469-xdzpn   1/1     Running   0          17m
apiserver-57474fd944-7zxmb      1/1     Running   0          18m
postgres-1                      1/1     Running   0          22m
postgres-2                      1/1     Running   0          21m
postgres-3                      1/1     Running   0          19m

Followed by getting all the metadata on the object, in json format

 $ kubectl get  pods postgres-1 -n shane -o json
{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": {
        "annotations": {
            "cnpg.io/nodeSerial": "1",
            "cnpg.io/operatorVersion": "1.21.0",
            "cnpg.io/podEnvHash": "fcffbc754",
            "cnpg.io/podSpec": "{\"volumes\":[{\"name\":\"pgdata\",\"persistentVolumeClaim\":{\"claimName\":\"postgres-1\"}},{\"name\":\"scratch-data\",\"emptyDir\":{}},{\"name\":\"shm\",\"emptyDir\":{\"medium\":\"Memory\"}},{\"name\":\"app-secret\",\"secret\":{\"secretName\":\"account-secret\"}}],\"initContainers\":[{\"name\":\"bootstrap-controller\",\"image\":\"ghcr.io/cloudnative-pg/cloudnative-pg:1.21.0\",\"command\":[\"/manager\",\"bootstrap\",\"/controller/manager\",\"--log-level=info\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/run\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"app-secret\",\"mountPath\":\"/etc/app-secret\"}],\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}],\"containers\":[{\"name\":\"postgres\",\"image\":\"ghcr.io/cloudnative-pg/postgresql:16.0\",\"command\":[\"/controller/manager\",\"instance\",\"run\",\"--log-level=info\"],\"ports\":[{\"name\":\"postgresql\",\"containerPort\":5432,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9187,\"protocol\":\"TCP\"},{\"name\":\"status\",\"containerPort\":8000,\"protocol\":\"TCP\"}],\"env\":[{\"name\":\"PGDATA\",\"value\":\"/var/lib/postgresql/data/pgdata\"},{\"name\":\"POD_NAME\",\"value\":\"postgres-1\"},{\"name\":\"NAMESPACE\",\"value\":\"shane\"},{\"name\":\"CLUSTER_NAME\",\"value\":\"postgres\"},{\"name\":\"PGPORT\",\"value\":\"5432\"},{\"name\":\"PGHOST\",\"value\":\"/controller/run\"}],\"resources\":{},\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/run\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"app-secret\",\"mountPath\":\"/etc/app-secret\"}],\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8000},\"timeoutSeconds\":5,\"periodSeconds\":10},\"readinessProbe\":{\"httpGet\":{\"path\":\"/readyz\",\"port\":8000},\"timeoutSeconds\":5,\"periodSeconds\":10},\"startupProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8000},\"timeoutSeconds\":5,\"periodSeconds\":10,\"failureThreshold\":360},\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}],\"terminationGracePeriodSeconds\":1800,\"serviceAccountName\":\"postgres\",\"securityContext\":{\"runAsUser\":26,\"runAsGroup\":26,\"runAsNonRoot\":true,\"fsGroup\":26,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}},\"hostname\":\"postgres-1\",\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"weight\":100,\"podAffinityTerm\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"cnpg.io/cluster\",\"operator\":\"In\",\"values\":[\"postgres\"]}]},\"topologyKey\":\"kubernetes.io/hostname\"}}]}}}"
        },
        "creationTimestamp": "2023-12-18T07:40:02Z",
        "labels": {
            "cnpg.io/cluster": "postgres",
            "cnpg.io/instanceName": "postgres-1",
            "cnpg.io/instanceRole": "primary",
            "cnpg.io/podRole": "instance",
            "role": "primary"
        },
        "name": "postgres-1",
        "namespace": "shane",
        "ownerReferences": [
            {
                "apiVersion": "postgresql.cnpg.io/v1",
                "blockOwnerDeletion": true,
                "controller": true,
                "kind": "Cluster",
                "name": "postgres",
                "uid": "261148e6-26ac-4d58-8c1e-b248f188aae1"
            }
        ],
        "resourceVersion": "9936",
        "uid": "1a003c2f-ee83-4d25-af0e-cf192495a235"
    },
    "spec": {
        "affinity": {
            "podAntiAffinity": {
                "preferredDuringSchedulingIgnoredDuringExecution": [
                    {
                        "podAffinityTerm": {
                            "labelSelector": {
                                "matchExpressions": [
                                    {
                                        "key": "cnpg.io/cluster",
                                        "operator": "In",
                                        "values": [
                                            "postgres"
                                        ]
                                    }
                                ]
                            },
                            "topologyKey": "kubernetes.io/hostname"
                        },
                        "weight": 100
                    }
                ]
            }
        },
        "containers": [
            {
                "command": [
                    "/controller/manager",
                    "instance",
                    "run",
                    "--log-level=info"
                ],
                "env": [
                    {
                        "name": "PGDATA",
                        "value": "/var/lib/postgresql/data/pgdata"
                    },
                    {
                        "name": "POD_NAME",
                        "value": "postgres-1"
                    },
                    {
                        "name": "NAMESPACE",
                        "value": "shane"
                    },
                    {
                        "name": "CLUSTER_NAME",
                        "value": "postgres"
                    },
                    {
                        "name": "PGPORT",
                        "value": "5432"
                    },
                    {
                        "name": "PGHOST",
                        "value": "/controller/run"
                    }
                ],
                "image": "ghcr.io/cloudnative-pg/postgresql:16.0",
                "imagePullPolicy": "IfNotPresent",
                "livenessProbe": {
                    "failureThreshold": 3,
                    "httpGet": {
                        "path": "/healthz",
                        "port": 8000,
                        "scheme": "HTTP"
                    },
                    "periodSeconds": 10,
                    "successThreshold": 1,
                    "timeoutSeconds": 5
                },
                "name": "postgres",
                "ports": [
                    {
                        "containerPort": 5432,
                        "name": "postgresql",
                        "protocol": "TCP"
                    },
                    {
                        "containerPort": 9187,
                        "name": "metrics",
                        "protocol": "TCP"
                    },
                    {
                        "containerPort": 8000,
                        "name": "status",
                        "protocol": "TCP"
                    }
                ],
                "readinessProbe": {
                    "failureThreshold": 3,
                    "httpGet": {
                        "path": "/readyz",
                        "port": 8000,
                        "scheme": "HTTP"
                    },
                    "periodSeconds": 10,
                    "successThreshold": 1,
                    "timeoutSeconds": 5
                },
                "resources": {},
                "securityContext": {
                    "allowPrivilegeEscalation": false,
                    "capabilities": {
                        "drop": [
                            "ALL"
                        ]
                    },
                    "privileged": false,
                    "readOnlyRootFilesystem": true,
                    "runAsNonRoot": true,
                    "seccompProfile": {
                        "type": "RuntimeDefault"
                    }
                },
                "startupProbe": {
                    "failureThreshold": 360,
                    "httpGet": {
                        "path": "/healthz",
                        "port": 8000,
                        "scheme": "HTTP"
                    },
                    "periodSeconds": 10,
                    "successThreshold": 1,
                    "timeoutSeconds": 5
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File",
                "volumeMounts": [
                    {
                        "mountPath": "/var/lib/postgresql/data",
                        "name": "pgdata"
                    },
                    {
                        "mountPath": "/run",
                        "name": "scratch-data"
                    },
                    {
                        "mountPath": "/controller",
                        "name": "scratch-data"
                    },
                    {
                        "mountPath": "/dev/shm",
                        "name": "shm"
                    },
                    {
                        "mountPath": "/etc/app-secret",
                        "name": "app-secret"
                    },
                    {
                        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                        "name": "kube-api-access-s9n8l",
                        "readOnly": true
                    }
                ]
            }
        ],
        "dnsPolicy": "ClusterFirst",
        "enableServiceLinks": true,
        "hostname": "postgres-1",
        "initContainers": [
            {
                "command": [
                    "/manager",
                    "bootstrap",
                    "/controller/manager",
                    "--log-level=info"
                ],
                "image": "ghcr.io/cloudnative-pg/cloudnative-pg:1.21.0",
                "imagePullPolicy": "IfNotPresent",
                "name": "bootstrap-controller",
                "resources": {},
                "securityContext": {
                    "allowPrivilegeEscalation": false,
                    "capabilities": {
                        "drop": [
                            "ALL"
                        ]
                    },
                    "privileged": false,
                    "readOnlyRootFilesystem": true,
                    "runAsNonRoot": true,
                    "seccompProfile": {
                        "type": "RuntimeDefault"
                    }
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File",
                "volumeMounts": [
                    {
                        "mountPath": "/var/lib/postgresql/data",
                        "name": "pgdata"
                    },
                    {
                        "mountPath": "/run",
                        "name": "scratch-data"
                    },
                    {
                        "mountPath": "/controller",
                        "name": "scratch-data"
                    },
                    {
                        "mountPath": "/dev/shm",
                        "name": "shm"
                    },
                    {
                        "mountPath": "/etc/app-secret",
                        "name": "app-secret"
                    },
                    {
                        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                        "name": "kube-api-access-s9n8l",
                        "readOnly": true
                    }
                ]
            }
        ],
        "nodeName": "minikube",
        "preemptionPolicy": "PreemptLowerPriority",
        "priority": 0,
        "restartPolicy": "Always",
        "schedulerName": "default-scheduler",
        "securityContext": {
            "fsGroup": 26,
            "runAsGroup": 26,
            "runAsNonRoot": true,
            "runAsUser": 26,
            "seccompProfile": {
                "type": "RuntimeDefault"
            }
        },
        "serviceAccount": "postgres",
        "serviceAccountName": "postgres",
        "terminationGracePeriodSeconds": 1800,
        "tolerations": [
            {
                "effect": "NoExecute",
                "key": "node.kubernetes.io/not-ready",
                "operator": "Exists",
                "tolerationSeconds": 300
            },
            {
                "effect": "NoExecute",
                "key": "node.kubernetes.io/unreachable",
                "operator": "Exists",
                "tolerationSeconds": 300
            }
        ],
        "volumes": [
            {
                "name": "pgdata",
                "persistentVolumeClaim": {
                    "claimName": "postgres-1"
                }
            },
            {
                "emptyDir": {},
                "name": "scratch-data"
            },
            {
                "emptyDir": {
                    "medium": "Memory"
                },
                "name": "shm"
            },
            {
                "name": "app-secret",
                "secret": {
                    "defaultMode": 420,
                    "secretName": "account-secret"
                }
            },
            {
                "name": "kube-api-access-s9n8l",
                "projected": {
                    "defaultMode": 420,
                    "sources": [
                        {
                            "serviceAccountToken": {
                                "expirationSeconds": 3607,
                                "path": "token"
                            }
                        },
                        {
                            "configMap": {
                                "items": [
                                    {
                                        "key": "ca.crt",
                                        "path": "ca.crt"
                                    }
                                ],
                                "name": "kube-root-ca.crt"
                            }
                        },
                        {
                            "downwardAPI": {
                                "items": [
                                    {
                                        "fieldRef": {
                                            "apiVersion": "v1",
                                            "fieldPath": "metadata.namespace"
                                        },
                                        "path": "namespace"
                                    }
                                ]
                            }
                        }
                    ]
                }
            }
        ]
    },
    "status": {
        "conditions": [
            {
                "lastProbeTime": null,
                "lastTransitionTime": "2023-12-18T07:40:08Z",
                "status": "True",
                "type": "Initialized"
            },
            {
                "lastProbeTime": null,
                "lastTransitionTime": "2023-12-18T07:40:22Z",
                "status": "True",
                "type": "Ready"
            },
            {
                "lastProbeTime": null,
                "lastTransitionTime": "2023-12-18T07:40:22Z",
                "status": "True",
                "type": "ContainersReady"
            },
            {
                "lastProbeTime": null,
                "lastTransitionTime": "2023-12-18T07:40:02Z",
                "status": "True",
                "type": "PodScheduled"
            }
        ],
        "containerStatuses": [
            {
                "containerID": "docker://56603914ff01f3b781174deac8e7d25fae9e5bd764139eea547715d0c1690351",
                "image": "ghcr.io/cloudnative-pg/postgresql:16.0",
                "imageID": "docker-pullable://ghcr.io/cloudnative-pg/postgresql@sha256:5baf99e0fb30a051d8e108eeecb558af65a7119e068a280641873a2b98e35457",
                "lastState": {},
                "name": "postgres",
                "ready": true,
                "restartCount": 0,
                "started": true,
                "state": {
                    "running": {
                        "startedAt": "2023-12-18T07:40:10Z"
                    }
                }
            }
        ],
        "hostIP": "192.168.49.2",
        "initContainerStatuses": [
            {
                "containerID": "docker://70792f2a6c3f5f40424d2a1bb9818d201a768d4c8f63935e80be1bbe700838ff",
                "image": "ghcr.io/cloudnative-pg/cloudnative-pg:1.21.0",
                "imageID": "docker-pullable://ghcr.io/cloudnative-pg/cloudnative-pg@sha256:44f6132d5335528417427ea12634c5e6c96371e25e00dc898c3b6d091a279c21",
                "lastState": {},
                "name": "bootstrap-controller",
                "ready": true,
                "restartCount": 0,
                "started": false,
                "state": {
                    "terminated": {
                        "containerID": "docker://70792f2a6c3f5f40424d2a1bb9818d201a768d4c8f63935e80be1bbe700838ff",
                        "exitCode": 0,
                        "finishedAt": "2023-12-18T07:40:06Z",
                        "reason": "Completed",
                        "startedAt": "2023-12-18T07:40:06Z"
                    }
                }
            }
        ],
        "phase": "Running",
        "podIP": "10.244.0.18",
        "podIPs": [
            {
                "ip": "10.244.0.18"
            }
        ],
        "qosClass": "BestEffort",
        "startTime": "2023-12-18T07:40:02Z"
    }
}

We can use almost anything in that, let’s say when the pod is assigned an IP. We can see the path is status.podIPs and it’s an array, with the field inside each podIP being ip

$ kubectl wait --for jsonpath='{.status.podIPs[0].ip}' --timeout=5s pod/postgres-1 -n shane
pod/postgres-1 condition met

Simple :)

Summary

Just a quick note for using what’s a very helpful feature of kubectl, wait, which is now firmly in my toolkit.

Troubleshooting your Kubernetes Cluster

Shane — Mon, 04 Dec 2023 20:03:22 GMT

When faced with a cluster that’s not doing what you expect it do, what should you do to find out why it’s not behaving?

Assuming that you have (tried to) configured your cluster to be something in the order of “Ingress listens for traffic on port 80, meant for domain name example.com” then passes that traffic to a service that passes the traffic to a pod which is your container, then the first job is to check that the ingress is receiving the traffic intended for the domain.

Start by looking at all the pods

$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                        READY   STATUS      RESTARTS        AGE
ingress-nginx   ingress-nginx-admission-create-ghh8z        0/1     Completed   0               23h
ingress-nginx   ingress-nginx-admission-patch-pxf2q         0/1     Completed   0               23h
ingress-nginx   ingress-nginx-controller-7c6974c4d8-2x5vq   1/1     Running     1 (5h39m ago)   23h
kube-system     coredns-5dd5756b68-rn4td                    1/1     Running     1 (19h ago)     30h
kube-system     etcd-minikube                               1/1     Running     1 (19h ago)     30h
kube-system     kube-apiserver-minikube                     1/1     Running     1 (5h39m ago)   30h
kube-system     kube-controller-manager-minikube            1/1     Running     1 (19h ago)     30h
kube-system     kube-proxy-kv2xd                            1/1     Running     1 (19h ago)     30h
kube-system     kube-scheduler-minikube                     1/1     Running     1 (19h ago)     30h
kube-system     storage-provisioner                         1/1     Running     3 (5h38m ago)   30h

Then check the status of the pods in your namespace

$ kubectl get pods -n shane
NAME                             READY   STATUS              RESTARTS     AGE
accountserver-56b469dbbd-9q492   0/1     ErrImageNeverPull   0            12h
apiserver-6687c679f-tzgsw        1/1     Running             1 (9h ago)   14h

We can see in that snippet that there’s a problem with the accountserver image, but the apiserver is claiming to be happy.

Assuming that we’re checking the apiserver we continue -

Check that the ingress is receiving traffic (requires two shells, one to send the command, one to log what’s happening)

The logging shell:

$ kubectl logs -n ingress-nginx ingress-nginx-controller-7c6974c4d8-2x5vq -f

The command shell: First I want the address that the ingress has

$ kubectl get ingress -n shane
NAME    CLASS    HOSTS       ADDRESS        PORTS   AGE
shane   <none>   example.com   192.168.58.2   80      3h15m

Test if that IP is receiving traffic

wget 192.168.58.2
--2023-12-04 15:17:53--  http://192.168.58.2/
Connecting to 192.168.58.2:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2023-12-04 15:17:53 ERROR 404: Not Found.

Well, wget reports connecting ok, but then it got a 404

So, let’s try with the domain name (For the record, I set the IP of the domain I want to pretend I have in /etc/hosts)

 $ wget example.com
--2023-12-04 15:27:03--  http://example.com/
Resolving example.com (example.com)... 192.168.58.2
Connecting to example.com (example.com)|192.168.58.2|:80... connected.
HTTP request sent, awaiting response... 502 Bad Gateway
2023-12-04 15:27:03 ERROR 502: Bad Gateway.

This tells us that the ingress is listening, and will only respond to requests on example.com which is fine. But it’s returning a 502 error, let’s see what the logs say (the other shell)

2023/12/04 04:27:03 [error] 577#577: *342388 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.58.1, server: example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.11:8080/", host: "example.com"
2023/12/04 04:27:03 [error] 577#577: *342388 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.58.1, server: example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.11:8080/", host: "example.com"
2023/12/04 04:27:03 [error] 577#577: *342388 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.58.1, server: example.com, request: "GET / HTTP/1.1", upstream: "http://10.244.0.11:8080/", host: "example.com"
192.168.58.1 - - [04/Dec/2023:04:27:03 +0000] "GET / HTTP/1.1" 502 150 "-" "Wget/1.21.2" 124 0.001 [example-api-service-8080] [] 10.244.0.11:8080, 10.244.0.11:8080, 10.244.0.11:8080 0, 0, 0 0.001, 0.000, 0.000 502, 502, 502 3cee74a2a4e97297f6ef379689bda677

The ingress is telling us that it the connection attempt upstream was refused.

We now turn our attention to the pod that the ingress is trying to connect to.

What have the logs been for apiserver?

$ kubectl logs apiserver-86c595ccfc-lpd2x -n shane

Ooo, empty, that’s curious.

From here it’s time to look at the configuration in kubernetes as well as in the container. It’s clear that kubernetes has directed traffic from the ingress to the accountserver service, but it’s not been accepted.

Upon inspection I found that the port configured in the service definition did not match the port configured in the container - so I have made a permanent fix, using an environment variable passed from the service to the container to define the port, meaning that the two will (from hence forth) have the same value for port number.

Observable systems

Shane — Sat, 02 Dec 2023 04:34:07 GMT

This post is mainly about instrumenting tracing in distributed systems with OpenTelemetry.

Before we get to that we need to understand what constitutes an observable system. Observable systems have three ‘pillars’ of observability:

Logs
Traces
Metrics

OpenTelemetry itself hasn’t got logging implemented, but it does have tracing and metric gathering capabilities (for Go).

On to the interesting stuff :)

I have an API container that has a RESTful interface for clients on the internet to call. The API container has a number of routes/endpoints that it handles by calling other containers via gRPC.

HTTP

I use Chi as my mux, and I use the Gorilla OpenTelemetry Middleware. Using the Server.go as an example I found it incredibly simple to instrument the HTTP server.

First, a trace initialisation:

package main

// required imports:
// "go.opentelemetry.io/otel"
//  stdout "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
// sdktrace "go.opentelemetry.io/otel/sdk/trace"

var tracer = otel.Tracer("api-server")

func initTracer() (*sdktrace.TracerProvider, error) {
        exporter, err := stdout.New(stdout.WithPrettyPrint())
        if err != nil {
                return nil, err
        }

        tp := sdktrace.NewTracerProvider(
                sdktrace.WithSampler(sdktrace.AlwaysSample()),
                sdktrace.WithBatcher(exporter),
        )
        otel.SetTracerProvider(tp)
        otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

        return tp, nil
}

Then tell our mux to use the middleware:

package main

// required imports:
// "github.com/go-chi/chi/v5"
// "github.com/go-chi/chi/v5/middleware"
// "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"
// "time" (not necessary for OpenTelemetry, but used for another middleware).

func createRouter() *chi.Mux {
        router := chi.NewRouter()

        router.Use(
                otelmux.Middleware("my-server"),
                // The rest are standard chi middleware (unnecessary for telemetry, but left in for example of other middleware).
                middleware.RequestID,
                middleware.RealIP,
                middleware.Logger,
                middleware.Recoverer,
                //nolint:gomnd
                middleware.Timeout(60*time.Second),
        )

        // All routes are in routes.go
        OnepageRoutes(router)

        return router
}

And a simple main - the only OpenTelemetry related code is the call to the initTracer function:

package main

// required imports:
// "log"
// "context"
// "log/slog"
// "net/http"
// "time"

func main() {
        tp, err := initTracer()
        if err != nil {
                log.Fatal(err)
        }

        defer func() {
                if err := tp.Shutdown(context.Background()); err != nil {
                        log.Printf("Error shutting down tracer provider: %v", err)
                }
        }()

        router := createRouter()

        server := &http.Server{
                Addr: ":80",
                ReadHeaderTimeout: 3 * time.Second,
                Handler:           router,
        }

        if err := server.ListenAndServe(); err != nil {
                slog.Warn("cannot listen and serve", "error", err.Error())
        }
}

The routes are untouched by the OpenTelemetry code, the next part is to ensure that the handlers are injecting traces, and spans, so that the call by a user can be ‘traced’ across the systems.

I keep a tracer.go that ensures that the tracer is accessible across the package:

package api

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("mux-server")

And, an example of handler (all that is required for OpenTelemetry is the first two lines):

package api

...

func GetSomething(response http.ResponseWriter, req *http.Request) {
        _, span := tracer.Start(context.Background(), "GetSomething")
        defer span.End()
        ...
}

That’s all that’s required for HTTP telemetry (tracing).

gRPC Server

gRPC initialisation on the server is a little more involved, allegedly there is an easier (less code) way to do this, but I couldn’t grasp how to enumerate which middleware I wanted to include.

These examples are largely based on Go gRPC examples

server:

package main

import (
        "bytes"
        "context"
        "fmt"
        "log"
        "log/slog"
        "net"
        "os"
        "runtime/debug"

        "shanehowearth.com/stuff"
        grpcProto "shanehowearth.com/stuff/grpc/proto/v1"

        grpcprom "github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus"
        "github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery"
        "github.com/prometheus/client_golang/prometheus"
        "github.com/prometheus/client_golang/prometheus/promauto"
        "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
        "go.opentelemetry.io/otel"
        stdout "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
        "go.opentelemetry.io/otel/propagation"
        sdktrace "go.opentelemetry.io/otel/sdk/trace"
        "go.opentelemetry.io/otel/trace"
        "google.golang.org/grpc"
        "google.golang.org/grpc/codes"
        "google.golang.org/grpc/reflection"
        "google.golang.org/grpc/status"
)

type server struct {
        grpcProto.UnimplementedAccountServiceServer
}

// package global
var stuffServer *stuff.Server

func main() {
        stuffServer = stuff.NewAccountServer()

        // gRPC service
        portNumber, ok := os.LookupEnv("PORT")
        if !ok {
                log.Fatalf("PORT not set in environment, cannot continue")
        }

        lis, err := net.Listen("tcp", "0.0.0.0:"+portNumber)
        if err != nil {
                log.Fatalf("Failed to listen: %v", err)
        }

        // Setup metrics.
        srvMetrics := grpcprom.NewServerMetrics(
                grpcprom.WithServerHandlingTimeHistogram(
                        grpcprom.WithHistogramBuckets([]float64{0.001, 0.01, 0.1, 0.3, 0.6, 1, 3, 6, 9, 20, 30, 60, 90, 120}),
                ),
        )

        reg := prometheus.NewRegistry()
        reg.MustRegister(srvMetrics)

        exemplarFromContext := func(ctx context.Context) prometheus.Labels {
                if span := trace.SpanContextFromContext(ctx); span.IsSampled() {
                        return prometheus.Labels{"traceID": span.TraceID().String()}
                }

                return nil
        }

        // Set up OTLP tracing (stdout for debug).
        exporter, err := stdout.New(stdout.WithPrettyPrint())
        if err != nil {
                slog.Error("exporter setup", "error", err.Error())
                os.Exit(1)
        }

        tp := sdktrace.NewTracerProvider(
                sdktrace.WithSampler(sdktrace.AlwaysSample()),
                sdktrace.WithBatcher(exporter),
        )
        otel.SetTracerProvider(tp)

        otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

        defer func() { _ = exporter.Shutdown(context.Background()) }()

        // Setup metric for panic recoveries.
        panicsTotal := promauto.With(reg).NewCounter(prometheus.CounterOpts{
                Name: "grpc_req_panics_recovered_total",
                Help: "Total number of gRPC requests recovered from internal panic.",
        })

        grpcPanicRecoveryHandler := func(p any) error {
                panicsTotal.Inc()
                slog.Error("msg", "recovered from panic", "panic", p, "stack", debug.Stack())

                return status.Errorf(codes.Internal, "%s", p)
        }

        grpcServer := grpc.NewServer(
                grpc.StatsHandler(otelgrpc.NewServerHandler()),
                grpc.ChainUnaryInterceptor(
                        // Order matters e.g. tracing interceptor have to create
                        // span first for the later exemplars to work.
                        otelgrpc.UnaryServerInterceptor(),
                        srvMetrics.UnaryServerInterceptor(grpcprom.WithExemplarFromContext(exemplarFromContext)),
                        recovery.UnaryServerInterceptor(recovery.WithRecoveryHandler(grpcPanicRecoveryHandler)),
                ),
                grpc.ChainStreamInterceptor(
                        otelgrpc.StreamServerInterceptor(),
                        srvMetrics.StreamServerInterceptor(grpcprom.WithExemplarFromContext(exemplarFromContext)),
                        recovery.StreamServerInterceptor(recovery.WithRecoveryHandler(grpcPanicRecoveryHandler)),
                ),
        )
        grpcProto.RegisterAccountServiceServer(grpcServer, &server{})
        reflection.Register(grpcServer)

        if err := grpcServer.Serve(lis); err != nil {
                panic(fmt.Sprintf("failed to serve: %v", err))
        }
}

again, I have a tracer.go that exposes the tracer to the whole package:

package main

import  "go.opentelemetry.io/otel"

var tracer = otel.Tracer("accounts-grpc-server")

And the functions exposed by the gRPC server look like this:

func (s *server) GetStuff(ctx context.Context) (*grpcProto.Stuff, error) {
        ctx, childSpan := tracer.Start(ctx, "Get Stuff")
        defer childSpan.End()
        
        ...

        return &grpcProto.Stuff{HtmlForm: "weee"}, nil
}

gRPC Client

The only thing happening in the client connection is the inclusion of the otel middleware

package stuff

import (
        "context"
        "log"
        "log/slog"
        "time"

        "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
        "google.golang.org/grpc"
        "google.golang.org/grpc/credentials/insecure"
        proto "shanehowearth.com/stuff/grpc/proto/v1"
)

type Client struct {
        Address string
}

func (ac *Client) newConnection() (proto.StuffServiceClient, *grpc.ClientConn) {
        conn, err := grpc.Dial(
                ac.Address,
                grpc.WithTransportCredentials(insecure.NewCredentials()),
                grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
        )
        if err != nil {
                log.Fatalf("did not connect: %v", err)
        }

        return proto.NewStuffServiceClient(conn), conn
}

And that’s it. I don’t include any traces in the client caller function, but there’s no reason that it cannot be done.

Summary

Turning a distributed system of microservices written in Go into an observable one with OpenTelemetry is a fairly simple affair, and these examples should mean you can do it too :)

Private repositories and Go

Shane — Sat, 25 Nov 2023 22:16:39 GMT

EDIT: I’ve discovered this morning that, in fact, the issue still exists. I suspect a bug in the VCS code for Go, and have raised an issue (Although the issue ~~bot~~ gnome has misread what was being said and claimed it was a discussion not an issue)

Almost every write up I have seen for working with private repositories and Go have taken the point of view that the repository is hosted on one of the major public Git repository holders (Github, or Gitlab). My problem is that I am hosting repositories on my own machines. I can use git without those websites ;).

I was having a lot of trouble with Go interacting with those self-hosted repositories however, which I thought was odd because Go uses Git under the hood for things like go get and go mod download.

After a lot of hunting around I discovered that, in fact, Go has been created to deal with several possible Version Control Systems (VCS). You can see how it interacts with them in the internal VCS package.

The general purpose nature of the VCS usage in Go means that some assumptions have been made that make go get and go mod download not quite work the same as (say) git clone, and that’s why I was having a bit of trouble :).

This StackOverflow question about Private Repository with SSH was invaluable for diagnosing the problems and suggestions for solutions, including what I have finally got working.

There are three things that are needed in order for you to use the code hosted non-publicly.

Gitconfig

The very first problem for me was that my repositories are accessible via ssh only, not https.

Both git and go get can handle this just fine, by adding a couple of lines to the local ~/.gitconfig

[url "git@github.com:"]
        insteadOf = https://github.com/
[url "git@shanehoweath.com:"]
        insteadOf = https://shanehowearth.com/

GOPRIVATE

The Go download tools will try to find the repository/package using the module proxy, but my modules are private, and I don’t want them published/acknowledged to even exist on the public proxy.

Go uses an environment variable GOPRIVATE to determine if the public proxy should be consulted.

I use GOPRIVATE when I run go mod download (including in Dockerfiles)

GOPRIVATE=shanehowearth.com go mod download

Repository naming.

This was the most difficult issue to diagnose. None of the tools were telling me what the problem was, and it’s a function of the fact that Go’s vcs system is trying to cater to multiple different types of version control systems (git, mercurial, etc).

Git will happily clone, pull, push, to any directory on any host that it’s directed to. Go will too, except when it’s a private repository over ssh, then the name of the repository/directory becomes important.

Firstly, the name of the host must have a ‘.’ (period) in the name. so localhost/path/to/repo will not work. The stackoverflow link above recommends to use the ip if the host doesn’t have a period in the name (eg. use 192.168.1.72 instead of zeus).

Secondly, and this is quite painful, the name of the repository must have a .git suffix.

So shanehowearth.com/path/to/repo must be renamed to shanehowearth.com/path/to/repo.git in order for it to work.

Note that this also means any references to the repo in the code must also have the .git suffix too, so I have package imports like shanehowearth.com/path/to/repo.git/package/name in my code, which I find ugly, but it is what it is.

This includes the go.mod file, which has to have .git in the package name:

module shanehowearth/path/to/module.git

I am exasperated by this issue, it took me a couple of days to discover that the three things were combining to make it really difficult to host my own repo and use it via ssh with go, but I can now.

Hopefully this blog post saves someone else the pain I went through, and, of course, a massive thanks to StackOverflow user nshuytsa for publishing their solution as well.

Combining templates with embed in Go

Shane — Wed, 22 Nov 2023 22:42:12 GMT

Go templates are a great way to build dynamic text or html with Go programs. They are very much like Jinja templates, but they come with a serious shortcoming, they are loaded at runtime, rather than compile time.

This is a problem because a developer has very little control on where the templates will be located on a filesystem when the application is in use (whether in a users machine, or on a cloud).

Go 1.16 added the embed package that meant that files needed by a binary could be embedded in the binary at compile time. This means that a developer now has absolute control on where a template is at compile time, and at runtime.

This is an example of templates being used with embed.

First, the filesystem for this example is thus:

$ tree
.
├── accounts.go
├── cmd
│   └── main.go
├── coreTemplates
│   └── base.tmpl
├── Dockerfile
├── go.mod
├── go.sum
├── Makefile
├── README.md
└──templates
    └── signup.tmpl

The accounts.go file needs to make use of the base.tmpl and signup.tmpl templates, so that is all the go code that needs to be seen.

Note that there is a form in signup.tmpl that requires a CSRF token in it (see the template file pasted at the bottom).

 $ cat accounts.go
package accounts

import (
        "embed"
        "io"
        "log/slog"
        "text/template"

        "github.com/gorilla/csrf"
)

var (
        //go:embed coreTemplates/*.tmpl
        baseTemplateFS embed.FS

        //go:embed templates/*.tmpl
        signupTemplateFS embed.FS
)

type Server struct{}

func NewAccountServer() *Server {
        return &Server{}
}

func (s *Server) ShowRegisterAccountForm(responseWriter io.Writer, csrfToken string) {
        baseTemplate := template.Must(template.New("base").ParseFS(baseTemplateFS, "coreTemplates/base.tmpl"))

        signupTemplate := template.Must(baseTemplate.ParseFS(signupTemplateFS, "templates/signup.tmpl"))

        if err := signupTemplate.ExecuteTemplate(responseWriter, "base.tmpl", map[string]interface{}{
                csrf.TemplateTag: csrfToken,
        }); err != nil {
                slog.Error("unable to execute template", "error", err.Error())
        }
}

The templates are loaded into the embed.FS filesystem, when the file is compiled. I’ve used two embed.FS one for coreTemplates which is a git submodule containing template files that are the basis for system wide web pages, containing common headers, footers, menus, etc.

The second embed.FS contains the templates relevant to this service/domain (for the Domain Driven Design aficionados).

Within the ShowRegisterAccountForm function the base.tmpl is parsed (the path and name of the template being parsed from the embed.FS needs to be supplied).

Similarly the signup.tmpl is parsed, but with the baseTemplate, because we want the signup.tmpl to have some of the base.tmpl attributes.

Finally, the templates are executed with the csrf token injected into the signup.tmpl.

The base.tmpl looks like this:

$ cat coreTemplates/base.tmpl
<html>
  <head>{{block "head" .}} Working Title {{end}}</head>
  <body>{{block "body" .}} Lorem ipsum dolor sit amet, consectetur adipiscing
  elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut
  enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut
  aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in
  voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint
  occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit
  anim id est laborum.{{end}}</body>
  <footer>{{block "footer" .}}TODO{{end}}</footer>

And the signup.tmpl looks like this.

$ cat templates/signup.tmpl
{{define "head"}}<title>Join Something</title>{{end}}
{{define "body"}}
<form method="post" action="/account">
  {{ .csrfField  }}
  <label for="fname">First name:</label><br>
  <input type="text" id="fname" name="fname" value="Lady"><br>
  <label for="lname">Last name:</label><br>
  <input type="text" id="lname" name="lname" value="Penelope"><br><br>
  <label for="lname">Company:</label><br>
  <input type="text" id="company" name="company" value="International Rescue"><br><br>
  <label for="lname">Email:</label><br>
  <input type="text" id="email" name="email" value="Lady.Penelope@example.com"><br><br>
  <input type="submit" value="Submit">
</form>
{{end}}

Benign Data Races in Go

Shane — Mon, 20 Nov 2023 23:12:53 GMT

Every so often, in one forum, or another, the subject of benign data races comes up. People want to know if a data race absolutely must have synchronisation guards applied (mutexes, semaphores, etc).

For those not familiar with the concept, a data race is when two or more goroutines/threads/processes/etc have access to a mutable resource, and at least one of them mutates that resource. A benign data race is when such a data race is claimed to have no ill effect on the application.

Benign Data races have been the subject of discussion in a number of technologies for some time, and are contentious.

Java documentation talks about benign data races not only being possible, but also advantageous, because of improved performance (no need to lock/unlock synchronisation tools). Note, from the documentation

In addition to benign data-races, a large class of applications allow data-races because they rely on lock-free and wait-free algorithms which are difficult to design correctly.

(My blog post isn’t going to delve into lock free programming, beyond Benign data races)

Hans-J. Boehm wrote a paper entitled How to miscompile programs with “benign” data races which concluded that

C or C++ data races are always unsafe in portable code

and

Although “benign” races in Java are not clearly incorrect, they nonetheless put the programmer into an uncertain and dangerous territory that can be avoided by avoiding data races.

Boehm’s findings largely rest on the fact that C and C++ compilers may be compiling for targets other than x86, which do not, necessarily, have atomic writes.

Further Boehm’s paper points to a number of usecases where data races can occur because of adjacency, that is, the compiler is treating two variables as one because they will lie so close together in memory.

For these reasons, it’s important to be aware of more than just the “correctness” of the code that the given language is written in, but, also the code that is emitted for a given CPU architecture, and that CPU’s atomicity when performing writes (it’s writes that cause the problem, not reads).

But we have a “benign data race” example, in computing, that is used all day, every day - we even have a name for it “Eventual consistency” - the state of the data may or may not be reflected accurately by the reader, but, we account for that by saying that, given some time, the reader will accurately reflect the actual state of the data.

Eventual consistency exists because of the CAP theorem. Once data is partitioned, that is, more than one copy exists, or multiple compute instances (threads, etc) access that data, then the end result will be consistent, or available, but never both.

For this reason, we have to accept that benign data races are both possible, and, genuinely safe for some applications. The principle is the same, the data is being shared by multiple threads/processes, and a mutation of that data can occur at any time. The readers will eventually be showing that new state, and as long as that is recognised, then that is an acceptable outcome.

For non-technical readers, eventual consistency has existed in the real world forever. If I say that I am going to pay a bill, and it takes some time for that payment to show up, even though the bill has been paid, it will show, for some time, as unpaid (the cheque is in the mail! :-)

Redux - state management (with React and Typescript)

Shane — Fri, 10 Nov 2023 00:17:08 GMT

I am building a tool that requires a great deal of state management on the client side. After all of 30 seconds of research I decided to use Redux for state management. (For the record, after I started down this path someone mentioned Zustand as an alternative, which they claimed had an easier interface. I haven’t investigated it at all, but someone might find it useful to know of alternaitves).

I largely followed the Redux Typescript tutorial, but there were a couple of things that I ended up asking for help with, because it wasn’t clear (to me) in the tutorial.

The basic concepts that are required to be understood is that redux has a store that components can Create, Read, Update, Delete (CRUD) items from.

In earlier releases a developer needed to understand that the store was immutable, so changes were made by creating a copy, making changes to the copy, then saving the new version of the store.

Redux-toolkit has abstracted that away, developers only need to tell redux what changes are being made, and the immer library is used under the hood to keep with the immutable nature of the redux store. (This is helpful to know, however, when debugging, because you cannot just console.log(state)).

Down to business, er, I mean, let’s look at how things are done in the code, with explanations.

First of all, access to the state is via reducers.

First, I put this code in features/location/locationSlice.ts

import { createSlice  } from '@reduxjs/toolkit';

// This import allows us to dump the state to console log
// for debugging
import { current } from "immer";

// This is the shape of the data being stored in redux.
// In this instance there is only one object, indexed by an id.
// But there is no reason that there couldn't be more objects.
// eg. If you wanted one for UI, and one for some
// other set of data.
interface EntityState {

      x: number;
      y: number;
  };
}

// Initialise the state for redux.
const initialState: EntityState = {};

// This is where the work is done.
// We name the object (this becomes important when reading from
// the state in components).
// We define the reducers, note that the reducers can be defined
// elsewhere, and called here in the reducers section.
export const positionSlice = createSlice({
    name: 'position',
    initialState,
  reducers: {
    // This is a modifier reducer, modifying the current state.
    moveItem: (state, action) => {
      // decompose the action payload.
      const { id, x, y  } = action.payload;

      // Update the nested object.
      if (state[id]) {
          state[id].x = x;
          state[id].y = y;
      }
    },

    // This is a Creation reducer. It's job is to create new
    // instances of objects being stored in the state.
    newItem: (state, action) => {
      // This is a debug statement, purely for demonstration.
      console.log(current(state));

      // Decompose the action payload.
      const { id, x, y, width, height, elementType  } = action.payload;

      // Create the new nested object.
      // Note that I am not doing any checking if an object with
      // the same name already exists.
      // I am using uuids for the name and am confident that
      // there will be no collisions, but defensive code would
      // be a good idea here.
      state[id] = {
        x: x,
        y: y,
      }
    },
  },
});

// Expose the actions to the world.
export const { newItem, moveItem  } = positionSlice.actions;

export default positionSlice.reducer;

I have a store, defined in store.ts, it’s not very complex

import { configureStore } from '@reduxjs/toolkit'

// When we build the store we use the previously created
// reducer.
import positionReducer from './features/position/positionSlice';

const store = configureStore({
  reducer: {
    position: positionReducer,
  },
})

// Make the RootState and Dispatch types easier to access
// in components.
export type RootState = ReturnType<typeof store.getState>

export type AppDispatch = typeof store.dispatch

export default store

edit: I forgot to add the following index.tsx (or App.tsx - this is purely to make the store available to all of the components.

import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import store from './store'
import { Provider } from 'react-redux'

const root = ReactDOM.createRoot(
  document.getElementById('root') as HTMLElement
);

root.render(
  <React.StrictMode>
    <Provider store={store}>
      <App />
    </Provider>
  </React.StrictMode>
);

That’s all that’s required for redux. The next parts are using the redux store in components.

This shows the creation reducer being called.

import React, { useState } from "react";

// import the creation reducer
import { newItem } from "../features/position/positionSlice";

// Import useDispatch - this is used to call the reducer.
import { useDispatch  } from 'react-redux'

const { v4: uuidv4  } = require('uuid');

const Editor = () => {
  // make the function a little bit nicer to call.
  const dispatch = useDispatch();

  // add an element
  const handleClick = (name: string, height: number, width: number) => {
    // Set the ID for the new item.
    const ID = uuidv4();

    // Add new item to global state.
    dispatch(newItem({
      id: ID,
      x: 0,
      y: 0,
    }));
  };

  const handleItemClick = () => { };

  return <div className="example">
    <span className="symbol" id="projectStarter" onClick={() => handleClick("project", 120, 120)}>
        [Product/System]
      </span>
</div>
};

export default Example;

The modifier access is exactly the same, but the read access is a little more tricky, so that’s what i will show here.

import React from "react";

// Import the tools needed to access the redux state.
import { useSelector } from 'react-redux';

// This was our way of making it easier to get the type of the state.
import { RootState } from "../store";

type ArrowProps = {
    startPoint: ElementItem;
    endPoint: ElementItem;
};

const Arrow = ({ startPoint, endPoint  }: ArrowProps) => {
  // Get start and end points from global state.
  // It's important to note the use of the name from the first file to access the nested object inside the state.
  // That is, the state itself is a complex object, and our data has to be accessed as a nested object.
  const start = useSelector((state: RootState) => state.position[startPoint.id]);
  const end = useSelector((state: RootState) => state.position[endPoint.id])
...
};

export default Arrow;

I have heavily commented the code because I feel that the explanation of what’s happening is best read in place, where something is being used.

It should be clear, though, that redux is very simple to use, when you know what you are doing ;)