The Discord bot's token, either for development or production depending on what you're setting up the cluster for.
You'll probably need at least 5 CPU cores (6 recommended) across all of your machines + 1 core per shard, 8 GB of ram + 512Mb of RAM per shard, and 30Gb of disk space available. It'll still run with less, but it'll be way slower. In multi-node environment, these specifications are cluster-wide, meaning that if you have 3 nodes with 3 cores, that makes it 9 cores in total and therefore suitable for a multi-shard deployment. Do not run nodes with less than 2 CPU cores, 2 Gb (ideally 4 Gb) of RAM, and 15 Gb of disk space available.

Kubernetes Setup

First of all, you need to install Kubernetes in the machine where you'll set up either a node or a whole cluster.

Make sure to have Docker installed in your machine before running the following steps. You can check how to install Docker in the Docker documentation. If you're setting up a new node for production, it's better if you only install the Docker Engine without Docker Desktop.

For Development

Ideally, use minikube for local development. This'll allow you to run a complete cluster locally. Make sure to have downloaded and installed minikube first. On Windows, it's available with chocolatey running choco install minikube. You can read the full set up guide for other installation methods at the official minikube documentation.

After minikube is installed, you need to set up a local cluster. This'll create a virtual machine in a Docker container to act as nodes in your cluster. Run the following command to start a 3-node local cluster:

minikube start --network-plugin=cni --cni=calico --nodes=3

This'll create 3 docker containers and start them, and set them up to use kubectl connecting them automatically. Management kubectl commands can only be ran from the master/control-panel node, but with kubectl you don't need to worry about that since running kubectl from your machine will work as if you were in a master node.

kubectl comes integrated with Docker Desktop. You can enable Kubernetes in Docker Desktop at Docker Desktop > Settings > Kubernetes > Enable Kubernetes.

For Production

These steps are required in order to set up a new microk8s node. These installation steps assume they're being ran in an Ubuntu 22.4+ server with at least 2 CPU cores, 4 GB RAM, and 25 GB disk space, and snap installed. You also need root privileges to follow these steps.

First of all, install microk8s.

sudo snap install microk8s --classic

After the installation, check that microk8s is up and running using the following command:

microk8s status --wait-ready

[!INFO] In case the last command fails to run because of missing permissions, you'll be guided on how to give your current user the required permissions to run the command. Follow those steps and re-run the command afterwards.

Then, you need to enable the required extensions for microk8s to work properly. Run the following commands to enable them:

microk8s enable dns
microk8s enable ha-cluster
microk8s enable community

sudo systemctl enable iscsid  # Needed to run the following command.

microk8s enable openebs

The rest of the guide uses kubectl, but microk8s has kubectl as a subcommand, so either:

You create an alias in .bash_aliases, appending alias kubectl='microk8s kubectl' to your ~/.bash_aliases file. If you don't have one, create an empty file and add the command in it. After updating the file, run source ~/.bash_aliases to apply the changes.
Use microk8s kubectl instead of kubectl in the following steps. This'll require you to type microk8s at the front of all kubectl commands that are run from now on.

To add an alias, run the following command:

echo "alias kubectl='microk8s kubectl'" >> ~/.bash_aliases && source ~/.bash_aliases

Add a Node to the Cluster

To add a new node to the cluster, run the following command on a master node:

microk8s add-node

You'll get a command to paste in your new node. Copy the one that ends with --worker and paste it in your new node. Wait a few minutes until it connects, and then you're ready to go! No need to follow the steps down below if the cluster is already up and running.

Deploying Payphone

The following steps show how to pull and install Payphone on a running cluster of nodes.

Building and Publishing the Payphone Image

Payphone's container images are hosted here at git.nyeki.dev. To login, run the following command:

docker login git.nyeki.dev

A new browser tab will open and you'll be prompted for log in. Once you accept, you'll be logged in to the container registry.

When you have made an update and want to update the production deployment's image, you need to build and push the image to the container registry:

docker build -t git.nyeki.dev/payphone/payphone:latest .
docker push git.nyeki.dev/payphone/payphone:latest

Starting Payphone in the Cluster

Setting Up Config Files

First of all, you'll need to set up the environment variables that Payphone needs. For this, you'll need:

A Discord bot token, base64 encoded. You can encode your token at www.base64encode.org.
A suggestions channel that the bot can write to (its ID).
A non-expiring invite to the official server.

First of all, open k8s/config.yml. At the bottom of the file, you'll find a data object with some environment variables definitions. Set SUGGESTIONS_CHANNEL_ID and SERVER_INVITE and save the file.

Then, after you've base64-encoded your token, open k8s/secrets.yml and in a similar data object at the bottom of the file, set TOKEN to the base64-encoded Discord bot token and save the changes.

After the files are updated with their new values, run the following commands to apply these changes:

kubectl apply -f k8s/config.yml
kubectl apply -f k8s/secrets.yml

If you see a ConfigMap and a Secret being created, that means you've successfully applied the changes.

Setting Up Redis

Now, you can initialize Redis. This step is pretty straightforward and you can forget about it later.

kubectl apply -f k8s/redis.yml

Setting Up Citus Database

This step requires some updates in the k8s/citus.yml file. Open the file and make sure that the default amount of replicas fits your needs. At least one is needed.

kubectl apply -f k8s/citus.yml

Once all the nodes are up and running, you have to connect them. The node citus-0 is always used as the coordinator node. Run the following command to set the coordinator host:

kubectl exec citus-0 -- psql -U payphone payphone -c "SELECT citus_set_coordinator_host('citus-0.citus-headless.default.svc.cluster.local', 5432);"

Once that command was run, you need to add the rest of the nodes to the cluster. To do that, run the following command replacing <node-index> with the index of the node you're adding, once per node.

kubectl exec citus-0 -- psql -U payphone payphone -c "SELECT * from citus_add_node('citus-<node-index>.citus-headless.default.svc.cluster.local', 5432);"

For example, if you had 3 replicas in total, you'd run the command 2 times, once for node 1 and once for node 2 (node 0 is the coordinator, so you don't have to add it).

As an optional step, you can check that the nodes were successfully connected running the following command, which will show all the nodes connected:

kubectl exec citus-0 -- psql -U payphone payphone -c "SELECT * FROM citus_get_active_worker_nodes();"

Setting Up Payphone

Finally, you need to start the bot's instance. First, log into your git.nyeki.dev account from kubectl with the following command:

kubectl create secret docker-registry regcred --docker-server=https://git.nyeki.dev/v1/ --docker-username=<your-username> --docker-password=<your-password>

After that command is ran, the next step is to update the amount of shards. To set the amount, open k8s/bot.yml and set the total number of shards in both spec.replicas and spec.template.spec.containers.bot.env.TOTAL_SHARDS.value.

Once it's done, you can save and close the file and run the final command.

kubectl apply -f k8s/tasks.yml
kubectl apply -f k8s/bot.yml