Payphone

Payphone is deployed using Kubernetes + Docker. To set up a local kubernetes cluster, you can use minikube.

Starting a Cluster

This section will guide you step by step to create a multi-node cluster or a local development cluster and start up Payphone. Make sure to have:

1. The Discord bot's token, either for development or production depending on what you're setting up the cluster for.
2. You'll probably need at least 5 CPU cores (6 recommended) across all of your machines + 1 core per shard, 8 GB of ram + 512Mb of RAM per shard, and 30Gb of disk space available. It'll still run with less, but it'll be way slower. In multi-node environment, these specifications are cluster-wide, meaning that if you have 3 nodes with 3 cores, that makes it 9 cores in total and therefore suitable for a multi-shard deployment. Do not run nodes with less than 2 CPU cores, 2 Gb (ideally 4 Gb) of RAM, and 15 Gb of disk space available.

Kubernetes Setup

First of all, you need to install Kubernetes in the machine where you'll set up either a node or a whole cluster.

Make sure to have Docker installed in your machine before running the following steps. You can check how to install Docker in the Docker documentation. If you're setting up a new node for production, it's better if you only install the Docker Engine without Docker Desktop.

For Development

Ideally, use minikube for local development. This'll allow you to run a complete cluster locally. Make sure to have downloaded and installed minikube first. On Windows, it's available with chocolatey running choco install minikube. You can read the full set up guide for other installation methods at the official minikube documentation.

After minikube is installed, you need to set up a local cluster. This'll create a virtual machine in a Docker container to act as nodes in your cluster. Run the following command to start a 3-node local cluster:

minikube start --network-plugin=cni --cni=calico --nodes=3

This'll create 3 docker containers and start them, and set them up to use kubectl connecting them automatically. Management kubectl commands can only be ran from the master/control-panel node, but with kubectl you don't need to worry about that since running kubectl from your machine will work as if you were in a master node.

kubectl comes integrated with Docker Desktop. You can enable Kubernetes in Docker Desktop at Docker Desktop > Settings > Kubernetes > Enable Kubernetes.

For Production

These steps are required in order to set up a new microk8s node. These installation steps assume they're being ran in an Ubuntu 22.4+ server with at least 2 CPU cores, 4 GB RAM, and 25 GB disk space, and snap installed. You also need root privileges to follow these steps.

First of all, install microk8s.

sudo snap install microk8s --classic

After the installation, check that microk8s is up and running using the following command:

microk8s status --wait-ready

[!INFO] In case the last command fails to run because of missing permissions, you'll be guided on how to give your current user the required permissions to run the command. Follow those steps and re-run the command afterwards.

Then, you need to enable the required extensions for microk8s to work properly. Run the following commands to enable them:

microk8s enable dns
microk8s enable ha-cluster
microk8s enable community

sudo systemctl enable iscsid  # Needed to run the following command.

microk8s enable openebs

The rest of the guide uses kubectl, but microk8s has kubectl as a subcommand, so either:

1. You create an alias in .bash_aliases, appending alias kubectl='microk8s kubectl' to your ~/.bash_aliases file. If you don't have one, create an empty file and add the command in it. After updating the file, run source ~/.bash_aliases to apply the changes.

2. Use microk8s kubectl instead of kubectl in the following steps. This'll require you to type microk8s at the front of all kubectl commands that are run from now on.

To add an alias, run the following command:

echo "alias kubectl='microk8s kubectl'" >> ~/.bash_aliases && source ~/.bash_aliases

Add a Node to the Cluster

To add a new node to the cluster, run the following command on a master node:

microk8s add-node

You'll get a command to paste in your new node. Copy the one that ends with --worker and paste it in your new node. Wait a few minutes until it connects, and then you're ready to go! No need to follow the deployment steps if the cluster is already up and running.

SSH Security

To secure the access to the server, you'll need to follow the following steps:

Disable Password Authentication

Having SSH password authentication leaves an open door for brute force attacks. Before disabling it, make sure that the server has your SSH key to prevent from getting locked out after proceeding.

After you have made sure your keys have been properly configured, edit the SSH configuration file:

nano /etc/ssh/ssh_config

In the file, find the following configurations:

1. ChallengeResponseAuthentication
2. PasswordAuthentication
3. UsePAM

If they're set in the file, set their value to no. After that, restart the SSH service:

systemctl restart ssh

Set Up fail2ban

The next step to secure SSH is to set up fail2ban, which will rate limit failed attempts to log in. First, install it in the server:

apt update
apt install fail2ban

Next, make a copy of the configuration file:

cd /etc/fail2ban
cp jail.conf jail.local

You now need to update it to enable SSH protection:

nano jail.local

Find the [ssh] section and add the following line (or update it if it already exists):

enabled = true

After updating that line, close the file, enable, and start the service:

systemctl enable fail2ban
systemctl start fail2ban

# Check the status of the service to make sure it was enabled properly. You can exit the status
# screen pressing "q".
systemctl status fail2ban

Deploying Payphone

The following steps show how to pull and install Payphone on a running cluster of nodes.

Building and Publishing the Payphone Image

Payphone's container images are hosted here at git.nyeki.dev. To login, run the following command:

docker login git.nyeki.dev

A new browser tab will open and you'll be prompted for log in. Once you accept, you'll be logged in to the container registry.

When you have made an update and want to update the production deployment's image, you need to build and push the image to the container registry:

docker build -t git.nyeki.dev/payphone/payphone:latest .
docker push git.nyeki.dev/payphone/payphone:latest

Starting Payphone in the Cluster

Setting Up Config Files

First of all, you'll need to set up the environment variables that Payphone needs. For this, you'll need:

1. A Discord bot token, base64 encoded. You can encode your token at www.base64encode.org.
2. A suggestions channel that the bot can write to (its ID).
3. A non-expiring invite to the official server.

First of all, open k8s/config.yml. At the bottom of the file, you'll find a data object with some environment variables definitions. Set SUGGESTIONS_CHANNEL_ID and SERVER_INVITE and save the file.

Then, after you've base64-encoded your token, open k8s/secrets.yml and in a similar data object at the bottom of the file, set TOKEN to the base64-encoded Discord bot token and save the changes.

After the files are updated with their new values, run the following commands to apply these changes:

kubectl apply -f k8s/config.yml
kubectl apply -f k8s/secrets.yml

If you see a ConfigMap and a Secret being created, that means you've successfully applied the changes.

Setting Up Redis and PostgreSQL

Now, you can initialize Redis. This step is pretty straightforward and you can forget about it later.

kubectl apply -f k8s/redis.yml
kubectl apply -f k8s/psql.yml

Setting Up Payphone

Finally, you need to start the bot's instance. First, log into your registry.nyeki.dev account from kubectl (using the same credentials that in git.nyeki.dev) with the following command:

kubectl create secret docker-registry regcred --docker-server=https://registry.nyeki.dev/v1/ --docker-username=<your-username> --docker-password=<your-password>

After that command is ran, the next step is to update the amount of shards. To set the amount, open k8s/bot.yml and set the total number of shards in both spec.replicas and spec.template.spec.containers.bot.env.TOTAL_SHARDS.value.

Once it's done, you can save and close the file and run the final command.

kubectl apply -f k8s/tasks.yml
kubectl apply -f k8s/bot.yml