# Payphone

Payphone is deployed using Kubernetes + Docker. To set up a local kubernetes cluster, you can use
`minikube`.

## Starting a Cluster

This section will guide you step by step to create a multi-node cluster or a local development
cluster and start up Payphone. Make sure to have:

1. The Discord bot's token, either for development or production depending on what you're setting
   up the cluster for.
2. You'll probably need at least 5 CPU cores (6 recommended) across all of your machines + 1 core
   per shard, 8 GB of ram + 512Mb of RAM per shard, and 30Gb of disk space available. It'll still
   run with less, but it'll be way slower. In multi-node environment, these specifications are
   cluster-wide, meaning that if you have 3 nodes with 3 cores, that makes it 9 cores in total and
   therefore suitable for a multi-shard deployment. Do not run nodes with less than 2 CPU cores, 2
   Gb (ideally 4 Gb) of RAM, and 15 Gb of disk space available.

### Kubernetes Setup

First of all, you need to install Kubernetes in the machine where you'll set up either a node or a
whole cluster.

Make sure to have Docker installed in your machine before running the following steps. You can
check how to install Docker in the
[Docker documentation](https://docs.docker.com/get-started/get-docker/). If you're setting up a new
node for production, it's better if you only install the Docker Engine without Docker Desktop.

#### For Development

Ideally, use `minikube` for local development. This'll allow you to run a complete cluster locally.
Make sure to have downloaded and installed minikube first. On Windows, it's available with
`chocolatey` running `choco install minikube`. You can read the full set up guide for other
installation methods at the [official minikube documentation](https://minikube.sigs.k8s.io/docs/).

After `minikube` is installed, you need to set up a local cluster. This'll create a virtual machine
in a Docker container to act as nodes in your cluster. Run the following command to start a 3-node
local cluster:

```sh
minikube start --network-plugin=cni --cni=calico --nodes=3
```

This'll create 3 docker containers and start them, and set them up to use kubectl connecting them
automatically. Management `kubectl` commands can only be ran from the master/control-panel node,
but with `kubectl` you don't need to worry about that since running `kubectl` from your machine
will work as if you were in a master node.

`kubectl` comes integrated with `Docker Desktop`. You can enable Kubernetes in Docker Desktop at
Docker Desktop > Settings > Kubernetes > Enable Kubernetes.

#### For Production

These steps are required in order to set up a new `microk8s` node. These installation steps assume
they're being ran in an Ubuntu 22.4+ server with at least 2 CPU cores, 4 GB RAM, and 25 GB disk
space, and `snap` installed. You also need root privileges to follow these steps.

First of all, install `microk8s`.

```sh
sudo snap install microk8s --classic
```

After the installation, check that `microk8s` is up and running using the following command:

```sh
microk8s status --wait-ready
```

> [!INFO] In case the last command fails to run because of missing permissions, you'll be guided
> on how to give your current user the required permissions to run the command. Follow those steps
> and re-run the command afterwards.

Then, you need to enable the required extensions for `microk8s` to work properly. Run the following
commands to enable them:

```sh
microk8s enable dns
microk8s enable ha-cluster
microk8s enable community

sudo systemctl enable iscsid  # Needed to run the following command.

microk8s enable openebs
```

The rest of the guide uses `kubectl`, but `microk8s` has `kubectl` as a subcommand, so either:

1. You create an alias in `.bash_aliases`, appending `alias kubectl='microk8s kubectl'` to your
   `~/.bash_aliases` file. If you don't have one, create an empty file and add the command in it.
   After updating the file, run `source ~/.bash_aliases` to apply the changes.

2. Use `microk8s kubectl` instead of `kubectl` in the following steps. This'll require you to type
   `microk8s` at the front of all `kubectl` commands that are run from now on.
   
To add an alias, run the following command:

```sh
echo "alias kubectl='microk8s kubectl'" >> ~/.bash_aliases && source ~/.bash_aliases
```

##### Add a Node to the Cluster

To add a new node to the cluster, run the following command on a master node:

```sh
microk8s add-node
```

You'll get a command to paste in your new node. Copy the one that ends with `--worker` and paste it
in your new node. Wait a few minutes until it connects, and then you're ready to go! No need to
follow the steps down below if the cluster is already up and running.

##### SSH Security

To secure the access to the server, you'll need to follow the following steps:

###### Disable Password Authentication

Having SSH password authentication leaves an open door for brute force attacks. Before disabling
it, make sure that the server has your SSH key to prevent from getting locked out after proceeding.

After you have made sure your keys have been properly configured, edit the SSH configuration file:

```sh
nano /etc/ssh/ssh_config
```

In the file, find the following configurations:
1. `ChallengeResponseAuthentication`
2. `PasswordAuthentication`
3. `UsePAM`

If they're set in the file, set their value to `no`. After that, restart the SSH service:

```sh
systemctl restart ssh
```

###### Set Up `fail2ban`

The next step to secure SSH is to set up `fail2ban`, which will rate limit failed attempts to log
in. First, install it in the server:

```sh
apt update
apt install fail2ban
```

Next, make a copy of the configuration file:

```sh
cd /etc/fail2ban
cp jail.conf jail.local
```

You now need to update it to enable SSH protection:

```sh
nano jail.local
```

Find the `[ssh]` section and add the following line (or update it if it already exists):

```toml
enabled = true
```

After updating that line, close the file, enable, and start the service:

```sh
systemctl enable fail2ban
systemctl start fail2ban

# Check the status of the service to make sure it was enabled properly. You can exit the status
# screen pressing "q".
systemctl status fail2ban
```

## Deploying Payphone

The following steps show how to pull and install Payphone on a running cluster of nodes.

### Building and Publishing the Payphone Image

Payphone's container images are hosted here at `git.nyeki.dev`. To login, run the following
command:

```sh
docker login git.nyeki.dev
```

A new browser tab will open and you'll be prompted for log in. Once you accept, you'll be logged in
to the container registry.

When you have made an update and want to update the production deployment's image, you need to
build and push the image to the container registry:

```sh
docker build -t git.nyeki.dev/payphone/payphone:latest .
docker push git.nyeki.dev/payphone/payphone:latest
```

### Starting Payphone in the Cluster

#### Setting Up Config Files

First of all, you'll need to set up the environment variables that Payphone needs. For this,
you'll need:

1. A Discord bot token, base64 encoded. You can encode your token at
   [www.base64encode.org](https://www.base64encode.org/).
2. A suggestions channel that the bot can write to (its ID).
3. A non-expiring invite to the official server.

First of all, open `k8s/config.yml`. At the bottom of the file, you'll find a `data` object with
some environment variables definitions. Set `SUGGESTIONS_CHANNEL_ID` and `SERVER_INVITE` and save
the file.

Then, after you've base64-encoded your token, open `k8s/secrets.yml` and in a similar `data` object
at the bottom of the file, set `TOKEN` to the base64-encoded Discord bot token and save the
changes.

After the files are updated with their new values, run the following commands to apply these
changes:

```sh
kubectl apply -f k8s/config.yml
kubectl apply -f k8s/secrets.yml
```

If you see a ConfigMap and a Secret being created, that means you've successfully applied the
changes.

#### Setting Up Redis and PostgreSQL

Now, you can initialize Redis. This step is pretty straightforward and you can forget about it
later.

```sh
kubectl apply -f k8s/redis.yml
kubectl apply -f k8s/psql.yml
```

#### Setting Up Payphone

Finally, you need to start the bot's instance. First, log into your `git.nyeki.dev` account from
`kubectl` with the following command:

```sh
kubectl create secret docker-registry regcred --docker-server=https://git.nyeki.dev/v1/ --docker-username=<your-username> --docker-password=<your-password>
```

After that command is ran, the next step is to update the amount of shards. To set the amount,
open `k8s/bot.yml` and set the total number of shards in both `spec.replicas` and
`spec.template.spec.containers.bot.env.TOTAL_SHARDS.value`.

Once it's done, you can save and close the file and run the final command.

```sh
kubectl apply -f k8s/tasks.yml
kubectl apply -f k8s/bot.yml
```