232 lines
8.6 KiB
Markdown
232 lines
8.6 KiB
Markdown
# Payphone
|
|
|
|
Payphone is deployed using Kubernetes + Docker. To set up a local kubernetes cluster, you can use
|
|
`minikube`.
|
|
|
|
## Starting a Cluster
|
|
|
|
This section will guide you step by step to create a multi-node cluster or a local development
|
|
cluster and start up Payphone. Make sure to have:
|
|
|
|
1. The Discord bot's token, either for development or production depending on what you're setting
|
|
up the cluster for.
|
|
2. You'll probably need at least 5 CPU cores (6 recommended) across all of your machines + 1 core
|
|
per shard, 8 GB of ram + 512Mb of RAM per shard, and 30Gb of disk space available. It'll still
|
|
run with less, but it'll be way slower. In multi-node environment, these specifications are
|
|
cluster-wide, meaning that if you have 3 nodes with 3 cores, that makes it 9 cores in total and
|
|
therefore suitable for a multi-shard deployment. Do not run nodes with less than 2 CPU cores, 2
|
|
Gb (ideally 4 Gb) of RAM, and 15 Gb of disk space available.
|
|
|
|
### Kubernetes Setup
|
|
|
|
First of all, you need to install Kubernetes in the machine where you'll set up either a node or a
|
|
whole cluster.
|
|
|
|
Make sure to have Docker installed in your machine before running the following steps. You can
|
|
check how to install Docker in the
|
|
[Docker documentation](https://docs.docker.com/get-started/get-docker/). If you're setting up a new
|
|
node for production, it's better if you only install the Docker Engine without Docker Desktop.
|
|
|
|
#### For Development
|
|
|
|
Ideally, use `minikube` for local development. This'll allow you to run a complete cluster locally.
|
|
Make sure to have downloaded and installed minikube first. On Windows, it's available with
|
|
`chocolatey` running `choco install minikube`. You can read the full set up guide for other
|
|
installation methods at the [official minikube documentation](https://minikube.sigs.k8s.io/docs/).
|
|
|
|
After `minikube` is installed, you need to set up a local cluster. This'll create a virtual machine
|
|
in a Docker container to act as nodes in your cluster. Run the following command to start a 3-node
|
|
local cluster:
|
|
|
|
```sh
|
|
minikube start --network-plugin=cni --cni=calico --nodes=3
|
|
```
|
|
|
|
This'll create 3 docker containers and start them, and set them up to use kubectl connecting them
|
|
automatically. Management `kubectl` commands can only be ran from the master/control-panel node,
|
|
but with `kubectl` you don't need to worry about that since running `kubectl` from your machine
|
|
will work as if you were in a master node.
|
|
|
|
`kubectl` comes integrated with `Docker Desktop`. You can enable Kubernetes in Docker Desktop at
|
|
Docker Desktop > Settings > Kubernetes > Enable Kubernetes.
|
|
|
|
#### For Production
|
|
|
|
These steps are required in order to set up a new `microk8s` node. These installation steps assume
|
|
they're being ran in an Ubuntu 22.4+ server with at least 2 CPU cores, 4 GB RAM, and 25 GB disk
|
|
space, and `snap` installed. You also need root privileges to follow these steps.
|
|
|
|
First of all, install `microk8s`.
|
|
|
|
```sh
|
|
sudo snap install microk8s --classic
|
|
```
|
|
|
|
After the installation, check that `microk8s` is up and running using the following command:
|
|
|
|
```sh
|
|
microk8s status --wait-ready
|
|
```
|
|
|
|
> [!INFO] In case the last command fails to run because of missing permissions, you'll be guided
|
|
> on how to give your current user the required permissions to run the command. Follow those steps
|
|
> and re-run the command afterwards.
|
|
|
|
Then, you need to enable the required extensions for `microk8s` to work properly. Run the following
|
|
commands to enable them:
|
|
|
|
```sh
|
|
microk8s enable dns
|
|
microk8s enable ha-cluster
|
|
microk8s enable community
|
|
|
|
sudo systemctl enable iscsid # Needed to run the following command.
|
|
|
|
microk8s enable openebs
|
|
```
|
|
|
|
The rest of the guide uses `kubectl`, but `microk8s` has `kubectl` as a subcommand, so either:
|
|
|
|
1. You create an alias in `.bash_aliases`, appending `alias kubectl='microk8s kubectl'` to your
|
|
`~/.bash_aliases` file. If you don't have one, create an empty file and add the command in it.
|
|
After updating the file, run `source ~/.bash_aliases` to apply the changes.
|
|
|
|
2. Use `microk8s kubectl` instead of `kubectl` in the following steps. This'll require you to type
|
|
`microk8s` at the front of all `kubectl` commands that are run from now on.
|
|
|
|
To add an alias, run the following command:
|
|
|
|
```sh
|
|
echo "alias kubectl='microk8s kubectl'" >> ~/.bash_aliases && source ~/.bash_aliases
|
|
```
|
|
|
|
##### Add a Node to the Cluster
|
|
|
|
To add a new node to the cluster, run the following command on a master node:
|
|
|
|
```sh
|
|
microk8s add-node
|
|
```
|
|
|
|
You'll get a command to paste in your new node. Copy the one that ends with `--worker` and paste it
|
|
in your new node. Wait a few minutes until it connects, and then you're ready to go! No need to
|
|
follow the steps down below if the cluster is already up and running.
|
|
|
|
## Deploying Payphone
|
|
|
|
The following steps show how to pull and install Payphone on a running cluster of nodes.
|
|
|
|
### Building and Publishing the Payphone Image
|
|
|
|
Payphone's container images are hosted here at `git.nyeki.dev`. To login, run the following
|
|
command:
|
|
|
|
```sh
|
|
docker login git.nyeki.dev
|
|
```
|
|
|
|
A new browser tab will open and you'll be prompted for log in. Once you accept, you'll be logged in
|
|
to the container registry.
|
|
|
|
When you have made an update and want to update the production deployment's image, you need to
|
|
build and push the image to the container registry:
|
|
|
|
```sh
|
|
docker build -t git.nyeki.dev/payphone/payphone:latest .
|
|
docker push git.nyeki.dev/payphone/payphone:latest
|
|
```
|
|
|
|
### Starting Payphone in the Cluster
|
|
|
|
#### Setting Up Config Files
|
|
|
|
First of all, you'll need to set up the environment variables that Payphone needs. For this,
|
|
you'll need:
|
|
|
|
1. A Discord bot token, base64 encoded. You can encode your token at
|
|
[www.base64encode.org](https://www.base64encode.org/).
|
|
2. A suggestions channel that the bot can write to (its ID).
|
|
3. A non-expiring invite to the official server.
|
|
|
|
First of all, open `k8s/config.yml`. At the bottom of the file, you'll find a `data` object with
|
|
some environment variables definitions. Set `SUGGESTIONS_CHANNEL_ID` and `SERVER_INVITE` and save
|
|
the file.
|
|
|
|
Then, after you've base64-encoded your token, open `k8s/secrets.yml` and in a similar `data` object
|
|
at the bottom of the file, set `TOKEN` to the base64-encoded Discord bot token and save the
|
|
changes.
|
|
|
|
After the files are updated with their new values, run the following commands to apply these
|
|
changes:
|
|
|
|
```sh
|
|
kubectl apply -f k8s/config.yml
|
|
kubectl apply -f k8s/secrets.yml
|
|
```
|
|
|
|
If you see a ConfigMap and a Secret being created, that means you've successfully applied the
|
|
changes.
|
|
|
|
#### Setting Up Redis
|
|
|
|
Now, you can initialize Redis. This step is pretty straightforward and you can forget about it
|
|
later.
|
|
|
|
```sh
|
|
kubectl apply -f k8s/redis.yml
|
|
```
|
|
|
|
#### Setting Up Citus Database
|
|
|
|
This step requires some updates in the `k8s/citus.yml` file. Open the file and make sure that the
|
|
default amount of replicas fits your needs. At least one is needed.
|
|
|
|
```sh
|
|
kubectl apply -f k8s/citus.yml
|
|
```
|
|
|
|
Once all the nodes are up and running, you have to connect them. The node `citus-0` is always used
|
|
as the coordinator node. Run the following command to set the coordinator host:
|
|
|
|
```sh
|
|
kubectl exec citus-0 -- psql -U payphone payphone -c "SELECT citus_set_coordinator_host('citus-0.citus-headless.default.svc.cluster.local', 5432);"
|
|
```
|
|
|
|
Once that command was run, you need to add the rest of the nodes to the cluster. To do that, run
|
|
the following command replacing `<node-index>` with the index of the node you're adding, once per
|
|
node.
|
|
|
|
```sh
|
|
kubectl exec citus-0 -- psql -U payphone payphone -c "SELECT * from citus_add_node('citus-<node-index>.citus-headless.default.svc.cluster.local', 5432);"
|
|
```
|
|
|
|
For example, if you had 3 replicas in total, you'd run the command 2 times, once for node 1 and
|
|
once for node 2 (node 0 is the coordinator, so you don't have to add it).
|
|
|
|
As an optional step, you can check that the nodes were successfully connected running the following
|
|
command, which will show all the nodes connected:
|
|
|
|
```sh
|
|
kubectl exec citus-0 -- psql -U payphone payphone -c "SELECT * FROM citus_get_active_worker_nodes();"
|
|
```
|
|
|
|
#### Setting Up Payphone
|
|
|
|
Finally, you need to start the bot's instance. First, log into your `git.nyeki.dev` account from
|
|
`kubectl` with the following command:
|
|
|
|
```sh
|
|
kubectl create secret docker-registry regcred --docker-server=https://git.nyeki.dev/v1/ --docker-username=<your-username> --docker-password=<your-password>
|
|
```
|
|
|
|
After that command is ran, the next step is to update the amount of shards. To set the amount,
|
|
open `k8s/bot.yml` and set the total number of shards in both `spec.replicas` and
|
|
`spec.template.spec.containers.bot.env.TOTAL_SHARDS.value`.
|
|
|
|
Once it's done, you can save and close the file and run the final command.
|
|
|
|
```sh
|
|
kubectl apply -f k8s/tasks.yml
|
|
kubectl apply -f k8s/bot.yml
|
|
```
|