137 lines
6.0 KiB
Markdown
137 lines
6.0 KiB
Markdown
# Payphone
|
|
|
|
Payphone is deployed using Kubernetes + Docker. To set up a local kubernetes cluster, you can use
|
|
`minikube`.
|
|
|
|
## Starting a Cluster
|
|
|
|
This section will guide you step by step to create a multi-node cluster or a local development
|
|
cluster and start up Payphone. Make sure to have:
|
|
|
|
1. A Docker Hub account with a slot for a private image. By default all free accounts allow 1
|
|
private image, so if you haven't uploaded any private images to your account, you do have an
|
|
available slot.
|
|
2. The Discord bot's token, either for development or production depending on what you're setting
|
|
up the cluster for.
|
|
3. You'll probably need at least 5 CPU cores (6 recommended) across all of your machines + 1 core
|
|
per shard, 8 GB of ram + 512Mb of RAM per shard, and 30Gb of disk space available. It'll still
|
|
run with less, but it'll be way slower. In multi-node environment, these specifications are
|
|
cluster-wide, meaning that if you have 3 nodes with 3 cores, that makes it 9 cores in total and
|
|
therefore suitable for a multi-shard deployment. Do not run nodes with less than 2 CPU cores, 2
|
|
Gb (ideally 4 Gb) of RAM, and 15 Gb of disk space available.
|
|
|
|
### Kubernetes Setup
|
|
|
|
First of all, you need to install Kubernetes in the machine where you'll set up either a node or a
|
|
whole cluster.
|
|
|
|
Make sure to have Docker installed in your machine before running the following steps. You can
|
|
check how to install Docker in the
|
|
[Docker documentation](https://docs.docker.com/get-started/get-docker/). If you're setting up a new
|
|
node for production, it's better if you only install the Docker Engine without Docker Desktop.
|
|
|
|
#### For Development
|
|
|
|
Ideally, use `minikube` for local development. This'll allow you to run a complete cluster locally.
|
|
Make sure to have downloaded and installed minikube first. On Windows, it's available with
|
|
`chocolatey` running `choco install minikube`. You can read the full set up guide for other
|
|
installation methods at the [official minikube documentation](https://minikube.sigs.k8s.io/docs/).
|
|
|
|
After `minikube` is installed, you need to set up a local cluster. This'll create a virtual machine
|
|
in a Docker container to act as nodes in your cluster. Run the following command to start a 3-node
|
|
local cluster:
|
|
|
|
```sh
|
|
minikube start --network-plugin=cni --cni=calico --nodes=3
|
|
```
|
|
|
|
This'll create 3 docker containers and start them, and set them up to use kubectl connecting them
|
|
automatically. Management `kubectl` commands can only be ran from the master/control-panel node,
|
|
but with `kubectl` you don't need to worry about that since running `kubectl` from your machine
|
|
will work as if you were in a master node.
|
|
|
|
`kubectl` comes integrated with `Docker Desktop`. You can enable Kubernetes in Docker Desktop at
|
|
Docker Desktop > Settings > Kubernetes > Enable Kubernetes.
|
|
|
|
#### For Production
|
|
|
|
These steps are required in order to set up a new `microk8s` node. These installation steps assume
|
|
they're being ran in an Ubuntu 22.4+ server with at least 2 CPU cores, 4 GB RAM, and 25 GB disk
|
|
space, and `snap` installed. You also need root privileges to follow these steps.
|
|
|
|
First of all, install `microk8s`.
|
|
|
|
```sh
|
|
sudo snap install microk8s --classic
|
|
```
|
|
|
|
After the installation, check that `microk8s` is up and running using the following command:
|
|
|
|
```sh
|
|
microk8s status --wait-ready
|
|
```
|
|
|
|
> [!INFO] In case the last command fails to run because of missing permissions, you'll be guided
|
|
> on how to give your current user the required permissions to run the command. Follow those steps
|
|
> and re-run the command afterwards.
|
|
|
|
Then, you need to enable the required extensions for `microk8s` to work properly. Run the following
|
|
commands to enable them:
|
|
|
|
```sh
|
|
microk8s enable dns
|
|
microk8s enable ha-cluster
|
|
```
|
|
|
|
The rest of the guide uses `kubectl`, but `microk8s` has `kubectl` as a subcommand, so either:
|
|
|
|
1. You create an alias in `.bash_aliases`, appending `alias kubectl='microk8s kubectl'` to your
|
|
`~/.bash_aliases` file. If you don't have one, create an empty file and add the command in it.
|
|
After updating the file, run `source ~/.bash_aliases` to apply the changes.
|
|
|
|
2. Use `microk8s kubectl` instead of `kubectl` in the following steps. This'll require you to type
|
|
`microk8s` at the front of all `kubectl` commands that are run from now on.
|
|
|
|
## Deploying Payphone
|
|
|
|
The following steps show how to pull and install Payphone on a running cluster of nodes.
|
|
|
|
### Building and Publishing the Payphone Image
|
|
|
|
To be able to deploy Payphone to your k8s cluster, you first need to upload the image to a private
|
|
registry. This documentation uses Docker Hub as a registry
|
|
|
|
```sh
|
|
# Only run this step if you need to build the source code and it is in your machine. Build and
|
|
# publish to the private registry. Use your own image name, for example `samantha/payphone:latest`.
|
|
docker login
|
|
|
|
docker build -t nekidev/payphone:latest .
|
|
docker publish nekidev/payphone:latest
|
|
|
|
# In the following steps, if you're using microk8s in production, use `microk8s kubectl` instead of
|
|
# `kubectl`. You can add this to your ~/.bash_aliases to avoid having to prepend `microk8s` to all
|
|
# commands:
|
|
# alias kubectl='microk8s kubectl'
|
|
|
|
# This step requires you to create a personal access token for Docker Hub. To create it:
|
|
# 1. Go to https://app.docker.com/settings/personal-access-tokens/create
|
|
# 2. Set the access permission to "Read-only"
|
|
kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v1/ --docker-username=<your-username> --docker-password=<your-personal-access-token>
|
|
|
|
|
|
# This will pull the images for citus and redis and start their containers.
|
|
kubectl apply -f redis.yml
|
|
kubectl apply -f citus.yml
|
|
|
|
# After applying citus.yml, you'll need to connect the nodes. We'll always use citus-0 as the
|
|
# master instance.
|
|
|
|
# Steps before running this command:
|
|
# 1. Update the image name in `bot.yml` to your own image name (the one that you have in your
|
|
# private docker hub registry)
|
|
# 2. Set the total number of shards in both `spec.replicas` and
|
|
# `spec.template.spec.containers.bot.env.TOTAL_SHARDS.value`.
|
|
kubectl apply -f bot.yml
|
|
```
|