feat(shard): 🎉 Initialize the rewrite.
This commit is contained in:
Nyeki.py
@@ -1,264 +0,0 @@
|
||||
# Payphone
|
||||
|
||||
Payphone is deployed using Kubernetes + Docker. To set up a local kubernetes cluster, you can use
|
||||
`minikube`.
|
||||
|
||||
## Starting a Cluster
|
||||
|
||||
This section will guide you step by step to create a multi-node cluster or a local development
|
||||
cluster and start up Payphone. Make sure to have:
|
||||
|
||||
1. The Discord bot's token, either for development or production depending on what you're setting
|
||||
up the cluster for.
|
||||
2. You'll probably need at least 5 CPU cores (6 recommended) across all of your machines + 1 core
|
||||
per shard, 8 GB of ram + 512Mb of RAM per shard, and 30Gb of disk space available. It'll still
|
||||
run with less, but it'll be way slower. In multi-node environment, these specifications are
|
||||
cluster-wide, meaning that if you have 3 nodes with 3 cores, that makes it 9 cores in total and
|
||||
therefore suitable for a multi-shard deployment. Do not run nodes with less than 2 CPU cores, 2
|
||||
Gb (ideally 4 Gb) of RAM, and 15 Gb of disk space available.
|
||||
|
||||
### Kubernetes Setup
|
||||
|
||||
First of all, you need to install Kubernetes in the machine where you'll set up either a node or a
|
||||
whole cluster.
|
||||
|
||||
Make sure to have Docker installed in your machine before running the following steps. You can
|
||||
check how to install Docker in the
|
||||
[Docker documentation](https://docs.docker.com/get-started/get-docker/). If you're setting up a new
|
||||
node for production, it's better if you only install the Docker Engine without Docker Desktop.
|
||||
|
||||
#### For Development
|
||||
|
||||
Ideally, use `minikube` for local development. This'll allow you to run a complete cluster locally.
|
||||
Make sure to have downloaded and installed minikube first. On Windows, it's available with
|
||||
`chocolatey` running `choco install minikube`. You can read the full set up guide for other
|
||||
installation methods at the [official minikube documentation](https://minikube.sigs.k8s.io/docs/).
|
||||
|
||||
After `minikube` is installed, you need to set up a local cluster. This'll create a virtual machine
|
||||
in a Docker container to act as nodes in your cluster. Run the following command to start a 3-node
|
||||
local cluster:
|
||||
|
||||
```sh
|
||||
minikube start --network-plugin=cni --cni=calico --nodes=3
|
||||
```
|
||||
|
||||
This'll create 3 docker containers and start them, and set them up to use kubectl connecting them
|
||||
automatically. Management `kubectl` commands can only be ran from the master/control-panel node,
|
||||
but with `kubectl` you don't need to worry about that since running `kubectl` from your machine
|
||||
will work as if you were in a master node.
|
||||
|
||||
`kubectl` comes integrated with `Docker Desktop`. You can enable Kubernetes in Docker Desktop at
|
||||
Docker Desktop > Settings > Kubernetes > Enable Kubernetes.
|
||||
|
||||
#### For Production
|
||||
|
||||
These steps are required in order to set up a new `microk8s` node. These installation steps assume
|
||||
they're being ran in an Ubuntu 22.4+ server with at least 2 CPU cores, 4 GB RAM, and 25 GB disk
|
||||
space, and `snap` installed. You also need root privileges to follow these steps.
|
||||
|
||||
First of all, install `microk8s`.
|
||||
|
||||
```sh
|
||||
sudo snap install microk8s --classic
|
||||
```
|
||||
|
||||
After the installation, check that `microk8s` is up and running using the following command:
|
||||
|
||||
```sh
|
||||
microk8s status --wait-ready
|
||||
```
|
||||
|
||||
> [!INFO] In case the last command fails to run because of missing permissions, you'll be guided
|
||||
> on how to give your current user the required permissions to run the command. Follow those steps
|
||||
> and re-run the command afterwards.
|
||||
|
||||
Then, you need to enable the required extensions for `microk8s` to work properly. Run the following
|
||||
commands to enable them:
|
||||
|
||||
```sh
|
||||
microk8s enable dns
|
||||
microk8s enable ha-cluster
|
||||
microk8s enable community
|
||||
|
||||
sudo systemctl enable iscsid # Needed to run the following command.
|
||||
|
||||
microk8s enable openebs
|
||||
```
|
||||
|
||||
The rest of the guide uses `kubectl`, but `microk8s` has `kubectl` as a subcommand, so either:
|
||||
|
||||
1. You create an alias in `.bash_aliases`, appending `alias kubectl='microk8s kubectl'` to your
|
||||
`~/.bash_aliases` file. If you don't have one, create an empty file and add the command in it.
|
||||
After updating the file, run `source ~/.bash_aliases` to apply the changes.
|
||||
|
||||
2. Use `microk8s kubectl` instead of `kubectl` in the following steps. This'll require you to type
|
||||
`microk8s` at the front of all `kubectl` commands that are run from now on.
|
||||
|
||||
To add an alias, run the following command:
|
||||
|
||||
```sh
|
||||
echo "alias kubectl='microk8s kubectl'" >> ~/.bash_aliases && source ~/.bash_aliases
|
||||
```
|
||||
|
||||
##### Add a Node to the Cluster
|
||||
|
||||
To add a new node to the cluster, run the following command on a master node:
|
||||
|
||||
```sh
|
||||
microk8s add-node
|
||||
```
|
||||
|
||||
You'll get a command to paste in your new node. Copy the one that ends with `--worker` and paste it
|
||||
in your new node. Wait a few minutes until it connects, and then you're ready to go! No need to
|
||||
follow the deployment steps if the cluster is already up and running.
|
||||
|
||||
##### SSH Security
|
||||
|
||||
To secure the access to the server, you'll need to follow the following steps:
|
||||
|
||||
###### Disable Password Authentication
|
||||
|
||||
Having SSH password authentication leaves an open door for brute force attacks. Before disabling
|
||||
it, make sure that the server has your SSH key to prevent from getting locked out after proceeding.
|
||||
|
||||
After you have made sure your keys have been properly configured, edit the SSH configuration file:
|
||||
|
||||
```sh
|
||||
nano /etc/ssh/ssh_config
|
||||
```
|
||||
|
||||
In the file, find the following configurations:
|
||||
1. `ChallengeResponseAuthentication`
|
||||
2. `PasswordAuthentication`
|
||||
3. `UsePAM`
|
||||
|
||||
If they're set in the file, set their value to `no`. After that, restart the SSH service:
|
||||
|
||||
```sh
|
||||
systemctl restart ssh
|
||||
```
|
||||
|
||||
###### Set Up `fail2ban`
|
||||
|
||||
The next step to secure SSH is to set up `fail2ban`, which will rate limit failed attempts to log
|
||||
in. First, install it in the server:
|
||||
|
||||
```sh
|
||||
apt update
|
||||
apt install fail2ban
|
||||
```
|
||||
|
||||
Next, make a copy of the configuration file:
|
||||
|
||||
```sh
|
||||
cd /etc/fail2ban
|
||||
cp jail.conf jail.local
|
||||
```
|
||||
|
||||
You now need to update it to enable SSH protection:
|
||||
|
||||
```sh
|
||||
nano jail.local
|
||||
```
|
||||
|
||||
Find the `[ssh]` section and add the following line (or update it if it already exists):
|
||||
|
||||
```toml
|
||||
enabled = true
|
||||
```
|
||||
|
||||
After updating that line, close the file, enable, and start the service:
|
||||
|
||||
```sh
|
||||
systemctl enable fail2ban
|
||||
systemctl start fail2ban
|
||||
|
||||
# Check the status of the service to make sure it was enabled properly. You can exit the status
|
||||
# screen pressing "q".
|
||||
systemctl status fail2ban
|
||||
```
|
||||
|
||||
## Deploying Payphone
|
||||
|
||||
The following steps show how to pull and install Payphone on a running cluster of nodes.
|
||||
|
||||
### Building and Publishing the Payphone Image
|
||||
|
||||
Payphone's container images are hosted here at `git.nyeki.dev`. To login, run the following
|
||||
command:
|
||||
|
||||
```sh
|
||||
docker login git.nyeki.dev
|
||||
```
|
||||
|
||||
A new browser tab will open and you'll be prompted for log in. Once you accept, you'll be logged in
|
||||
to the container registry.
|
||||
|
||||
When you have made an update and want to update the production deployment's image, you need to
|
||||
build and push the image to the container registry:
|
||||
|
||||
```sh
|
||||
docker build -t git.nyeki.dev/payphone/payphone:latest .
|
||||
docker push git.nyeki.dev/payphone/payphone:latest
|
||||
```
|
||||
|
||||
### Starting Payphone in the Cluster
|
||||
|
||||
#### Setting Up Config Files
|
||||
|
||||
First of all, you'll need to set up the environment variables that Payphone needs. For this,
|
||||
you'll need:
|
||||
|
||||
1. A Discord bot token, base64 encoded. You can encode your token at
|
||||
[www.base64encode.org](https://www.base64encode.org/).
|
||||
2. A suggestions channel that the bot can write to (its ID).
|
||||
3. A non-expiring invite to the official server.
|
||||
|
||||
First of all, open `k8s/config.yml`. At the bottom of the file, you'll find a `data` object with
|
||||
some environment variables definitions. Set `SUGGESTIONS_CHANNEL_ID` and `SERVER_INVITE` and save
|
||||
the file.
|
||||
|
||||
Then, after you've base64-encoded your token, open `k8s/secrets.yml` and in a similar `data` object
|
||||
at the bottom of the file, set `TOKEN` to the base64-encoded Discord bot token and save the
|
||||
changes.
|
||||
|
||||
After the files are updated with their new values, run the following commands to apply these
|
||||
changes:
|
||||
|
||||
```sh
|
||||
kubectl apply -f k8s/config.yml
|
||||
kubectl apply -f k8s/secrets.yml
|
||||
```
|
||||
|
||||
If you see a ConfigMap and a Secret being created, that means you've successfully applied the
|
||||
changes.
|
||||
|
||||
#### Setting Up Redis and PostgreSQL
|
||||
|
||||
Now, you can initialize Redis. This step is pretty straightforward and you can forget about it
|
||||
later.
|
||||
|
||||
```sh
|
||||
kubectl apply -f k8s/redis.yml
|
||||
kubectl apply -f k8s/psql.yml
|
||||
```
|
||||
|
||||
#### Setting Up Payphone
|
||||
|
||||
Finally, you need to start the bot's instance. First, log into your `git.nyeki.dev` account from
|
||||
`kubectl` with the following command:
|
||||
|
||||
```sh
|
||||
kubectl create secret docker-registry regcred --docker-server=https://git.nyeki.dev/v1/ --docker-username=<your-username> --docker-password=<your-password>
|
||||
```
|
||||
|
||||
After that command is ran, the next step is to update the amount of shards. To set the amount,
|
||||
open `k8s/bot.yml` and set the total number of shards in both `spec.replicas` and
|
||||
`spec.template.spec.containers.bot.env.TOTAL_SHARDS.value`.
|
||||
|
||||
Once it's done, you can save and close the file and run the final command.
|
||||
|
||||
```sh
|
||||
kubectl apply -f k8s/tasks.yml
|
||||
kubectl apply -f k8s/bot.yml
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user