(｡◕‿‿◕｡) Sylvain's blog

Setup a complex infra

Dec 19, 2023

Required

infra

Intro

In this blog post I will introduce how I have set up my own infrastructure with around 20 services self hosted. The result of my work is hosted on my gitlab here. For the rest of this post remember that when I chose to use a VPS, it is that I want the service to run even if my infra is down. Also, you need an IP range to allocate an IP to the new router.

My ansible methodology

All my playbooks use roles to do actions, allowing me to reuse them for other services. All roles are store here.

Setting for all the VMs

I won’t precise it for everything, but every VMs should deny root login, and deny password authentication, only allowing connection with ssh key.

Setting up a Vault

First thing first, you need a place to host your secrets. I chose ansible vault because I was using it professionaly, and liked it a lot, and it is made by Hashicorp so it work with ansible, and I couldn’t find any down side of using it nor a better alternative. I also chose to host it on a VPS hosted separetely from everyting else, as it doesn’t consume a lot of resources, and by doing so you can still set up your whole infrastructure if everything else is down. So I made an ansible playbook to create a vault on a server here Once the Vault is created you will be able to create secrets for all your other services. For the initial key of vault, what I did is store it locally, and then store it in my own bitwarden (and remove it from my local storage).

Now you should have a server that can hold your secrets.

Setting up your storage server

I personnaly use a server with huge disk space and then apply the following restrictions. As every VMs you have can be destroyed at any moment, you need to setup a storage server to hold the important datas (obviously the partition storing everything have to be encrypted, using luks for exemple). Install nfs-kernel-server and configure /etc/exports with the following entries for your VMs using the following format

/local/path/dir          {client_ip}/32(rw,async,no_subtree_check,no_root_squash)

You can create as much entry as you want. But you have to be as restrictive as possible.

Setting up your DNS

For my DNS I chose to use two VPS, and I have setup a bind9 on both of them and deploy them using ansible. The files are here For the DNS I highly suggest using servers outside your servers hosting your website, as it will still be able to resolve domain name.

Setting up your reverse proxy

As for the vault server, I chose to use a VPS dedicated to be a reverse proxy and set up all my config there. My current nginx configs can be found here. You can use the templates I created, but I highly suggest creating one file per service, and setting up security headers for each of them. Currently my default security headers are here As for the DNS, I highly suggest using a reverse proxy in front of all your service on another server.

In the future I would like to have multiple RP, but it’s still in the idea phase.

Setting up your hypervisor

I made the choice of renting a beafy server and then making VMs on it using XCP-NG but you can user another hypervisor or use a miriad of VPS. My terraform config is located here, there is nothing too fancy there, I just created VMs for all of my services. I used a module source I created for my VMs allowing me to not duplicate code. The module is here and is made to create VM on XCP-NG. (Please note that at the time of writing this there is an issue with the auto_poweron feature in this plugin)

For XCP-NG I had to do the following:

On XCP-NG

sysctl -w net.ipv4.ip_forward=1
echo 1 > /proc/sys/net/ipv4/ip_forward

#!/bin/sh

/usr/sbin/sysctl -w net.ipv4.ip_forward=1
/usr/sbin/iptables-restore < /root/iptables

XAPI_INTERFACE="xapi1"
XEN_IP="{replace with your xen IP}"
NETWORK_IP="{replace with your network IP}"
NETWORK_RANGE="{replace with your range}"
/usr/sbin/ip link set "${XAPI_INTERFACE}" up
/usr/sbin/ip addr add "${XEN_IP}/${NETWORK_RANGE}" dev "${XAPI_INTERFACE}"
/usr/sbin/ip route add "${NETWORK_IP}/${NETWORK_RANGE}" dev "${XAPI_INTERFACE}"

* * * * * /root/create_route.sh

xe network-create name-label=Public

On XOA VM

#!/bin/bash

systemctl stop xo-server
certbot -q renew
systemctl start xo-server

0 0 1 * * /home/xoa/renew_certbot.sh

(Optional if you have a storage server) On XOA config

Setup the storage server in XOA settings.

Then you are done with the hypervisor part.

Setting up your router

Now that you have an hypervisor ready to go you need to setup a router to pass requests from your hypervisor to the services VMs. If you’ve followed the steps so far your should create a router with an IP in the range you had allocated by your service provider. Your router need to be in two network interfaces at least, one with the hypervisor and one for your VMs you want to allocate. I set up a DHCP and a DNS to resolve my services by names instead of having to remember every IP.

Installing your services

Here you can use my repo directly and remove / add services you want.

For the addition of a service you need to do the following:

Setting up your monitoring

I use two other VPS for my monitoring, one that need to be a bit beafy to host your ELK, and a smaller one used for alerting. As I have lots of logs I have to host my data on my storage server. Every server as an elastic agent installed and reporting to my ELK. When alert is detected on the monitoring of ELK I add an entry in my errors index with a boolean notified set as false, and a description of the issue. I then use an homemade script to detect changes on the error index using the notified boolean, notifies me with an email then set the boolean as true.

Now you should be able to run your own infrastructure !