Jekyll2023-05-18T18:56:35+02:00https://www.onemarcfifty.com/feed.xmlonemarcfiftyMarc's personal blogMarc AhlgrimRouter On A Stick with OpenWrt and Raspberry Pi2023-05-08T00:00:00+02:002023-05-08T19:00:00+02:00https://www.onemarcfifty.com/blog/video/router-on-a-stick<h2 id="router-on-a-stick-with-openwrt-and-raspberry-pi">Router On A Stick with OpenWrt and Raspberry Pi</h2>
<p>We will use a Raspberry Pi and a managed Switch in order to extend the Pi’s Ethernet Port to a LAN and WAN Port using VLANs with OpenWrt. This will effectively turn the Pi into a Router with a WAN port and multiple LAN ports. I will be using a Netgear GS308E Switch (ASIN: B07PDHVZNS), but you could use the following switches as well:</p>
<ul>
<li>Zyxel GS1200 (recommended): <strong>ASIN B0798PKGFQ</strong> (8 Port) or <strong>B08K2KKK86</strong> (5 Port)</li>
<li>Netgear GS305E: <strong>ASIN B07PHNTV45</strong></li>
<li>TP-Link TL-SG108E: <strong>ASIN B00JKB63D8</strong>, 5 Ports (TL-SG105E): ASIN <strong>B00A128S24</strong></li>
</ul>
<p><a href="https://www.youtube.com/watch?v=fOYmHPmvSVg"><img src="/assets/images/thumbnails/fOYmHPmvSVg.png" />Watch the video on YouTube</a></p>
<details>
<summary>Click to view the entire transcript</summary>
A router usually has at least two Ethernet ports – in the Consumer segment that’s usually one for LAN and one for WAN. This little Raspberry Pi 2B here only has one Ethernet interface. So what options do we have if we wanted to turn it into a router? We could of course just add another Ethernet interface over USB. Or – we could do the following: We use VLANs and a managed switch in order to extend that one Ethernet port to multiple ports. Our Pi could then have not only two but three or four ports on different networks.
Just before we do that - to make it clear from the very beginning – I am not advocating in any way that this would be a good or viable solution as an Internet or Wifi router for your home. The Pi is difficult to get, it’s very expensive at the moment of making this video in 2023 and there are many limitations which we will look at throughout the video. However, the solution shows nicely how you can extend any device with one Ethernet port to multiple ports using a switch. In other words - that could as well be an old PC or Laptop or thin client for example. It doesn’t have to be a Pi.
Let’s go step by step. First let’s install OpenWrt on the Pi. We could as well do this without OpenWrt – with plain Linux – but – with OpenWrt we have a nice GUI – and also it’s made for routers.
(Installing OpenWrt)
Let’s first browse to OpenWrt.org and select the right image. I klick on the Link for the Firmware selector where I can then chose the Raspberry Pi Model and also the desired Version of OpenWrt. Chose the Squashfs Factory Image or the Ext4 Image. With the Squashfs Image you can be reset to Factory Settings at any time. With the ext4 image that’s not possible. Flash the Image to the SD Card with dd or with the Raspberry Pi Imager. Let me quickly show you both methods. In the imager you would click on “Choose OS”, then scroll down to “Use Custom” and select the img file that you have downloaded from OpenWrt. I did not have to unzip it first. Selecting the gzip file directly did work well. Click on Storage in order to select the SD card that you want to write on. Under Linux this wants to run as root. If you are in the sudoer group then you will just be asked for your password. Klicking on Write starts the process. When it’s finished it will tell you so. Now for dd from the command line. This also needs to be ran as root so I’ll become root quickly. Now just to be safe – I’ll do an fdisk -l in order to list all disks. I don’t want to accidentally erase my hard drive. In my case it’s sdb. Make sure you double check on that one. As dd is just writing bit by bit, I need to gunzip the image file first and then run dd. The infile is the image from OpenWrt and the output file is /dev/sdb plus I chose a blocksize like 4096 for example in order to have dd write larger blocks. Here we go. Smooth. In either case I can now put the SD card into the Raspberry Pi and power it on.
(Logging into OpenWrt)
Once I have connected the Ethernet interface of my Laptop to the Pi, I can then connect to the web Interface on 192.168.1.1. OpenWrt by default acts as a DHCP Server. The default login is root without a password – so I’ll quickly change that under system – then administration in order to get rid of that warning. If I go to network – then Interfaces – I can see the only interface here is the LAN interface. But before we move on with the Pi, let’s have a look at the Switch.
(Configuring the Switch)
I will be using a Netgear GS308E switch. That’s an 8 Port managed Gigabit switch. It’s important to use a managed switch for what we want to do as we need VLAN-awareness. This switch family comes in various sizes. There is a 5 port version which you can buy for roughly 20 Dollars. There are unmanaged versions without the E in the name or POE versions with 5 or 8 ports such as the GS308EP that sells for roughly 80 dollars. TP-Link has a comparable switch family, the SG switches. They have a similar naming convention and also comparable prices really. If you don’t have a managed switch and if you want to buy one, then I can recommend the Zyxel GS1200 in that same price range. It does have additional features such as Link Aggregation plus you can chose the VLAN for the Web interface. I’ll come to that one later. So as you can see – you have the choice.
Now let’s configure it. For starters we would need at least three ports here. One Port for the LAN, one for the WAN and one for the Pi itself. By default every port on this switch is untagged on VLAN 1 – let’s change that. I want to use the following VLAN numbers. 1000 for the WAN and let’s say 99 for the LAN. In order to do this, I click on VLAN, then 802.1Q, then on “VLAN Configuration” here under “Advanced”. Here I tick the “Enable” Radio button. As you can see here – all ports are currently on VLAN 1. Under “VLAN Membership” we can see that they are “Untagged” – that means that they are participating in this VLAN but they are not adding a Tag to the Ethernet packets containing the ID of the VLAN. In Cisco terminology that would be called “Port VLAN” as opposed to “Trunk VLAN” for a tagged port. The “U” means that everything for that VLAN will be dispatched to that port and also be stripped of any tag. So first I need to add the VLANs that I want to use. I just type them in under VLAN configuration and confirm with Apply. Next I want to assign the ports to the VLANs under VLAN membership. So let’s say Port 8 is going to be the WAN port and port 6 will be the Pi. Therefore Port 8 will be untagged and port 6 tagged or trunked. Apply. Now the LAN. That’s VLAN 99. Let’s say that should go to port 7 untagged and again tagged to the Pi. We do also need to assign the PVID – the Port VLAN ID to the untagged ports. Why? When a packet comes in untagged, then we don’t know which VLAN it should belong to. The “U” for untagged in the previous screen is relevant for egress – meaning we dispatch the corresponding VLAN to that port. The PVID here is relevant for ingress – for incoming packets that do not have a tag. It’s kind of an assumption we make. If ever a packet comes in on port 8, let’s assume or rather define that it should belong to the VLAN 1000. We would not expect any untagged packets on port 6 as we will configure the Pi to tag everything. So in theory we could leave the PVID at its current value. However – I am not a friend of VLAN 1. So many hardware products have this hard coded, I really don’t want any undefined traffic here. Problem is that I can’t remove the Port 6 from VLAN 1 because it still has 1 as the PVID. So what I tend to do in my network is that I define an additional VLAN ID which will actually never be used. Let’s say 55. So let me add this and then add port 6 to that VLAN tagged. While I am at it I can do this with all other ports as well. Let me add them to VLAN 55 and then set the PVID to that unused VLAN. I’m not changing port 1 for the moment because that’s the one I am connected to. Last but not least I will remove all of them from VLAN 1. You could as well add all those unused ports to let’s say the LAN port by assigning them untagged to VLAN 99 and setting their PVID to 99 as well. The important part is that Port 6 is tagged on everything and that ports 7 and 8 are untagged. Port 6 should be the only port that participates in 99 and 1000 at the same time. This is the port where we will connect the Pi to. Now let’s tell the Pi about those VLANs.
(configuring the VLANs on the Pi)
In LuCi on the Pi I click on Network – Interfaces, then I select the devices tab. Here I have a bridge called br-lan. I will be using DSA and Bridge VLAN filtering here. So edit that bridge, go to the Bridge VLAN filtering Tab and enable VLAN filtering. Now I add VLANs 99 and 1000 and tag them. Save. Very important: Do not click “Save and apply” during the steps we perform here. We first need to do all changes and only then we apply them. Otherwise we will lock ourselves out. If ever you lock yourself out, please do nothing – just wait for LuCi to revert the changes after 90 seconds. So just wait a minute and a half and LuCI will give you the option to revert. Next, let’s add the WAN interface as a DHCP client and of course assign it to VLAN 1000 on the bridge. Just double checking that the “Use Default Gateway” is checked which means that OpenWrt will use this interface as a default gateway. As we called the interface “wan” – it is now already assigned to the “wan” firewall zone. Again. Click on Save BUT NOT on “Save and Apply”. Last but not least we need to tell the LAN interface that it should be on VLAN 99. Let’s edit the interface properties and change the device to br-lan.99 – save. Now we could save and apply but I want to double check everything by clicking on this “Unsaved Changes” icon up here. Yes – we add the VLAN 99, the VLAN 1000, we assign WAN to 1000 and LAN to 99. Looks good. Save and Apply.
Now I will lose Connection. This is expected. LAN and WAN now expect tagged packets. That means that I now have 90 seconds to connect everything to the switch. My Laptop to port 7, the Pi to port 6 – here we go – back in business. Now I can connect the WAN to port 8 – yep – it gets an IP address. Let me quickly see if DNS and the like are working. Network – Utilities and ping Openwrt.org – that works – we are connected.
OK – this has been a lot of click here, click there – what have we done here? Both the Pi and the Switch on port 6 now add VLAN tags to all packets. There are two tagged VLANs – 1000 for the WAN and 99 for the LAN. The switch removes the tags from the packets coming in on port 6 and patches them to port 7 (if the tag was 99) or to port 8 (if the tag was 1000). The other way round, any untagged or “normal” packet coming in on port 7 will be tagged with VLAN 99 and sent out tagged on port 6 – to the Pi. Same for the port 8 with VLAN 1000. We have successfully turned one Ethernet port into two by using tagged VLANs and Port VLAN IDs. Awesome – now we do have a working router with a WAN port and up to 6 LAN or other ports – depending on your switch really. Now here’s two more things that I want to show you and also a couple of remarks with regards to performance and security. Let’s start with security.
(Config Summary)
Here’s again a short summary of the config that we wanted to achieve. If you got lost or locked out at any time, then alternatively you can mount the sd card of the Pi on a Linux Workstation and open the /etc/config/network file – this is what it should contain. I’ll put a copy on Github for you.
(Security remarks)
Using the same wire for WAN and LAN traffic as such is not a big problem really as long as you avoid using VLAN 1 and as long as you avoid untagged traffic. We did that – that should actually be quite secure – at least by home user standards. What can pose a serious security problem though is that switch. The switch has a web interface. And that web interface is bound to the IP address of the switch of course. Now – unfortunately – I can’t tell the switch which VLAN it should be listening on. That means that potentially the web interface of the switch could be bound to the WAN interface. In other words, the Web interface of your switch could be accessible from the WAN. That’s definitely a show stopper for me. I would not recommend using this config as an Internet router. You would of course still be somehow protected by your ISPs router if you connected the WAN interface of the Switch to the LAN interface of your ISPs router, so if you would create a double NAT with a DMZ – a demilitarized zone – between the ISP and the Pi. But still – you have no control which interface the switch would bind its GUI to. That’s sad. The Zyxel GS1200 does NOT have that problem. That’s the reason why I can recommend it. So – again – If I needed to buy a new switch, then I’d rather use the Zyxel.
(Performance remarks)
Now let’s have a look at some performance benchmarks. While everyone is talking about 10 Gbit connections, This Raspberry Pi has a 100Mbit interface. So realistically we can’t get much higher than let’s say 80 or 90 Mbits even with full duplex because we use the same wire for WAN and LAN. I did some tests here with iperf3 and as you can see it’s not only maxing out at 80 or 90 Mbits really, but also there are a lot of retries here in iperf3. Now – that might be related to pause frames created on the switch as I have connected everything to a Gigabit backbone. Now I wanted to make some more tests with dedicated USB Ethernet adapters. In theory I should be able to get to something in the 300 Mbit range with a USB Ethernet adapter. I can’t get higher on this Pi, because it only has USB2. The Rasperry Pi 3 has Gigabit Ethernet but still internally wired to the USB2 controller. So that has more or less the same limitations really. If you want to get close to Gigabit Speed, then you would have to use a Raspberry Pi 4 or any of the more modern versions such as the Compute Module etc. Anyhow, Here is how to add USB adapters. Just plug in the USB Ethernet adapter, then check the Syslog in order to get an idea which driver you might need. In my case it seems to be ASIX, so go over to system-software and search for kmod-usb-net and see if there is a module available for that hardware. I’m lucky, there is one for the ASIX. So let me install it. Checking back in the syslog, I can see that it has now been recognized as an ethernet adapter. Now I can change the WAN adapter to eth1. Or rather add a new interface and disable the old one. Then I’ll do the same game for the LAN adapter. Here I just change the device. Ready to go? Let’s test again. And – as you can see – this is better than before, slightly better bandwidth. Few if any retries, but still we seem to be stuck at around 100 Mbits really. I tried offloading as well but that didn’t help. Looks like the drivers only use one CPU really.
(Additional Configuration)
There is some additional configuration that you might want to do on the Raspberry Pi. For example if you look at the available disk size here, then you can see that the partition is just 104 Mbytes in size even though I have an 8 Gbyte SD card here. We can easily fix this. First I need to ssh into the router, then I’ll update the opkg sources and then install the fdisk software package. The first step is to increase the size of the partition. So let’s get a list of all partitions. This is the SD card apparently. /dev/mmcblk0p1 and p2. In order to resize the partition we launch fdisk on the block device and print out the current partition table. We need to take note of the start sector of this 2nd partition here. Now we delete the partition. Yes – we delete it. No worries, before we write the changes onto the disk, we recreate it. So “d” for delete, partition 2, then immediately “n” for new partition, “p” for primary”, again partition number two, and this is the critical part – we need to fill in the same start sector as before. Keep the end sector because by default this is the maximum size. Now fdisk tells me that there is already an ext4 partition. And I do for sure NOT want to remove this – so No. “P” Prints out the current layout – as you can see, the second partition is now 7.4 Gigs in size. And the start sector is the same like before. So let’s write this by typing “W”. Now we need to resize the file system on this partition. There is a nice article on the OpenWrt Forum that describes how to do this on X86. The procedure is more or less the same on the Pi really. If we scroll down, it describes how to do things on UEFI and the like, it shows the fdisk maneuver which I have just done and here’s the interesting part. Normally we can’t resize the file system if it is mounted. But what we can do is that we create a loopback device to that partition and do the operation on the loop device. Let me give this a shot and just copy paste the code. This fails badly. Apparently it can’t identify the devices properly. So let me do this by hand. First I set up the loop device, then run a file system check, resize the file system to the maximum of the partition and then reboot. Again – remember – 102 Mbytes here. Reboot. And after the reboot I have more than 7 GB available. Ready to install some software on this box. I could as well have taken the sd card out, attach it to a USB reader and do the resize on a different system using gparted or the like.
(Some remarks on Wi-fi)
We haven’t talked about Wifi yet. This Raspberry Pi 2 does not have any Wi-fi. Of course I could use a Raspberry Pi 4 – or I could add Wifi hardware over USB. But – let’s face it – none of these makes a good access point. The drivers and the hardware are designed to be used in STA mode – as Wi-fi clients. Just because there is a linux driver for a Wifi module, does not mean that it is great in Access point mode – look, you would not get features like MU-MIMO or the like – plus, the total amount of money that you need to spend will be much more than let’s say the price of a really good Wi-fi 6 router such as the Belkin RT 3200. You can get that one at around 80 to 100 Dollars and it’s a really nice device.
(What can we use this for?)
So -whats left? What can we use this for? In my opinion the real strength of this setup as opposed to a commercial Wi-fi router for example is the amount of RAM that we have available plus the fact that we can extend the file storage considerably by adding a larger sd card or an external ssd drive. So I would rather see this as an application server that needs multiple Ethernet ports – VOIP for example. You could add asterisk or yate as software packages and run the thing as a PBX. You could hook up multiple phones to it and it would really not use a lot of energy to run. But still – I would not go and buy any of that equipment new at the current price points. If you have them in a drawer or if you plan to use them for something else in the future – then maybe go for it. In one of the next episodes I will also show you a possible application of a Raspberry Pi with OpenWrt as a phone tethering gateway. We will not even need a switch for that. So – here’s another reason to stay tuned to the channel right? Please do leave me a comment on YouTube! Many thanks for watching! Stay safe, stay healthy, bye for now.
</details>Marc AhlgrimRouter On A Stick with OpenWrt and Raspberry Piexample.com in Proxmox VE2023-04-08T00:00:00+02:002023-04-08T19:00:00+02:00https://www.onemarcfifty.com/blog/video/example_com<h2 id="running-the-examplecom-domain-in-proxmox-ve">Running the example.com domain in Proxmox VE</h2>
<p>Let’s create a fully blown example.com domain on Proxmox including Certificates and e-Mail in less than 20 minutes! You can use this domain to test drive software or as a staging environment.</p>
<p>The blog article w/r to the certificates <a href="https://www.onemarcfifty.com/blog/Portainer_TLS/">is here</a></p>
<p>The deployment scripts are <a href="https://github.com/onemarcfifty/example.com-proxmox">on my github</a></p>
<p><a href="https://www.youtube.com/watch?v=jZjysgVk9wU"><img src="/assets/images/thumbnails/jZjysgVk9wU.jpg" />Watch the video on YouTube</a></p>
<details>
<summary>Click to view the entire transcript</summary>
(the use case)
The example.com domain is a special domain that is owned by the Internet Assigned Numbers Authority, the IANA. It has specifically been designed for documentation purposes. What does that mean? Whenever you look at software for self-hosting, then you will find that the config files of that software usually are using the example.com domain. This is great. Everyone knows that they have to replace that with their own domain name. However – if you just want to test drive a Software and see if you like it or not, then you will have to go through a more or less painful process of using nano or vi and adapt the config files before you can do that. If the software needs certificates or uses e-mail, then you will need to handle that as well before you can even start it up. So I thought – why not just build a real example.com domain? Isolated – with its own network – here on my Proxmox server. This way I would not have to change config files or the like if I just wanted to test drive. Now – creating a domain as such is no rocket science. You just need an isolated network and a DNS Server. We’ve done that before on this channel. But there are three things that would be real game changers. First – have the whole setup in a ready-to use, repeatable or “canned” environment. Click and Go. Second – readily deployed certificates. And - last but not least – a fake e-mail server intercepting any mail and delivering it to a mailbox that can be used with your favorite e-mail client. As an entry point into that platform. I’ll use a container and install a graphical Linux environment that I can access with RDP. I will show later how to use a MACOS or a Windows Machine for that. You can use Linux but you don’t have to. DNS can be done by a small router VM that will also act as a Gateway to my real LAN and from there to the internet. For maximum Flexibility I’ll also add a Docker host inside the environment. The e-mail server is a Docker Container running on the Docker host. I’ll also put Portainer here and of course I’ll have certificates everywhere. I will create and distribute them at install time. Now all I need on the client is a Browser and a Mail client. I’ll use Firefox and Thunderbird for that. To make editing of config files easier, I’ll also add filezilla and vs code to the client.
(Installation Instructions)
So – here is how to build this. There is very few manual interaction required. You can find the required software and instruction on my Github Server. Log into your Proxmox GUI, go to the Network node and create a Linux Bridge. This will be our virtual network. Give it a name – for example vmbr111. Tick the boxes “Autostart” and “VLAN aware” – that’s it. Leave all others blank. Click on Create and then on “Apply Configuration” up here. Next – open a shell on the Proxmox PVE server either from the GUI or over ssh and copy paste the commands from the README on my Github Server. If you have git installed, then you can clone the repo, if not – download it as ZIP and unzip it. Do a cd into the subfolder where you have put the software and quickly review or edit the config file. Little Tip here – you may of course run nano or vi inside the shell. Or – you connect to your Proxmox using WinSCP or FileZilla and edit the file there with your favorite editor. I like to use Visual Studio Code for this. The config file contains very few settings. Basically the names of the network adapters that will act as WAN and LAN inside our environment, the name of the domain – for instance example.com, but you could use any other domain if you wanted. Next – the storage where the containers and VM will be put on your Proxmox, the name of the templates to use and a list of available Container or VM IDs. Last but not least a download URL where we can pull an OpenWrt image from.
Cool – once you are happy with everything, just type ./deploy-sandbox.sh and the script should run. It will first give you a short summary and ask you to press Enter. You may stop it at any time by typing CTRL-C. Next, it’s asking for a password for the root user in the test box. Please do not use any characters here that might interfere with the script such as Dollar, Exclamation Mark, Quotes or the like – I do not do any filtering here. The password will not be shown as you type. Next, it asks for a non-root user – thats the name of the user that we will use to log in to the platform later. And the last question is the password for that user. Now you will see a lot of text fly by. The first thing that the script installs is the perimeter router. Once it has done that it will actually stop so that you can check if your router has internet access. We will need internet access for the next steps actually. So go to the Proxmox GUI again and open a shell on the newly created Router. Just type enter to activate the shell and see if you can ping www.google.com. That’s it. Go back to the script and press Enter. Next – the script will set up the containers. After it has done that, it will stop again so that you can double check if the containers have the right settings. Make changes as needed. The Client container should have two network interfaces. eth0 in your real LAN and eth1 in the virtual LAN. As I am using VLANs at home I can just add the VLAN tag. But that’s related to my specific config here at home. Back to the script, press Enter. Now it will run through unattended until the end and install all required components for you. At the end it prints out the IP address of the Client container. Just to be safe, restart the router and the two containers to make sure they’ve picked up everything.
(How to use the platform)
You can now connect to the client container by using let’s say Remmina from Linux or Remote Desktop Connection from Windows with the Non root user and password we specified. This is what you should see. A MATE desktop running on Debian. If you don’t like MATE and rather want xfce4 or cinnamon or the like, then just replace mate-desktop-environment with the one you want in the client init script. If you don’t like Linux at all, then just use any other VM and hook it up to the virtual bridge that you created in the beginning. You can then use it as a client. For now, let’s use this client. You may want to browse to the portainer interface of the docker host first. That’s https://exc-docker.example.com:9443 – You should be prompted by Portainer to create a new admin account. If this timed out for security reasons, then just either restart the docker host from the Proxmox GUI or log into it and type “docker restart portainer”. Specify a name and a password and then connect to the local environment. That’s it. We’re in. If you click on “Containers” then you should see Portainer, the Portainer agent and the imap server up and running. Please note that little padlock here – all TLS. If we examine the Certificate then you can see that this is actually a Wildcard certificate and a self-signed CA which the script has generated and also imported into the Firefox Browser so that you don’t get any nasty warnings here. I’ve made the certificates valid for ten years. Please only use them in this test environment. A copy of all the keys and certificates can be found in the /etc/certificates subfolder of the client and of the docker host.
Now let’s check the e-Mail integration. I have installed Thunderbird as a mail client here. When you start it for the first time, then it will ask you for a Name, a Mail address and a password here. You are free to chose the name. For the mail address, type in bob@example.com. The password is literally just “password” – all minor letters. If you click on “continue”, then Thunderbird should actually find all the server names. Still – we need to click on “Configure manually” because the user name is not just bob but rather bob@example.com. Apply that by clicking on “Done” and it should go through. The IMAP and SMTP Servers do have TLS certificates from the same CA like Firefox has. The script has also deployed those certificates to the Thunderbird client. You can now send mail to any example.com mail address. Like for example alice@example.com. The imap server has a catch all mailbox. Let’s open that one as a second profile. Click on the top right menu, then “Account Settings”. Here under “Account Actions” click on “Add mail account”. Do the same steps like before. Just this time the mailbox and account are debug@example.com and the password is “debug” – all in minors. Whatever address you now send mail to, it will end up in this catch all mailbox here. I found this a great way to have e-mail without needing to go through the internet.
The container that I am using by the way is a fork of Antonio Espinosas project on Github. He did all the heavy lifting. All I had to do was adapt the container to debian 11 and add some bells and whistles really. Antonio – many thanks for the great project.
(Additional use cases)
Awesome – that’s more or less it – please let me know in the comments if this is useful for you. Also – if you could think of any useful extensions, maybe a code server on the docker host or the like. Do leave me a comment on YouTube please. Thank you very much.
(How to use Windows client)
If you wanted to use a Windows or MAC client, then this is how you could do it. Let’s say I want to use this Windows 11 machine here. All I have to do is to go to the hardware node, select the Network Device, and change it to the virtual example.com bridge. When I now boot the VM, it will be inside the example.com Test box. If I now use let’s say Microsoft Edge to browse to the Portainer then of course I’ll get the certificate warning here, because I don’t have the CA installed. But no problem – we can easily download it from the web page itself. If you click on the “Not secure” icon here, then on the red warning, you see this little certificate icon up here? Click on it. That shows the details of the self signed certificate. Click on “Details”. Here we select the AAA_TestCA entry and click on Export. Now close that window and click on the three dots in the upper right corner. Then settings. Search for “cert” – now click on the “Manage certificates” link here. That will bring you to the Certificates applet. On Windows, the certificates are managed by the OS, not by the browser. Select the “Trusted Root CA” tab, then click on “Import”. Browse to the file that you have just downloaded and click through with Next-Next. You’ll get a security warning – confirm with yes – Now we can close everything. You’ll have to close and reopen Edge so that the changes are taken into account. If we browse to the site again – no more warning. Beautiful. Now for email. Add a new account, select the advanced setup. Then Internet eMail. Enter the information as shown on the screen, untick the SSL boxes – we are using STARTTLS. Click on OK. That should all go through. If you want to add the Catch All Mailbox then do the same thing again with the debug account.
(MacOS)
With a MacOS VM – same game – hook it up to the virtual network interface and boot. Once I had logged into MACOS, I used FileZilla to download a copy of the certificates from the Docker Host. The install script has placed a copy in /etc/certificates/example.com on the server. I then launched the Keychain App and imported the rootCA, the IMAP Full chain and the Wildcard full chain into the system node. I then did a double click on each one of them and set the Trust level to “Always Trust”. Now I could just add the email Account as before and also browse to the Portainer interface with no issues.
(The network layout)
Just a quick remark with regards to the IP address range of the test environment. By default, OpenWrt uses the 192.168.1.1 address. If you are using the same address range in your real LAN, then you would have to move the example.com environment to another subnet. In order to do this, you need to change the ROUTERIP setting in the config file and also run this one-liner on the router once it has come up, when the install script halts to check if the router has internet access.
(Summary)
Awesome – we now have a working example.com domain with an isolated virtual network, and all bells and whistles that we need in order to test drive software without having to change too much in config files etc. Here are just some more ideas what you could do with it. Maybe you want to have two sites and test things like wireguard connections between the sites. In this case you could just spawn the example.com domain twice, with different container ID’s of course and have the routers talk to each other or port forward or the like.
Another idea I had was to actually use a second environment with my own domain name as kind of a staging environment. That means, I could test in the example.com domain and then see what I need to change in order to run it in my own domain. The nice thing with Proxmox is that you can take a backup of a container or VM and then restore it with a different ID. I could therefore test everything let’s say with container ID 100 in the example.com domain, take a backup and restore it with let’s say container ID 200 and hook that one up to the production environment. Or I’d do it the other way round. If I want to test something on a production container, then I would just back it up and restore it in the staging environment and run my tests on the copy. Once I’m happy with everything, then I could do the changes on the production container. This way I would have minimum interruption in the real environment and achieve a good Family Acceptance Factor.
That’s all I wanted to show you today. Thank you so much for watching. Please do not forget to leave a comment and a like. Stay safe, stay healthy, bye for now
</details>Marc AhlgrimRunning the example.com domain in Proxmox VEAuthelia on Proxmox - 2FA SSO with Nextcloud, Proxmox, Portainer Gitea OpenID Connect Single Sign On2023-03-07T00:00:00+01:002023-01-30T18:00:00+01:00https://www.onemarcfifty.com/blog/video/Authelia_Proxmox_Video<h1 id="running-authelia-on-proxmox">Running Authelia on Proxmox</h1>
<p>This video is really following up to <a href="https://www.onemarcfifty.com/blog/Authelia_Proxmox/">this blog article</a> which I published earlier here on my blog.</p>
<h2 id="video-content">Video content</h2>
<p>I will show how to Self-host Authelia in a Proxmox Container and use it as an OpenID Connect (OIDC) Identity Provider for 2FA Single sign On (SSO) with Nextcloud, Proxmox, Portainer and Gitea.</p>
<p>#nextcloud #proxmox #sso #portainer #gitea #authelia #openid #oidc #selfhosted</p>
<p>The Github Repo is <a href="https://github.com/onemarcfifty/authelia-proxmox-SSO">here</a></p>
<details>
<summary>Click to view the entire transcript</summary>
(Intro – VLAN / NGINX / SSO)
I am self hosting a bunch of applications. Nextcloud, Proxmox, Portainer, Gitea and so on. The more applications you have, the more user names and passwords you need to manage. Quite a challenge. Or – you use SSO – Single Sign on. Let me show you this. I browse to my NextCloud instance. But rather than logging in with username and password, I click on this link and am redirected to Authelia for login. Authelia asks me for a user name and password as well as a second factor. Here we go, I am logged in. How does that help or make anything better than before? Let me browse to my Proxmox Server UI. This is where the beauty happens. As I am already logged in to Authelia, I can now log into Proxmox without having to type anything again. Just need to consent and I am in. Also – Up here I am not root or admin, but I am Marc. Same with Portainer. Click on SSO, consent. Logged in. One single account for all my applications. Very convenient.
But first - here’s the breakdown of this episode. Feel free to browse to the corresponding time marker. I’ll explain how everything works, why I chose Authelia over anything else, explain the necessary setup steps in detail, then we’ll configure Nextcloud, Proxmox, Portainer and Gitea and I will then outline a couple more use cases. If you already have Authelia and you’re just interested in installing the SSO bit then go to that time marker here.
(How does that work?)
Here’s the blue print. The core is Authelia running in a container on Proxmox. You can run it in Docker as well. Authelia has a Database or a simple file with all the user accounts in it. That could be LDAP – in my case it’s just a simple text file. I only have a bunch of users, like ten or so. If I had hundreds or thousands, then I’d use LDAP for that. If I wanted to add Authentication to a web site that is not protected, then I’d just add a reverse Proxy in front of it and require Authentication prior to Access. But after that I would still have to log in to that Application. So here’s the second feature of Authelia. It can act as an OpenID Connect (OIDC) Identity provider. So if my Web App can use that, like Nextcloud, Proxmox, Gitea or Portainer can do – Rather than asking for a user name and password, they would just check with Authelia if I am already logged in. If I can present a valid cookie from Authelia, then I am logged in – if not, then I would be redirected to the Authelia Login page. Once I am logged in there, I receive a new Cookie and am then redirected to the Application again. The login is therefore done by Authelia and the Application just trusts it. That brings a couple of advantages. I can log in with the same account everywhere – I have Single Sign on, SSO. Once I am logged in to one application, the login is valid for other applications. Last but not least, I can require a second factor for additional security. That can be a Time based One Time password (TOTP) with Microsoft or Google Authenticator or Duo or the like. Or it could be a push Authentication with Duo as a service provider. They offer ten free accounts. Or – it could be a Webauthn FIDO2 key like those Yubi Keys.
(Why Authelia?)
Why did I chose Authelia over other solutions? Well, first off – it’s free and Open Source under the Apache License. I am a big fan of Open Source. Second – I can easily self host it in a Container. I don’t have to rely on 3rd party service providers. And Last - but by no means least – They do have a clear and concise Road Map! They do have a clear view where they want to go and when they will do things – that’s really essential and unfortunately not true for many other open source projects. If we look at the OpenID Connect Roadmap, we can see that they plan 8 (!) beta phases before GA. That’s quite thorough planning I’d say. I like that. Also – there’s a bunch of other functionalities in the pipe – like multi domain, multi device WebAuthn registration and passwordless login. I personally feel quite confident about this.
(Setup Steps – Overview)
Here is an outline of the necessary setup and configuration steps. We will first do what I call a rudimentary installation – without start up checks. Just to see if we can get the software up and running. In a second step we will then adapt some settings - Domain names, e-mail server parameters, the SSL certificates. Then we enable the startup checks and get everything running with a fully blown config. Next, we will register a 2FA device and then hide Authelia behind an NGINX reverse proxy. Last but not least, we will add the OpenID connectors to our applications and then use SSO happily ever after ;-) By the way – there is also an article on my blog site www.onemarcfifty.com describing the installation. I have just packed everything into an automated script on github for you.
(1. rudimentary install)
OK, let’s start with a rudimentary install of Authelia. After this step, we will have authelia up and running, but not configured yet. First we need an LXC container in Proxmox. It needs to be a privileged container because we need the /dev/random and the /dev/urandom devices really. I have given it 2 GB of RAM and 2 Cores plus 8 GB of Disk. This is probably even a bit over-sized. As a template I used the latest Debian 11 template. Right – once you have the container up and running, just pull the installation sources from my github using this one-liner here. That should take care of all installation steps and give you an up and running authelia server. In order to test this, just browse to http port 9091 of that server and you should be presented with the authelia login screen. Use http here, not https. We don’t have certificates yet. Just we can’t really use Authelia yet because we need to tweak some more settings.
(2. Adapt settings)
Now comes the hardest part. We need to adapt the config file. Authelia heavily relies on e-mail. This is how users reset their passwords and also how they register 2FA devices. In other words – we need a real e-mail account. The second barrier are certificates. You will need TLS/SSL Server certificates. If you don’t know how to do that then please watch the second episode of my X.509 Certificate series. Plus we will need to adapt the domain and server names. Now – all these settings can be changed in the config file /etc/authelia/configuration.yaml. However – editing a yaml file with nano or vi is – well – painful at least. That’s because YAML is very picky with regards to indentation. So here are some alternatives. Either you install vs code server in the authelia container – like we did in the rundeck video. Or you use winscp or Filezilla and access the server over ssh. In order to do this, let’s temporarily enable ssh root access with password by changing the /etc/ssh/sshd_config file. Restart the sshd service and now we are able to edit the file using a GUI editor like vs code or notepad++ or the like. The settings we need to change are outlined in my blog article as well. I’ll change the default redirection URL and the TOTP issuer. I review the policies and adapt them to my needs. I’ll just request 2FA for everything in my domain and deny everything else. Then I put in the SMTP Server details. The passsword for the SMTP Server has been put into the file smtp in the .secrets subfolder of the /etc/authelia directory. It has been randomly generated at install time. You can either copy paste that and set the mail password of your provider to that secret or you overwrite the secret with your mail password. Once we have that in, then we can enable the startup checks by setting the disable_startup_check value to false. Let’s save everything and see if authelia still starts. It now tries to connect to the mail server at startup. If it can’t do that then it will not start. You will need to review the settings. Perfect. Once we have this running, then we can activate TLS. For this we need to add this TLS block here. Just point it to the key and the full certificate chain of the server certificate. I am using a self signed certificate in this test sandbox. Again, save and restart Authelia. If everything went well until here, then we should now be able to browse to port 9091 using https rather than http. Perfect. Just one more step and we’re done with the basic config here. The installation script has created a file called users_database.yml in the .users subfolder and added four example users – Bob, Alice, Dave and Frank. Of course we want to edit this and point it to real user accounts with real mail addresses. Don’t worry about the password. We will reset it in a second. You can also add additional parameters such as groups. Right – again, save and restart authelia. You should now be able to use the “reset password” link, receive an email that is valid for five minutes btw, and set a new password. Some helpful troubleshooting tips if anything goes wrong in this phase: Check the authelia log and the service status using these commands.
(3. Test 2FA)
Cool – now let’s test 2FA. If I click on register device here, I do again receive an email with instructions. The link in the mail again is valid for 5 minutes. Now I can either register an Authenticator app such as Microsoft Authenticator, Google Authenticator, Okta or Duo or the like – it doesn’t matter, they should all be able to scan that QR code and add the Authelia Secret to their config. Alternatively – if you have a Yubikey or any other Fido2 Webauthn key then you can as well register that. Depending on the policy that you set in the config file, you will or will not be prompted for a second factor once you log in. If you want to use DUO push notification then you need to register on their site and get the details like hostname, integration key and secret_key. The first ten users are free of charge at the time of making this video.
(4. Hide behind NGINX)
Awesome – now we have a working Authelia instance using https and we can now hide everything behind NGINX. The installation script should already have installed NGINX in your container. All we need to change now is the listen address in the authelia config file and point it to the local host 127.0.0.1 and also adapt the server name in the NGINX config file which you can find in this path here. Again, use Filezilla or Winscp. Once we restart both services, we should then be able to access Authelia using https on the standard port 443.
(5. Add OpenID Connect to the apps)
Perfect – now let’s add Authelia as an OpenID Connect provider to our apps. Let’s start with Nextcloud, then we’ll do Proxmox, Portainer and Gitea.
The steps for the different platforms are clearly outlined in the authelia documentation. There is a whole section with examples for many platforms. In general, we need to tell the App that it should use OpenID and we also need to tell Authelia about it. On the Authelia side this is done in the configuration YAML File. We first specify some generic settings under the identity_providers section, such as the validity duration of a login and who we accept requests from. Then we create an entry for each app under the clients subsection. Each client app needs to have a unique ID, a Secret Token, an Authorization policy and we need to tell Authelia which URLs are valid to be redirected to. This can be multiple destinations. Also, depending on the app, different scopes are possible. Usually you should always have openid and profile in here, plus maybe email and groups. The hardest part here is really double-checking on all the information for typos etc. plus of course the indentation of the YAML syntax. Cool, now let’s add the client apps. Quick remark here – please do never use the Client secrets provided in the examples, but always create your own and make sure they are long – like at least 30 characters long. You will never have to type them by hand anyhow. This simple one liner from my cheat sheet repo can help you create them on the fly.
For Nextcloud, we first need to install the OpenID Connect Login App. Let’s do that here in the Nextcloud GUI under the active apps
The configuration for the OIDC provider is done in the config.php file which by default is located in /var/www/html/nextcloud/config.
The only settings you need to change from the template on this page really are the login provider URL which should point to the FQDN of your authelia server and the client secret. Please use a long random character chain for the secret. This one liner here from the setup script or from my cheat sheet repo on Github can help you create one.
On the Authelia side we can also just copy paste the config section into the config file. The secret needs to be the same like the one that we just created on the Nextcloud side. Just – it needs to be preceeded by the term $plaintext$ - I will need to check if we can specify this encrypted as well.
Once you restart Nextcloud and Authelia, then you should be able to use this link in Nextcloud and be prompted for login by Authelia. On successful login you would be redirected to your Nextcloud instance. Please keep in mind that Authelia will only forward you to secure web sites. That means Web sites that use https and have valid certificates. If you don’t use https then this will not work. Authelia will categorically refuse to forward you.
When I did all this, I encountered a couple of problems. First I had to remove this line for the alternate login page from the authelia template – that page does not exist on my system. Error symptom was just a white page instead of the login screen. Second – As I am using self signed certificates, I got this error page here complaining about the certificates. Solution was to give php on the Nextcloud Server a copy of the root CA and add this line to /etc/php/7.4/apache2/php.ini. Third – After clicking the login button I received this error – turned out that the redirect URL contained the index.php parameter where it shouldn’t. I first just added that as a valid redirect. But the real “proper” solution is to add this line into the nextcloud config.php and then launch the nextcloud occ php with the maintenance:update:htaccess argument. Once I restarted Apache and reloaded the page, then the index.php was gone from the URL.
(Proxmox)
Right, now let’s do ProxMox.
The process is quite similar. Just this time we can use the Proxmox GUI in order to tweak the settings on the Proxmox side. Under Datacenter – Permission - then Realms we can add an OpenID Connect Server. The issuer URL is the URL of our Authelia Server. The realm is a name that you can freely chose, I chose authelia. Same for the Client ID and the client Key. You just need to specify the same ID and Key on the Authelia side in the next step. Username Claim and Scopes need to be entered exactly as shown here. If you tick this box, then users will be automatically created once they log in. They will not have admin capabilities by default. Proxmox does not seem to honor groups. You will need to manually log into Proxmox with a PAM account, then add the user to the admin group manually. If you want to create users before thy log in, just create them here under “Users” and specify the new realm.
So far for the Proxmox side, now let’s tell Authelia about it. You can again copy-paste the snippet from the Authelia documentation and just adapt the values. The id and secret correspond to the Client ID and Client Key on the Proxmox side. The redirect URL needs to be the URL of your Proxmox Server.
Once this is working, we can again log into Proxmox, just this time we select the authelia realm and – tadaa – we can log in using SSO. There are just a handful of limitations with such an account, namely you can’t open a shell on the Proxmox VE Server itself. Also, you can’t change Privilege information on Containers. You need to be root for this. Everything else should work without a problem.
I did hit a couple of road blocks here as well. First I forgot to specify the port number 8006 in the proxmox redirect URL. Second – again because I use self-signed certificates, I had to copy the rootCA over to the /etc/ssl/certs folder on the proxmox server and then install it to OpenSSL with this command.
(Portainer)
Awesome – we’re on a roll here – let’s do Portainer next. Here, we can do all the settings in the GUI again. We go to Settings-Authentication, then chose OAUTH as the method and enter all information like in the other two examples. Again – guys, use a different secret for each client app. Use a long key. Do not use the example keys. Portainer wants a lot more URLS from us so make sure you double check the Authorization, Token and Resource URL. All other values are similar to the other clients.
Just a quick tip with regards to those three URLs here – how would I know what to put here? Actually, we can query them from the Authelia Server. If we open the “.well-known/openid-configuration” URL on the Authelia Server, then Authelia will provide us with all of them. Let me copy this into a text editor and beautify it for better readability. Here we have all the settings we need. Authorization Endpoint, Token Endpoint, Supported Claims, Userinfo Endpoint. If any of these still show example.com then you will have to revisit your Authelia Config File.
Before I can log into portainer I do however have to create the user there.
Nothing exiting to configure on the Authelia side, just this time we can add groups to the scopes.
Once this is all done, then Portainer will come up with this Login screen, basically asking you if you want to use OAUTH or use the internal authentication. If I am already logged in, then I can just consent and will be logged into Portainer.
When I did this in my lab I found again some issues with regards to self-signed certificates. Please check my blog www.onemarcfifty.com for Details. Basically I had to install the Root CA to the Docker host and give Portainer a copy of the Root CA.
Let’s wrap this up with configuration for Gitea. Here, the config is done under Site Administration, then Authentication Sources. Click on “Add Authentication Sources”, chose “OAUTH2” as a type and OpenID Connect as a Provider. Fill in Client ID and Secret. Just this time – gitea can actually use the Auto Discover URL directly. So let’s put in the “.well-known” URL here.
On the Authelia side, nothing new. Same Procedure like for the other three. Just the callback URI looks a bit different.
Just – here in Gitea we have the possibility to enable automatic user creation. I’ll just copy paste the snippet here from the Authelia Documentation into the app.ini file on my gitea server and then restart gitea.
When I now log into gitea the next time, then there is this button here “Sign in with OpenID”. And – beautiful. SSO everywhere.
On gitea I had the same issue like on Portainer with the self-signed certs. Again – check my blog article on that one. Also – I had forgotten to specify port 3000 in the config which lead to an error message. Adding the port correctly in the authelia config fixed the issue.
Chaps, we could do many more – I think you get the idea. It’s always the same procedure. Many other Apps can work with Authelia such as SeaFile, Apache Guacamole, CloudFlare ZeroTrust and so on.
(What if App does not support OpenID)
What can we do if the App we want to secure does not support OpenID? Can we still use Authelia? Basically we have three possibilities here. Either the app supports header based authentication. Rather than having the app listen on the generic IP 0.0.0.0, we let it listen on the local host address only. That means, it will not be reachable from the outside. We then install a reverse proxy on the same machine that listens to the outside world and forwards to the local app once Authentication has occurred. Authelia and NGINX can add a couple of X-Headers to the forward request which the app can then read out. For eaxample like I did here with my Shinobi Video surveillance. I actually had to change the code so that it reads out the “remote-user” header and if that header exists, it will trust that Authelia has already done the authentication. There are some examples in the Authelia docs on how to do this with Jira, Organizr and SeaFile.
Some Apps don’t support OAUTH2 or OpenID Connect. But they do support SAML. One of my next projects is to actually use a gateway application that would act as a SAML provider and use Authelia as a backend in order to authenticate. Kind of a SAML to OpenID Proxy really. Watch my blog and github for this. I will publish as soon as I get to grips with it.
Now what if the app can’t do Header based, can’t do SAML, can’t do OAUTH? Well, then you can still put NGINX or Traefik or Caddy in front of the app, integrate the reverse proxy with Authelia and then redirect – just – you would have to login to the app after you logged into Authelia. Not very nice but maybe still a viable solution if you are exposing stuff to the internet such as the web interface of a webcam or the like.
(Some last thoughts)
Here are some last thoughts and potential enhancements. The scripts I provide do not use MySQL but rather a simple SQLite file. This is not dimensioned for large scale but rather for small deployments. I’ve put some examples on the github on how to extend this to MySQL or MariaDB rather. Also, you might want to enable REDIS for larger deployments. Redis is an in-memory Database that can drastically speed up things.
Also, an important reminder especially if you are using OIDC to log into Proxmox. If you are running Authelia in a container on Proxmox and your Proxmox has a problem bringing the container up, then you will not be able to log into the GUI, hence you can’t fix the problem. So please make sure you still have the possibility to log into the Proxmox UI using PAM authentication. You still need this for some admin tasks on Proxmox anyhow.
For full disclosure – I am not the inventor of all this, but I have used mainly three resources here. First and foremost – during my research for this episode, I came across a blog by Florian Müller from Hamburg. On his blog site he describes in great detail how he set up Authelia for a CloudFlare Tunnel with OIDC on Proxmox. Florian, shout out to Hamburg and many thanks for this – your blog really helped me a lot in order to get to grips with Authelia. I have adapted and modified his code snippets in order to give you guys an automated installation experience. The code is on my github repository. Link in the description. Second – fellow YouTube Creator Techno Tim has a nice video about Authelia on Docker. You may want to watch this as well if you intend to use Authelia with Docker. Last but not least – the Authelia Documentation contains a lot of ready-to-use samples on how to configure OIDC with Authelia.
That’s it guys – I hope you liked the episode. If so, please give it a like on YouTube and leave me a comment! Many thanks for watching. Happy SSO – stay safe, stay healthy, bye for now!
</details>Marc AhlgrimRunning Authelia on ProxmoxPortainer with self-signed certificates2023-02-25T00:00:00+01:002023-02-25T18:00:00+01:00https://www.onemarcfifty.com/blog/Portainer_TLS<h2 id="portainer-and-self-signed-certificates">Portainer and self-signed certificates</h2>
<p>I am currently preparing videos in a test environment with self-signed certificates. One of the tests was to use Authelia with Portainer for SSO. I just couldn’t get it to work until I realized that Portainer (as of Version 2.17) does not seem to honor the <code class="language-plaintext highlighter-rouge">--tlscacert</code> flag.</p>
<p>We can give Portainer self-signed <em>server</em> certificates with the <code class="language-plaintext highlighter-rouge">--sslcert</code> and <code class="language-plaintext highlighter-rouge">--sslkey</code> flags. With this, we can connect to Portainer without Certificate warnings if we have the Root CA imported into our browser. However, Portainer would just not trust any other site (such as Authelia).</p>
<h2 id="the-solution">the Solution</h2>
<p>the solution is more kind of a workaround really. What I ended up doing was to install the Root CA on the Docker host and provide a copy of the complete RootCA crt file to Portainer. Assuming that your Server key and certificate as well as the rootCA.crt are in /etc/certificates/example.com, run the following on your Docker host:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># copy the Root CA file to the system certificate subfolder</span>
<span class="c"># (on Debian in /etc/ssl/certs)</span>
<span class="nb">cp</span> /etc/certificates/example.com/rootCA.crt /etc/ssl/certs
<span class="c"># create a Symlink from the hash to the cert</span>
<span class="nb">ln</span> <span class="nt">-s</span> /etc/ssl/certs/rootCA.crt /etc/ssl/certs/<span class="sb">`</span>openssl x509 <span class="nt">-hash</span> <span class="nt">-noout</span> <span class="nt">-in</span> rootCA.crt<span class="sb">`</span>.0
<span class="c"># this will update the /etc/ssl/certs/ca-certificates.crt</span>
update-ca-certificates
</code></pre></div></div>
<p>We could as well just have copied the original <code class="language-plaintext highlighter-rouge">/etc/ssl/certs/ca-certificates.crt</code> file and could have done <code class="language-plaintext highlighter-rouge">cat rootCA.crt >> ca-certificates.crt</code> , thus appending our Root CA to the file.</p>
<p>Now we can run Portainer with the following command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">-d</span> <span class="nt">-p</span> 9000:9000 <span class="nt">-p</span> 9443:9443 <span class="se">\</span>
<span class="nt">--name</span><span class="o">=</span>portainer <span class="se">\</span>
<span class="nt">--restart</span><span class="o">=</span>always <span class="se">\</span>
<span class="nt">-v</span> /var/run/docker.sock:/var/run/docker.sock <span class="se">\</span>
<span class="nt">-v</span> portainer_data:/data <span class="se">\</span>
<span class="nt">-v</span> /etc/certificates/example.com:/certs <span class="se">\</span>
<span class="nt">-v</span> /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt <span class="se">\</span>
portainer/portainer-ce:latest <span class="se">\</span>
<span class="nt">--sslcert</span> /certs/wildcard_fullchain.crt <span class="se">\</span>
<span class="nt">--sslkey</span> /certs/wildcard.key <span class="se">\</span>
<span class="nt">--tlscacert</span> /cert/rootCA.crt
</code></pre></div></div>
<p>(basically we are mounting the modified Root CA File to the container)</p>Marc AhlgrimPortainer and self-signed certificatesAuthelia as OpenID Server on Proxmox2023-02-11T00:00:00+01:002023-02-11T18:00:00+01:00https://www.onemarcfifty.com/blog/Authelia_Proxmox<h2 id="how-i-use-authelia">How I use Authelia</h2>
<p>I use Authelia as an Identity Provide in my network. That means that everyone who wants to use resources (such as Jellyfin or Gitea or Nextcloud) or who wants to “traverse” VLANs (e.g. to manage an OpenWrt router or switch or my Proxmox VE) has to login first. Authelia was my product of choice because</p>
<ul>
<li>It is free and Open Source Software (FOSS)</li>
<li>It can act as an OpenID Connect (OIC) Identity provider</li>
<li>It can be integratet into an NGINX reverse proxy</li>
<li>It supports Two Factor Authentication (2FA) with Time based One Time Passwords (TOTP) or Fido2 compatible Keys such as the Yubikey.</li>
</ul>
<h2 id="installing-authelia-on-proxmox">Installing Authelia on Proxmox</h2>
<p>I set up a small LXC container running Debian Bullseye (Debian 11) on Proxmox. It needs to be privileged because it needs to have access to <code class="language-plaintext highlighter-rouge">/dev/urandom</code> or <code class="language-plaintext highlighter-rouge">/dev/random</code> for the generation of random numbers.<br />
Then we need to install some software, add the authelia repos and finally install authelia:<br /></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># run as root</span>
apt update
apt <span class="nb">install</span> <span class="nt">-y</span> curl gnupg apt-transport-https <span class="nb">sudo
</span>curl <span class="nt">-s</span> https://apt.authelia.com/organization/signing.asc | <span class="nb">sudo </span>apt-key add -
<span class="nb">echo</span> <span class="s2">"deb https://apt.authelia.com/stable/debian/debian/ all main"</span> <span class="o">>></span>/etc/apt/sources.list.d/authelia.list
apt-key <span class="nb">export </span>C8E4D80D | <span class="nb">sudo </span>gpg <span class="nt">--dearmour</span> <span class="nt">-o</span> /usr/share/keyrings/authelia.gpg
apt update
apt <span class="nb">install</span> <span class="nt">-y</span> authelia
</code></pre></div></div>
<h2 id="configuration-of-authelia">Configuration of Authelia</h2>
<p>I mostly followed <a href="https://florianmuller.com/setup-authelia-bare-metal-with-openid-and-cloudflare-tunnel-on-a-hardened-proxmox-lxc-ubuntu-22-04-lts-container#configureauthelia">this guide here by Florian Mueller</a></p>
<p>for the configuration. Basically we create sub-directories for all secrets and auto-generate them, create keys and add the secrets to the environment of authelia (the below is a shortened version of Florian’s scripts)</p>
<p><strong>important note</strong> The scripts below use a single SQLITE file rather than mysql! Also, no OIDC provider is configured - just a dummy entry. Please see the implications of this <a href="https://www.authelia.com/configuration/storage/sqlite/">here</a></p>
<h3 id="create-the-secrets-and-service-systemd-unit-file">create the secrets and service systemd unit file</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for </span>i <span class="k">in</span> .secrets .users .assets .db <span class="p">;</span> <span class="k">do </span><span class="nb">mkdir</span> /etc/authelia/<span class="nv">$i</span> <span class="p">;</span> <span class="k">done
for </span>i <span class="k">in </span>jwtsecret session storage smtp oidcsecret redis <span class="p">;</span> <span class="k">do </span><span class="nb">tr</span> <span class="nt">-cd</span> <span class="s1">'[:alnum:]'</span> < /dev/urandom | <span class="nb">fold</span> <span class="nt">-w</span> <span class="s2">"64"</span> | <span class="nb">head</span> <span class="nt">-n</span> 1 | <span class="nb">tr</span> <span class="nt">-d</span> <span class="s1">'\n'</span> <span class="o">></span> /etc/authelia/.secrets/<span class="nv">$i</span> <span class="p">;</span> <span class="k">done
</span>openssl genrsa <span class="nt">-out</span> /etc/authelia/.secrets/oicd.pem 4096
openssl rsa <span class="nt">-in</span> /etc/authelia/.secrets/oicd.pem <span class="nt">-outform</span> PEM <span class="nt">-pubout</span> <span class="nt">-out</span> /etc/authelia/.secrets/oicd.pub.pem
<span class="o">(</span><span class="nb">cat</span> <span class="o">></span>/etc/authelia/secrets<span class="o">)</span> <span class="o"><<</span><span class="no">EOF</span><span class="sh">
AUTHELIA_JWT_SECRET_FILE=/etc/authelia/.secrets/jwtsecret
AUTHELIA_SESSION_SECRET_FILE=/etc/authelia/.secrets/session
AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE=/etc/authelia/.secrets/storage
AUTHELIA_NOTIFIER_SMTP_PASSWORD_FILE=/etc/authelia/.secrets/smtp
AUTHELIA_IDENTITY_PROVIDERS_OIDC_HMAC_SECRET_FILE=/etc/authelia/.secrets/oidcsecret
AUTHELIA_IDENTITY_PROVIDERS_OIDC_ISSUER_PRIVATE_KEY_FILE=/etc/authelia/.secrets/oicd.pem
</span><span class="no">EOF
</span><span class="nb">chmod </span>600 <span class="nt">-R</span> /etc/authelia/.secrets/
<span class="nb">chmod </span>600 /etc/authelia/secrets
<span class="o">(</span><span class="nb">cat</span> <span class="o">></span>/etc/systemd/system/authelia.service<span class="o">)</span> <span class="o"><<</span><span class="no">EOF</span><span class="sh">
[Unit]
Description=Authelia authentication and authorization server
After=multi-user.target
[Service]
Environment=AUTHELIA_SERVER_DISABLE_HEALTHCHECK=true
EnvironmentFile=/etc/authelia/secrets
ExecStart=/usr/bin/authelia --config /etc/authelia/configuration.yml
SyslogIdentifier=authelia
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span>systemctl daemon-reload
</code></pre></div></div>
<h3 id="create-the-user-file">create the user file</h3>
<p>Next, we create a rudimentary User database yaml file with randomly generated passwords (the users can reset them with the “forgot password” link):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"users:"</span> <span class="o">></span> /etc/authelia/.users/users_database.yml
<span class="k">for </span>user <span class="k">in </span>bob alice dave frank <span class="p">;</span> <span class="k">do
</span><span class="nv">randompassword</span><span class="o">=</span><span class="si">$(</span><span class="nb">tr</span> <span class="nt">-cd</span> <span class="s1">'[:alnum:]'</span> < /dev/urandom | <span class="nb">fold</span> <span class="nt">-w</span> <span class="s2">"64"</span> | <span class="nb">head</span> <span class="nt">-n</span> 1 | <span class="nb">tr</span> <span class="nt">-d</span> <span class="s1">'\n'</span><span class="si">)</span>
<span class="nv">encryptedpwd</span><span class="o">=</span><span class="si">$(</span>authelia hash-password <span class="nt">--no-confirm</span> <span class="nt">--</span> <span class="nv">$randompassword</span> | <span class="nb">cut</span> <span class="nt">-d</span> <span class="s2">" "</span> <span class="nt">-f</span> 2<span class="si">)</span>
<span class="o">(</span>
<span class="nb">echo</span> <span class="s2">" </span><span class="k">${</span><span class="nv">user</span><span class="k">}</span><span class="s2">:"</span>
<span class="nb">echo</span> <span class="s1">' displayname: "First Last"'</span>
<span class="nb">echo</span> <span class="s2">" password: </span><span class="nv">$encryptedpwd</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="s2">" email: </span><span class="k">${</span><span class="nv">user</span><span class="k">}</span><span class="s2">@yourdomain.com"</span>
<span class="o">)</span> <span class="o">>></span> /etc/authelia/.users/users_database.yml
<span class="k">done
</span><span class="nb">chmod </span>600 <span class="nt">-R</span> /etc/authelia/.users/
</code></pre></div></div>
<h3 id="create-the-configurationyml">create the configuration.yml</h3>
<p>Now we need to create a configuration file for authelia in <code class="language-plaintext highlighter-rouge">/etc/authelia/configuration.yml</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /etc/authelia
<span class="c"># save the old version of the file</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-e</span> configuration.yml <span class="o">]</span> <span class="p">;</span> <span class="k">then
</span><span class="nb">mv </span>configuration.yml configuration.yml.old
<span class="k">fi</span>
<span class="c"># Now let's use Marc's version of Florian's Template File for the new config:</span>
wget https://raw.githubusercontent.com/onemarcfifty/cheat-sheets/main/templates/authelia/configuration.yml
<span class="nb">chmod </span>600 configuration.yml
</code></pre></div></div>
<h2 id="starting-authelia-for-the-first-time">Starting Authelia for the first time</h2>
<p>There we go - authelia should be able to run already - if you do</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl start authelia
systemctl status authelia
</code></pre></div></div>
<p>you should see all green and be able to browse to <code class="language-plaintext highlighter-rouge">http://localhost:9091</code> and see the authelia login prompt. Even though this is an important checkpoint (in the sense that we can see if authelia will run at all), we can’t really use it yet. We need to take care of the following things:</p>
<ol>
<li>All domain names still point to example.com</li>
<li>The start up checks are disabled (especially checking for an e-mail Server)</li>
</ol>
<p><strong>You will need a real e-Mail account / Server for Authelia to work correctly</strong> (Users need this to reset their password and to register 2FA devices)</p>
<ol>
<li>We have no TLS enabled</li>
<li>The policies need to be adapted</li>
<li>We need to increase the security and hide Authelia behind an NGINX Server</li>
<li>We need to harden the server</li>
<li>The user accounts are not real</li>
</ol>
<h2 id="adapting-and-securing-authelia">Adapting and securing authelia</h2>
<p>to get a first idea of how much you need to change do <code class="language-plaintext highlighter-rouge">grep example.com /etc/authelia/configuration.yml</code> - that shows you all the lines where we specified the example.com domain.</p>
<h3 id="enabling-the-startup-checks-and-change-the-domain-names">Enabling the startup checks and change the domain names</h3>
<p>The first one is the mail server. We need to edit the configuration.yml and give it the login data of a real mail server. This is done in the <code class="language-plaintext highlighter-rouge">notifier</code> section of the config:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">notifier</span><span class="pi">:</span>
<span class="na">disable_startup_check</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">smtp</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">smtp.domain.com</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">465</span>
<span class="na">timeout</span><span class="pi">:</span> <span class="s">5s</span>
<span class="na">username</span><span class="pi">:</span> <span class="s">noreply@auth.example.com</span>
<span class="na">sender</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Authentication</span><span class="nv"> </span><span class="s">Service</span><span class="nv"> </span><span class="s"><noreply@auth.example.com>"</span>
<span class="na">subject</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{title}"</span>
<span class="na">startup_check_address</span><span class="pi">:</span> <span class="s">noreply@auth.example.com</span>
</code></pre></div></div>
<p>Change all the settings to reflect a real mailbox that you control. Once you have done that, change the <code class="language-plaintext highlighter-rouge">disable_startup_check: true</code> to <code class="language-plaintext highlighter-rouge">disable_startup_check: false</code> and restart authelia:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl restart authelia
systemctl status authelia
</code></pre></div></div>
<p>If you see errors, i.e. Authelia idn’t start then that’s because now it checks if it can log into the mailbox at startup. Reminder: The password it uses for the mailbox Server is in <code class="language-plaintext highlighter-rouge">/etc/authelia/.secrets/smtp</code></p>
<p>Onc Authelia starts, move on to the next step.</p>
<h3 id="tls">TLS</h3>
<p>Before we can <em>really</em> use Authelia, we need to provide it with <em>real</em> SSL certificates. Use your own or get them from letsencrypt. <a href="https://youtu.be/Z81jegMCrfk">This video on my channel</a> shows more.</p>
<p>Once you have copied the certificate and key to the server, adapt the configuration.yml file:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">server</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">0.0.0.0</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">9091</span>
<span class="na">asset_path</span><span class="pi">:</span> <span class="s">/etc/authelia/.assets/</span>
<span class="na">tls</span><span class="pi">:</span>
<span class="na">key</span><span class="pi">:</span> <span class="s">/etc/authelia/certs/server.key</span>
<span class="na">certificate</span><span class="pi">:</span> <span class="s">/etc/authelia/certs/server.crt</span>
</code></pre></div></div>
<p>Basically I’ve just added the tls section and point it to the certificates. Again - restart Authelia, check the status. If it starts and if you can browse to <code class="language-plaintext highlighter-rouge">https://...:9091</code> rather than <code class="language-plaintext highlighter-rouge">http...</code> then you can move to the next step.</p>
<h3 id="bind-port">bind port</h3>
<p>For the moment Authelia listens on <em>any</em> interface, i.e. we can browse to port 9091 from the outside. We will however hide it behind an NGINX server. For this, Authelia should only listen to the localhost interface. Change the configuration.yml from</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">server</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">0.0.0.0</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">9091</span>
</code></pre></div></div>
<p>to</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">server</span><span class="pi">:</span>
<span class="na">host</span><span class="pi">:</span> <span class="s">127.0.0.1</span>
<span class="na">port</span><span class="pi">:</span> <span class="m">9091</span>
</code></pre></div></div>
<p>Restart and check - you should not be able to browse to it from the outside.</p>
<h2 id="hiding-authelia-behind-nginx">Hiding Authelia behind NGINX</h2>
<p>In this step we install NGINX, let it listen to the outside world on port 443 and forward all requests to Authelia on the local host. You can use the templates from <a href="https://github.com/onemarcfifty/cheat-sheets/tree/main/templates/nginx/authelia">My cheat sheet repo on Github</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># install nginx</span>
apt <span class="nb">install</span> <span class="nt">-y</span> nginx
<span class="c"># stop NGINX</span>
systemctl stop nginx
<span class="c"># remove the default site</span>
<span class="nb">rm</span> /etc/nginx/sites-enabled/<span class="k">*</span>
<span class="c"># download the templates from Marc's cheat sheets</span>
wget https://raw.githubusercontent.com/onemarcfifty/cheat-sheets/main/templates/nginx/authelia/siteconf <span class="nt">-O</span> /etc/nginx/sites-available/authelia.conf
wget https://raw.githubusercontent.com/onemarcfifty/cheat-sheets/main/templates/nginx/authelia/proxy-snippet <span class="nt">-O</span> /etc/nginx/snippets/proxy.conf
wget https://raw.githubusercontent.com/onemarcfifty/cheat-sheets/main/templates/nginx/authelia/ssl-snippet <span class="nt">-O</span> /etc/nginx/snippets/ssl.conf
<span class="c"># link back the authelia site as enabled to NGINX </span>
<span class="nb">ln</span> <span class="nt">-s</span> /etc/nginx/sites-available/authelia.conf /etc/nginx/sites-enabled/authelia.conf
<span class="c"># restart NGINX</span>
systemctl start nginx
</code></pre></div></div>
<h2 id="adapting-the-policies">adapting the policies</h2>
<p>The template file contains three sample policies for bypass, one factor and two factor. You will want to adapt these to your needs. My Server only has one policy:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">access_control</span><span class="pi">:</span>
<span class="na">default_policy</span><span class="pi">:</span> <span class="s">deny</span>
<span class="na">rules</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">domain</span><span class="pi">:</span> <span class="s1">'</span><span class="s">*.mydomain.com'</span>
<span class="na">policy</span><span class="pi">:</span> <span class="s">two_factor</span>
</code></pre></div></div>
<h2 id="last-but-not-least">Last but not least</h2>
<p>The last steps are</p>
<ul>
<li>harden the server</li>
<li>make the user accounts real</li>
</ul>
<p>Server hardening is not in the scope of this article. Basically at least you should do the following:</p>
<ol>
<li>Lock down the firewall to only let pass port 443 tcp incoming (optionally port 67 UDP if you use dhcp and port 22 tcp if you want to access the server via ssh)</li>
<li>disable / expire the password of the root account</li>
<li>Create a non-root user for login who has sudo capabilities with a loooong password</li>
<li>switch off password authentication and root login for sshd</li>
</ol>
<p>Now just review the settings in the <code class="language-plaintext highlighter-rouge">/etc/authelia/.users/users_database.yml</code> and make sure that the user accounts are real accounts with real mail addresses.</p>
<p>That’s it - you’re all set and can now use Authelia in front of your servers with NGINX/Traefik/Caddy or the like and/or add OIDC providers for Proxmox, Gitea, Portainer, Nextcloud and so on….</p>Marc AhlgrimHow I use AutheliaTLS X.509 Client Certificates2023-01-30T00:00:00+01:002023-01-30T18:00:00+01:00https://www.onemarcfifty.com/blog/video/Certificates_Episode3<h2 id="tls-x509-client-certificates">TLS X.509 Client Certificates</h2>
<p>How to secure Internet Servers with X.509 Client Certificates? How to deploy X.509 Client Certificates ? How does a Certificate Signing Request (CSR) work ? In this hands-on video we will run a little nodejs Server that requests Authentication with an X.509 Client Certificate, we will Sandbox a CSR with XCA and we will have a look at OpenXPKI which is a great Software to automate processes around TLS and Certificate Generation, Key Management and the like. Last but not least I show a Blueprint on how to securely link a hosted MQTT into your home automation Software.</p>
<p>The nodejs Server Example is <a href="https://github.com/onemarcfifty/client-cert-test">on my github</a></p>
<p>The XCA Tool can be obtained <a href="https://hohnstaedt.de/xca/">here</a></p>
<p>More Info on my <a href="https://github.com/onemarcfifty/cheat-sheets/tree/main/Certificates">Cheat Sheet Repo on Github</a></p>
<p><a href="https://www.youtube.com/watch?v=5lYQRuzdZr0"><img src="/assets/images/thumbnails/5lYQRuzdZr0.jpg" />Watch the video on YouTube</a></p>
<details>
<summary>Click to view the entire transcript</summary>
(the use case)
Let me show you how I open the door to my
house or to my garage at home with my mobile
phone.
I open this app where I can type in a pin,
click on this button and then the door opens.
That app is in fact just a web page running
on a web server in the cloud.
Now I hear you say – “Marc, are you mad
– do you really allow everyone to open the
door to your house at night?” - of course
not – the site is protected with a client
certificate.
That means that my phone is the only device
on this planet that can browse to that web
site.
Here is how I did it.
We had talked about certificates and about
this nice GUI here called XCA in the last
episode.
I showed you how to generate a Certificate
Authority, a CA, and how to generate server
certificates.
Now the great thing with X.509 certificates
is, that we can also use them for client authentication.
That means that a web server would only answer
requests coming from a device that has the
right certificate.
All others will be rejected.
Imagine how you could secure for example your
Nextcloud instance, your personal photo collection
or – like I do – simple apps to interface
with your IOT at home in a way that only you
can use them from your tablet or phone.
(Demo server from my Github)
If you browse to my github repository, then
you can find an example for nodejs that does
exactly what I showed here – it shows a
web page with the two input lines for a pin
and two buttons.
In order to use it, you will need a host with
nodejs installed.
Windows, MacOS or Linux – it doesn’t matter.
Just copy the files to a subdirectory and
start node testserver.js.
The server now answers requests on port 8443.
So if you now browse to let’s say https://testserver:8443/form
then you will most probably get a certificate
warning.
In the ca subdirectory you can find a self-signed
certificate authority (CA) that you can import
into your browser.
In Chrome and Firefox this is done under the
security settings.
Alternatively - if you want to run this on
a server with valid Let's encrypt certificates
or the like, just replace the files server.crt
and server.key with the certificate and private
key of your "real" server.
Don't replace the ca.crt file as it's used
for client authentication!
The files server.crt and server.key in the
ca subdirectory have been signed with the
ca.crt for testing purposes.
If you want to test without certificate warning,
then you could just make an entry into your
host file and point testserver to any host,
for example your localhost by adding the following
into your /etc/hosts file (on Linux) or your
C:\Windows\System32\drivers\etc\hosts file
(on Windows).
If you now browse to the URL again, then your
browser should tell you that you have been
rejected as you don't have the right client
certificate installed.
That's the purpose of this exercise ;-) - In
order to gain access, you need to import the
file testuser.p12 from the xca.db sub directory
into your browser.
Let me do this.
I’ll go again into the security settings
of my browser, then import the certificate.
The password for this is hello.
Let me open the page again – and now the
magic happens.
My browser tells me that the server is requesting
authentication.
It asks me which certificate I should provide.
If I now select the testuser certificate,
then the page opens.
I can use it as desired.
Without having to type any username or password,
without using a VPN.
Just plain TLS certificates.
Very transparent.
On my phone, it would just use the certificate
without asking.
(How to generate client certificates)
So – how can we generate our own client
certificates then?
Just like in the last episode, we will use
XCA for this (you could do this on the command
line as well with OpenSSL of course).
Launch XCA and open the xca.db file from my
github – this already contains some sample
data.
As you can see, there is a CA, a server certificate
and also a client certificate.
How did I generate the client certificate?
Well, pretty much like the server certificate
in the last episode.
I clicked on “New Certificate”, just this
time I chose and applied the TLS_client template.
As a common name this time I just entered
the name “testuser” And of course this
one also needs a key.
So I generated a new one.
In order to be able to use this with my web
browser or phone, I exported it as PKCS#12
file, because that file format contains both
the key and the certificate and also I know
that my iPhone can read that type of certificate
files.
As there is a private key inside, I now needed
to protect it with a password.
The password for all the sample data on my
github is just “hello”.
(how secure is this)
You might have doubts on the security of this.
And you are right to challenge that.
We should not take security lightly.
In a nutshell, the certificate is just another
authentication factor.
An authentication factor can be something
you know – such as a user name and password
– it can be something that you are – so
biometric data, or it can be something you
have – like a key, a token or a certificate.
So if you wanted to let’s say make the admin
frontend of your nextcloud or photo sharing
app or anything else super secure, then I
would not ask you to remove or replace the
password authentication, but rather add the
client certificate requirement on top of it.
That would reduce the surface that you potentially
expose to hackers by a huge percentage – I’d
say above 99% really.
Why?
If you have services running in the web – be
it a password protected web site or an ssh
server or whatever – then you will see hundreds
if not thousands of log entries from people
or bots attempting to connect to it.
Every single day.
They get to the login prompt and now they
can start guessing passwords or using cross
site scripts or injection attacks to your
frontend.
If you had that site locked down with a client
certificate, then all these attempts would
not even get to that page on the application
layer because they would be rejected by TLS
a layer below.
On the presentation layer, between the TCP
protocol and the application.
Still, there are remaining attack vectors.
If someone gained control over the server,
then they could of course do whatever they
wanted there.
You still need to secure the server.
If someone had gained control over your client,
then they could potentially steal the client
certificate and use it.
If someone is able to get into the middle
between you and the server, then they can
potentially intercept data as well and exploit
weaknesses of the protocol.
Last but not least, if someone had a day 0
exploit to TLS, or if you used an old version
of TLS that has known vulnerabilities, then
they could use that as well.
But still – the security level is not worse
with certificates than it was without certificates.
All these attack vectors can be exploited
with or without certificates.
(how to deploy client certificates)
The main challenge with this concept however
is the distribution of the client certificates.
Not a big thing if you need to put it into
one single web browser.
But how would you distribute these let’s
say to 10 or 100 clients?
And how would you distribute them to let’s
say an iPhone?
In an enterprise environment those devices
would be managed by a central Mobile Device
Management platform such as intune or altiris
or the like.
In the home environment this is a challenge
though.
Let me first quickly show you how it should
NOT be done.
I’ll attach the PKCS#12 file (which includes
the certificate _and_ the private key) to
an email and send that email to my mobile
phone.
Now this is great in the sense that I can
now easily open and install the certificate.
But even though the PKCS#12 format is password
protected, this is a weak protection.
Especially if you chose a super weak password
like 12345 or hello.
This would actually open an additional attack
vector – the interception of the certificate.
(Certificate Signing Requests, CSR)
So how to deal with this?
In a real life secure scenario, the private
keys should never ever leave the system where
they have been generated on.
Now – I do not have an app that allows me
to generate keys on my iphone, so let me show
you this with two instances of XCA.
One instance would be the client who wants
to have a client certificate signed by the
CA, let’s call him Marc – that’s me,
and the other instance would be the site that
has the private key for the CA, let’s call
it onemarcfifty.com.
So first, I would download the ca certificate
from onemarcfifty.com.
The public part.
Without the private key.
Then I would generate a private key which
I would want to use with that certificate.
Now – rather than creating a certificate
(which I could not sign anyhow because I don’t
have onemarcfifty.com’s private key), I
would now generate a Certificate Signing request,
a CSR.
That CSR contains the public key that complements
my private key.
And I would then submit that CSR to onemarcfifty.com.
So let me export this on Marc’s side and
then import it on onemarcfifty.com’s side.
On the Onemarcfifty.com side, this CSR can
now be signed with their CA.
That generates a certificate below the CA
here.
Now Onemarcfifty would export that ready made
certificate which now contains the public
keys of Marc and of the onemarcfifty.com CA.
But still no private key has been transferred.
Onemarcfifty.com send me that certificate
and I’ll import it.
Because I do have my own private key plus
the public key of the CA, the certificate
will now be properly linked to the CA and
I can use it.
You could in theory do that workflow over
e-mail – I would really not recommend it,
but you could.
No secret information was transmitted at any
point in time.
Just - a thoroughly forged man-in-the-middle
attack could still be possible.
A malicious attacker who has access to your
e-Mail could replace your request with their
own and hence get granted access to onemarcfifty.com.
Also – some information that you transfer
with the CSR could be used to lower the effort
for brute forcing the encryption.
Please remember that most successful attacks
do not attack the encryption.
They rather try to circumvent it.
And many popular attacks in the past aimed
at the authentication process.
That means the part of the process where you
prove that you are authorized to obtain a
certificate.
If you send the CSR via e-mail then the only
authentication is your e-mail address.
The CA can’t verify your key.
There is actually free and open source software
available for that type of workflow.
Have a look at openxpki for example.
You can self host this software.
For testing purposes, there is a demo site
where we can actually log in with demo accounts
and submit CSRs and they’ll sign it for
us.
So I create a CSR
- export it
- Generate a Request on the openxpki platform
-
Now – here I am at the same time the officer
who approves this request – normally that
would be a 3rd person.
- And now I can download the signed certificate
from here.
So – if I made the access to that site quite
secure with let’s say an additional 2nd
factor such as a one time password or a temporary
code sent via text message, or the site would
only be available in the intranet for example,
then I could make that quite secure – by
home or small business standards.
If I go back to the initial application that
I have shown in the beginning, then we could
even take this blue print further.
Many home automation products can use MQTT
as a protocol.
Let’s say you have your MQTT Server running
here at home in a container or on your router.
And here on the other end we are in the internet,
so outside of the LAN.
In order to open the door at home we need
to publish a command to MQTT.
Now here we could have a VPS, a cloud server
for a dollar or a free oracle instance or
the like that has Letsencrypt Server certificates.
If we add a self-signed CA here then we can
do the following.
Our MQTT at home could communicate over TLS
with an MQTT instance here on the VPS.
Our phone could securely connect to a little
nodejs server here using client certificates
and issue the command to the local MQTT which
would in turn relay the command to the MQTT
at home which would then instruct the home
automation to open the door.
No passwords, no VPN, no hole in the firewall,
no port forwarding.
Nothing.
Just the right certificates at the right places.
Guys, we’re running out of time.
Let me know if you liked this episode.
Leave me a comment if you want follow-ups.
All links and the example code are in the
description and on my github server.
In any case – thank you so much for watching.
Stay safe, stay healthy, bye for now.
</details>Marc AhlgrimTLS X.509 Client CertificatesSelf Signed and LetsEncrypt Server Certificates for the LAN2023-01-23T00:00:00+01:002023-01-23T18:00:00+01:00https://www.onemarcfifty.com/blog/video/Certificates_Episode2<h2 id="self-signed-and-letsencrypt-server-certificates-for-the-lan">Self Signed and LetsEncrypt Server Certificates for the LAN</h2>
<p>How to use Certificates in the LAN? What are our options? We can use self-signed certificates, but we can also use public Let’s Encrypt Certificates LOCALLY - in the LAN. In this video we will look at the options such as self-signed CA and Certificates, Let’s Encrypt Server Certificates and Wildcard Certificates</p>
<p>The XCA Tool can be obtained <a href="https://hohnstaedt.de/xca/">here</a></p>
<p>More Info on my <a href="https://github.com/onemarcfifty/cheat-sheets/tree/main/Certificates">Cheat Sheet Repo on Github</a></p>
<p><a href="https://www.youtube.com/watch?v=Z81jegMCrfk"><img src="/assets/images/thumbnails/Z81jegMCrfk.jpg" />Watch the video on YouTube</a></p>
<details>
<summary>Click to view the entire transcript</summary>
(about certificates)
We all know this nasty warning here when we
open web pages in our own intranet.
We don’t have a certificate.
Today I want to show you three alternative
ways to get rid of that.
How can we do that?
First option – self signed certificates.
We will use this great free and open source
software called XCA to generate our own Certification
Authority or CA and our own certificates.
No fiddling around with command line OpenSSL
or the like.
Nice software.
I love it.
Second option – We can use Let’s Encrypt
Server certificates in our LAN – and the
third option that I want to show you are Wildcard
certificates with a dedicated sub-domain.
Let’s go step by step
(self signed certificates)
In the first episode I have already shown
you this very cool software called XCA by
Christian Hohnstädt.
It’s awesome and it’s free open source.
As a reminder – if you browse to https://hohnstaedt.de/xca,
then you can download it for various platforms
such as Linux, MacOS or Windows plus there
is a portable version.
After you start the software, you first need
to generate a database where we will store
all the certificates.
So you click on File – New Database, then
you give the database file a name and after
you have created the file, XCA asks you to
assign a password to that database.
Make sure you use a password you can remember
– there is no way to recover it if ever
you forget it.
(create CA)
So now, we can create our own CA – just
like we did in the first episode.
Click on “New Certificate”, select and
apply the default CA template down here at
the bottom, then move over to the “Subject”
tab and fill out at least the fields Internal
Name, countryName, organizationName and commonName.
Last but not least, generate a new key for
the CA and confirm everything.
(create Server certificate)
Here we go, we have our own CA.
Now let’s generate a Server certificate
that we sign with this CA.
Again, I click on “New Certificate”, this
time I select the existing CA for signing
and I apply the default TLS_Server template.
As an internal and commonName, I now type
in the name of the server that I want to create
the certificate for.
Again – fill out the country and organizationName
fields and generate or use a key.
If you want this certificate to be valid for
multiple names – for example short names
and fully qualified domain names or IP addresses,
then you can add these here under Extensions
as Subject Alternative names.
You could as well generate a wild card certificate,
if you wanted to use one single certificate
for a whole domain for example *.onemarcfifty.com.
(export components)
Cool – now we just have to export the various
components in the right formats and then distribute
them to the right places.
The web server which we create the certificate
for needs the server certificate and the server
key.
Our browser just needs the CA certificate
(again without the private key).
So let’s export everything.
There are various export formats.
Some of them contain the certificate and the
private key, others do only contain the certificate
or the key in separate files.
Let’s start with the CA.
Click on export – the most common format
without the key here is PEM – that’s basically
a base64 encoded or packed version of the
text certificate or chain.
The file extension .crt just makes it easier
for some software to recognize it as a certificate.
So let’s call this one ca.crt.
Next we need the server certificate.
So this is the same game really, click on
export, select PEM as the format, let’s
call this one server.crt.
Last but not least, we need the key on the
server.
So move over to private keys, make sure you
select the right key, Export in pem format.
Just this time let’s call it server.key.
The file extensions and names are not really
standardized anywhere.
The PEM format itself is a standard but the
naming of the files is entirely up to you
or to the software that you are using.
I just like .crt and .key because it clearly
shows what it is.
(distribute components)
I want to use this certificate with an OpenWrt
router.
If I browse to the router’s address then
you can see that I get that ugly security
warning.
Here’s the detail of the current certificate.
I will now replace the original certificate
with my own.
I just need to copy the two server files to
/etc on the router and name them uhttpd.key
and uhttpd.crt.
Before I can use them, I need to quickly restart
the uhttpd daemon on the router.
If I now open the page, then I get the same
warning.
Just this time the certificate is my own.
In order to trust it, I need to import the
CA to my web browser.
I go to settings then security and down here
I can manage the certificates and also import
the CA.
Let me do that.
If I now refresh the page, then it shows up
secure and without a warning.
Mission accomplished.
Let me quickly view and check the certificate
– here you can see all the details like
the issuer, the subject and also all the alternate
names that it is valid for.
Where exactly you have to put these certificates
on your server and how you have to name them
really depends on the server software.
Check my cheat sheet repository on Github.
I will try and make a short overview of the
most common products and the place where they
expect the certificates to be.
So far for self-signed certificates.
The advantage of this mechanism is, that you
can generate any certificate really.
The disadvantage is that you really need to
import the CA certificate to any client and
to all clients that want to browse to that
server.
Now for another method – Let’s encrypt
certificates.
(Let’s encrypt server certificates)
Here is how you would typically use Let’s
Encrypt certificates.
You have a web server running in the internet
and on this server you run the certbot software.
That software periodically connects to Let’s
Encrypt and requests a new certificate using
the ACME protocol.
Let’s Encrypt would then do a DNS lookup
of your server and check if this is the server
that made the request.
If yes, then they would either say – “no
need to renew, your certificate is still valid”
or they would give you a new one.
So how could we use this in our LAN then – in
our local network?
Let’s first have a look how we can use server
certificates and then look at wildcard certificates.
(How to use Let’s Encrypt in the LAN)
If you own a domain – like I do with onemarcfifty.com
– then you could do the following.
You define your internal domain name, so the
DNS domain that you use inside your lan, to
be the same domain like the one that you use
in the internet.
I could therefore use the domain onemarcfifty.com
inside my LAN.
Why not?
I have control over the DNS inside my LAN
because that’s running on my router.
So my proxmox server for example would then
be called pve.onemarcfifty.com.
It’s not reachable from the outside, it’s
just a name I give it internally.
So how would I give this server a public certificate
then?
Well, I would just make an entry for the server
in my public DNS as well and let that entry
point to the same server that has the certbot
agent for let’s say www.mydomain.com.
That entry could be an A record pointing to
the server’s IP address or it could be an
Alias or CNAME pointing to the name of that
server.
The certbot on that server can now request
certificates for that name.
If I browse to that server name from inside
my LAN, then I would not be redirected to
that www server, but rather the machine that
I have running inside my LAN.
So all I’d have to do is copy those certificates
along with the server’s private key from
the WAN machine to the server in my LAN and
that server would then have a valid public
certificate.
Those certificates are located under /etc/letsencrypt
on the public server.
This solution has a couple of challenges though.
Of course, the certificate would only work
if I called the server page by it’s fully
qualified domain name.
Because the certificate had been issued to
pve.onemarcfifty.com but not just to pve.
So I would still get a certificate warning
if I used the short name.
Also – I would potentially need to make
a lot of entries in my public DNS if I wanted
to use this for multiple servers.
Plus – I either need a cloud server or I
have to open ports on my firewall so that
the certbot can be checked by Let’s encrypt.
Here’s another solution: Wildcard certificates.
(Let’s Encrypt wild card certificates)
Wild card certificates from Let’s Encrypt
work like this: your certbot requests a wildcard
certificate let’s say for *.onemarcfifty.com
– now, rather than connecting back to a
specific host, Let’s Encrypt now give you
a challenge: They tell you to put a txt record
named something like _acme_challenge into
your dns.
They give you an arbitrary value to put into
that record.
If they can successfully query that txt record
and if it does contain the value that they
have requested you to put in there, then they
know that you are the owner of that domain.
They issue the wild card certificate.
Yo do NOT need to open any ports.
You do NOT need a server in the cloud for
this.
This method of course does have challenges
as well.
If you want to automate it, then you would
need an API for your DNS provider.
But there’s a lot of howtos on the net for
digitalocean, cloudflare and many others.
I am using a German Web space provider who
do not offer an api.
In this case I can use a manual authentication
hook script that basically logs into their
web ui and makes the entry as requested.
So now I have a certificate that is valid
for any host inside onemarcfifty.com and also
for any domain under onemarcfifty.com – I
could therefore use a subdomain inside my
lan – such as local.onemarcfifty.com.
And I do not have to create any public DNS
entries for my hosts inside my lan.
Any host in that sub domain would show as
valid in the browser if I provided a copy
of the wildcard certificate to it.
I am using this in my network on a reverse
proxy server running nginx.
Whenever I cross network or rather VLAN boundaries,
then the traffic is redirected over this proxy
server in order to ask for a second factor
of authentication.
That reverse proxy has a wildcard certificate
for my domain.
That means that all SSL or TLS traffic that
I have running via that nginx would automatically
be shown as secured and OK.
Just to round this subject up: If you are
using the Let’s Encrypt Certificate method,
you should be aware that due to Certificate
Transparency (CT) or rather Audit requirements
the CA Authorities keep publicly available
logs of every single certificate that they
issued.
That means that everyone on this planet may
know that I have a server called pve.onemarcfifty.com
and also everyone may get a copy of that certificate.
Of course without the private key.
But still I am exposing the naming scheme
of my servers.
May or may not be an issue for you.
Perfect – I hope I could give you a couple
of ideas on how to use Server certificates
in your LAN.
Let me know if you want to have a follow up
on anything – leave me a comment here on
YouTube or start a discussion on my discord
server.
In the next episode I want to show you how
you can secure access to web servers by using
x.509 client certificates.
This does not seem to be used so frequently.
However it can shrink the attack surface of
your servers exposed to the internet considerably.
This is going to be a very interesting episode.
Don’t miss out on it.
Until then – thank you so much for watching,
liking and leaving comments.
Stay safe.
Stay healthy, bye for now.
</details>Marc AhlgrimSelf Signed and LetsEncrypt Server Certificates for the LANX.509 Certificates explained2023-01-16T00:00:00+01:002023-01-16T18:00:00+01:00https://www.onemarcfifty.com/blog/video/Certificates_Episode1<h2 id="certificates-from-scratch---x509-certificates-explained">Certificates from Scratch - X.509 Certificates explained</h2>
<p>What are X.509 Certificates? What is a “Certification Authority” or CA? How can we create our own CA? How can we sign our own Server certificates? How does LetsEncrypt work? How do private and public keys work? What is a certificate Chain or a Chain of Trust? The answers are in this video.</p>
<p>The XCA Tool can be obtained <a href="https://hohnstaedt.de/xca/">here</a></p>
<p>More Info on my <a href="https://github.com/onemarcfifty/cheat-sheets/tree/main/Certificates">Cheat Sheet Repo on Github</a></p>
<p><a href="https://www.youtube.com/watch?v=kAaIYRJoJkc"><img src="/assets/images/thumbnails/kAaIYRJoJkc.jpg" />Watch the video on YouTube</a></p>
<details>
<summary>Click to view the entire transcript</summary>
(about certificates)
We all know certificates, and we are using them on a daily basis. Even if we don’t necessarily call them certificates. Let me give you two examples. First one – if I browse to my own web site www.onemarcfifty.com then I see this little padlock icon here in the address bar. If I click on it, then my browser tells me that my connection is secure and that I have a valid certificate. Hmm. Yeah that’s great but what does it actually mean? What is secure? And what am I protected from? We’ll come to that – but first let me tell you a little story.
This is my nephew. He’s five years old and he has a kick scooter. One day I visit my brother in law and we are all sitting on the veranda. My nephew drives by and starts making car driving noises. I was up to the game, stood up and said “hold it citizen – may I see your driver’s license please?”. Now – my nephew is smart and seemed to be prepared. So he shows me this. I try to maintain a strict face and say “Thank you sir, your license does check out. Please continue and have a nice day”.
I invented this story. But it shows nicely what a certificate is. A certificate is a document that is linked to an identity. In this case “Marc’s Nephew”. So Marc’s Nephew can use this to prove his identity, but no one else can. A certificate has a subject and/or a purpose. In this case it proves that the person using it is authorized to drive a kick scooter. It could not be used to fly an airplane. It also has an issuer. In this case it’s my Nephew. It is a so called self-signed certificate. Because he issued it to himself. Anyone who trusts the issuer would in turn trust the certificate. But would this license be accepted in the real world – if he drove a real car on a real road? Presumably no. If we compare this license to a real driver’s license in the real world however, then there’s actually only one single difference here. The issuer. A driver’s license in the real world would be issued by the government typically. Or by any authority that is authorized by the government, such as a state or a county or a city and so on. Plus - those certificates have a certain number of features which makes it difficult to falsify or copy them. This example from the swiss government shows some of these features on the swiss driver's license.
(certificate chains – Certificate Autority/CA)
Back to the certificate on my web site onemarcfifty.com. How can it be that everyone on this planet trusts this certificate? Let me examine it. For this, I will use a cool software called XCA by Christian Hohnstädt. It’s awesome and it’s free open source. If we browse to www.hohnstaedt. HYPERLINK "http://www.hohnstaedt.de/"de, then we can download it for various platforms such as linux, MacOS or Windows. There’s even a portable version of this. Guys – all links to the software are in the description of the video. So let me go to my blog site, then click on the padlock and view the details of the certificate. I will then export it into a file. As a file format with Google Chrome I use PKCS#7 certificate chain. Because that format includes the most information. In Firefox you can save it as a full chain PEM file by just clicking on that link that it shows on the certificate detail page.
Next, I’ll use XCA to import it. Just before I can use XCA I need to quickly generate a new database for certificates and keys – that can be any file anywhere. And of course it needs to be protected with a password. Now I click on import, then PKCS#7 and select the file that I had just exported from my browser. Here we go. On the certificates tab I can now see the details and – equally important – I can see the chain of trust. The full certificate chain. My certificate www.onemarcfifty.com had been issued by someone or something called R3. And that certificate had been issued by a corpus called ISRG Root X1. Let’s go bottom up. I am interested in two fields here. The subject and the issuer. The subject of my certificate is www.onemarcfifty.com. The issuer is R3 who – surprise – belong to Let’s Encrypt. Let me go one level up by clicking on this green R3 in the Signature field. Here the subject of the R3 certificate is identical to the issuer of the onemarcfifty.com certificate. And the issuer of this one is of course the Internet Security Research Group or short ISRG. Again one level up to the root certificate, the so called Certificate Authority or CA. And – surprise – this CA certificate which is at the root level of the chain appears to be self signed! The issuer and the subject are identical. But wait! That one is self signed just like my nephew’s kick scooter license, right? And everyone trusts that one? How can that be? That sounds wrong.
Well, the secret is lifted if we check on the Root Certificate Authorities that my browser or my operating system trusts out of the box. On my Linux machine I can find them here in /usr/share/ca-certificates/mozilla. And there we go, there are a lot of certificates and of course there is the ISRG Root X1. So it’s part of the well known companies or organizations that are trusted by the common browser products. They are commonly trusted. And – we just learned one more thing – a CA or Certification authority is just a self signed certificate really. In your browser, you can find them in the settings under the security options. Here in Chrome you go to settings, then Privacy and security, click on security, then scroll down to Manage certificates and then select the “Authorities” tab. You can as well export them from there with the context menu. In Firefox, you can find this under Settings, Privacy and Security, then scroll all the way down to Certificates and click on “View Certificates”. Again, select the “Authorities” tab. Select a certificate and you can export it using the Export button at the bottom.
In other words – the “trust” for those self signed root CA certificates is built into the products that use them. Typically, that’s your browser.
(private keys)
A Server certificate that is signed or issued by such a trusted CA can prove to everyone that the server that I am connecting to actually is who it pretends to be. But – could I go around this? Could I use these certificates let’s say in an isolated environment? I mean, I have a valid certificate here for ibm.com. I have just saved it from Chrome like I did with my own certificate. Obviously I can’t fake IBM’s identity in the web because I don’t control their DNS servers. And I have no interest doing this. But - could I now let’s say define ibm.com to be my internal domain name here in my LAN with my local DNS Server, then set up a web server – still in my local LAN and set up a host called www, then browse to that internal web server – would I get one of these nasty security warnings or would I get a certificate for free? You may have guessed the answer – this is not possible. But why? And how?
Well, this is how certificates are supposed to work. Only the owner can use them but anyone and everyone can verify them. Just like Photo ID in the real world. Let’s go back to XCA. The onemarcfifty.com certificate has been signed by the R3 certificate. That’s why the R3 appears green here in the signature field. And there is a public key stored inside the onemarcfifty certificate that allows me to verify that signature coming from the private key of the R3 certificate. Watch what happens when I delete the R3 certificate. I can still see the public key but I can’t verify the signature because the signing certificate is not there any more. Let me quickly explain how private and public keys work.
(How private/public keys work)
In a nutshell, with algorithms like RSA we can do four or rather five things with private or public keys. We can encrypt, decrypt, sign and verify. And as a consequence, we can authenticate. The trick is now that you can only verify the signature that has been made with one key by using the other key. The same is true for encryption and decryption. You can only decrypt data with one key that had been encrypted with the other key. In real life we use the private key to sign something. That means that only the public key can verify the signature. One person can sign and everyone can verify. The other way round everyone can encrypt with the public key and only the owner of the private key can decrypt. How is that possible? Let’s do a little bit of math. Don’t worry. It will not get too complicated.
(a tiny bit of math)
In fact, we would need an operation that can easily be one in one direction but is impossible or let’s say hard to do in the other direction. Let me give you one example. If I was to calculate let’s say 3 times 7 times 17 times 23 – that’s quite easy to do. The result is 8211. The numbers that I have used for this multiplication are prime numbers. Now if I would ask you to do this the other way round – so if I would ask you to tell me which prime numbers I have multiplied in order to get to 8211, then this is much much harder to do. If the calculation in one direction would need let’s say 4 CPU cycles, then the reverse calculation could use hundreds or maybe even thousands of cycles. It would take much much more time. Now of course, this is feasible with 8211. But what if I had not used 7 or 17 as prime numbers but really huge prime numbers such as 2 to the power of 255 minus 19? That’s a number with 77 digits!!! Virtually impossible. It would take the fastest computer on this planet thousands of years to calculate it. Maybe it could go faster if it had a huge hash table with all known prime numbers and a really big number of pre-calculated hashes. The algorithms that are used for public and private keys are of course much more complex than this – but I think you get the idea. In a nutshell, it is not impossible to let’s say calculate or find a private key if you only have the public key. It would just take incredibly long. We therefore say that the cipher is computationally secure.
Back to the certificates.
In order to use a certificate for all operations, I would need the public key (which I have, because it’s stored inside the certificate) and the private key (which I don’t have – in my ibm.com example, only ibm have it). That’s the reason why I can’t fake their identity. I can’t build a chain of trust without having both keys. So when and how would a public CA issue a certificate to me? To Marc from Germany? They would need to be sure that I am who I pretend to be – or rather that my server is who it pretends to be.
(How are Let’s Encrypt doing it?)
How are ISRG and Let’s Encrypt doing it? I mean, all the browser vendors are trusting the Let’s Encrypt certificates. And they have issued a certificate to me – how would they make sure that they can stand up to Firefox, Chrome, Edge or whoever and say “Trust us, we have verified Marc’s identity” – I mean, I have never spoken to them? Well, Let’s encrypt are using DNS for this. They don’t actually have to verify my personal identity, just the one of my server. The thought process is as follows. If a host – let’s say www.onemarcfifty.com requests a certificate from ISRG, then all they’d have to do is do a DNS lookup of that address. And if that DNS lookup points to the host that made the request then it’s authentic. Because the person initiating the request needs to have control over the domain’s DNS entries. That process can easily be automated. And they do provide me with a signed certificate and a private key for that single specific identity. If I am able to put some arbitrary text entry on demand into my DNS, then they are even willing to issue a so called wild card certificate to me. That means that I could request a certificate that would be valid for any host in the onemarcfifty.com domain. For free. This type of authentication is called Domain validation (DV) – I authenticate by proof of control over a domain, in my case onemarcfifty.com.
(Let’s create our own CA and certificates)
So far so good. We now know that there are different types of certificates. Some of these are called a Certificate Authority or short “CA”. A CA can sign other certificates or even other CA’s just like ISRG signed the R3 CA which in turn signed my onemarcfifty.com Server certificate. So how would I proceed if I wanted to use certificates for my servers at home? How could I generate my own certificates? Well, basically I have three alternatives here. Either I use a paid or free service that issues Server or wildcard certificates to me, such as Let’s Encrypt. Or – I ask one of the root CA’s to sign my own CA which in turn I could then use to sign as many certificates as I please. A so called “vanity CA”. Or – that’s alternative 3 – I use my own self-signed certificate. Each of these mechanisms has advantages and challenges. Let’s Encrypt and the like require me to use an agent and to be in control of DNS. Not a big thing. There’s many videos on that.
And in the next episode I’ll show you a couple of tricks around that such as automating wildcard certificate requests with Ansible or using Let's Encrypt certificates for your internal network.
The limitation of these services is usually that you only get server certificates. The second method – the vanity CA – has one big challenge: The price. I checked on multiple provider pages for pricing and all of them just stated “ask for quote” or “on demand” – that doesn’t sound cheap. Also – you would presumably have to provide a lot of security procedures in order to prove to them that you don’t issue certificates for just anybody. Remember – if your CA is signed by a trusted CA, then you become part of the globally recognized chain of trust and – as such – a high value target. I wouldn’t actually want that. So what’s left? Actually, we can easily create our own Certificate Authority or CA. A self-signed CA. Just no one else would trust it – so probably not a good solution for a public web server but there are valid use cases for this.
(When and how to use self-signed CA)
A CA is really just a normal certificate which has a couple of specific attributes. Here in XCA we can add them down here with the default CA template. That does set a couple of things here in the Extensions tab and the Key usage tab. As a minimum I need to give the CA a name and a common name. I strongly advice to also assign it an organizationalName and a country code. And I need to generate a private key. Cool, so now I have a CA that is – of course – trusted by no one on this planet.
However, we can now generate more certificates. Here we see the first difference. Now I can not only create self-signed certificates, but I can actually sign this certificate with my CA. Let me first apply the TLS Server template here. Again – I need to give it a name and a common name, preferrably an organization and a country code and I need to generate a key for it. I’ll use a server or router name in my sandbox as the common name here because I want to use it for that server. Let me export that server certificate in pem format, that means just the plain certificate. Without the key. There are other formats available. Some of these do contain the key and others don’t. The web server that I want to use the certificate with needs both in separate files. So next, I’ll export the corresponding key – again in pem text format.
That’s it – I can now use that certificate for that server. But in order to have my browser trust it, I need it to recognize my CA certificate. But no problem, I own it so I can just import it to my web browser. In XCA I export the CA certificate. We don’t need the key for this – just like we don’t have the key for the public ones. Then next – in my browser I go back to the certificate settings or info page and there I can import the new CA. My browser will now trust any certificate that has been issued by that CA.
So when can and when would we use self signed certificates? In general, we can use this mechanism when we have control over both parties, that means the client and the server. Just like I showed here. I control the server and the client (my web browser). Or – maybe I don’t trust the public CA’s for some reason. However, there are challenges when you have many clients. As your CA is not built into the browsers, you would need to import or roll out the ca certificate to all of them. But let’s have a look at options for that scenario in the next episode when we will see how we can use free Let’s encrypt certificates for servers for our local network and also more options for self-signed CA’s.
Just a last remark with regards to security, vulnerabilities and so on. You may have heard that the security gets better with longer keys. That’s true. Also there are different cipher and hashing algorithms that you can use. Some of them are compromised, in other words proven to be insecure. MD5 is an example for this. However – please keep in mind that most successful attacks against crypto do NOT break the underlying crypto primitives but rather get around them. In other words – why would I want to brute force your private key if I can easily steal it from your workstation? So TLS, Certificates, encryption, authentication – all these things do not only rely on a good algorithm and a long key. It is equally important to have a look at the whole process of generating and distributing keys and certificates and also how you manage your keys. In other words – the real challenge is not encryption or decryption but rather Key management. But more on this in later episodes.
Until then – like always – thank you so much for watching. Stay safe, stay healthy, bye for now.
</details>Marc AhlgrimCertificates from Scratch - X.509 Certificates explainedIPv6 with OpenWrt2023-01-05T00:00:00+01:002023-01-05T18:00:00+01:00https://www.onemarcfifty.com/blog/video/IPv6_with_OpenWrt<h2 id="ipv6-with-openwrt">IPv6 with OpenWrt</h2>
<p>There are at least three ways to use IPv6 ith OpenWrt: Prefix Delegation, NDP Proxy and 6in4 tunnel with Hurricane Electric or the like. In this video we will walk through the configuration for each of them.</p>
<p>The IPv6 from Scratch Episodes are here:</p>
<p><a href="https://youtu.be/oItwDXraK1M">Episode 1</a>
<a href="https://youtu.be/jlG_nrCOmJc">Episode 2</a></p>
<p><a href="https://www.youtube.com/watch?v=LJPXz8eA3b8"><img src="/assets/images/thumbnails/LJPXz8eA3b8.jpg" />Watch the video on YouTube</a></p>
<details>
<summary>Click to view the entire transcript</summary>
(IPv6 with OpenWrt)
I have some good news and some bad news for
you.
First the good news: IPv6 is fully supported
by OpenWrt.
Because OpenWrt is Linux and Linux does support
it.
Now the bad news: If and to what extend you
can really use IPv6 with OpenWrt is largely
determined by your ISP – that’s the people
who provide internet access to you.
Not by OpenWrt.
Just because you have an IPv6 address on your
router does not necessarily mean that you
can do IPv6 in the way it had been intended
by its inventors.
But we’ll see into that in detail.
There is however some more good news: Even
if you only get IPv6 in a crippled way, then
OpenWrt can help you to get the most out of
it.
Either with prefix delegation or with NDP
proxy or with a 6in4 tunnel.
Let’s walk through our options.
(Option 1 – Prefix delegation)
In an ideal world, your ISP would delegate
an IPv6 prefix to you.
That means that the first 40 or 48 or 56 bits
of the IPv6 address are fix and you could
do whatever you wanted with the remaining
bits.
How can you find out if you have prefix delegation
or not?
Look at the status page of Openwrt.
If you have IPv6 enabled, then you should
see an IPv6 Upstream here.
And if your ISP does delegate a prefix, then
there would be an entry called “Prefix delegated”.
In my case it’s a 59 bit prefix.
My ISP delegates a 56 bit prefix but this
router is inside a test lab sandbox and there
are some more routers in between.
If you don’t see IPv6 at all, then either
your ISP is not giving it to you or you have
previously disabled it.
Let’s check the relevant settings quickly.
And this is on an OpenWrt router with default
settings.
First we go to network – interfaces.
There should be interfaces for LAN and for
WAN and also for WAN6.
The relevant settings on the WAN6 interface
are DHCPv6 client as a protocol, then Request-IPv6
address should not be set to disabled and
in order to actually get assigned a prefix
you should either select a fixed length setting
or chose Automatic from the drop down that
is titled “Request IPv6-prefix of length”.
I have set it to Automatic.
If we want IPv6 prefixes to be delegated further
down the chain, for example to the LAN interface,
then we would also tick this box on the advanced
tab that is titled “Delegate IPv6 prefixes”.
Watch what happens when I untick that box.
You can see that my LAN interface has an IPv6
address in the 2003:de range.
That’s Deutsche Telekom.
Now let me go to the advanced settings of
the WAN interface and remove the tick from
the IPv6 prefix delegation box.
Save and apply.
Checking back on the LAN interface, you can
see that the 2003:de address has now disappeared.
The only remaining IPv6 address on this interface
is the Unique Link (ULA) Address in the fd
something range.
If I go back to the advanced settings of the
WAN interface and tick the box back in, then
save and save and apply, then the global IPv6
address comes back on the LAN interface.
Before we move on let’s quickly talk about
that fd something address here.
You may remember the IPv6 from scratch video
where we talked about scopes.
This unique local address comes from a ULA
scope that we can define here on the network
interfaces on the Global Options tab.
There is an IPv6 ULA prefix tab here.
This is randomly generated at install time.
You could either set it to something more
simple such as fddd::/48 or you can as well
safely delete it – in my opinion there is
no point in having ULA addresses if you have
prefix delegation.
You can safely do everything with global and
link local addresses.
So let me delete it.
Save and Apply – see?
The address has disappeared from the LAN interface.
(Special case: PPPOE)
Before we dig deeper into the LAN configuration,
let’s quickly check two more possible situations.
Maybe you had deleted your WAN6 interface
in the past or your WAN interface is a PPPOE
interface.
So let me delete my WAN6 interface and recreate
it.
Add new Interface, select DHCPv6 client as
a protocol and now – rather than selecting
the same eth1 interface like the WAN interface
has, I now tell OpenWrt that this is a so
called alias interface.
It’s the same interface like the wan interface
but we do not need to do PPOE again, just
DHCPv6.
This way you can run multiple protocols over
one interface.
Just don’t forget to assign it to the right
firewall zone, for instance WAN in my case.
Save and apply and – here we go – we get
everything back.
IPv6 prefix and address on WAN6 as well as
on LAN.
(Firewall rules WAN side)
You remember that IPv6 needs the ICMPv6 protocol
to function correctly?
That means that we might need to tweak the
firewall settings a little bit – especially
if you have deleted the corresponding rules
in the past because you didn’t want to have
IPv6 in your network.
Let’s head over to Network- Firewall.
Here on the traffic rules tab there are just
three or four rules that we need to check
on really.
First of course we need to let in traffic
on UDP port 546 to the router itself for DHCPv6.
The next rule lets in any ICMPv6 traffic from
any device on the local link.
That would be the ISP’s router really.
The next two rules actually let in ICMPv6
traffic from any device in the internet and
will even forward it to any zone.
But we will limit it in two ways.
First, we only need a certain number of ICMPv6
types to be let in.
And second, we will limit the number of packets
to 1000 per second in order to be protected
against flooding attacks.
I will put a dump of the relevant firewall
configuration into the cheat sheet repository
on my github site.
The link is in the description of the video.
(LAN interface settings)
Nice – now let’s quickly walk through
the settings of the LAN interface.
After all, what we want is that OpenWrt assigns
or suggests IPv6 addresses to our clients
in the LAN.
If you remember the IPv6 from scratch videos
then you know that we have essentially three
possibilities to assign IPv6 addresses to
our nodes.
We could give them a static address, we could
use DHCPv6 and/or we could use SLAAC.
Let’s check the settings for this.
I edit the LAN interface and go to the advanced
tab.
We already know this tick box from the WAN
interface.
Just now – we don’t want to delegate IPv6
prefixes further down the stream as presumably
you don’t have any more routers behind the
LAN interface.
So let’s untick that box here.
Next – the IPv6 assignment length.
The network part of the IPv6 addresses are
the first 64 bit – you remember?
As we don’t want to delegate further down,
we therefore set the assignment length to
64 bits.
That means that the LAN network will now have
its own full /64 network but we can’t subnet
any further behind that interface.
But which subnet will the LAN interface get?
I have tuned the WAN interface to have a /60
prefix which means that I can use the last
4 bits of the network mask for various interfaces.
That’s what the IPv6 assignment hint does.
Let’s say I wanted the LAN to be subnet
5 – then I would set the IPv6 assignment
hint to 5.
The IPv6 suffix here defines which address
my router will get inside that subnet.
Usually this is set to ::1, but let me set
it to 99 so that we see what it does.
Save.
Now I save and apply.
Look what this did.
My LAN interface now has a /64 IPv6 address.
The subnet is 74d5 (it was 74d0 before) – that’s
because of the assignment hint which I had
set to 5.
And the node address is ::99.
Let’s add another interface, for example
guest.
I give that one another IPv4 address range,
then I go to the advanced tab and put in the
same values except this time I use let’s
say “a” as IPv6 assignment hint.
I would then expect the guest network to have
the subnet 74da rather than 74d0 or 74d5.
Save and apply.
I did some more things here like define a
device and a firewall zone and I had to restart
the networking service so that OpenWrt applied
the changes.
But once I did that, as you can see - the
subnet on the guest interface is now 74da
as expected.
By the way, when I stop the WAN6 interface,
then the IPv6 addresses of the LAN and guest
interface go away.
Once I restart it, then they come back.
In other words – the easiest way to shut
down IPv6 in your whole network is to just
stop the WAN6 interface.
When you bring it back up, IPv6 comes back.
Nice and clean.
Cool, now let’s have a look at the DHCP
settings.
How do we tell our clients in the LAN that
they can get IPv6 addresses from OpenWrt ?
(Configuring DHCPv6 options)
If we want to have global IPv6 addresses in
our LAN, then we need to let our clients in
the LAN know which IPv6 prefix they should
use, which IPv6 router and a couple more things
like the address of the DNS server.
We can do this either via router advertisement
or via DHCPv6 or – we can mix both mechanisms.
Here is how.
I go to Network-Interfaces, then I click on
“Edit” next to the desired interface – let
me use the LAN interface for starters.
Under “DHCP Server” I can see the IPv4
DHCP settings.
If the DHCP Server is not yet configured,
then you get a button here to enable it.
What I am interested in, are the IPv6 settings.
Here I can tell OpenWrt that I want to have
two things.
First, I want this router to be advertised
using ICMPv6 router advertisement.
For this I select “server mode” in the
RA-Service dropdown.
But I also want this router to be an IPv6
DHCP Server.
Therefore I also select “server mode”
in the DHCPv6-Service dropdown.
My router shall also serve DNS so I do check
the “Local IPv6 DNS server” tick box.
Please do not tick the designated master tick
box and also set the NDP-Proxy dropdown to
disabled.
We don’t need this here, we’ll talk about
this in the second scenario.
On the next tab (that is titled IPv6 RA settings)
I can define some more details.
First – do I want my router to be the default
router?
I set this value to automatic.
Now for the interesting part.
The tick box “enable SLAAC” actually decides
whether we allow our clients to do stateless
automatic address configuration.
If we tick this box, then the corresponding
flag in the router advertisement packet gets
set.
In the next box we can configure some more
flags for the router advertisement.
If we set the managed config flag, then this
tells our clients that they should request
an IPv6 address via DHCPv6.
If we tick the “other config” box here
then this means that we let our clients know
that they can have more info such as DNS servers
over DHCPv6, even if they don’t request
an IPv6 address via DHCPv6.
I know that this part may sound a bit confusing,
especially when you are new to IPv6.
Please refer to the second episode of my IPv6
from scratch videos and also have a look at
the comparison table on my IPv6 cheat sheet
from my github repository.
The table here helps you decide which mechanism
to use.
How am I using this at home?
You may remember from other videos that I
have a dedicated management VLAN here.
This is where all my routers, servers, switches
– generally speaking the whole infrastructure
– provides https and or ssh interfaces.
These devices need to be identifiable, therefore
I don’t use SLAAC on that interface.
On my LAN interface however, where all the
laptops are in, I do not need fixed addresses
and therefore I only have SLAAC on that interface.
Still – even if I do not set the “Managed
config flag”, I can still use DHCPv6 even
in my LAN by forcing the client to do so.
For example these Proxmox containers here.
They do get an IPv6 address from the LAN range
over DHCPv6, not SLAAC.
I can do this because I set the IPv6 address
to DHCP in Proxmox and also I told the container
to NOT use SLAAC by setting this autoconf
value to 0.
As you can see, it does have a global IPv6
address, but its a nice clean address coming
from DHCPv6.
In a nutshell – the fact that we set the
DHCPv6 Service to server mode on the first
tab means that the router will answer to DHCPv6
requests on UDP port 546 to the multicast
address ff02::1:2.
The RA-Service dropdown determines whether
we answer Router Solicitation requests on
ICMPv6 or not.
If I set this to disabled, then the second
tab disappears.
(static leases and DNS)
Perfect.
Now we have IPv6 configured on the WAN and
LAN side.
Just – we have a handful of little challenges
here.
Let’s say we don’t have a fixed IPv6 prefix,
we rather get a new one from the ISP every
24 hours or so (like I do with Deutsche Telekom).
How would we manage DNS and firewall rules
for nodes that have ever changing IPv6 addresses?
Let me show you the problem.
I go to the firewall Traffic rules and I want
to make a rule for this sandbox-adguard server
here.
If I made a rule now and then later the prefix
or the address of this adguard server changes,
then my rule would be useless.
How can I solve this?
First, I would need to make sure that this
server gets a constant or static node address.
I am not talking about the prefix.
Just the lower 64 bits that identify the node
itself.
Also – I know that the server will always
have the value 5 as the last 4 bits of the
subnet – you remember?
So – let’s start with the node address.
Going to the status page of my router, I can
see all the clients that have DHCP leases
at the moment.
For both IPv4 and IPv6.
Using this “Set static” button I can make
sure that this same device gets the same IP
address the next time it requests one.
The unique identifier for this would be the
MAC address in the IPv4 world.
In the IPv6 world, the machine has sent a
unique identifier, the so called DUID to the
DHCPv6 server.
The client is responsible for generating this
and needs to make sure that it is always the
same value.
You can actually see portions of the MAC address
in the DUID.
Unfortunately, if I now make both the IPv6
and the IPv4 address static, then OpenWrt
will create two entries here under Network-DHCP
and DNS- Static leases.
That would work but I want to make sure that
the IPv6 address also resolves correctly with
DNS.
Therefore, let me delete the IPv6 entry and
rather edit the IPv4 entry.
Here I have two parameters that I can tweak
for IPv6.
In the DUID dropdown box I can actually select
the adguard server.
I can see that’s the same MAC address.
Or – if I have many machines or if I can’t
find it here, then I could also just copy
paste it in here.
The next parameter is interesting and actually
lets me define the node address portion of
that machine.
Let me put in 245, because the IPv4 address
ends in 245.
I know it’s not the same thing.
The one is decimal and the other one is hex,
but they look the same.
Save, then Save and apply.
The next time that this machine requests an
IPv6 address, that address will end in 245.
I can accelerate that a bit if I restart the
odhcpd service on the router and down and
up the network interface on the client.
Takes a while.
Here we go.
Quick cross check with nslookup.
Yep.
That comes back with both addresses.
IPv4 and IPv6.
Beautiful.
So – how does that help me with my firewall
rule now ? Let’s have a closer look at this
server’s address.
We have the prefix which is 60 bits long.
Then we have the 4 bit subnet which I can
manage on my own, then we have the node address
which is many zeros and then 245.
If I apply a bitmask to this in the sense
that I set all bits of the relevant (fixed)
part to 1 and all bits of the changing prefix
to 0, then I can write this address in subnet
mask notation like this.
Not beautiful, but it works.
So rather than having a firewall rule for
the whole IPv6 address, I now have one for
any IPv6 address in subnet 5 that ends in
000... 245.
I could now even have one single rule for
IPv4 and IPv6.
Of course I could have used the MAC address
as well.
(NDP proxy – no prefix delegation)
So far so good.
This all works if we have prefix delegation
by our ISP.
Unfortunately, many ISPs do not offer prefix
delegation, but they rather just give you
a /64 subnet.
What can we do here?
In this case, you would not see any delegated
prefix here on the status page or on the WAN
interface.
Just a /64 address really.
And no IPv6 address on the LAN or guest interface,
because there are no more subnets to be delegated
really.
The whole /64 space is already “eaten up”
by the WAN interface.
If you still want to use IPv6 in your LAN,
then you can configure OpenWrt as an IPv6
NDP proxy between the WAN and LAN zone.
NDP stands for the Neighbor discovery protocol.
Here is how.
First, we need to edit the WAN6 interface
and add DHCP settings.
Important – we don’t want to run a DHCP
Server on the WAN zone here, so make sure
that this “Ignore Interface” Checkbox
is ticked.
On the IPv6 Settings tab, we tick the box
“Designated Master” and we set all other
drop downs to “relay mode”.
On the interfaces in our LAN or guest network
we now set all drop downs to relay as well.
Once we save and apply, we won’t really
notice any difference on this screen.
But watch what happens if I turn the network
connection on my client off and on again.
Tadaaa – I get an IPv6 address.
My OpenWrt router now proxies all my requests
through to the WAN side.
Unfortunately, our possibilities what we can
do with IPv6 LAN side are quite limited with
this setup.
In a nutshell we get a SLAAC or DHCPv6 address
for our nodes and that’s pretty much it.
(6in4 tunnel)
So what can we do if we want to enjoy full
IPv6 functionality but our provider doesn’t
give it to us?
There is a third alternative called 6in4.
This basically tunnels IPv6 through IPv4.
In order to use this you have to sign up to
a tunnel broker on one side and create a 6in4
interface on your side.
The tunnel broker will now give you ipv6 connectivity
through a tunnel inside ipv4.
Pretty similar to a VPN really.
There are IPv6 tunnel brokers who give you
a whole /48 prefix for free like Hurricane
Electric for example.
All that you have to do is sign up on their
site tunnelbroker.net.
From here, you can create a new tunnel to
many locations on various continents.
Let me add one for New York here.
All that you have to give them in order to
be able to generate a tunnel is a valid IPv4
address which they can ping in this IPv4 endpoint
field here.
So I am not sure if this works with double
Nat (Carrier grade NAT / CGN).
Leave me a comment if you have it up and running.
Also you might need to add a firewall rule
to your router so that they can actually ping
you.
Don’t forget to select /48 prefix.
Now for the configuration on the OpenWrt side.
You go to system-software and install the
6in4 software package.
You will need to reboot after this so that
LuCI shows the new protocol.
Once you’re back in, go to network-interfaces
and add a new interface with the IPv6-in-IPv4
protocol.
Enter all the information from the tunnelbroker
website.
Just down here where it says HE.net password,
enter the dynamic update key.
This way hurricane electric will automatically
update the IPv4 endpoint for you.
Don’t forget to assign the interface to
the WAN firewall zone.
And – there we go.
Beautiful /48 prefix delegated to us.
Full IPv6 functionality.
Awesome.
You might wonder – why does Marc not talk
about IPv6 NAT?
Well, there’s two reasons for that.
First – I have never used it.
I am only using IPv6 for a couple of months
now and really want to understand the whole
thing fully before I examine things like IPv6
NAT.
Second, OpenWrt have just recently moved from
iptables to nftables as their firewall and
currently it looks like IPv6 NAT relies on
Iptables.
So I guess we will just have to wait a bit.
Even though IPv6 NAT might not sound right
to the IPv6 purists among us, I do actually
have a use case for it which is reverse proxy
between network segments.
So – there’s probably more to come.
Stay tuned.
Until then – many many thanks for watching,
liking and commenting.
Stay safe, stay healthy, bye for now.
</details>Marc AhlgrimIPv6 with OpenWrtUpgrade and Revert using Grub and efibootmgr2022-10-31T00:00:00+01:002022-10-31T18:00:00+01:00https://www.onemarcfifty.com/blog/video/Proxmox_Upgrade<h2 id="proxmox-upgrade">Proxmox Upgrade</h2>
<p>I wanted to upgrade my <a href="https://www.proxmox.com">Proxmox VE</a> Server from version 6 to version 7. But I did not want to do this without a <strong>Plan B</strong>, a <strong><em>fail back plan</em></strong>. For this I used GRUB and efibootmgr. I converted a small swap partition into a bootable Linux partition and pivoted the Version 6 to Version 7. When I noticed that things did not work as expected, I was able to revert to version 6 in less than a minute using GRUB, UEFI Bios and efibootmgr.</p>
<h2 id="situation-before-the-upgrade">Situation before the Upgrade</h2>
<p>I have Proxmox VE (Version 6.x) running on my Server. My Proxmox Server has two hard disks, but for this exercise we will only look at the first one. In fact, as the server boots with EFI, there are three partitions on that disk:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: xxx-xxx-xxx-xxx-xxx
Device Start End Sectors Size Type
/dev/sdb1 34 2047 2014 1007K BIOS boot
/dev/sdb2 2048 1050623 1048576 512M EFI System
/dev/sdb3 1050624 937703054 936652431 446.6G Linux LVM
</code></pre></div></div>
<p>The second partition is the EFI boot partition, and the third partition contains all the LVM volumes. In order to see the pools and volumes, I type <code class="language-plaintext highlighter-rouge">lvs</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>LV VG Attr LSize Pool
data pve twi-aotz-- 340.00g
root pve -wi-a----- 82.00g
swap pve -wi-a----- 8.00g
vm-1000-disk-0 pve Vwi-aotz-- 70.00g data
vm-1000-disk-1 pve Vwi-aotz-- 4.00m data
</code></pre></div></div>
<p>(More volumes are listed but are not important here). All these volumes are mappd to /dev/pve/(swap, root) and also to /dev/mapper/(pve-root,pve-swap,pve-vm….)</p>
<p>The output is a bit tricky to understand. Basically, there is a pool called “data” which contains all the volumes for my VMs and containers (“t” - thin provisioned). Plus there is an 82 GB root partition and an 8 GB swap partition. As I wanted to be able to go back after the upgrade if things would go wrong, here’s the idea:</p>
<h2 id="the-plan">The Plan</h2>
<ol>
<li>Turn off swap usage</li>
<li>Convert the swap volume to a bootable partition</li>
<li>Boot into the (former) swap partition</li>
<li>shrink the root volume to roughly 50% (41 GB)</li>
<li>create a new root2 volume</li>
<li>copy all the data from the root volume to the new root2 volume</li>
<li>boot into root2 and do the upgrade in the new root2 volume</li>
<li>If things go wrong, revert back to the old root volume</li>
</ol>
<h2 id="step-1-converting-the-swap-volume-into-a-bootable-linux">Step 1: Converting the swap volume into a bootable Linux</h2>
<p>The swap volume is listed in your <code class="language-plaintext highlighter-rouge">/etc/fstab</code> - just comment out the line</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vi /etc/fstab
# /dev/pve/swap none swap sw 0 0
</code></pre></div></div>
<p>Then we switch off swapping on the running system, convert the swap volume into a file system and create a copy of the running linux to the new file system:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># check if swap is used
swapon -v
# switch swap off
swapoff
# remove the entry from fstab
vi /etc/fstab
# see if anything else is mounted
mount -a
swapon -v
# create an ext4 file system on the former swap volume
# and mount it in /mnt/swap
mke2fs -j /dev/mapper/pve-swap
mkdir /mnt/swap
mount -t ext4 /dev/mapper/pve-swap /mnt/swap
</code></pre></div></div>
<p>before we can rsync the files over to the (former) swap volume, we need to stop all services that have files open (I forgot to stop the firewall service but the copy was still “good enough” to temporarily boot from it)</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># check for running services and stop them
systemctl status
systemctl stop lxc-monitord.service
systemctl stop lxcfs.service
systemctl stop pve-lxc-syscalld.service
# now rsync the files over from root to the former swap
# we need to exclude pseudo file systems like dev, sys, proc and the like
rsync -aAXv / --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /mnt/swap/
# now rsync again
</code></pre></div></div>
<p>Now we have a good enough copy on the former swap volume - next we need to create a new EFI boot entry.</p>
<h2 id="step-2-copy-grub-and-change-the-uefi-boot-order">Step 2: Copy grub and change the UEFI boot order</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># the grub entries for pve are in /boot/efi/EFI
# we just copy the proxmox entry to a new entry called "pve6"
cd /boot/efi/EFI
mkdir pve6
cp proxmox/grubx64.efi pve6/
# now check the UEFI boot entries
efibootmgr -v
# see the partition that grubx64.efi would boot from
strings proxmox/grubx64.efi
# create a new UEFI bios boot entry for the new one
# please note the double back slashes!
efibootmgr --create --disk /dev/sdb --part 2 --label "pve6" --loader \\EFI\\pve6\\grubx64.efi
# change the boot order
efibootmgr -o 0000,0003,0001,0002
# bind mount dev etc.
mount /dev/sdb2 /mnt/swap/boot/efi
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt/swap$i; done
# chroot into the new root
chroot /mnt/swap
# now mount boot/efi and change the Grubx64.efi
mount /dev/sdb2 /boot/efi
grub-install --target=x86_64-efi /dev/sdb2
# last but not least update the grub config
update-grub
# exit, unmount and reboot
exit
for i in /dev /dev/pts /proc /sys /run; do sudo umount /mnt/swap$i; done
reboot
</code></pre></div></div>
<p>Phew! That’s it. The system should now boot into the (old) swap volume. We can now take an offline copy of the old system onto a second disk (/mnt/disk2) - because the next steps (shrinking the file system etc.) are risky.</p>
<h2 id="step-3-take-a-safety-copy-of-the-old-system">Step 3: Take a safety copy of the old system</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># system boots into /dev/mapper/pve-swap
# now take another offline copy
mount -t ext4 /dev/mapper/pve-root /mnt/pveroot/
mkdir /mnt/disk2/safetycopy
rsync -aAXv /mnt/pveroot/ /mnt/disk2/safetycopy/
</code></pre></div></div>
<h2 id="step-4-shrink-the-original-file-system-and-volume">Step 4: Shrink the original file system and volume:</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># unmount, check the file system and shrink it
umount /mnt/pveroot/
e2fsck -f /dev/mapper/pve-root
resize2fs /dev/mapper/pve-root 40G
# now reduce the Logical Volume
lvreduce -L 41G /dev/pve/root
# mount and check
mount -t ext4 /dev/mapper/pve-root /mnt/pveroot/
ls -al /mnt/pveroot/
</code></pre></div></div>
<h2 id="step-5-create-a-new-volume-root2-for-pve7-and-copy-the-pve6-volume-over">Step 5: Create a new Volume (“root2”) for pve7 and copy the pve6 volume over</h2>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># looks good, create the new volume for pve 7
lvcreate -n root2 -L41G /dev/pve
# check it with lvs
lvs
# the "o" attribute shows it's open so let's unmount the old one and check again
umount /dev/mapper/pve-root
lvs
# Now create the file system on the the new one, mount it and offline rsync everything over
mke2fs -j /dev/mapper/pve-root2
mkdir /mnt/pveroot2
mount -t ext4 /dev/mapper/pve-root2 /mnt/pveroot2/
rsync -aAXv /mnt/pveroot/ /mnt/pveroot2/
</code></pre></div></div>
<h1 id="step-6-same-story-as-before---copy-efi-update-uefi-grub-install-and-update-grub">Step 6: same story as before - copy efi, update uefi, grub-install and update-grub</h1>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>efibootmgr -v
cd /boot/efi/EFI/
cp -av proxmox/ swap
ls -al
efibootmgr --create --disk /dev/sdb --part 2 --label "swap" --loader \\EFI\\swap\\grubx64.efi
efibootmgr -o 0000,0003,0004,0001,0002
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt/pveroot2$i; done
chroot /mnt/pveroot2
mount /dev/sdb2 /boot/efi
grub-install --target=x86_64-efi /dev/sdb2
update-grub
# don't foregt to modify the new fstab
vi /etc/fstab
exit
for i in /dev /dev/pts /proc /sys /run; do sudo umount /mnt/pveroot2$i; done
reboot
</code></pre></div></div>
<h1 id="step-7-boot-into-root2-and-do-the-in-place-upgrade-of-proxmox-to-v7">Step 7: boot into root2 and do the in-place upgrade of Proxmox to V7</h1>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pve6to7
apt update
apt dist-upgrade
sed -i -e 's/buster/bullseye/g' /etc/apt/sources.list
apt update
apt dist-upgrade
reboot
</code></pre></div></div>
<h1 id="step-8-revert-to-version-6">Step 8: Revert to Version 6</h1>
<p>Once I realized that Version 7 did not work as expected, I reverted to Version 6 simply by changing the boot order back to the “pve6” entry:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>efibootmgr -v
efibootmgr -o 0003,0000,0004,0001,0002
reboot
</code></pre></div></div>
<p><a href="https://www.youtube.com/watch?v=3UMx7P5n91Y"><img src="/assets/images/thumbnails/3UMx7P5n91Y.jpg" />Watch the video on YouTube</a></p>
<details>
<summary>Click to view the entire transcript</summary>
In this video I want to show you how I used the GRUB boot loader and the EFI bios of my Proxmox server in order to upgrade Proxmox from Version 6 to Version 7 and how I went back in just a minute. This video is all Linux command line. Proxmox is just a debian Linux that runs additional software such as LXC,KVM, QEMU and the like. What I’m showing you today is how to pivot from one Linux installation to another by using a temporary linux installation and shrinking the size of the original filesystem.
( The scenario )
Here’s the volume layout of my Proxmox Server system which is actually a PC which is located in a closet downstairs in the basement. There’s a root volume with Proxmox 6 in it and a swap volume. I will turn that little swap volume into a bootable Linux volume, boot into it with GRUB, shrink the original file system and volume, then add a second root volume, copy the first volume onto it, then I’ll instruct the BIOS to EFI boot from the second volume and I will then run the Proxmox upgrade there. After the upgrade I’ll figure out that some things are not working as expected. And as I have kept the original volume, I will revert the system back to Version 6 using GRUB in less than a minute. And the best: I will do all this remotely on a real headless system – even though I will reboot the server three or four times from different volumes, I will not enter the bios a single time. I have no screen or keyboard attached to that system. I’ll do everything from Linux over ssh. Hooray for command-line!
(Things to consider before an upgrade )
When you have a server and you need to upgrade it then you want to take care of a couple of things. If you are the only user of the server and of the services that it provides, then you don’t really care if things go wrong – after all, you are the only person that is affected. On the other side – if there are other people using that server then you want to take two basic things into consideration. One is downtime (in other words the time that the services will be offline while you do the upgrade) and the other one is the way back – the Plan B if things go wrong. Today we will not talk too much about downtime. With Proxmox you could have a second server or – even better – a cluster of three or more nodes and migrate the containers and VMs over to another node before you do the upgrade in order to minimize down time. I really want to focus on the Plan B scenario today.
Oh – and just want to add one thing at this point. I will be typing a lot of commands in the command line in this video. But please do not focus too much on those commands. I will provide a written blog version of all this on my blog on www.onemarcfifty.com. So if you want to go through this in detail later or copy paste commands then you can do it from there. The link to my blog is in the description of the video.
(Step 1 : Clean up of the old system)
Here’s the LVM disk layout of my system again. This is actually what Proxmox installs as a default if you only have one disk. It’s got a fast SSD that has three partitions. One for non-EFI boot (that’s the old school Master boot record or MBR boot for 32 bit systems), one for 64 bit EFI boot and one with the real data on it. We will only look into UEFI today. I think MBR is gone – thankfully. This third partition is the only disk in a Volume Group called pve that has basically the following Volumes and pools on it. There is my Linux root volume, there is a small swap volume and then there is a thin provisioned pool called data that in turn contains all the volumes of the containers and VMs that I am running on that system. The root volume is roughly 100 GB in size. However – once I cleaned it up – that means – once I removed all the downloads, ISO images etc. from it – it turned out that it really only contained 4 to 5 GB of real Linux system data. And that actually led me to this crazy idea to use that Linux swap volume here as a kind of pivot installation. We’ll see in a minute what I mean by that.
Before I started of course I stopped all containers and VMs running on that Proxmox node. I should also have stopped and disabled a couple of services. But more on this later. Now lets see how we can turn a swap volume into a bootable Linux.
(Step 2 – Create a temporary boot installation on the Swap Volume)
A swap volume or partition can be switched on or off using the swapon and swapoff command in linux. Swapon -v shows the currently used volumes and swapoff switches them off. The system is now not using them any more. I can now just convert it into a “normal” Linux ext4 file system using the mke2fs command – That creates an ext2, ext3 or ext4 file system inside the volume. I can now mount this new empty file system using the mount command. Just need to create a directory somewhere and then I can mount it there. So my plan is to boot from this. Therefore I need a Linux on it. What I want to do is just take a copy of the currently running Linux system and then boot from that copy by modifying the settings for GRUB. In order to take a copy I am using the rsync command. This copies over all the files, directories, links and so on from one volume to another. Just – if we do that we need to exclude a couple of things – the so called pseudo file systems. Those are file systems that only exist during runtime and are not physically present on the disc. Such as sys, proc, run and a couple of others. I got some errors during the copy process. Basically I don’t care too much because I will only use this volume temporarily. Nevertheless I figured out that the files which it couldn’t copy were related to LXC so I stopped all LXC related services and launched the rsync again. I should have taken the firewall down as well because that had some files in use as well.
On that note – taking a good copy of a running system is not that trivial. Basically you need to handle two things here. Files that are locked by processes and pseudo file systems that are mounted by processes. You can see the running processes by typing systemctl status and you can list open files by using the lsof command. In order to make this copy as good as possible I should have stopped all services on the machine first. But again – I am not planning to run Proxmox from that volume. I just need it temporarily.
(Step 3 – instruct the BIOS to EFI boot GRUB from the new system)
Cool. So now I have a kind of OK copy which I am quite confident that I can boot from. But how can I tell the BIOS to actually boot from that former swap volume and not from the pve root volume? Let’s have a look at how UEFI Bioses boot Linux. The EFI BIOS would first check the disks for an EFI partition. That’s actually a rather small partition, typically 100 or 200 MB in size that contains a FAT16 file system – the old MS DOS style file system, right?. Inside that file System we have a folder called EFI. Inside that folder typically you have subdirectories for each Operating system or Linux distribution or boot loader manager that is installed on the system, so you might have Windows, Ubuntu, Debian, Proxmox, Clover, Apple and so on. And in each of these subdirectories there are x64 efi binaries which the bios then executes. You can do that from an EFI shell as well. Those binaries then load necessary drivers in order to be able to access an NTFS or HPFS+ or ext4 file system, depending on the OS and then try to boot the OS from that volume. But which one should it boot ? Apple, Ubuntu or Windows ? In the bios typically you can chose one to be the default boot loader. We will do this from Linux without being in front of the machine and hitting DEL or F2 or whatever at boot time. Here is how.
Under a Linux that has been booted using EFI, the EFI boot partition is mounted in /boot/efi. In my case that points to /dev/sdb2. Here in the EFI subdirectory we can find the binaries which the UEFI Bios will call in order to boot the system. That could be a Windows Boot Manager or Clover or in my case Grub. Let me take a copy of it into a separate subdirectory. Now we use the efibootmgr command to visualize the BIOS boot entries. And – we can also use it to add a new entry and to change the boot order. In my case I add the copy of the grubx64.efi which I have just taken as a new entry which I labeled pve6. My boot disk is /dev/sdb and the EFI partition is partition number 2. By default the newly added entry boots first, but I just want to keep that as a fall back solution and actually modify the original one which should then point to the former swap partition. I do this by launching the efibootmgr with the -o option. One reason why you can find so many reports on the internet of Linux installations going FUBAR after they have been moved from one disk or system to another is that the location of the partition where grub will look for the next step in the boot process is actually stored inside that grubx64.efi file. The next step in fact for the grub boot loader would be to load the config file that tells it which menu options to show – that’s the boot manager portion of grub. From there it would then load the initial ram disk where it can actually boot a basic Linux from. In order to change that entry or rather to make grubx64.efi point to another volume we need to use the program grub-install. Unfortunately grub-install needs to be ran from the volume that you want to boot to. In Linux we can use a little trick for that. We can actually change the root of the running installation by doing a chroot. But before we can chroot, we need to mount those pseudo file systems that I have been talking about. At least we need dev, proc, sys and run. We can mount these as so called bind mounts in the new root with the -B option and actually use the ones from the real system. Now I can chroot. I am now in the new root in a new Linux and can now install another version of the Grub loader onto the EFI partition. Just need to mount the efi partition inside the chroot cage before I can do that. As I want to make sure that Grub gets installed as EFI loader, I specify the target=x86_64-efi option here. Now I have taken care of the first two steps – I have a bios entry that loads the right boot manager and that boot manager points to the right volume. Now we need to look after the third step in that boot process – that’s the grub config files and those initramfs images here. That’s done by launching update-grub. So again – we need to do both – grub-install and update-grub. Once grub has mounted one of the initramfs images and loaded the kernel from there, it will then pivot the root to the volume that is specified in the config file and launch the /sbin/init process there. That init or systemd process will then do stuff like start services and mount the file systems and so on. So before we reboot we also need to make sure that the last step after the boot process is OK, that’s actually the /etc/fstab file which tells the new init process which file systems it needs to mount. We also need that to point to the new location.
So – to sum up – there are five locations or pointers that you need to change in order to EFI boot to a new partition or volume. The bios entry, the grub efi boot loader, then the grub boot manager config files, the initramfs images and last but not least the fstab file. Cool – now we should be all set to reboot the system into the new Linux on the former swap partition. Let’s reboot.
(Why I did it the way I did)
At this point – you might wonder why I did not simply boot from a Linux CD like Knoppix or the like. There’s actually four reasons for that. First – this machine has a quite complex network configuration with tagged VLANs. If I booted from a CD, then I would have had to spend a lot of time to get network connectivity back. Second – I had to chroot in order to reinstall GRUB. Doing a chroot is quite simple if the chroot Linux is similar or identical to the base Linux that you are running but can be a challenge if the versions of glibc or libc etc. are different. So using the same Linux makes this much easier. Third – as Proxmox installs everything into LVM volumes by default, I was sure that if I used the same Linux for emergency boot then I would have all the volumes readily mounted. But the decisive argument was the fourth reason. My Proxmox Server is headless. That means there is no screen or keyboard. Plus it’s in the basement. Somewhere between the ironing board and the washing machine. I would therefore have had to connect a Screen and Keyboard first and stand in front of the server. Not very nice. And guys – even though I am showing everything with LVM – it would work exactly the same with disk partitions.
(Step 4 – Take an offline copy and shrink the original Volume)
The system has now rebooted into the new volume which I can actually verify by typing mount without any arguments and here I see that the swap volume is my current root. The next thing that I want to do is take an offline copy of the original Linux. The copy which I had made was just good enough to boot from it – it is not consistent in order to run Proxmox. So what I do is that I mount the old Proxmox 6 in /mnt/pveroot and rsync the content into a subdirectory on a second disk. I do not need to exclude the pseudo file systems because they are not mounted as that Linux system is not running. Why do I take a copy at all? Because the next step is going to be risky. I will now shrink the file system inside the original volume and then shrink the volume itself. I have shrinked the file system to 40 GB and I am adding a bit of safety margin to the volume by shrinking it to 41 GB. Just mounting it quickly in order to have a quick check if everything looks OK. So far so good.
(Step 5 – Create a new Volume and boot from it)
Now I have the free space to create a second volume which I call root2. I do this with lvcreate – the name is root2, the size is 41 GB and the volume group I create it in is pve. If I check with lvs then I can see both logical volumes. The o here just indicates that the original root is open. So let me unmount it and here we go – the o goes away. Now I do the same like I did with the swap partition in the beginning. Create a file system with mke2fs, mount it, rsync the files from the original root over. Take a copy of the efi boot loader, label the original one “swap” and create an efi boot entry for it. Change the boot order. Chroot into the new volume, grub-install, update-grub and modify /etc/fstab. Unmount proc,sys, run and reboot.
(Step 6 – Run the upgrade)
At this point I have completed all preparation steps and I can now run the Proxmox upgrade in this partition. Proxmox comes with a program called pve6to7 that does a lot of checks if your environment is ready for the upgrade. All I have to do then is apt update, apt dist-upgrade, replace all occurences of buster with bullseye in the apt sources and do the dist-upgrade again. Reboot.
(Step 7 – fail back to the old volume)
Let me check the Proxmox Interface – and yes, here it shows that I am now on version 7. I told you in the beginning that I had to revert back – but why ? The reason is that one of the machines that I am running on this Proxmox actually has the GPU passed through. And that pass through did not work as expected in Proxmox 7 at the time. If you search on the internet then you’ll find a lot of info about things working on some kernel versions and not on others. So I decided that I would not pursue this for the time being but rather revert back to version 6. And this is where the beauty of all this happens. I have spent a lot of time preparing this upgrade and the plan B, the fall back plan and so on. But as I have done this before, it now only takes me a minute to revert back. All I have to do is change the boot order in the efi bios with efibootmgr and reboot the machine. Bam. I am back on 6. And as I now have the second volume ready, I can retry this at a later point without the stress of going through all this. I just have to change the boot order back in order to check back if things have been fixed in Proxmox 7 or rather the underlying Kernel.
(Step 8 – clean up)
Last step of course- I should now turn the swap volume back into being swap and also fix the fstab files. This can be done using the mkswap command which will overwrite the ext4 file system on the volume.
Cool – that’s all I wanted to show you guys today. Please do leave me a comment on YouTube if you liked this episode and if it was useful for you. A like on YouTube is always appreciated of course. Having said that – many thanks for watching. Stay safe, stay healthy, bye for now.
</details>Marc AhlgrimProxmox Upgrade