www/ Infrastructure

freedesktop.org infrastructure

Our infrastructure falls into two categories.

GitLab

gitlab.fd.o is hosted on the Google Cloud Platform within us-east1, run on Kubernetes with Helm charts. The Helm charts and the Helm configuration used to deploy are both publicly available.

Repositories and the PostgreSQL database are both stored on persistent disks mapped into the Kubernetes service pods.

Docker Registry images, file uploads and attachments, and backups, are all stored within Google Cloud Storage buckets. Backups are retained for one week.

There are several shared runners available for all GitLab CI jobs:

  • fdo-packet-m1xl-* execute 8 concurrent jobs, and are 24 vCPU, 256GB RAM workers hosted in Equinix Metal's EWR1, with 2.8TB of SSD storage, sponsored by Equinix Metal
  • gst-gitlab-htz-runner* execute 6 concurrent jobs, and are a 48 vCPU, 128GB RAM workers hosted in Hetzner Germany, with 1.8TB of SSD storage, sponsored by the GStreamer Foundation
  • AArch64: fdo-packet-arm-* execute 4 concurrent jobs, and are 32 vCPU, 128GB RAM workers hosted in Equinix Metal SJC1, with 480GB of SSD storage, under the aarch64 tag, sponsored by Equinix Metal
  • macOS: gst-mac-catalina* are macOS Catalina workers, under the gst-macos-10.15 tag, sponsored by the GStreamer Foundation
  • Windows: gst-gitlab-htz-runner4-windows-1809 executes 6 concurrent jobs, and is a 24 vCPU, 128GB RAM worker hosted in Hetzner Germany, with 256GB of SSD storage, under the windows and 1809 tags, sponsored by the GStreamer Foundation

Given the level of concurrency, you should aim for each job to parallelise with 4 concurrent processes.

It is possible to add runners for your own projects, however this has significant caveats. Most obviously, your runners should be faster than the shared runners, unless your goal is just to remove load from the shared runners. Speed of storage is quite important here, and you should also have a fast network to Google us-east1, especially if your pipeline has multiple stages, as the build contents are saved, uploaded, and re-downloaded, between each stage. If you need to run Docker-in-Docker (e.g. to build Docker images to execute in), your runner must have the privileged flag set, which means you must consider the host machine to be completely compromised as the root user. Using tools such as buildah, kaniko or img may help you avoid the need for a privileged runner.

If you would like to donate execution time for shared runners, please file an issue or contact the admins and we can discuss this. Our main considerations when accepting new runners are performance (should be similar to existing runners), reliability (should not unduly cause failures, or huge spikes in build times due to external load), and security.

How Git itself is served is in the Git page.

Everything else

All the other services are run on a small fleet of machines hosted at Portland State University. These machines all run Debian. They are all (with the exception of fruit?) virtual machines.

This list is not quite complete.

Primary service hosts:

  • gabe.fd.o (restricted access): serves DNS with Bind for fd.o and X.Org; MX with Postfix for all incoming fd.o/x.org mail and the only outgoing mail relay; runs Apache for web access to Mailman lists; hosts some confidential X.Org Board of Directors information; is still somehow 32-bit
  • kemper.fd.o (restricted access): git.fd.o host for master Git repo storage; users can push with Git but not get a shell; exports read-only NFS mountpoint for git
  • annarchy.fd.o (open access): serves HTTP with Apache for most project-specific pages, the www.fd.o wiki, and people.fd.o; general shell host; much abused for arbitrary user file storage and exchange; serves home directories over NFS
  • molly.fd.o (admin only): serves anongit with git-daemon over NFS mounts from kemper/annarchy; serves cgit via Apache; runs Postfix for incoming bug mail
  • culpepper.fd.o (admin only): serves Bugzilla over Apache
  • emeril.fd.o (restricted access): serves Patchwork (in Python venv) and members.x.org (in Docker container); runs Postfix for incoming Patchwork mail

Virtual machines:

  • lyle.fd.o (amdin only): Ganeti VM cluster control host, Ganeti VM host
  • teodor.fd.o (admin only): Ganeti VM host
  • cornelius.fd.o (admin only): Bacula backup master
  • fruit.fd.o (admin only): runs OpenLDAP and a very ancient fork of Debian's userdir-ldap, used to produce our user account database which is then distributed to all the other machines with rsync
  • patrick.fd.o (admin only): admin VM
  • ray.fd.o (admin only): ?
  • todd.fd.o (admin only): ?

Plans

Most of our planning is discussed in freedesktop/freedesktop issues on GitLab.

Contacts

The admins are reachable through GitLab issues or the sitewranglers list.