Retiring our Gentoo platform – Sundown until September 2018

Over the last years we have moved our managed service offerings from a Gentoo-based Linux system over to a distribution called NixOS.

Since almost two years this has been the platform of choice for new projects and even within existing projects we started to add NixOS VMs where possible. We have also migrated some projects or moved them partially to NixOS where newer components were required.

Today, it’s time to start saying goodbye to our old Gentoo platform. Of course, we won’t leave anyone behind who is still using Gentoo-based VMs. Here’s our schedule for the next months and it’s impact for customers using the Gentoo platform:

Phase Dates Impact for Gentoo VMs
Announced  Immediately
  • No further feature development
  • No major updates
Sundown period May 2018
to
August 2018
  • No new VMs
  • Security updates only
  • Migration to NixOS VMs depending on individual agreements
Grace period September­ 2018
  • No further security updates
  • Remaining VMs will stay online.
End of Life September 2019
  • Remaining Gentoo VMs will be shut down.

Note: Customers already using the NixOS platform will not be affected by this.

I’m still using Gentoo-based VMs. What do I do now?

If you’re a customer with a support contract in the “Guided” or “Managed” service classes  then we’ll approach you directly and discuss how to move your remaining Gentoo VMs to NixOS.

If you’re a customer in the “Hosted” service class then we recommend you contacting our support team to discuss setting up new VMs and migrating your services over. We’ll help you with any information and coordination that you might need, but you’ll be responsible to migrate your data and services to new machines.

And lastly, rest assured that we won’t shut off any remaining Gentoo VMs for at least another 18 months. However, as the old platform will not receive further updates and as there will be a hard limit in September 2019, we advise you to take the time and move to the new platform as early as possible.

How do I know which VMs are still using Gentoo?

You can look at the VMs of your projects on my.flyingcircus.io. Select a project (“More details”) and then choose “Manage” on the box titled “Virtual Machines”. You’ll see a listing like the one in this screenshot. The VMs have different labels. If a VM has a label “Puppet” then it is still running on Gentoo. If the VM has a label “NixOS” then it is already running on the NixOS platform.

Screen Shot 2018-03-15 at 13.03.33

Why are we moving to NixOS?

A big part of our service is that we want to have as few “breaking” updates as possible – after all, we want to deliver small and continuous updates. When we started out with out Gentoo-based platform more than 10 years ago, we envisioned that we would profit from Gentoo’s rolling nature.

However, with the rising complexity, Gentoo has shown conceptual issues that has hindered us to efficiently manage the balance between stability and progress.

NixOS has been around for a while but wasn’t ready until around 2015 when we started to investigate alternatives to Gentoo. Since then we’ve been achieving great improvements to our service that would be impossible on our old platform. Due to that, we decided that it’s time to make the transition for everyone.

Aside from the larger motivation, there are also a number of direct benefits for you when moving to our NixOS based platform:

  • VMs now run a 64-bit kernel which provide better performance for many languages (Python, Java, …) and allow larger RAM allocations to be used effectively.
  • Service users can install custom (Nix) packages and versions without requiring pre-defined roles from our platform and still have them monitored within our security update tools.
  • Improved logging (Graylog), monitoring (Sensu), and telemetry (Telegraf/Prometheus/Grafana) services that have a higher flexibility and allow more direct interaction without needing our personal assistance. (Even though we’re always happy to help!)
  • Overall a newer set of versions for many components like nginx (HTTP 2!), MySQL, PostgreSQL, Python, PHP, …
  • A better release process that is much much more robust and more flexible to provide you with early releases of customizations.
  • Faster installation of changes, updates, rollback capability, and local versioning of all configuration.

If you’d like to know more about NixOS and its benefits, we recommend talking to us or visiting the NixOS homepage. Similar to the effect that Gentoo has been a comparatively “exotic” Linux distribution, we know that NixOS may look even more so. However, our documentation has been extended with a NixOS-specific area that will help you discover the relevant parts for you to interact with. On every other account: it’s a Linux environment that will run your applications well and we hope that you’ll enjoy that platform that we’ve built using it.

If you have questions …

As always: if you have any questions or comments then let us know and send us an email to support@flyingcircus.io and we will follow up quickly.

Greetings from a new employee

DSC_0084_cs

Hello, My name is Christian Schmidt, the new Application Operations Engineer at Flying Circus.

Since about two weeks I augment the crew and dive into a very interesting environment. While getting in touch I currently migrate our Jenkins from Gentoo to NixOS platform.

I am already feeling at home here and the staff here has been great with giving me all of the necessary information to perform the necessary tasks.

On a personal level I’m 31 years old and, since I moved to Halle (Saale) in 2011, I compaign at the local hacker and makerspace Eigenbaukombinat, organizing workshops and events to provide knowledge of especially privacy, coding, lockpicking, linux and many more.

I look forward to working with all of you and providing the quality service and support that Flying Circus is known for providing. See you on the web!

CPU hardware security issues: Meltdown and Spectre (updated 2018-03-08)

Researchers have published serious and widespread security issues relevant for all users of Intel (and other) CPUs for all products from the last decade. The bugs are known as “Meltdown” and “Spectre”. Both bugs have massive implications for the security of all applications both within an operating system as well as on hosted virtualised platforms like Amazon AWS, Google Compute Engine or the Flying Circus.

The security issues were intended to be under an embargo for another week but a couple of news outlets have already started reporting about them and forced the security researchers to publish the issues earlier than intended.

We’re watching the in-progress security patches as they arrive and will take appropriate measures. We’ll update our customers with more specific information over time but want you to know that we are aware of the issue and its implications.

Update Monday 2018-01-08

There is still progress happening and the most relevant security issue (Spectre, Variation 2, CVE 2017-5715) has no patch available yet. Some vendors and distributions are providing undocumented (and not publicly tested) patches that we are refraining from rolling out into our infrastructure. We’re in contact with Qemu and Linux kernel developers who are still working on reliable patches on both levels. We’ll keep you updated.

Update Monday, 2018-01-29

The situation remains complex. We have identified a small Linux kernel change that will ensure proper KVM/Qemu guest/host isolation. However, there are a number of other patches that keep finding their wait into this part of the Linux kernel code and Intel is still communicating very unclear messages and has retracted (some or all) µCode updates for their CPUs last week as well as other Vendors like Ubuntu, VMware, etc. Intel has announced another update for 2018-01-31 which we will review and consider after waiting for industry feedback on the performance and stability.

We are then planning to roll out an update Linux kernel on VM hosts and will likely enable the additional countermeasures (like KPTI) on the hosts. To validate that this does not have drastic performance impacts we are reviewing our baseline system and application performance using the Phoronix Test Suite.

More updates will follow here as the situation develops.

Update Thursday, 2018-03-08

A few weeks ago we have reviewed the status of the vanilla kernel fixes Spectre and Meltdown and have decided to update our Gentoo-based Linux 4.9 series hardware and virtual machines with the recent 4.9.85 update. The upstream developers have implemented sufficient mechanics at this point to selectively enable mitigations depending on hardware support and balance performance versus security. We started to roll out those updates in yesterday’s release and VMs and servers will perform required reboots within regular maintenance windows over the next days. Our host servers will enable mitigations for all variants (Meltdown and Spectre 1 and 2). Our guest systems will enable mitigations against Spectre 1 and 2 but not Meltdown, due to missing support for PCID and thus avoid KPTI which would have a big performance impact.

Due to the extent of mitigations available in the Linux kernel and a rough history of stability of Intel’s patches  we also decided to not apply µCode updates for the Intel CPUs at this point in time.

Support during Christmas and New Year’s 2017/2018

It’s that time again: Christmas is around the corner.

To ensure that all your applications in the Flying Circus are running smoothly we will monitor all regular support during business hours* and emergency support as usual. We won’t be performing non-critical work in this time and catch up with any backlog early in January 2018.

Here’s an overview of the next days and our support availability. The highlighted days are national or local holidays and are only covered for SLA customers:

  • 2017-12-20 (Wednesday): regular support
  • 2017-12-21 (Thursday): regular support
  • 2017-12-22 (Friday): regular support
  • 2017-12-23 (Saturday): SLA-covered emergency support only
  • 2017-12-24 (Sunday): SLA-covered emergency support only
  • 2017-12-25 (Monday): SLA-covered emergency support only
  • 2017-12-26 (Tuesday): SLA-covered emergency support only
  • 2017-12-27 (Wednesday): regular support
  • 2017-12-28 (Thursday): regular support
  • 2017-12-29 (Friday): regular support
  • 2017-12-30 (Saturday): SLA-covered emergency support only
  • 2017-12-31 (Sunday): SLA-covered emergency support only
  • 2018-01-01 (Monday): SLA-covered emergency support only
  • 2018-01-02 (Tuesday): regular support
  • 2018-01-03 (Wednesday): regular support
  • 2018-01-04 (Thursday): regular support
  • 2018-01-05 (Friday): regular support

Happy holidays to everybody and see you in 2018!

  • business hours = Mo-Fr, 8-16 CE(S)T