CPU hardware security issues: Meltdown and Spectre (updated 2018-03-08)

Researchers have published serious and widespread security issues relevant for all users of Intel (and other) CPUs for all products from the last decade. The bugs are known as “Meltdown” and “Spectre”. Both bugs have massive implications for the security of all applications both within an operating system as well as on hosted virtualised platforms like Amazon AWS, Google Compute Engine or the Flying Circus.

The security issues were intended to be under an embargo for another week but a couple of news outlets have already started reporting about them and forced the security researchers to publish the issues earlier than intended.

We’re watching the in-progress security patches as they arrive and will take appropriate measures. We’ll update our customers with more specific information over time but want you to know that we are aware of the issue and its implications.

Update Monday 2018-01-08

There is still progress happening and the most relevant security issue (Spectre, Variation 2, CVE 2017-5715) has no patch available yet. Some vendors and distributions are providing undocumented (and not publicly tested) patches that we are refraining from rolling out into our infrastructure. We’re in contact with Qemu and Linux kernel developers who are still working on reliable patches on both levels. We’ll keep you updated.

Update Monday, 2018-01-29

The situation remains complex. We have identified a small Linux kernel change that will ensure proper KVM/Qemu guest/host isolation. However, there are a number of other patches that keep finding their wait into this part of the Linux kernel code and Intel is still communicating very unclear messages and has retracted (some or all) µCode updates for their CPUs last week as well as other Vendors like Ubuntu, VMware, etc. Intel has announced another update for 2018-01-31 which we will review and consider after waiting for industry feedback on the performance and stability.

We are then planning to roll out an update Linux kernel on VM hosts and will likely enable the additional countermeasures (like KPTI) on the hosts. To validate that this does not have drastic performance impacts we are reviewing our baseline system and application performance using the Phoronix Test Suite.

More updates will follow here as the situation develops.

Update Thursday, 2018-03-08

A few weeks ago we have reviewed the status of the vanilla kernel fixes Spectre and Meltdown and have decided to update our Gentoo-based Linux 4.9 series hardware and virtual machines with the recent 4.9.85 update. The upstream developers have implemented sufficient mechanics at this point to selectively enable mitigations depending on hardware support and balance performance versus security. We started to roll out those updates in yesterday’s release and VMs and servers will perform required reboots within regular maintenance windows over the next days. Our host servers will enable mitigations for all variants (Meltdown and Spectre 1 and 2). Our guest systems will enable mitigations against Spectre 1 and 2 but not Meltdown, due to missing support for PCID and thus avoid KPTI which would have a big performance impact.

Due to the extent of mitigations available in the Linux kernel and a rough history of stability of Intel’s patches  we also decided to not apply µCode updates for the Intel CPUs at this point in time.

Support during Christmas and New Year’s 2017/2018

It’s that time again: Christmas is around the corner.

To ensure that all your applications in the Flying Circus are running smoothly we will monitor all regular support during business hours* and emergency support as usual. We won’t be performing non-critical work in this time and catch up with any backlog early in January 2018.

Here’s an overview of the next days and our support availability. The highlighted days are national or local holidays and are only covered for SLA customers:

  • 2017-12-20 (Wednesday): regular support
  • 2017-12-21 (Thursday): regular support
  • 2017-12-22 (Friday): regular support
  • 2017-12-23 (Saturday): SLA-covered emergency support only
  • 2017-12-24 (Sunday): SLA-covered emergency support only
  • 2017-12-25 (Monday): SLA-covered emergency support only
  • 2017-12-26 (Tuesday): SLA-covered emergency support only
  • 2017-12-27 (Wednesday): regular support
  • 2017-12-28 (Thursday): regular support
  • 2017-12-29 (Friday): regular support
  • 2017-12-30 (Saturday): SLA-covered emergency support only
  • 2017-12-31 (Sunday): SLA-covered emergency support only
  • 2018-01-01 (Monday): SLA-covered emergency support only
  • 2018-01-02 (Tuesday): regular support
  • 2018-01-03 (Wednesday): regular support
  • 2018-01-04 (Thursday): regular support
  • 2018-01-05 (Friday): regular support

Happy holidays to everybody and see you in 2018!

  • business hours = Mo-Fr, 8-16 CE(S)T

Announcing fc-userscan

NixOS manages dependencies in a very strict way—sometimes too strict? Here at Flying Circus, many users prefer to compile custom applications in home directories. They link them against libraries they have installed before by nix-env. This works well… until something is updated! On the next change anywhere down the dependency chain, libraries get new hashes in the Nix store, the garbage collector removes old versions, and user applications break until recompiled.

In this blog post, I would like to introduce fc-userscan. This little tool scans (home) directories recursively for Nix store references and registers them as per-user roots with the garbage collector. This way, dependencies will be protected even if they cease to be referenced from “official” Nix roots like the current-system profile or a user’s local Nix profile. After registering formerly unmanaged references with fc-userscan, one can fearlessly run updates and garbage collection.

Continue reading Announcing fc-userscan

How to renew Puppet CA and server certificates in place

It used to run fine for years… but now the (deprecated) Puppet infrastructure at the Flying Circus is showing signs of aging. It’s not about server hardware or something like this (fully virtualized anyway) – it’s about SSL certificates of Puppet’s own SSL infrastructure. Time for a face lift.

In the following, I will describe what we did to renew both CA and Puppet server certificates. Despite that this problem should occur on every Puppet server running for a prolonged amount of time, I found remarkably few resources on the net (that did not involve completely replacing the CA) – so I’m going to share our findings.

Continue reading How to renew Puppet CA and server certificates in place

Release 2017_010 with many security updates

During the last weeks we have prepared a larger update for our Gentoo based VMs. It will include many basic libraries as well as added support for Python 3.5 and 3.6. A detailed list of affected packages and changes can be found as usual in our ChangeLog. Please review the list of updated packages for libraries and tools that may have compiled into your applications. While we have tried to avoid link-level compatibility issues, a small chance remains that applications will not start afterwards due to dynamic linkage problems. Recompiling usually  solves this kind a problem.

As the update is a bit bulky, we opted for a staged roll out during the week. Each VM will get an individual scheduled maintenance slot according to the agreed pre-announcement period. Development and staging VMs will already receive updates during this weekend.

Please feel free to contact our support if you need assistance.