All posts by Christian Kauhaus

About Christian Kauhaus

Christian is a systems engineer working with Flying Circus Internet Operations.

Improving Ceph OSD start-up behaviour with vmtouch

We have a love/hate relation ship with Ceph. On one hand, it is probably the best open source distributed storage around. On the other hand, Ceph repeatedly exhibits unexpected behaviour under high load. And it is absolutely correct that you expect Flying Circus VMs to perform evenly. That is something we keep revisiting regularly. In the following article, I will describe an improvement we have applied on a common pain point: I/O hangs during OSD restarts.

Restarting an OSD (Object Storage Daemon) places additional load on its backing disks. Flying Circus business growth led to increasing storage I/O demand. While this is generally a good thing, it brought our main Ceph cluster near its throughput limit for several times. Danger ahead: The storage cluster is running fine as long as nothing special happens. But if something unusual happens, the cluster suddenly goes over the tipping point and performance becomes shaky.

Continue reading Improving Ceph OSD start-up behaviour with vmtouch

VENOM’s little brother is here – another Qemu security upgrade required

A new Qemu vulnerability has been discovered recently. We are going to proactively reboot all VMs during the next days.

Update 2015-08-05: The VM restarts will be performed during maintenance windows according to every customers’ schedule tonight.  We decided to skip the regular lead time due to the importance of this update and to speed up another important update to our storage and backup infrastructure. We are paying close attention to keep your applications and your data safe, especially after the events in recent months. The current and upcoming changes belong to the promised updates, upgrades, and improvements to our infrastructure in response to those outages.

Continue reading VENOM’s little brother is here – another Qemu security upgrade required

Improving periodic data import jobs in 3 steps

Unimposing, less-than-fashionable, often hacked together without passion—yet, these little periodic data import jobs are still ubiquitous in any sizable datacenter. They often provide the glue that make data flow from one system to another. If they break, important stuff may get stuck. It’s time to pay them the attention they deserve. Continue reading Improving periodic data import jobs in 3 steps