Holiday Notice: October 2018

This really was a hot summer in the city. Now its gone, time flies so fast.
Here in Germany we have some public holidays coming up in October.
Our office is closed on:

Wednesday October 3rd – German Unification Day
Wednesday October 31st – Day of Reformation

To all projects covered by an SLA we provide emergency support as usual.

Have a wonderful autumn everybody.

We have moved into our new office

As previously announced, the Flying Circus has a new office address since September 1st. If you’d like to write us a letter or visit us, please be sure to use our new address:

Flying Circus Internet Operations GmbH
Leipziger Str. 70/71
06108 Halle (Saale)
Germany

Map excerpt showing new office location

It’s only a few blocks away from our old office. Why the move? We’re teaming up with other companies from Halle to form SaltLabs, a loosely coupled collective of IT-related companies under one roof. Details on this will follow, so stay tuned!

2018-08-30 – Umzug unseres Rechenzentrums-Labors / Moving our data center laboratory

Unser Büro ist letzte Woche wie geplant umgezogen und wir sind jetzt in der Leipziger Str. 70/71 beheimatet.

In der nächsten Woche werden wir unser Rechenzentrums-Labor ebenfalls aus der Forsterstraße in das neue Gebäude umziehen. Unser Labor besteht aus zwei Racks die unser produktives Rechenzentrum einmal zur Entwicklung und zur Qualitätssicherung detailliert nachbilden. Dort betreiben wir auch einige sekundäre Dienste (z.B. Build-Server, unseren Chat oder ein paar ausgewählte Monitoring-Skripte) durch deren Downtime der Produktiv-Betrieb nicht gestört wird.

Beim Umzug werden wir im Laufe des Donnerstags (30.08.2018) das Labor vollständig offline nehmen, umziehen und im Laufe des Nachmittags wieder in Betrieb nehmen.

Die Auswirkungen während der Umzugsphase werden voraussichtlich sein:

  • Deployments über unseren zentralen Build-Server (builds.flyingcircus.io) nicht möglich.
  • Konfigurationsänderungen von NixOS-VMs werden möglich sein, können jedoch beim Anfordern neuer Pakete mit erhöhtem Compiler-Aufwand verbunden sein.
  • Unser Support wird per E-Mail und Telefon aber nicht per Chat erreichbar sein.
  • Unser Support wird keinen Zugriff auf die zentrale Telemetrie haben und es werden für diesen Zeitraum auch keine zentralen Telemetrie-Daten erfasst.
  • Einzelne “Frühwarn-Checks” auf Infrastruktur-Ebene und die im Kundenportal definierten HTTP-Checks werden nicht verfügbar sein. Tests zur externen Verfügbarkeit der Anwendungen, sowie VM-spezifische Checks und die Statuspages werden wie gewohnt verfügbar sein.

Bei Fragen sind wir wie gewohnt über alle Kanäle ansprechbar.


 

Last week we moved into our new office building and are now located in the Leipziger Str. 70/71 in Halle (Saale).

Next week we’ll also move our data center laboratory to the new premises. The lab consists of two racks that fully model our productive data center environment for development and quality assurance. The lab also runs a couple of secondary services  (e.g. CI/CD servers, our chat, a couple of selected monitoring scripts, …) whose absence will not have an impact on the productive data center applications.

During the move on Thursday (2018-08-30) we’ll shut down the lab during the morning and rebuild and power everything up in the new location during the afternoon.

The projected impact of the secondary services during the move will be:

  • Deployments through our central build server (builds.flyingcircus.io) won’t be possible.
  • Configuration changes to NixOS VMs will be possible but may require substantial local package compilation.
  • Our support will be available by email and phone but not per chat.
  • Our support will not have access to central telemetry and our central telemetry will not record data for this period.
  • A couple of “canary checks” on the infrastructure level will be offline and HTTP checks defined through our customer portal will not be executed. External availability checks, regular VM checks and the status pages will be available as usual.

If you have any questions don’t hesitate to contact us through the usual channels.

Auf zu neuen Ufern

Ab 1. September hat der Flying Circus seine Heimatbasis im Areal der Saltlabs. Ihr findet uns in der Leipziger Str. 70/71, auch bekannt als Oberer Boulevard von Halle (Saale). Gemeinsam mit unseren Freunden von gocept und weiteren Unternehmen, die auch alle „irgendwas mit IT“ machen, freuen wir uns darauf mit den Saltlabs die IT-Welt in Halle öffentlicher und mit gemeinsamen Veranstaltungen auch spannender zu machen.

Noch wird in den neuen Räumlichkeiten kräftig Staub aufgewirbelt – an allen Ecken arbeiten die Handwerker mit Hochdruck. Und bevor es soweit ist, heißt es auch für uns in den nächsten Tagen erst mal Dinge sichten, bewerten und alles wichtige und relevante sicher in den Umzugskisten verstauen.

Das Team vom Support ist während des Büro-Umzugs am Donnerstag den 16. August 2018 ohne Unterbrechung über die bekannten Kanäle (E-Mail, Chat, SLA-Support-Telefon) für euch erreichbar. Ende August werden wir noch unsere Labor-Server in die neuen Gebäude umziehen und dafür eine eigene Ankündigung schreiben.

2018-05-29 – 2018-05-31: Major data center upgrade

TL;DR We are consolidating our hardware and racks in the data center and will perform a longer period of maintenance over multiple days. We have prepared thoroughly for the migration to avoid any downtimes and will use this opportunity to further improve our network.

Over the last years our data center setup has grown from a few machines in a single rack to three racks that are completely filled up with servers and additional customer-specific racks in our vicinity.

One of our basic tenets has always been to grow organically to avoid unnecessary waste. Now we have reached the limit of renting individual racks and our next organic step is to move to a separate row (and room!) of multiple consecutive racks. This gives us and you enough room to grow in the future while maintaining tight control over our network structure. It also gives us the chance to nicely clean up some smaller annoyances have accumulated over the years.

As this maintenance requires us to move all of our machines, we are leveraging this opportunity by having reviewed and improved all our technology layers:

  • We are introducing a redundant spine/core into our switching setup and are upgrading to 40G on all backbone connections.
  • Simplify our network infrastructure by reducing it to single-vendor components.
  • Our routers get upgraded with 10 Gbit/s on internal and external interfaces.
  • Our DNS is now more reliable by running it on the routers and having it included in automatic failover.
  • We improved our VM migration code to better support large migration tasks like moving whole racks around.
  • Our overall resource usage has around 40% or more free capacity on all of CPU, RAM and storage.
  • We are keeping a set of SSDs and HDDs on hand just in the case that disks should experience failures after turning the servers off. of them should experience issues when turning them back on.
  • Virtualisation hosts that have not yet been upgraded to 10G storage interfaces will be upgraded at that time.

The maintenance itself will be performed during regular business and evening hours as all involved components are fully redundant and have been tested recently. We will perform all steps slowly and carefully, leaving enough capacity and time to verify individual steps to reduce the chance for critical mishaps.

Nevertheless, our back office personnel will be monitoring the situation closely and will be able to respond to any issues immediately.

If you have any questions or feedback – let us know through your usual contact channels or by email to support@flyingcircus.io.

Cover photo by Tristan Schmurr, © 2012 CC-BY-2.0

Support during Ascension and Pentecost 2018

In Germany we have a few further public holidays coming up next time resulting in shorter-than-usual work weeks. Here’s an overview of our support availability. The highlighted days are national holidays and only covered for SLA customers:

  • 2018-05-07 (Monday): regular support
  • 2018-05-08 (Tuesday): regular support
  • 2018-05-09 (Wednesday): regular support
  • 2018-05-10 (Thursday): Ascension: SLA-covered emergency support only
  • 2018-05-11 (Friday): regular support
  • 2018-05-12 (Saturday): SLA-covered emergency support only
  • 2018-05-13 (Sunday): SLA-covered emergency support only
  • 2018-05-14 (Monday): regular support
  • 2018-05-15 (Tuesday): regular support
  • 2018-05-16 (Wednesday): regular support
  • 2018-06-17 (Thursday): regular support
  • 2018-06-18 (Friday): regular support
  • 2018-06-19 (Saturday): SLA-covered emergency support only
  • 2018-06-20 (Sunday): SLA-covered emergency support only
  • 2018-05-21 (Monday) Pentecost: SLA-covered emergency support only
  • 2018-05-22 (Tuesday): regular support

To ensure that all your applications in the Flying Circus are running smoothly we will monitor all regular support during business hours: Monday to Friday , 8-16 CE(S)T and SLA-covered emergency support as usual.

Have a good time, a safely flight – and May the 4th be with you!