All posts by Alexander Bittner

About Alexander Bittner

I'm currently working as a systems administrator at http://flyingcircus.io/

DevOpsDays 2015 Berlin – good principles start gaining ground

devopsdays_berlin_2015

It was a joy to be at DevOpsDays 2015 in Berlin. The community did a great job in organizing the conference. The location was nice and the evening event, taking place in a beautiful old Wilhelminian style flat, provided a good atmosphere for enlightening discussions.

I was impressed by the program as well as the mindset of the audience because it was different from what you might expect from a typical conference where technology is the focus. Continue reading DevOpsDays 2015 Berlin – good principles start gaining ground

How we protect against Poodlebleed (CVE-2014-3566)

Operations is driven by chaos. Yet we have another brilliant example: Poodlebleed. This is how we at the Flying Circus face it.

Yesterday, an exploit to a severe security issue in SSLv3 has been released. This exploit is known as Poodlebleed. SSLv3 is involved in encrypting communication over networks, e.g. when securing websites with HTTPS or encrypting email traffic using IMAPs/POP3s and STMPs. More information about the exploit can be read here and downloaded here (more details).

The bug

The exploit allows an attacker to expose data over time that is encrypted between SSLv3 compatible servers and clients. SSLv3 is quite old but still widely supported by nearly any software. Although using the exploit is not trivial and stealing information takes some time, we take Poodlebleed very serious.

We have verified that our managed components do not support SSLv3 by default. Further, we have scanned our whole platform for customer installations that have overridden these defaults and informed customers accordingly. Right now, our systems are reasonably safe again.

In the following, we will explain in more detail which measures have been taken and what our customers and others can do to come up against Poodlebleed in their own installations.

The fix in brief

Because SSLv3 is a general protocol, the issue can’t be fixed by updating a specific software to a non-vulnerable version. The approach is to avoid the usage of SSLv3 and vulnerable ciphers in any software that supports them.

HTTPS via nginx and Apache2

Our managed nginx configuration was not vulnerable since SSLv3 is forbidden as a valid protocol by default. Customers that have overridden this default have been informed.

Some of our managed Apache2 installations were vulnerable and have been fixed by disabling SSLv3 and limiting the available cipher suites (Thanks Hynek!) in every Virtual Host configuration block that has SSL enabled:

<VirtualHost _default_:443> 
... 
SSLProtocol ALL -SSLv2 -SSLv3 
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
... 
</VirtualHost>

Vulnerable Apache2 customer configurations have not been found.

In the future, we will provide configuration snippets for both, nginx and Apache2 that can easily be included in own configurations. It will contain a safe set of SSL configuration options which we will maintain. This way, customers do have to worry less about safe defaults.

SMTPs via Postfix

We have decided not to disable SSLv3-Support on our central mail server. In our opinion, mail servers (not clients!) have to be treated differently because emails need to reach their recipients, even if the next mail server we pass them on to does not apply the same safety measures as we do.

Customers who want to disable SSL support in their installations can find further information here.

POP3s/IMAPs via Cyrus and dovecot

We have disabled SSLv3 support in our central IMAP and POP3 services (Cyrus) by limiting available ciphers (Thanks again Hynek!) in imapd.conf to:

tls_cipher_list: ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

However, tools judge differently when determining whether we are still vulnerable. We are in contact with Hynek to further track this down.

We do not provide managed Cyrus installations and therefore no customer action is necessary (unless you have a self-compiled Cyrus).

For our managed dovecot installations we have also limited available ciphers in /etc/dovecot/conf.d/10-ssl.conf like we did for Cyrus:

ssl_protocols = !SSLv2 !SSLv3
ssl_cipher_list = ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

LDAPs via OpenLDAP

We have limited the cipher suites that our managed OpenLDAP installations allow in /etc/openldap/slapd.10tls.conf:

TLSCiphersuite SECURE256:-VERS-SSL3.0:+VERS-TLS1.2:+VERS-TLS1.0:+VERS-TLS1.1

Customer configurations have already been adjusted by hand and will be permanently fixed upon the next Flying Circus release which we expect in the next days.

NRPE

In NRPE, which we heavily use to monitor our infrastructure, SSL configuration options unfortunately can’t be changed and even worse, supporting SSL is the fixed default. Luckily, SSLv3 has been fixed-disabled, too. Our NRPE installations are therefore not vulnerable.

MySQL

Our managed MySQL-Server installation has never been vulnerable because it is not configured to support OpenSSL by default. This can easily be checked by:

litdev81 ~ # mysql
mysql> SHOW VARIABLES LIKE 'have_openssl';
| Variable_name | Value |
| have_openssl | DISABLED |
1 row in set (0.02 sec)
mysql>

PostgreSQL

Our PostrgreSQL installations are basically not vulnerable since they are not listening on public interfaces. However, customers should check whether they have changed the default listening ports and addresses.

Nevertheless, we will try to disable SSLv3 support in our managed PostgreSQL by default. Tomorrow is another day…

Aftermath and future improvements

In the next days we will roll out a Flying Circus release which contains permanent fixes for adjustments that have for now been made by hand.

Further, we have dependency issues on our platform which prohibit us from upgrading openssl to version 1.0.1 on our current Gentoo snapshot. This will change when rolling out our yearly OS update in the next weeks. openssl-1.0.1 brings support for TLSv1.2. At the moment we run on openssl-1.0.0m which only supports earlier versions of TLS that are no longer recommended. Having TLSv1.2 on board will allow us to disable SSL support completely (in favor of TLS) in a lot of our managed components.

Providing a safe set of options for a piece of software in a snippet file that can easily be included in own configurations is a good idea in general and we will provide this w.r.t. SSL-related options for nginx and Apache2. We will also evaluate how to bring this to other components.

Language support: Python, Ruby, and PHP versions after the OS update

In the near future, we are going to roll out our yearly OS update. It will bring a lot of software packages to a recent version – and discontinue older versions. Today, we will give you an overview of how this applies to our platform support for Python, Ruby, and PHP.

Python

The newest and greatest Python stuff will be there: we will add platform support for Python 3.3 and Python 3.4. We have also tuned the versions of related packages like virtualenv, pip, and setuptools so that everything will play nicely together.

We will discontinue the platform support Python 2.4, Python 2.5, and Python 3.1. Supported versions will be: Python 2.6, Python 2.7 (default Python 2), Python 3.2, Python 3.3 (default Python 3), and Python 3.4.

Version 2.4 has been discontinued upstream since 2008 and does not receive security updates any longer. Running it as platform package seems not to be a good idea anymore. However, since Python 2.4 is still involved in a few applications running Plone 3.x, we encourage those users to either go for a newer Plone version or switch to a self-compiled Python 2.4.

We see version 2.5 being very rarely used on our platform and, due to the fact that we still have 2.6 and 2.7 on board, we will drop platform support for Python 2.5. Further, upon its last release in 2011 it has already been said that no security issues will be fixed in 2.5 any longer. We encourage users that still rely on this version to upgrade or use a self-compiled Python 2.5.

Further, we will discontinue platform support for Python 3.1 as we provide versions 3.2, 3.3, and 3.4. Users of Python 3.1 may use a self-compiled version of Python 3.1 or migrate their projects to use a newer version of Python.

Ruby

We have added Ruby 2.0 to our platform. We will discontinue platform support for Ruby 1.8.7. Supported versions will be Ruby 1.9 (default) and Ruby 2.0.

Support for Ruby 1.8 has been discontinued upstream, which is why we will not provide it on platform level any longer.

PHP

We will discontinue the platform support for PHP 5.4. The only version supported will be PHP 5.5.

Users of PHP 5.4 may have to adjust their application code. According to the PHP core developers, however, migrating projects from PHP 5.4 to PHP 5.5 should not be too much work since there are only a handful of backward incompatible changes in PHP 5.5.