Heartbleed

The recently exposed heartbleed bug in the OpenSSL library has surprised everyone with a catastrophic vulnerability in many of the world’s secure systems.

In common with many other SSL-exposed services, some freenode servers were running vulnerable versions of OpenSSL, exposing us to this exploit. Consequently, all of our affected services have been patched to mitigate the vulnerability, and we have also regenerated our private SSL keys and certificates.

In an unrelated event, due to service disruption & the misconfiguration of a single server on our network, an unauthorised user was allowed to use the ‘NickServ’ nickname for a short period Sunday morning. Unfortunately there is a possibility that your client sent data (including your freenode services password) to this unauthorised client. Identification via SASL, certfp or server password were not affected, but any password sent directly to the “NickServ” user might have been.

Because of these two recent issues, we would like to make the following recommendations to all of our users. It would also be good practice to follow them at regular intervals.

  • Though we are not aware of any evidence that we have been targeted, or our private key compromised, this is inevitably a possibility. SSL sessions established prior to 2014/04/12 may be vulnerable. If your current connection was established prior to this date via ssl then you should consider reconnecting to the network.
  • We would advise that users reset their password (after reconnecting) using instructions returned by the following command:

/msg nickserv help set password

This should help ensure that if your password was compromised through an exploitation of the Heartbleed vulnerability, the damage is limited.

  • In line with general best practice, we would always recommend using separate passwords on separate systems – if you shared your freenode services password with other systems, you should change your password on all of these systems; preferably into individual ones.
  • If you use CertFP, you should regenerate your client certificate (instructionsand ensure that you update NickServ with the new certificate hash. You can find out how to do this using the following command:

/msg nickserv help cert

  • Having changed passwords and/or certificate hashes, it cannot hurt to verify your other authentication methods (such as email, ACCESS or CERT). It is possible you have additional access methods configured either from past use or (less likely) due to an account compromise.
  • Finally, it is worth noting that although probably the least likely attack vector, Heartbleed can also be used as client-side attack, i.e. if you are still running a vulnerable client a server could attack you. This could be a viable attack if, for instance, you connect to a malicious IRC server and freenode at the same time; hypothetically the malicious IRC server could then attack your client and steal your IRC password or other data. If affected, you should ensure your OpenSSL install is updated and not vulnerable then restart your client.

As ever, staff are available in #freenode to respond to any questions or concerns.

New TLS/SSL Channel Modes & Webchat Features

We’ve recently enabled some new functionality in our ircd to further help you manage your channels:

Channel mode +S

This ensures only users that have connected via TLS/SSL (and so have user mode +Z) are able to join; you can not /invite them through it. It will not prevent the use of the channel by any non-TLS/SSL users already present.

Extended ban $z

Documented in ‘/help extban’ for some time, this has also been enabled and matches all TLS/SSL users. Usage is similar to the ‘$a’ type (which matches all identified users) and could for example be set as ‘+q $~z’ to to quiet any users not connected over an ssl connection.

Webchat

WEBIRC has been enabled so that behind their hostmask, users can now be considered to be connecting from their real address. This means that a single ban format can apply to both direct connections and webchat connections.

For example, a user connecting from 171.205.18.52 will still appear as ‘nickname!abcd1234@gateway/web/freenode/ip.171.205.18.52′ but ban masks of the form ‘*!*@171.205.18.52′ will match! This is now the most effective method of matching users using webchat but the realname and hexip username are still available.

Although freenode’s webchat is available over SSL, the webchat’s localhost connection to the ircd is not SSL, so webchat users do not get user mode +Z. Webchat users will not be able to join a +S channel and will not match the $z extban, even if they are using webchat over SSL.

Security considerations

These channel modes can not guarantee secure communication in all cases; if you choose to rely on them, please understand what they can and can’t do, and what other security considerations there are.

There are a variety of known security problems with SSL, and reasons why the +S mode may not guarantee transport security on freenode. Some of these are:

  • These modes may be unset by channel operators at any time, allowing non-TLS/SSL users to join, and the mode may subsequently be reapplied;
  • If network splits occur it may also be possible for users to bypass +S intentionally or by chance;
  • Clients may be compromised or malicious, or using a malicious shared host;
  • Clients may have traffic intercepted as part of a Man In The Middle (MITM) attack and then transparently forwarded via SSL, invisibly to channel users;
  • There may be issues with TLS/SSL itself in server or client configuration or architecture which compromise its ability to provide effective transport security at the network level (there have been several published attacks against SSL recently – see here).

This is not an authoritative list, so before using +S as part of any channel which requires strong anonymity, please ensure you understand what it does and its drawbacks.

There are other security tools you may want to look at – you may want to consider using client plugins that provide additional encryption or route your connection through Tor. Tor also allows you to create spurious traffic to hide real traffic patterns. freenode provides its own hidden Tor node which means you can trust this connection as much as you trust freenode. Your IRC traffic with freenode via Tor is end-to-end encrypted from your Tor client to our Tor node. It does not pass through any third party nodes in unencrypted form.

Finally, unless you can trust everyone in a channel and are sure it is configured properly and you understand the other technical risks, do not rely on these channel modes exclusively. Security is generally layered; ensure you have good defense in depth and don’t rely on individual controls which may be a single point of failure.

Using other websites or services via Tor

Remember to always encrypt your traffic when using Tor as you have no control over who is running exit nodes and if they are doing traffic analysis on them. While your traffic to the exit node is encrypted and the ingress node can not read it, the exit node will always need to be able to remove Tor encryption. If your traffic is clear-text said exit node will be able to read it.

Upgrade and database prune completed

The planned services upgrade and database prune went ahead today as planned and has completed successfully. Approximately 300000 nicks were removed from the database, and we’ve moved to Atheme 7, so hopefully response times from services should be improved, with less of the lag that was sometimes noticeable before.

In addition, certificate based authentication is now available. We’ll hopefully get the docs for this up online shortly.

Services upgrade and database prune

Hi
The long-awaited upgrade of services which we blogged about a while ago is now planned for this coming weekend, the 16th/17th June.

We anticipate up to an hour of services outage for this upgrade and prune to take place. We will notify the connected users closer to the time through the use of WALLOPS and/or globals, but please do plan ahead accordingly for a period of services unavailability.

We will be moving to Atheme 7, so, amongst other improvements, this will see the introduction of certificate-based authentication to services.

To use certificate based authentication, you would need firstly to generate a certificate, then add the certificate to your client, then tell nickserv about your certificate fingerprint. We’ll explain more about this in a future blog entry or on the freenode website in the near future.

Database prune

Every couple of years, freenode likes to get out the shears and prune the services database. Recently we broke the 80,000 usercount barrier, but the services stats are way ahead:

Sat 13:35:46 -OperServ(OperServ@services.)- Registered accounts: 446777
Sat 13:35:46 -OperServ(OperServ@services.)- Registered nicknames: 557497
Sat 13:35:47 -OperServ(OperServ@services.)- Registered channels: 141373

We’ve noticed that nearly half of the accounts shown there haven’t been used in the past 6 months! More importantly, over the past few months many people have noticed significant waits when issuing certain services commands – and we’d like to fix that.

Hopefully, the services upgrade should help with this, but we’re going to coincide this with a database prune.

As of the services upgrade date, any nicks unused for > 150 days are at risk of being dropped. This includes grouped nicks. The easy way to avoid this happening is to use each of your grouped nicks (while identified to the appropriate account) within the next few weeks – and to drop those that you don’t need anymore!

The testnet (testnet.freenode.net, port 9002. 9003 for SSL) is running a database snapshot from mid-March and will be periodically updated from the production network. This database instance is being regularly pruned – so check there to see how your account will be affected (use /msg nickserv info on both the production and test networks to see the differences).

Remember that testnet isn’t running a real-time duplicate of the production network, so when you use nicks which would be expired on the production network, they will still appear expired on testnet until the next database snapshot is migrated. Don’t worry though – the actual pruning will only occur on the current database at the time of upgrade.

On which note.. an upgrade date hasn’t been formally fixed but we’re aiming for mid-May.

Thanks, and don’t forget to test the testnet!

Help us test our services upgrade!

Very soon we will be upgrading your favourite network helpers… (no not erry…): NickServ, ChanServ, Alis etc. They’re currently connected to our testnet and we need your help with testing, looking for any issues which may affect the production network.

You can connect to our testnet at testnet.freenode.net port 9002 (or 9003 for SSL)

The full changelog is rather long and not all of the features offered by atheme are loaded on freenode. So to help you out, we’ve pulled out the highlights which we think deserve attention:

  • NickServ’s certfp module. (see /msg nickserv help cert and this link.)
  • NickServ will now notify you in real time of failed logins.
  • NickServ’s previous limit on password lengths has been increased.
  • ChanServ will still hand over single-# channels to freenode-staff on expiration of the channel founders, but the method has changed.
  • NickServ & ChanServ’s ‘set’ commands have had a general reorganisation behind the scenes. Nothing should be visibly different but it won’t hurt to check them!

Please note that the services database on the testnet is probably more than a few days old. Don’t be surprised if recent changes you have made on the production network aren’t replicated there.

We’re all in in #freenode on the testnet so please come find us there if you have any questions or bugs.

Finally, look out for a followup blogpost (hopefully quite soon) with some important information on the upgrade itself and our planned database cleanup!

Thanks for using freenode!

P.s. a full list of changes from atheme ~5.1 to ~6 can be found here

ircd upgrades

We’ve got some ircd upgrades in the works!

You may remember several weeks ago that we upgraded our ircd on the production network. Since then, we’ve wanted to fine-tune some changes and make sure that the upgrade is more consistent with the old version.

Over the next few weeks, we’ll be looking to perform upgrades on the production network again. This will mean every server will reboot. A programme for the upgrades can be found at the end of this post (updated 13th Nov 2011).

In the meantime, please continue to help us to test the ircd at testnet.freenode.net port 9002 or 9003 for SSL (if you don’t get onto the first server that the DNS roundrobin gives you, keep trying!). Look for anything broken, inconsistent with previous versions (especially in terms of information release) or illogical. If serious issues are reported, we’ll aim to fix before upgrading, rather than having a further later upgrade. Please report issues to #freenode-seven on the production network.

Thanks!

NB: this list does not include servers invisible to users (eg hubs).
Week 1: Sun 13th Nov
-!- kornbluth.freenode.net Frankfurt, Germany
-!- zelazny.freenode.net Corvallis, OR, USA
-!- stross.freenode.net Corvallis, OR, USA (webchat backup)

Week 2: Sun 20th Nov
-!- barjavel.freenode.net Paris, FR
-!- wolfe.freenode.net Manchester, England
-!- hubbard.freenode.net Pittsburgh, PA, US

Week 3: Sun 27th Nov
-!- adams.freenode.net Budapest, HU, EU
-!- holmes.freenode.net London, UK
-!- sendak.freenode.net Vilnius, Lithuania, EU
-!- rowling.freenode.net Corvallis, OR, USA (webchat)

Week 4: Sun 4th Dec
-!- pratchett.freenode.net Rennes, France
-!- calvino.freenode.net Milan, IT
-!- leguin.freenode.net Ume?, SE, EU
-!- niven.freenode.net Corvallis, OR, USA

Week 5: Sun 11th Dec
-!- hitchcock.freenode.net Sofia, BG, EU
-!- gibson.freenode.net Oslo, Norway
-!- card.freenode.net Washington, DC, USA
-!- asimov.freenode.net TX, USA
-!- verne.freenode.net Newark, NJ, US

Unscheduled.
-!- roddenberry.freenode.net
-!- bartol.freenode.net
-!- brown.freenode.net
-!- anthony.freenode.net


Update: all upgrades are now complete.

Further webchat issues

Unfortunately, it seems the box our webchat is on has decided to fall out with the Internet again. We’re working on setting up a reserve instance which shouldn’t be affected by this sort of issue in the future. When we have more details, we’ll update this post. We’re really sorry for the inconvenience this causes, and guarantee it will be less in future.

Update: The host’s issues appear to have been resolved. We now also have a backup instance running which can easily be switched to in the event of downtime in the future.

Webchat downtime

Hi everyone.
Currently the freenode webchat instance (webchat.freenode.net) is down. This is due to maintenance by the host of the box upon which the service sits, and looks set to continue for up to a further 6 hours.
This is maintenance that we, as staff, were not previously aware of.
We’re very sorry for the inconvenience and are doing what we can to reduce it.

Update: resolved.

freenode webchat changes

Webchat has always presented an interesting problem, mostly for the staff of various channels as well as the network itself, but indirectly for all our users as well.  All webchat connections come from the IP address of the webchat service.  This results in them having to be handled a little bit differently from other connections.

To begin with, there needs to be a way for network or channel staff to identify individual connections, as well as where they originated from.  The way this has previously been handled is by encoding the IP of the source (the IP someone uses to connect to the webchat) in hexadecimal form in the ident field of the user.  The webchat users are “cloaked” (that is, their real hostname, which would be that of the webchat server, is replaced) with a unique string identifying the connection.  This method allows channel staff to ban or quiet a webchat user via the unique connection string, or via the ident information.

While this works, it’s confusing to many. The unique connection string changes every time a user makes a new connection through webchat. Therefore, we’ve changed how we do the cloaking so IPs are shown in cloaks. This makes it much simpler for channel staff to see what is going on, and who is who. For now, this change only applies to those using the freenode webchat at http://webchat.freenode.net. The effect is to change a cloak of the form “gateway/web/freenode/x-iiqzrxiqfnnglqji” to the form “gateway/web/freenode/ip.171.205.239.16“.

We would like to point out that this does not in any way reduce the privacy of users of webchat: it has always been possible for anyone to directly convert the encoded ident string back to an IP address. In addition, the real hostnames of clients have always been visible unencoded in the “whois” output for the user.

In addition, we have made a small but potentially significant change to how the “ident” is shown. This has become necessary so that, with future versions of our ircd, we can properly limit connections per IP address via webchat. For a typical freenode webchat user, the full hostmask previously had the form “~abcdef1@gateway/web/freenode/...“. Many historical webchat bans and quiets are set as “*!~abcdef1@gateway/web/freenode/*“. The change that we are making will break these bans. We have removed the ~ from the ident for all webchat connections (not just freenode’s webchat), giving a full mask of the form “abcdef1@gateway/web/freenode/ip.171.205.239.16“.

As such, channel ops are advised to adjust their bans into the form of either “*!abcdef1@gateway/web/freenode/*” or “*!*@gateway/web/freenode/ip.171.205.239.16” as soon as possible.

A further result of this change is that those hosts from which a large number of legitimate users connect to freenode through the webchat service may suffer refused connections due to breaching the limits. If you find youself faced by an error of the form “Too many connections”, please email iline at freenode dot net with details of the IP address affected (which can be obtained from www.whatismyip.org), the name of the organisation, and the number of connections expected, so that we can place a limit exemption. Please note that if you have a message of the form “Gateway connections are currently blocked” or “Gateway connections are currently being throttled”, this is a different matter for which an I:line cannot help.

We hope that these changes make connections through the freenode webchat easier to manage for channel ops and more transparent for all users.