Heartbleed

The recently exposed heartbleed bug in the OpenSSL library has surprised everyone with a catastrophic vulnerability in many of the world’s secure systems.

In common with many other SSL-exposed services, some freenode servers were running vulnerable versions of OpenSSL, exposing us to this exploit. Consequently, all of our affected services have been patched to mitigate the vulnerability, and we have also regenerated our private SSL keys and certificates.

In an unrelated event, due to service disruption & the misconfiguration of a single server on our network, an unauthorised user was allowed to use the ‘NickServ’ nickname for a short period Sunday morning. Unfortunately there is a possibility that your client sent data (including your freenode services password) to this unauthorised client. Identification via SASL, certfp or server password were not affected, but any password sent directly to the “NickServ” user might have been.

Because of these two recent issues, we would like to make the following recommendations to all of our users. It would also be good practice to follow them at regular intervals.

  • Though we are not aware of any evidence that we have been targeted, or our private key compromised, this is inevitably a possibility. SSL sessions established prior to 2014/04/12 may be vulnerable. If your current connection was established prior to this date via ssl then you should consider reconnecting to the network.
  • We would advise that users reset their password (after reconnecting) using instructions returned by the following command:

/msg nickserv help set password

This should help ensure that if your password was compromised through an exploitation of the Heartbleed vulnerability, the damage is limited.

  • In line with general best practice, we would always recommend using separate passwords on separate systems – if you shared your freenode services password with other systems, you should change your password on all of these systems; preferably into individual ones.
  • If you use CertFP, you should regenerate your client certificate (instructionsand ensure that you update NickServ with the new certificate hash. You can find out how to do this using the following command:

/msg nickserv help cert

  • Having changed passwords and/or certificate hashes, it cannot hurt to verify your other authentication methods (such as email, ACCESS or CERT). It is possible you have additional access methods configured either from past use or (less likely) due to an account compromise.
  • Finally, it is worth noting that although probably the least likely attack vector, Heartbleed can also be used as client-side attack, i.e. if you are still running a vulnerable client a server could attack you. This could be a viable attack if, for instance, you connect to a malicious IRC server and freenode at the same time; hypothetically the malicious IRC server could then attack your client and steal your IRC password or other data. If affected, you should ensure your OpenSSL install is updated and not vulnerable then restart your client.

As ever, staff are available in #freenode to respond to any questions or concerns.

Server hosting and trust

For the purpose of disclosure we have had to make the difficult decision to discontinue a long-standing relationship with a server sponsor.

As a freenode user you may be aware that our set-up is somewhat untraditional and differs from that of many other IRC networks; servers are sponsored by various companies and educational institutions across the globe and all our infrastructure is centrally managed by the freenode infrastructure team. Generally speaking we do not provide o:lines or other privileges to server sponsors. Whilst it is possible for a sponsor contact to also volunteer as a staffer on the network such recruitment is independent of any server hosting.

Our staff are expected to work together closely and communication is key in any freenode relationship, be that with users, among staff or with sponsor contacts. It is important to us to be consistent in the way we provide support and apply policy and we expect all volunteers to be intimately familiar with our policies, procedures and philosophies — which in turn means that senior staff invest a lot of time in ensuring that any new recruits are given adequate support when getting to know the ins and outs of the network and what being a freenode volunteer entails.

Unfortunately one of our server sponsors added an o:line for themselves on the server they sponsored and whilst we do not believe that this was done with any malicious intent, more through thoughtlessness/negligence and having forgotten the expectations set out on our “Hosting a Server” page we feel that we are unable to comfortably and confidently continue the relationship.

Our number one priority has to be our target communities, the Free and Open Source Software communities that have chosen to make use of freenode in their internet activities.

Whilst we do not believe and have no evidence to indicate that any user traffic or data has been compromised, we would of course encourage you to change your passwords if you feel that this would make you more comfortable in continuing to use our services.

We can only apologise for this happening and we’d like to assure you that trust is incredibly important to us and that we are incredibly embarassed that this situation arose in the first place.

As a result of this we have just replaced our SSL certificates, so if you notice that these have changed then this is the reason why.

We will of course take this opportunity to remind all our sponsors of our expectations when it comes to providing services to freenode and our target communities.

Again, we apologise for any inconvenience and we hope that any loss of trust in the network that may have resulted from this incidence can be restored and that your projects will continue to feel comfortable using the network in future.

 

 

New TLS/SSL Channel Modes & Webchat Features

We’ve recently enabled some new functionality in our ircd to further help you manage your channels:

Channel mode +S

This ensures only users that have connected via TLS/SSL (and so have user mode +Z) are able to join; you can not /invite them through it. It will not prevent the use of the channel by any non-TLS/SSL users already present.

Extended ban $z

Documented in ‘/help extban’ for some time, this has also been enabled and matches all TLS/SSL users. Usage is similar to the ‘$a’ type (which matches all identified users) and could for example be set as ‘+q $~z’ to to quiet any users not connected over an ssl connection.

Webchat

WEBIRC has been enabled so that behind their hostmask, users can now be considered to be connecting from their real address. This means that a single ban format can apply to both direct connections and webchat connections.

For example, a user connecting from 171.205.18.52 will still appear as ‘nickname!abcd1234@gateway/web/freenode/ip.171.205.18.52′ but ban masks of the form ‘*!*@171.205.18.52′ will match! This is now the most effective method of matching users using webchat but the realname and hexip username are still available.

Although freenode’s webchat is available over SSL, the webchat’s localhost connection to the ircd is not SSL, so webchat users do not get user mode +Z. Webchat users will not be able to join a +S channel and will not match the $z extban, even if they are using webchat over SSL.

Security considerations

These channel modes can not guarantee secure communication in all cases; if you choose to rely on them, please understand what they can and can’t do, and what other security considerations there are.

There are a variety of known security problems with SSL, and reasons why the +S mode may not guarantee transport security on freenode. Some of these are:

  • These modes may be unset by channel operators at any time, allowing non-TLS/SSL users to join, and the mode may subsequently be reapplied;
  • If network splits occur it may also be possible for users to bypass +S intentionally or by chance;
  • Clients may be compromised or malicious, or using a malicious shared host;
  • Clients may have traffic intercepted as part of a Man In The Middle (MITM) attack and then transparently forwarded via SSL, invisibly to channel users;
  • There may be issues with TLS/SSL itself in server or client configuration or architecture which compromise its ability to provide effective transport security at the network level (there have been several published attacks against SSL recently – see here).

This is not an authoritative list, so before using +S as part of any channel which requires strong anonymity, please ensure you understand what it does and its drawbacks.

There are other security tools you may want to look at – you may want to consider using client plugins that provide additional encryption or route your connection through Tor. Tor also allows you to create spurious traffic to hide real traffic patterns. freenode provides its own hidden Tor node which means you can trust this connection as much as you trust freenode. Your IRC traffic with freenode via Tor is end-to-end encrypted from your Tor client to our Tor node. It does not pass through any third party nodes in unencrypted form.

Finally, unless you can trust everyone in a channel and are sure it is configured properly and you understand the other technical risks, do not rely on these channel modes exclusively. Security is generally layered; ensure you have good defense in depth and don’t rely on individual controls which may be a single point of failure.

Using other websites or services via Tor

Remember to always encrypt your traffic when using Tor as you have no control over who is running exit nodes and if they are doing traffic analysis on them. While your traffic to the exit node is encrypted and the ingress node can not read it, the exit node will always need to be able to remove Tor encryption. If your traffic is clear-text said exit node will be able to read it.

Bye bye PDPC

Sadly, we were forced to dissolve PDPC, freenode’s parent organisation.

When the organisation transferred across from the US to the UK we wanted to keep the organisational structure as close to what it had been before (change is scary, right?) — however, we made the conscious decision to no longer have any paid employees after Rob Levin passed away. This meant that everyone involved with the organisation were volunteers and we no longer had anyone who could invest the time and effort required to do fundraising and similar tasks, meaning that the organisation was unable to sustain the levels of donations required to obtain and maintain charitable status in the UK.

Due to the massive reduction in financial support we found ourselves in a position where being an incorporated organisation cost more than what we were able to bring in in donations and after years of operating at a loss it was decided that we would apply for the dissolution of the corporation in order to drastically reduce costs. The application has been processed and the organisation has been dissolved; to further reduce costs we have also discontinued the majority of infrastructure services for which the organisation paid, together with the reduced administration and organisational fees this means that we are now in a position where our outgoings are restricted to domain renewals! We would like to thank everyone who has contributed to the organisation in the past, users, organisations and staff in particular, who have always been (begrudgingly?) happy to contribute towards the difference in order to cover the deficit.

What does this organisational change mean for freenode?

In practise it means very little, the PDPC has never been involved in the day to day operations of the network and there will be no changes to the way in which the network is run. freenode is staffed entirely by volunteers from all over the globe who contribute their time and expertise to keep the network up and running in between contributing to various other FOSS projects.

What about other PDPC projects, such as fosscon, geeknic, and the fossevents site?

These projects will continue as they have before, and we invite you to attend fosscon for real world talks and collaboration, to join a geeknic picnic or plan your own at http://geeknic.org, and to check out http://fossevents.org for events in your neighbourhood and around the world.

I appreciate the work you do and I still want to contribute

The best way in which to help the network is to contribute time — help out in #freenode or elsewhere on the network, assist users in finding answers to their questions and help us try keep the channel and network temperature at a nice, comfortable level which encourages collaboration!
If you are low on time but still want to help out you might be able to help us through your company or organisation by becoming a server sponsor (See “Hosting a server” for more information).
If you feel that one particular volunteer has helped you out and you want to say thank you — ask them if they have a preferred charity to which you could make a small donation! With time we might update our website to provide links and information of such preferences.
Alternatively, you may consider donating to one of the following projects:

Existing PDPC donor cloaks

Existing PDPC donor cloaks will remain valid for a full year, after which they will be converted to unaffiliated cloaks. Ongoing donations will be cancelled by us. If you have previously donated to PDPC you’ll still qualify for your donor cloak as normal. If you believe you’re due a cloak and we haven’t processed it yet please contact us.

Upgrade and database prune completed

The planned services upgrade and database prune went ahead today as planned and has completed successfully. Approximately 300000 nicks were removed from the database, and we’ve moved to Atheme 7, so hopefully response times from services should be improved, with less of the lag that was sometimes noticeable before.

In addition, certificate based authentication is now available. We’ll hopefully get the docs for this up online shortly.

Services upgrade and database prune

Hi
The long-awaited upgrade of services which we blogged about a while ago is now planned for this coming weekend, the 16th/17th June.

We anticipate up to an hour of services outage for this upgrade and prune to take place. We will notify the connected users closer to the time through the use of WALLOPS and/or globals, but please do plan ahead accordingly for a period of services unavailability.

We will be moving to Atheme 7, so, amongst other improvements, this will see the introduction of certificate-based authentication to services.

To use certificate based authentication, you would need firstly to generate a certificate, then add the certificate to your client, then tell nickserv about your certificate fingerprint. We’ll explain more about this in a future blog entry or on the freenode website in the near future.

Database prune

Every couple of years, freenode likes to get out the shears and prune the services database. Recently we broke the 80,000 usercount barrier, but the services stats are way ahead:

Sat 13:35:46 -OperServ(OperServ@services.)- Registered accounts: 446777
Sat 13:35:46 -OperServ(OperServ@services.)- Registered nicknames: 557497
Sat 13:35:47 -OperServ(OperServ@services.)- Registered channels: 141373

We’ve noticed that nearly half of the accounts shown there haven’t been used in the past 6 months! More importantly, over the past few months many people have noticed significant waits when issuing certain services commands – and we’d like to fix that.

Hopefully, the services upgrade should help with this, but we’re going to coincide this with a database prune.

As of the services upgrade date, any nicks unused for > 150 days are at risk of being dropped. This includes grouped nicks. The easy way to avoid this happening is to use each of your grouped nicks (while identified to the appropriate account) within the next few weeks – and to drop those that you don’t need anymore!

The testnet (testnet.freenode.net, port 9002. 9003 for SSL) is running a database snapshot from mid-March and will be periodically updated from the production network. This database instance is being regularly pruned – so check there to see how your account will be affected (use /msg nickserv info on both the production and test networks to see the differences).

Remember that testnet isn’t running a real-time duplicate of the production network, so when you use nicks which would be expired on the production network, they will still appear expired on testnet until the next database snapshot is migrated. Don’t worry though – the actual pruning will only occur on the current database at the time of upgrade.

On which note.. an upgrade date hasn’t been formally fixed but we’re aiming for mid-May.

Thanks, and don’t forget to test the testnet!

Help us test our services upgrade!

Very soon we will be upgrading your favourite network helpers… (no not erry…): NickServ, ChanServ, Alis etc. They’re currently connected to our testnet and we need your help with testing, looking for any issues which may affect the production network.

You can connect to our testnet at testnet.freenode.net port 9002 (or 9003 for SSL)

The full changelog is rather long and not all of the features offered by atheme are loaded on freenode. So to help you out, we’ve pulled out the highlights which we think deserve attention:

  • NickServ’s certfp module. (see /msg nickserv help cert and this link.)
  • NickServ will now notify you in real time of failed logins.
  • NickServ’s previous limit on password lengths has been increased.
  • ChanServ will still hand over single-# channels to freenode-staff on expiration of the channel founders, but the method has changed.
  • NickServ & ChanServ’s ‘set’ commands have had a general reorganisation behind the scenes. Nothing should be visibly different but it won’t hurt to check them!

Please note that the services database on the testnet is probably more than a few days old. Don’t be surprised if recent changes you have made on the production network aren’t replicated there.

We’re all in in #freenode on the testnet so please come find us there if you have any questions or bugs.

Finally, look out for a followup blogpost (hopefully quite soon) with some important information on the upgrade itself and our planned database cleanup!

Thanks for using freenode!

P.s. a full list of changes from atheme ~5.1 to ~6 can be found here

ircd upgrades

We’ve got some ircd upgrades in the works!

You may remember several weeks ago that we upgraded our ircd on the production network. Since then, we’ve wanted to fine-tune some changes and make sure that the upgrade is more consistent with the old version.

Over the next few weeks, we’ll be looking to perform upgrades on the production network again. This will mean every server will reboot. A programme for the upgrades can be found at the end of this post (updated 13th Nov 2011).

In the meantime, please continue to help us to test the ircd at testnet.freenode.net port 9002 or 9003 for SSL (if you don’t get onto the first server that the DNS roundrobin gives you, keep trying!). Look for anything broken, inconsistent with previous versions (especially in terms of information release) or illogical. If serious issues are reported, we’ll aim to fix before upgrading, rather than having a further later upgrade. Please report issues to #freenode-seven on the production network.

Thanks!

NB: this list does not include servers invisible to users (eg hubs).
Week 1: Sun 13th Nov
-!- kornbluth.freenode.net Frankfurt, Germany
-!- zelazny.freenode.net Corvallis, OR, USA
-!- stross.freenode.net Corvallis, OR, USA (webchat backup)

Week 2: Sun 20th Nov
-!- barjavel.freenode.net Paris, FR
-!- wolfe.freenode.net Manchester, England
-!- hubbard.freenode.net Pittsburgh, PA, US

Week 3: Sun 27th Nov
-!- adams.freenode.net Budapest, HU, EU
-!- holmes.freenode.net London, UK
-!- sendak.freenode.net Vilnius, Lithuania, EU
-!- rowling.freenode.net Corvallis, OR, USA (webchat)

Week 4: Sun 4th Dec
-!- pratchett.freenode.net Rennes, France
-!- calvino.freenode.net Milan, IT
-!- leguin.freenode.net Ume?, SE, EU
-!- niven.freenode.net Corvallis, OR, USA

Week 5: Sun 11th Dec
-!- hitchcock.freenode.net Sofia, BG, EU
-!- gibson.freenode.net Oslo, Norway
-!- card.freenode.net Washington, DC, USA
-!- asimov.freenode.net TX, USA
-!- verne.freenode.net Newark, NJ, US

Unscheduled.
-!- roddenberry.freenode.net
-!- bartol.freenode.net
-!- brown.freenode.net
-!- anthony.freenode.net


Update: all upgrades are now complete.

Sponsorship Roundup

As you may know, the network operations of freenode are fully supported by donations – of hosting and other resources – from both companies and individuals. We acknowledge all sponsors on our website, but it is nice from time to time to provide a round-up of recent changes on the sponsorship scene!

If you’re currently connected to freenode, you will be connected to a donated server – look at the “MOTD” (delivered to you on connection or by passing the command /motd) to see who has provided your particular server.

Our newest servers include roddenberry.freenode.net (Brisbane, Australia) and asimov.freenode.net (Dallas, Texas, USA), provided by On Q Telecom and by Rackspace, respectively.

Worthy of mention indeed are those companies who support the network in ways other than providing servers. Gandi provides our SSL certificate and acts as our domain registrar, and Simtec Electronics recently generously supported the network with a donation of entropykeys. Look out for a later technical blog post as we roll these out!

While this post focuses on recent additions to the sponsorship team, it’s important not to forget the ongoing contributions of all our sponsors – take another look at our acknowledgements section and give these groups the kudos they deserve!

Cheers,
Martin