Server issues

Earlier today the freenode infra team noticed an anomaly on a single IRC server. We have since identified that this was indicative of the server being compromised by an unknown third party. We immediately started an investigation to map the extent of the problem and located similar issues with several other machines and have taken those offline. For now, since network traffic may have been sniffed, we recommend that everyone change their NickServ password as a precaution.

Before changing your password, please check your email address in /msg nickserv info and, if needed, update it – see /msg nickserv help set email (remember to check your new email for the verification key). This will ensure that we can send you a password reset email should, for whatever reason, your password change not work properly. If you have no email set on your account or an email set that you cannot access, we cannot send password resets to you, so do please keep this up-to-date.

To change your password use /msg nickserv set password newpasshere

Since traffic may have been sniffed, you may also wish to consider any channel keys or similar secret information exchanged over the network.

We’ll issue more updates as WALLOPS and via social media!

Turbulence

As many of you will be aware, freenode has been experiencing intermittent instability today, as the network has been under attack. Whilst we have network services back online, the network continues to be a little unreliable and users are continuing to report issues in connecting to the network.

We appreciate the patience of our many wonderful users whilst we continue to work to mitigate the effects this has on the network.

We also greatly appreciate our many sponsors who work with us to help minimise the impact and who are themselves affected by attacks against the network.

We’ve posted on this subject before, and what we said then remains as true as ever – and for those of you who didn’t read the earlier blogpost first time round, it’s definitely worth perusing it now if this subject interests or affects you.

Thank you all for your patience as we continue to work to restore normal service!

[UPDATE 04/02/2014]

At the moment SASL authentication works only on PLAINTEXT, *not* BLOWFISH. We’ve checked and TOR should be working too. Sadly wolfe.freenode.net will be taken off the rotation, so those users who’ve connected specifically to it, please make sure that your client points to our recommended roundrobin of chat.freenode.net!

Reminder: Keep your NickServ email up to date.

If you’ve registered with NickServ within the last few years then you’ll have used an email address and we’ll have sent you a mail to verify it. That will probably be the last time you heard from us…

…until you forget your password and find yourself unable to identify to your account. When that happens we can send an email (only to that same address) to verify your identify and reset your password.

You aren’t stuck with the email you originally used though! We’d very strongly recommend you take 5 minutes to double check the set email address is current, especially in light of recent service closures. You don’t need access to your old inbox to change your registered email, just your NickServ password.

To view the current state of your account, while identified type:

/msg nickserv info

If you’d like to then change the registered email address, first…

/msg nickserv set email [email protected]

… then check your email inbox. We’ll have sent you another email with instructions to verify this new address.

Your email address is hidden from other users by default. You can ensure this by setting:

/msg nickserv set hidemail on

Thanks for using freenode!

Server hosting and trust

For the purpose of disclosure we have had to make the difficult decision to discontinue a long-standing relationship with a server sponsor.

As a freenode user you may be aware that our set-up is somewhat untraditional and differs from that of many other IRC networks; servers are sponsored by various companies and educational institutions across the globe and all our infrastructure is centrally managed by the freenode infrastructure team. Generally speaking we do not provide o:lines or other privileges to server sponsors. Whilst it is possible for a sponsor contact to also volunteer as a staffer on the network such recruitment is independent of any server hosting.

Our staff are expected to work together closely and communication is key in any freenode relationship, be that with users, among staff or with sponsor contacts. It is important to us to be consistent in the way we provide support and apply policy and we expect all volunteers to be intimately familiar with our policies, procedures and philosophies — which in turn means that senior staff invest a lot of time in ensuring that any new recruits are given adequate support when getting to know the ins and outs of the network and what being a freenode volunteer entails.

Unfortunately one of our server sponsors added an o:line for themselves on the server they sponsored and whilst we do not believe that this was done with any malicious intent, more through thoughtlessness/negligence and having forgotten the expectations set out on our “Hosting a Server” page we feel that we are unable to comfortably and confidently continue the relationship.

Our number one priority has to be our target communities, the Free and Open Source Software communities that have chosen to make use of freenode in their internet activities.

Whilst we do not believe and have no evidence to indicate that any user traffic or data has been compromised, we would of course encourage you to change your passwords if you feel that this would make you more comfortable in continuing to use our services.

We can only apologise for this happening and we’d like to assure you that trust is incredibly important to us and that we are incredibly embarassed that this situation arose in the first place.

As a result of this we have just replaced our SSL certificates, so if you notice that these have changed then this is the reason why.

We will of course take this opportunity to remind all our sponsors of our expectations when it comes to providing services to freenode and our target communities.

Again, we apologise for any inconvenience and we hope that any loss of trust in the network that may have resulted from this incidence can be restored and that your projects will continue to feel comfortable using the network in future.

 

 

New TLS/SSL Channel Modes & Webchat Features

We’ve recently enabled some new functionality in our ircd to further help you manage your channels:

Channel mode +S

This ensures only users that have connected via TLS/SSL (and so have user mode +Z) are able to join; you can not /invite them through it. It will not prevent the use of the channel by any non-TLS/SSL users already present.

Extended ban $z

Documented in ‘/help extban’ for some time, this has also been enabled and matches all TLS/SSL users. Usage is similar to the ‘$a’ type (which matches all identified users) and could for example be set as ‘+q $~z’ to to quiet any users not connected over an ssl connection.

Webchat

WEBIRC has been enabled so that behind their hostmask, users can now be considered to be connecting from their real address. This means that a single ban format can apply to both direct connections and webchat connections.

For example, a user connecting from 171.205.18.52 will still appear as ‘nickname!abcd1234@gateway/web/freenode/ip.171.205.18.52′ but ban masks of the form ‘*!*@171.205.18.52′ will match! This is now the most effective method of matching users using webchat but the realname and hexip username are still available.

Although freenode’s webchat is available over SSL, the webchat’s localhost connection to the ircd is not SSL, so webchat users do not get user mode +Z. Webchat users will not be able to join a +S channel and will not match the $z extban, even if they are using webchat over SSL.

Security considerations

These channel modes can not guarantee secure communication in all cases; if you choose to rely on them, please understand what they can and can’t do, and what other security considerations there are.

There are a variety of known security problems with SSL, and reasons why the +S mode may not guarantee transport security on freenode. Some of these are:

  • These modes may be unset by channel operators at any time, allowing non-TLS/SSL users to join, and the mode may subsequently be reapplied;
  • If network splits occur it may also be possible for users to bypass +S intentionally or by chance;
  • Clients may be compromised or malicious, or using a malicious shared host;
  • Clients may have traffic intercepted as part of a Man In The Middle (MITM) attack and then transparently forwarded via SSL, invisibly to channel users;
  • There may be issues with TLS/SSL itself in server or client configuration or architecture which compromise its ability to provide effective transport security at the network level (there have been several published attacks against SSL recently – see here).

This is not an authoritative list, so before using +S as part of any channel which requires strong anonymity, please ensure you understand what it does and its drawbacks.

There are other security tools you may want to look at – you may want to consider using client plugins that provide additional encryption or route your connection through Tor. Tor also allows you to create spurious traffic to hide real traffic patterns. freenode provides its own hidden Tor node which means you can trust this connection as much as you trust freenode. Your IRC traffic with freenode via Tor is end-to-end encrypted from your Tor client to our Tor node. It does not pass through any third party nodes in unencrypted form.

Finally, unless you can trust everyone in a channel and are sure it is configured properly and you understand the other technical risks, do not rely on these channel modes exclusively. Security is generally layered; ensure you have good defense in depth and don’t rely on individual controls which may be a single point of failure.

Using other websites or services via Tor

Remember to always encrypt your traffic when using Tor as you have no control over who is running exit nodes and if they are doing traffic analysis on them. While your traffic to the exit node is encrypted and the ingress node can not read it, the exit node will always need to be able to remove Tor encryption. If your traffic is clear-text said exit node will be able to read it.

The good, the bad, and the ugly…

Firstly, I would like to apologise for the interruptions the network has experienced in the last week (and continues to experience as we speak). I would also like to thank our incredible server sponsors for the time and dedication they have shown in helping us attempt to deal with the situation.

Sponsors — sponsors are the lifeblood of the network; without sponsors there would be no freenode. Unfortunately, the recent attacks have been significant enough for some of our sponsors to pull the plug as they were unable to continue providing the same level of assistance to the network as they had in the past. These kind of attacks can be costly for our sponsors; the disruptions soon have a financial impact for sponsors and their paying clients when service is disrupted. They are also costly in time and resources spent trying to alleviate the issues caused within their networks. To those of our sponsors who have had to discontinue sponsorship, in part or in full, I would like to thank you for the years of support. Not just for freenode but for the Free and Open Source Software Communities and we can only apologise for the difficulties your organisations have experienced as a result of these recent attacks.

Free and Open Source Software Communities — whilst sponsors may be the lifeblood of the network, the FOSS communities are our reason for being. Unfortunately they, along with our sponsors, are the ones suffering at the hands of the attacker(s) — it is their projects that are disrupted and affected and we can only apologise for the instability and disruption experienced by projects on the network in this last week.

freenode — ironically freenode is the puzzle piece that gets off lightly. We’re just a bunch of people passionate about FOSS — the network itself is devoid of feeling and whilst our volunteers do their best dealing with the aftermath of the attacks and try to keep the network up and running the reality is that in the grand scheme of things freenode is nothing. freenode is just a means to an end; the projects that have chosen to use freenode could easily go elsewhere, the volunteers who staff the network… well, they could easily go wherever their projects went — we volunteer for freenode because we’re passionate about FOSS, and the majority of us also contribute to one or several FOSS projects or have done in the past. For us it has never been about “freenode” — it has been about FOSS; and the projects we, as individuals, care about. We are all freenode users first, and staffers second.

If there was no freenode, there would be other alternatives — perhaps similar alternatives, perhaps very different alternatives. The FOSS communities are full of talented, passionate people and I have no doubt that we’d all find different ways to stay in touch and work on our projects even if there was no freenode.

That’s not to say we’re about to throw in the towel — we’ve all invested a lot of time and effort in the network and I am sure we will continue to do so for as long as there are projects wanting to use it and sponsors willing to help us.

I wish I could provide you with detailed information about the attacks and the cause of them — but these details are but a mystery to us and with nearly 90,000 users I’d be loathe to speculate as to who we might have annoyed… or how. For the time being, we intend to continue mitigating attacks where possible and continuing to endeavour to provide service as usual!

Once more, thank you for the support and the faith in the project — and thank you for the patience whilst our infrastructure team desperately tries to juggle our infrastructure around to bring back as much of our normal services as is possible at this point in time.

Insert witty title here

Like every year, we would like to invite our users to take part in the April 1st quiz and have the chance to win an April Fools’ cloak.

Good luck, lots of fun, and thanks for flying freenode trebutchet!

What does Dorothy wish she was on the other side of?
###>++++++++++ [>++++++++++>+++++++++++>+++++++++++>+++++++++++>++++++++++><<<<<<-] >+>++>++++>+>+++> <<<<<< >.>.>.>.>.
##bhggbyhapu

Group Registration Closure

Our group registration system has been around for some time, in various guises.  Over that time, our small but dedicated team of staff has attempted to keep up with demand for groups.  Unfortunately, in the early years of GRF, this generated a substantial backlog of processing, since the system was very manual, a lot of data was processed (restricting the staff who had access) and each group can take some time to properly investigate.  To address this issue, we’ve tried a number of alternatives, such as priority group emails, and, lately, a streamlined group registration system known as “grf-f”.

For various reasons, these replacements haven’t worked quite how we’d want or need them to in order to achieve our objective of registering groups in a timely fashion.  Meanwhile, development of GMS, our automated replacement GRF system, continues.

For these reasons, we have taken the decision to temporarily close the group registration system.

What this means is that –

  • No new group registrations will be accepted from this point onwards, until further notice.
  • Outstanding grf-f applications already in the queue will be processed in due course.
  • Outstanding “old style” GRF applications will *not* be processed (most of the applications in this set are very old now, and the people who submitted them should have seen at least one reminder to refile under grf-f)
  • All existing registered groups are unaffected and continue as normal.

Please bear with us whilst we work on where we want to go next with this system.  Meanwhile, if there are channels currently owned by freenode-staff that your project could make use of, please contact a staffer to see if (provisional) op rights can be granted to your account. (These would likely be done on a somewhat temporary basis, until such time as registration re-opens.)  Note also that this is *only* to gain ops on channels, *not* for obtaining group cloaks.

As always, thanks for flying freenode!

mrmist.

April 1st, the aftermath

It’s been two weeks since our April Fools quiz; thanks to everybody who participated. Several people managed to complete the quiz; the first eight are:

  1. dwfreed
  2. bikcmp
  3. divVerent
  4. sbp
  5. meridion
  6. heftig
  7. JoshuaA
  8. [bjoern]

Here are the riddles and their solutions, in the original order:

Stage 1, the picture linked on the blog:

ITucplOwnTShozIfVT1cM2u0VTWyVPZwp3EupaD=


This was base64 + rot13 encryption,
##start

Stage 2, topic in ##start

###### MLG UZHGVI GSZM


Encryption A-Z Z-A,
“NOT FASTER THAN”

With a bit of thinking, this led the users to ######neutrinos

Stage 3, topic in ######neutrinos

#### C0LEMAK 43WRXC 34PTCXRW XSF34GDVC BHJ6 CTP4GHKNL7 87JHKMEN


Solvable with a picture of the Colemak keyboard layout and a bit of tracing with your fingers. Following the path between the keys on the keyboard gave you C, O, D, I, N, G.

Stage 4, topic in ####coding

3. #### Find a five-digit number in which the last number is the sum of the first, second, and third; the third is four less than the last; the fourth is two less than the last; and the first and fourth added are one less than the last. The last number is also three times the second.


Plain math, 13579

Stage 5, topic in ####13579

#### This “huge” work, written over twen ty years, contains ten books. What colour is it?


The Great Book of Amber
####amber

Last Stage, topic in ####amber

i yam what i yam but i yam ####

which lead to ####popeye, the final channel.

We hope you enjoyed our little quiz and had a great time.