Server issues

Earlier today the freenode infra team noticed an anomaly on a single IRC server. We have since identified that this was indicative of the server being compromised by an unknown third party. We immediately started an investigation to map the extent of the problem and located similar issues with several other machines and have taken those offline. For now, since network traffic may have been sniffed, we recommend that everyone change their NickServ password as a precaution.

Before changing your password, please check your email address in /msg nickserv info and, if needed, update it – see /msg nickserv help set email (remember to check your new email for the verification key). This will ensure that we can send you a password reset email should, for whatever reason, your password change not work properly. If you have no email set on your account or an email set that you cannot access, we cannot send password resets to you, so do please keep this up-to-date.

To change your password use /msg nickserv set password newpasshere

Since traffic may have been sniffed, you may also wish to consider any channel keys or similar secret information exchanged over the network.

We’ll issue more updates as WALLOPS and via social media!

New extban: $j

We have loaded a new module on the network which provides the $j extban type:

$j:<chan> – matches users who are or are not banned from a specified channel

As an example…

/mode #here +b $j:#timbuktu

…would ban users from #here that are banned (+b) in #timbuktu.

Please note that there are a couple of gotchas:

  • Only matching +b list entries are checked. Quiets (+q) Exemptions (+e) & invexes (+I) are NOT then considered. As such, the following mode change would not alter the behaviour of the first example:

/mode #timbuktu +e *!*@*

  • Quiets and the quieting effect of bans may not immediately take effect on #here when #timbuktu’s ban list changes due to caching by the ircd.
  • $j isn’t recursive. Any $j extbans set in #timbuktu are ignored when matching in #here.

We imagine you’ll have some more useful use cases than the above.

Thanks for flying freenode!

April 1st 2014, Followup

It’s been almost too long for this blog post to arrive here after the April Fools quiz this year. Thanks to everyone who participated!

The first ten people who completed the challenges are, in descending order of aprilness:

(times are listed in UTC)

  1. 2014-04-02T18:25:17 booto
  2. 2014-04-02T23:36:53 Fuchs *

  3. 2014-04-03T00:29:29 furry
  4. 2014-04-03T01:34:18 mniip
  5. 2014-04-03T09:41:38 jojo
  6. 2014-04-03T16:29:51 redi
  7. 2014-04-03T18:57:21 BlueShark
  8. 2014-04-04T15:33:24 larinadavid
  9. 2014-04-04T22:27:20 Omniflux
  10. 2014-04-04T23:02:19 apoc
  11. 2014-04-04T23:13:02 thommey

(*) user opted out of any prizes
There were 25 additional nicks who completed the quiz and made it to the winner’s circle but weren’t fast enough to place in the top 10.

The prizes were cloaks for those in the top-10. In addition to the top-10 cloaks everyone else who finished the challenge that ‘opted-in’ were eligible for the cloak lottery. This was a lottery for 3 runnerup cloaks.

Out of the 25 additional people that completed the challenge, the following 3 won a cloak through the cloak lottery:

  • skasturi
  • danielg4
  • jojoa1997

Here are the riddles and their solutions, in the original order:

  • Level 0
    • The clue was given in the April 1st blog post: IyMjI3hrY2Q=
    • That is the string "####xkcd" encoded using base64.
    • The answer: ####xkcd, which was the first channel in the quiz.
  • Level 1
    • Clue: Tnl2cHItbmFxLU9iby1qbnl4LXZhZ2Itbi1vbmU=
    • This is a rot13‘ed and base64’ed string.
    • In Python: "Tnl2cHItbmFxLU9iby1qbnl4LXZhZ2Itbi1vbmU=".decode('base64').decode('rot13')
    • The answer: ####Alice-and-Bob-walk-into-a-bar
  • Level 2
    • Clue: MKWkpKMa
    • This is another string that is encoded with a series of base64 and rot13 transformations.
    • In Python: "MKWkpKMa".decode('rot13').decode('base64').decode('rot13')
    • The answer: ####reddit
  • Level 3
    • Clue: SHg5RkR4SUpIeHFGSnlXVUlJSVFJeHFKCg== | Save this for a later level: https://i.imgur.com/87cX9y4.jpg | 4 decodes needed
    • Yet another string encoded with a series of base64 and rot13 transformations.
    • In Python: "SHg5RkR4SUpIeHFGSnlXVUlJSVFJeHFKCg==".decode('base64').decode('rot13').decode('base64').decode('rot13')
    • This yields: EBEORIETEMETHHPITI
    • Contestants were expected to do a web search for this and find out it is the end of the Zodiac Killer’s infamous message.
    • The answer: ####zodiac
  • Level 4
    • Clue: https://i.imgur.com/x4nejBh.png | LaTeX right direction | Google! | No maths needed
    • The topic changed several times as contestants seemed pretty stumped on this level, the topic line above was its final form.
    • The answer: ####exner – this was expected from figuring out what the equation is. Simply put, the equation in the image is Exner’s Equation.
  • Level 5
  • Level 6
    • Clue: https://www.dropbox.com/s/emz7xy3p9r2ivxe/wat.unknown (verify the file, sha256sum: 0efade1bb29d1b7fdd65e5612159e262cbd41a2e27ed89a0144701a5556da68f)
    • This file is more stenography:
      • Use ‘file‘ to determine what the file type is.
      • Un-7zip the .unknown file
      • Base64 decode the output
      • Use ‘file’ to determine that the output is a .jpg
      • Unzip the .jpg
      • Untar two.tar.gz
      • Open the surprised.txt file.
    • The content of surprised.txt is: ####ImSoMetaEvenThisAcronym
    • The answer: ####ImSoMetaEvenThisAcronym
  • Level 7
    • Clue: AQwPfPN1ZBXNfvNj4bPmVR4fVQYPfPNlZBXNfvNkAP4jZhXNflOS and “Da Vinci” | Jules Verne | s/.02/.03/ in the decrypted text
    • The clue is base64’ed and rot13’ed. To decode it in Python: print "AQwPfPN1ZBXNfvNj4bPmVR4fVQYPfPNlZBXNfvNkAP4jZhXNflOS".decode('rot13').decode('base64')
    • This yields: 48° 50′ 0″ N, 2° 20′ 14.02″ E
    • These are GPS coordinates for the Paris meridian.
    • From this and the “Da Vinci” clue contestants were expected to find the Wikipedia page about the Rose Line.
    • The specific quote that contestants were suppose to find:
      "Dan Brown simply invented the 'Rose Line' linking Rosslyn and Glastonbury. The name 'Roslin' definitely does not derive from any 'hallowed Rose Line'. It has nothing to do with a 'Rose Bloodline' or a 'Rose Line meridian'. There are many medieval spellings of 'Rosslyn'. 'Roslin' is certainly not the 'original spelling': it is now the most common spelling for the village."[18]

      Source

    • The “Jules Verne” clue is suppose to reaffirm to contestants that they were on the right track:
      The competition between the Paris and Greenwich meridians is a plot element in Jules Verne's "Twenty Thousand Leagues Under the Sea", published just before the international decision in favor of the British one.

      Source

    • The answer: ####roslin
  • Level 8
  • Level 9
    • Clue: ZCLVLLCOIUTKKJSCEKHHHSMKTOOPBA | OGUCSSGAPVGVLUMBTVOGICUNJDHSTB | RUTJJGNXUNTY | Letters that would repeat in a typical word do not repeat in the key(s), example ‘freenode’ would be ‘frenod’ | https://i.imgur.com/pGIBjEE.png | http://is.gd/TgNsvm
    • Alright this one is really really really tricky. The topic changed several times.
    • The three strings are encoded with Four-square from the previous level with the same keys.
    • Contestants were expected to use ‘UVB’ and ‘RUSSIA’ as keys for the Four-square cipher.
    • It was expected that contestants arrive at ‘UVB’ from the channel name, ####POVAROVOSOLNECHNOGORSKRUSSIA
    • The former transmitter[27] was located near Povarovo, Russia[28] at 56°5′0″N 37°6′37″E which is about halfway between Zelenograd and Solnechnogorsk and 40 kilometres (25 mi) northwest of Moscow, near the village of Lozhki.

      Source

    • The is.gd link points to a file that has the “No Q” image from a previous level hidden in it.
    • The “RUTJJGNXUNTY” decrypts to AaronHSwartz
    • The answer: ####AaronHSwartz
  • Level 10
    • Clue: HKGJSUOJVRLGSBELAUHOUIGLVRURWMGTUGJGWTKN
    • Originally this channel (####AaronHSwartz) was suppose to be the winner’s circle, however due to too many people leaking answers and channel names, one more challenge was added.
    • Same cipher as before, this time the keys were ‘DEMAND’ and ‘PROGRESS’
    • Demand Progress is an Internet activist-related organization specializing in petitions to help gain traction for legal movements against Internet censorship and related subjects, started by Aaron Swartz, source.
    • The clue decrypts to JOINUSNOWANDSHARETHESOFTWAREWRITTENBYRMS
    • RMS is Richard Matthew Stallman, and ‘Join Us Now and Share the Software’ is an openly licensed song by Richard Stallman.
    • The answer: ####JOINUSNOWANDSHARETHESOFTWAREWRITTENBYRMS

The topic in ####JOINUSNOWANDSHARETHESOFTWAREWRITTENBYRMS was: Congratulations on solving the freenode’s April Fools 2014 Crypto Challenge | Want MOAR? #ircpuzzles

Congratulations to those who participated this year!

The 25 additional people that completed the challenge:

  • 2014-04-05T04:06:53 knivey
  • 2014-04-05T10:00:12 Tordek
  • 2014-04-05T15:40:50 jacob1 *
  • 2014-04-05T15:48:48 stac
  • 2014-04-05T16:24:01 Changaco *
  • 2014-04-05T17:30:01 Arch-TK *
  • 2014-04-05T17:35:05 ar *
  • 2014-04-05T18:16:20 Weetos *
  • 2014-04-05T18:38:39 nyuszika7h
  • 2014-04-05T18:56:26 vi[NLR]
  • 2014-04-05T19:06:38 tkd *
  • 2014-04-05T21:54:56 Chiyo
  • 2014-04-05T22:46:01 slidercrank
  • 2014-04-05T22:54:10 jojoa1997
  • 2014-04-06T00:55:51 Pixelz *
  • 2014-04-06T02:53:25 Transfusion
  • 2014-04-06T02:58:15 DonkeyHotei
  • 2014-04-06T03:04:01 sdamashek *
  • 2014-04-06T03:07:49 Cypi *
  • 2014-04-06T03:36:03 FXOR
  • 2014-04-06T13:44:35 pad
  • 2014-04-06T19:22:06 skasturi
  • 2014-04-06T19:37:13 Bloodhound
  • 2014-04-07T08:16:22 molly *
  • 2014-04-07T14:42:32 Bijan-E

(*) user opted out of the cloak lottery

Heartbleed

The recently exposed heartbleed bug in the OpenSSL library has surprised everyone with a catastrophic vulnerability in many of the world’s secure systems.

In common with many other SSL-exposed services, some freenode servers were running vulnerable versions of OpenSSL, exposing us to this exploit. Consequently, all of our affected services have been patched to mitigate the vulnerability, and we have also regenerated our private SSL keys and certificates.

In an unrelated event, due to service disruption & the misconfiguration of a single server on our network, an unauthorised user was allowed to use the ‘NickServ’ nickname for a short period Sunday morning. Unfortunately there is a possibility that your client sent data (including your freenode services password) to this unauthorised client. Identification via SASL, certfp or server password were not affected, but any password sent directly to the “NickServ” user might have been.

Because of these two recent issues, we would like to make the following recommendations to all of our users. It would also be good practice to follow them at regular intervals.

  • Though we are not aware of any evidence that we have been targeted, or our private key compromised, this is inevitably a possibility. SSL sessions established prior to 2014/04/12 may be vulnerable. If your current connection was established prior to this date via ssl then you should consider reconnecting to the network.
  • We would advise that users reset their password (after reconnecting) using instructions returned by the following command:

/msg nickserv help set password

This should help ensure that if your password was compromised through an exploitation of the Heartbleed vulnerability, the damage is limited.

  • In line with general best practice, we would always recommend using separate passwords on separate systems – if you shared your freenode services password with other systems, you should change your password on all of these systems; preferably into individual ones.
  • If you use CertFP, you should regenerate your client certificate (instructionsand ensure that you update NickServ with the new certificate hash. You can find out how to do this using the following command:

/msg nickserv help cert

  • Having changed passwords and/or certificate hashes, it cannot hurt to verify your other authentication methods (such as email, ACCESS or CERT). It is possible you have additional access methods configured either from past use or (less likely) due to an account compromise.
  • Finally, it is worth noting that although probably the least likely attack vector, Heartbleed can also be used as client-side attack, i.e. if you are still running a vulnerable client a server could attack you. This could be a viable attack if, for instance, you connect to a malicious IRC server and freenode at the same time; hypothetically the malicious IRC server could then attack your client and steal your IRC password or other data. If affected, you should ensure your OpenSSL install is updated and not vulnerable then restart your client.

As ever, staff are available in #freenode to respond to any questions or concerns.

+freenode

UPDATE: This was of course an April Fool… you can “/msg nickserv set property GOOGLE+” to remove the property from your account. There might still be other secrets within the message though…

freenode4

Edit: Previous versions of the post contained an incorrect NickServ command. We have corrected this and apologise for the inconvenience.

Turbulence

As many of you will be aware, freenode has been experiencing intermittent instability today, as the network has been under attack. Whilst we have network services back online, the network continues to be a little unreliable and users are continuing to report issues in connecting to the network.

We appreciate the patience of our many wonderful users whilst we continue to work to mitigate the effects this has on the network.

We also greatly appreciate our many sponsors who work with us to help minimise the impact and who are themselves affected by attacks against the network.

We’ve posted on this subject before, and what we said then remains as true as ever – and for those of you who didn’t read the earlier blogpost first time round, it’s definitely worth perusing it now if this subject interests or affects you.

Thank you all for your patience as we continue to work to restore normal service!

[UPDATE 04/02/2014]

At the moment SASL authentication works only on PLAINTEXT, *not* BLOWFISH. We’ve checked and TOR should be working too. Sadly wolfe.freenode.net will be taken off the rotation, so those users who’ve connected specifically to it, please make sure that your client points to our recommended roundrobin of chat.freenode.net!

Reminder: Keep your NickServ email up to date.

If you’ve registered with NickServ within the last few years then you’ll have used an email address and we’ll have sent you a mail to verify it. That will probably be the last time you heard from us…

…until you forget your password and find yourself unable to identify to your account. When that happens we can send an email (only to that same address) to verify your identify and reset your password.

You aren’t stuck with the email you originally used though! We’d very strongly recommend you take 5 minutes to double check the set email address is current, especially in light of recent service closures. You don’t need access to your old inbox to change your registered email, just your NickServ password.

To view the current state of your account, while identified type:

/msg nickserv info

If you’d like to then change the registered email address, first…

/msg nickserv set email [email protected]

… then check your email inbox. We’ll have sent you another email with instructions to verify this new address.

Your email address is hidden from other users by default. You can ensure this by setting:

/msg nickserv set hidemail on

Thanks for using freenode!

Server hosting and trust

For the purpose of disclosure we have had to make the difficult decision to discontinue a long-standing relationship with a server sponsor.

As a freenode user you may be aware that our set-up is somewhat untraditional and differs from that of many other IRC networks; servers are sponsored by various companies and educational institutions across the globe and all our infrastructure is centrally managed by the freenode infrastructure team. Generally speaking we do not provide o:lines or other privileges to server sponsors. Whilst it is possible for a sponsor contact to also volunteer as a staffer on the network such recruitment is independent of any server hosting.

Our staff are expected to work together closely and communication is key in any freenode relationship, be that with users, among staff or with sponsor contacts. It is important to us to be consistent in the way we provide support and apply policy and we expect all volunteers to be intimately familiar with our policies, procedures and philosophies — which in turn means that senior staff invest a lot of time in ensuring that any new recruits are given adequate support when getting to know the ins and outs of the network and what being a freenode volunteer entails.

Unfortunately one of our server sponsors added an o:line for themselves on the server they sponsored and whilst we do not believe that this was done with any malicious intent, more through thoughtlessness/negligence and having forgotten the expectations set out on our “Hosting a Server” page we feel that we are unable to comfortably and confidently continue the relationship.

Our number one priority has to be our target communities, the Free and Open Source Software communities that have chosen to make use of freenode in their internet activities.

Whilst we do not believe and have no evidence to indicate that any user traffic or data has been compromised, we would of course encourage you to change your passwords if you feel that this would make you more comfortable in continuing to use our services.

We can only apologise for this happening and we’d like to assure you that trust is incredibly important to us and that we are incredibly embarassed that this situation arose in the first place.

As a result of this we have just replaced our SSL certificates, so if you notice that these have changed then this is the reason why.

We will of course take this opportunity to remind all our sponsors of our expectations when it comes to providing services to freenode and our target communities.

Again, we apologise for any inconvenience and we hope that any loss of trust in the network that may have resulted from this incidence can be restored and that your projects will continue to feel comfortable using the network in future.

 

 

Fosscon, an open source conference in Philadelphia PA, Saturday August 10th

FOSSCON 2013 will be held on August 10th, 2013.  Several of our very own staff here at freenode will be attending this year and we are really looking forward to it.

FOSSCON was spawned from the depths of freenode and this will be the 4th event so far.

We are very excited about this year’s keynote speaker, Philadelphia’s own Jordan Miller, who leads a research team at The University of Pennsylvania. Jordan makes heavy use of open source software and is doing amazing work with 3D printing as it pertains to transplant organs.  http://www.upenn.edu/pennnews/news/penn-researchers-improve-living-tissues-3d- printed-vascular-networks-made-sugar.

Listed below is a just a quick peek at some of our confirmed speakers and their topics:

  • Bhavani Shankar will be speaking on how to bring in new developers to open source projects.
  • Elizabeth Krumbach Joseph will be speaking on Open Source Systems Administration.
  • Corey Quinn will be speaking on configuration management with Salt.
  • Brent Saner will be speaking on Project.Phree, a wireless mesh project.
  • Dru Lavigne will be speaking on FreeNAS 9.1.
  • Jérôme Jacovella-St-Louis will be hosting a workshop on cross-platform development with the Ecere SDK.
  • John Ashmead will be speaking on the math and science of invisibility.
  • John Stumpo will be offering a workshop on the Challenges facing FOSS game projects.
  • Walt Mankowski will be speaking on Scientific Programming with NumPy and SciPy.
  • Chris Nehren will be speaking on bridging the gap between development and operations.
  • Christina Simmons will be speaking on starting and managing open source events/projects.
  • Hector Castro will be offering a hands-on workshop on the Riak database engine.
  • Dan Langille will be hosting a workshop on Bacula: The Networked Backup Open Source Solution

If you haven’t registered yet, please do so here: https://www.wepay.com/events/fosscon-2013!  We’ve had such an awesome response so far and are so excited to see how far we can go this year! Invite your friends, your partners, your business associates, and everyone else you know!  We’ll see you soon!

New TLS/SSL Channel Modes & Webchat Features

We’ve recently enabled some new functionality in our ircd to further help you manage your channels:

Channel mode +S

This ensures only users that have connected via TLS/SSL (and so have user mode +Z) are able to join; you can not /invite them through it. It will not prevent the use of the channel by any non-TLS/SSL users already present.

Extended ban $z

Documented in ‘/help extban’ for some time, this has also been enabled and matches all TLS/SSL users. Usage is similar to the ‘$a’ type (which matches all identified users) and could for example be set as ‘+q $~z’ to to quiet any users not connected over an ssl connection.

Webchat

WEBIRC has been enabled so that behind their hostmask, users can now be considered to be connecting from their real address. This means that a single ban format can apply to both direct connections and webchat connections.

For example, a user connecting from 171.205.18.52 will still appear as ‘nickname!abcd1234@gateway/web/freenode/ip.171.205.18.52′ but ban masks of the form ‘*!*@171.205.18.52′ will match! This is now the most effective method of matching users using webchat but the realname and hexip username are still available.

Although freenode’s webchat is available over SSL, the webchat’s localhost connection to the ircd is not SSL, so webchat users do not get user mode +Z. Webchat users will not be able to join a +S channel and will not match the $z extban, even if they are using webchat over SSL.

Security considerations

These channel modes can not guarantee secure communication in all cases; if you choose to rely on them, please understand what they can and can’t do, and what other security considerations there are.

There are a variety of known security problems with SSL, and reasons why the +S mode may not guarantee transport security on freenode. Some of these are:

  • These modes may be unset by channel operators at any time, allowing non-TLS/SSL users to join, and the mode may subsequently be reapplied;
  • If network splits occur it may also be possible for users to bypass +S intentionally or by chance;
  • Clients may be compromised or malicious, or using a malicious shared host;
  • Clients may have traffic intercepted as part of a Man In The Middle (MITM) attack and then transparently forwarded via SSL, invisibly to channel users;
  • There may be issues with TLS/SSL itself in server or client configuration or architecture which compromise its ability to provide effective transport security at the network level (there have been several published attacks against SSL recently – see here).

This is not an authoritative list, so before using +S as part of any channel which requires strong anonymity, please ensure you understand what it does and its drawbacks.

There are other security tools you may want to look at – you may want to consider using client plugins that provide additional encryption or route your connection through Tor. Tor also allows you to create spurious traffic to hide real traffic patterns. freenode provides its own hidden Tor node which means you can trust this connection as much as you trust freenode. Your IRC traffic with freenode via Tor is end-to-end encrypted from your Tor client to our Tor node. It does not pass through any third party nodes in unencrypted form.

Finally, unless you can trust everyone in a channel and are sure it is configured properly and you understand the other technical risks, do not rely on these channel modes exclusively. Security is generally layered; ensure you have good defense in depth and don’t rely on individual controls which may be a single point of failure.

Using other websites or services via Tor

Remember to always encrypt your traffic when using Tor as you have no control over who is running exit nodes and if they are doing traffic analysis on them. While your traffic to the exit node is encrypted and the ingress node can not read it, the exit node will always need to be able to remove Tor encryption. If your traffic is clear-text said exit node will be able to read it.