staffblog

We are currently experiencing heavy DDoS against several locations at which some of our servers are hosted. The attack is ongoing and cause a lot of disruption, both to users of the network and unfortunately to projects/companies/individuals whose infrastructure is hosted at the same locations as us. Our sponsors and our sponsors’ upstreams are working hard to try curb the attacks as best they can.

We will try keep this page updated with any significant information as and when we receive it, however, users of the network will also be able to receive (infrequent) status updates via global notice and slightly more frequent updates via wallops for those who have chosen to go +w (/umode +w or /mode yournick +w) will enable wallops in your irc client should you wish to see these. Global notices do not work on a opt-in basis, and are restricted to information we deem important, however for those of you who have absolutely no interest in what’s going on with the network you may /ignore *!*@freenode/staff/* notices in order to prevent global notices from displaying in your client.

We apologise for the inconvenience this no doubt causes for you and your project(s) and we would like to thank you all (in particular, our very generous and dedicated sponsors) for the patience and support while the issues are still ongoing.

As you are probably aware, we’ve been facing some fairly major splits today as there have been issues between some of our major hubs. We’ve rerouted these and are working on tracing down the cause of any other splits.  Please be aware, our staff are already hard at work on these issues and will resolve them as quickly as possible. Included here are 2 global notices about this matter.
-christel(i=christel@freenode/staff/exherbo.christel)-
[Global Notice] Hi all, we appear to be having some
connectivity issues with our main US hub, as a result of
this we are temporarily without  services, if this affects
your channel please contact staff in #freenode for
assistance. We’re looking into the issues as we speak. Thank
you for your patience.

-christel(i=christel@freenode/staff/exherbo.christel)-
[Global Notice] Hi all, we’re having some major issues with
connectivity at the DC hosting one of our hidden hubs, I’m
going to re-route around it,  which will cause about twice
as much noise as the splits already made. Apologies for the
inconvenience.

These and other issues are a large part of the reason for the upcoming migration to ircd-seven, and we still need your help in that regard.  We are still in need help testing the new ircd and working out the bugs.  If you would like to help out, have a look at this posting for information on how you can test the new ircd.

Thank you for using freenode, and have a great day!

We are aware of the recent DCC spam attempts, and we are working on this issue.  In the meantime, please do not paste the full DCC text you recieve in #freenode (or any other channel) as in many cases it can cause you to appear as a problem yourself.

Instead, feel free to report it by first verifying (using  the /whois command) that the sender is still online, and then reporting the sender in #freenode.  Please be aware that #freenode is a general help channel and we need to keep it clear of general chatter in order to support the many users of freenode.  If you wish to discuss anything other than an immediate support request, please find a more appropriate channel for the topic.

On this topic, please be sure to only accept DCC requests from trusted users and to be cautious about them at all times.  You may wish to consider filtering out the DCC requests using your clients ignore functions, or enabling umode +C to prevent CTCP messages from being received. If you would like help with these topics feel free to stop by #freenode or your clients support channel.

As always, thank you for using freenode.

We’re in the late stages of testing our new ircd, ircd-seven, which is intended to replace our current hyperion ircd, and we need your help!

While we have been testing it regularly against some basic loads, nothing really replaces users, and we need as many as possible to connect, try it out, and report back with any issues.  Please note that ircd-seven differs from our current software; hyperion — and some bits of behaviour may differ, if your project/channels rely on the use of bots on the production network we encourage you to also test these on the testnet!  We would really like to stress, there are some significant changes in the new ircd so please do test the full functionality of any bots you require at this time, as we’ll be moving forward with this new ircd in the future.

To connet, use testnet.freenode.net for ipv4 or ipv6. Port 9002 listens for regular connections, while 9003 listens for SSL.

We want a lot of traffic, and while we don’t normally encourage it — you are welcome to bring bots and drones en-masse! So bring in the bots, simulate traffic, join your regular channels, talk or spout nonsense. You can find us all in #freenode when you’ve connected to the testnet.

To connect from irssi: “/connect -ssl testnet.freenode.net 9003″ for ssl and “/connect testnet.freenode.net 9002″ for non-ssl.
To connect from xchat, first open a new tab, then “/server -ssl testnet.freenode.net 9003″ for ssl and “/server testnet.freenode.net 9002″ for non-ssl.

Thank you for using freenode and for helping us out, and freel free to drop by #freenode on either network to report any issues you might have with the testnet.

–update–

Some of you are asking about user and channel modes, many of which will have changed.  You can get a listing of the user and channel modes and what they mean with “help umode” and “help cmode” respectively.  Some clients will allow this directly, using “/help umode” or “/help cmode” but in many you will need to instead use “/quote help umode” or “/quote help cmode”  Some clients also use /raw in place of /quote.

First off, allow me to apologise to all users affected by the recent “client killing” rampage of our utility bot; Syn. She appears to have gotten into the Halloween spirit a bit too much!

You may have noticed a large number of people disconnecting from freenode with the reason ‘Nick collision from syn.’ We feel we should explain what happened.

For those of you not already familiar with her, syn is a utility bot that, amongst other duties, regulates gateway access to the network. This could be web gateways such as CGI:IRC or our own webchat, NAT gateways, or some conferences and shell services. One of the things that she does, for web gateways in particular, is to match the reported IP address (hex-encoded in the ident field) against network bans, and deny the connection if a match is found.

It was this particular part that had an unfortunate pair of bugs resulting in the incident you observed. Firstly, in using sscanf() to detect a hex-encoded IP address in the ident field, the validation was not quite strict enough — any ident that *began* with a series of valid hexadecimal characters (the digits 0-9 and letters a-f) would result in a number being decoded. In normal circumstances, this would be relatively harmless as the resulting IP is clearly invalid — in most cases, it would begin 0.0.0., and not match any network bans.

Unfortunately, there was a second bug introduced more recently as part of a performance fix. This meant that in certain cases, a K:line whose host part contained wildcards would incorrectly match against these invalid IP addresses.

Each of these, taken in isolation, would be relatively innocuous, and so they slipped under the radar and made it into production. The combination of the two, however, had rather disastrous results.

We apologise, and welcome you to castigate our developers and staff for our incompetence and for allowing these bugs to make it into production.