Server hosting and trust

For the purpose of disclosure we have had to make the difficult decision to discontinue a long-standing relationship with a server sponsor.

As a freenode user you may be aware that our set-up is somewhat untraditional and differs from that of many other IRC networks; servers are sponsored by various companies and educational institutions across the globe and all our infrastructure is centrally managed by the freenode infrastructure team. Generally speaking we do not provide o:lines or other privileges to server sponsors. Whilst it is possible for a sponsor contact to also volunteer as a staffer on the network such recruitment is independent of any server hosting.

Our staff are expected to work together closely and communication is key in any freenode relationship, be that with users, among staff or with sponsor contacts. It is important to us to be consistent in the way we provide support and apply policy and we expect all volunteers to be intimately familiar with our policies, procedures and philosophies — which in turn means that senior staff invest a lot of time in ensuring that any new recruits are given adequate support when getting to know the ins and outs of the network and what being a freenode volunteer entails.

Unfortunately one of our server sponsors added an o:line for themselves on the server they sponsored and whilst we do not believe that this was done with any malicious intent, more through thoughtlessness/negligence and having forgotten the expectations set out on our “Hosting a Server” page we feel that we are unable to comfortably and confidently continue the relationship.

Our number one priority has to be our target communities, the Free and Open Source Software communities that have chosen to make use of freenode in their internet activities.

Whilst we do not believe and have no evidence to indicate that any user traffic or data has been compromised, we would of course encourage you to change your passwords if you feel that this would make you more comfortable in continuing to use our services.

We can only apologise for this happening and we’d like to assure you that trust is incredibly important to us and that we are incredibly embarassed that this situation arose in the first place.

As a result of this we have just replaced our SSL certificates, so if you notice that these have changed then this is the reason why.

We will of course take this opportunity to remind all our sponsors of our expectations when it comes to providing services to freenode and our target communities.

Again, we apologise for any inconvenience and we hope that any loss of trust in the network that may have resulted from this incidence can be restored and that your projects will continue to feel comfortable using the network in future.

 

 

The good, the bad, and the ugly…

Firstly, I would like to apologise for the interruptions the network has experienced in the last week (and continues to experience as we speak). I would also like to thank our incredible server sponsors for the time and dedication they have shown in helping us attempt to deal with the situation.

Sponsors — sponsors are the lifeblood of the network; without sponsors there would be no freenode. Unfortunately, the recent attacks have been significant enough for some of our sponsors to pull the plug as they were unable to continue providing the same level of assistance to the network as they had in the past. These kind of attacks can be costly for our sponsors; the disruptions soon have a financial impact for sponsors and their paying clients when service is disrupted. They are also costly in time and resources spent trying to alleviate the issues caused within their networks. To those of our sponsors who have had to discontinue sponsorship, in part or in full, I would like to thank you for the years of support. Not just for freenode but for the Free and Open Source Software Communities and we can only apologise for the difficulties your organisations have experienced as a result of these recent attacks.

Free and Open Source Software Communities — whilst sponsors may be the lifeblood of the network, the FOSS communities are our reason for being. Unfortunately they, along with our sponsors, are the ones suffering at the hands of the attacker(s) — it is their projects that are disrupted and affected and we can only apologise for the instability and disruption experienced by projects on the network in this last week.

freenode — ironically freenode is the puzzle piece that gets off lightly. We’re just a bunch of people passionate about FOSS — the network itself is devoid of feeling and whilst our volunteers do their best dealing with the aftermath of the attacks and try to keep the network up and running the reality is that in the grand scheme of things freenode is nothing. freenode is just a means to an end; the projects that have chosen to use freenode could easily go elsewhere, the volunteers who staff the network… well, they could easily go wherever their projects went — we volunteer for freenode because we’re passionate about FOSS, and the majority of us also contribute to one or several FOSS projects or have done in the past. For us it has never been about “freenode” — it has been about FOSS; and the projects we, as individuals, care about. We are all freenode users first, and staffers second.

If there was no freenode, there would be other alternatives — perhaps similar alternatives, perhaps very different alternatives. The FOSS communities are full of talented, passionate people and I have no doubt that we’d all find different ways to stay in touch and work on our projects even if there was no freenode.

That’s not to say we’re about to throw in the towel — we’ve all invested a lot of time and effort in the network and I am sure we will continue to do so for as long as there are projects wanting to use it and sponsors willing to help us.

I wish I could provide you with detailed information about the attacks and the cause of them — but these details are but a mystery to us and with nearly 90,000 users I’d be loathe to speculate as to who we might have annoyed… or how. For the time being, we intend to continue mitigating attacks where possible and continuing to endeavour to provide service as usual!

Once more, thank you for the support and the faith in the project — and thank you for the patience whilst our infrastructure team desperately tries to juggle our infrastructure around to bring back as much of our normal services as is possible at this point in time.

Merry Christmas

It was the day before Christmas, when all across the network not a creature was stirring, not even a troll…

Another year is coming to a close, a year of FOSS, a year of collaboration, a year of getting to know exciting projects and contributors, a year plentiful in conferences and events. A year in which community was at the heart of everything freenode and the PDPC did.

I am sat here in a candle lit room that smells of Christmas spice whilst sipping eggnog and stealing a moment to myself to reflect… when I first started using freenode, the network had around 1200 users — some years on we’ve passed 76,000. It is great to see so many projects make use of the service! The PDPC is doing some exciting stuff, FOSSCON was arranged for the second year running in 2011, FOSSEVENTS is still going strong and Geeknics are being held across the globe. Sometimes it is a bit daunting, everyone involved with freenode (and other PDPC projects) volunteer their time and skills to help the communities — and whilst this is great, the volunteer roster isn’t growing at a rate matching the increase in users. We will be doing another “Call for volunteers” in the New Year and we’re looking to find some awesome people to add to the team.

On a more personal basis, I’ve had the pleasure of participating in some brilliant “real life” events this year – from the always brilliant OGGCAMP, arranged by the amazing people behind the Linux Outlaws and Ubuntu UK podcasts. OGGCAMP is the sort of event which attracts “my sort of geek” — people with which I really enjoy spending time and socialising and I am already looking forward to next years! I’ve also enjoyed a number of Ubuntu-UK events, from the recent release party to smaller happy hours to the rather interesting Christmas meal at Dans Le Noir, where we ate a surprise menu (consisting of glow in the dark scallops, ostrich and blue shark to name some items!) in complete darkness before wandering off to enjoy a few pints of ale. Now, this is where freenode communities are great — I am an avid fan of the Ubuntu project, however, I have never used the distribution (though as a Debian user I am sure I’d get on with it just fine.. right?) but through freenode I have, over the years, come to know a lot of Ubuntu contributors and users, and over time discovered that these are people I really enjoy spending my time and people I am proud to call my friends! And I love their social events, they are a lot of fun! So thank you for letting me be a part of your community despite not really being “one of you” ;)

On the subject of freenode, communities and Christmas parties — as freenode volunteers are scattered across the globe, few of us meet on a regular basis but we tend to try make an effort to get together for dinner and drinks at events such as fosdem. This year, we decided to have a Christmas party, surprisingly we ended up with 21 people attending, most of which had never met each other in person before. Our volunteers flew over from America and Europe and we all had a smashing weekend filled with good food, nice drink and much laughter. Thank you all for coming!

In 2012 I look forward to learning about more projects, old and new. I look forward to attending yet more conferences and meeting more of you in person! I look forward to another year together with our volunteers and our sponsors, and of course our users.

On which note I shall wrap this entry up, thank you for using freenode and wish you all a Merry Christmas!

 

 

IRCD Upgrades

Over the next weeks we will be upgrading our servers to the next version of ircd-seven. This means restarting all our servers. Downtime should be minimal, and as we will not upgrade all servers at the same time this should not be as noisy as the upgrade from hyperion to ircd-seven was. When the server you are on is upgraded you will be disconnected, but should be able to reconnect immediately (most clients will do this automatically).

The following user-visible changes have been made since the versions in production:

– The channel quiet list is now sent using the new numerics RPL_QUIETLIST(728) and RPL_QUIETLISTEND(729) instead of overloading the same numerics as for ban lists. You may find that clients have to be updated before they will display this in a user-friendly format.
– Users who cannot send to a channel are now prevented from changing its topic, even when mode +t is not set.
– Sending a private message to another user while user mode +g is active will now automatically add an accept-list entry so that they can reply.
– Account names are now displayed in WHOWAS entries.
– Two new client capabilities are available: ACCOUNT-NOTIFY and EXTENDED-JOIN. These two together with the existing extended WHO syntax allow clients to track account names of other users who share a channel.
– Client flood control settings have been made configurable; you may notice them being stricter than before.

Once all ircds have upgraded we also plan to re-enable the +S channel mode, which only allows users connected using SSL to join.

Some more features will become available once we upgrade services, which will happen at some point after we have upgraded all irc servers:

– It will be possible to identify to services using SSL client certificates.
– ChanServ mode locks will be (mostly) enforced by the server. Instead of setting a mode and having ChanServ revert it immediately, you will not be able to change a locked mode.

Fixing #defocus

We want #defocus, the network social channel, to be a welcoming place for relaxed, easy conversation. Of late, we’re seeing that staff are acting in a policing role more than we are able to take an active part in the flow of the channel. This is something of a circle, as the more we are seen to be police, the harder it is for us to achieve what we want with the channel.

We’ve had some extensive discussions with #defocus regulars and ops on various channels around what we can do to improve the day-to-day atmosphere of the channel and bring it around to be what we want it to be, and one of the more immediate outcomes of these discussions was that we need to introduce more specific channel ops for #defocus, to work alongside staff in shaping the channel’s future.

To that end, the following users are now channel ops in #defocus, and are ready and able to asist when moderation might be required. _msh, jbroome , rww , Ttech, bburhans, frantk, gry, Fuchs , TheEvilPhoenix, Zuu.

Please help us in welcoming them to the #defocus team.

Happy Holidays!

Just a quick note to wish each and every one of our users, sponsors, donors, volunteers and projects for making freenode great.

It’s amazing being able to communicate and collaborate with such a variety of people and projects. YOU make the network what it is and help us provide a fantastic resource for FOSS communities.

I’d also like to say a special “Thank You” to Martin (Martinp23) Peeks and Richard (RichiH) Hartmann for squeezing into my size 7’s and keeping a tight grip on the steering wheel ensuring that we don’t weer off track and crash into too many icebergs during my leave from active freenode management duties. You’re doing a tremendous job, and I’m thrilled to see that you’ve got the support from the fantastic volunteer base and our exceptional sponsors.

Have a happy Christmas (or whichever holiday you do or do not celebrate) and a fantastic New Year, I hope it brings you everything you wish for!

Oh. And don’t forget, tis the season for giving!

Cheers,

Christel x

freenode is dead, long live freenode

After much time in development and testing, the move to ircd-seven is finally complete. The migration took place in the early hours of today, Saturday January 30th 2010.

I would like to express thanks to everyone who has helped us get here — those staff and users who have helped find and squash bugs, those who have done extensive load testing and those who have helped finalising documentation in preparation for the migration earlier today.

In particular I would like to thank the Charybdis development team and the ratbox contributors whose work left us with a brilliant ircd platform to build upon to create the more freenode specific ircd-seven. In no particular order my thanks go to:

dwr, Valery Yatsko <dwr -at- shadowircd.net>
gxti, Michael Tharp <gxti -at- partiallystapled.com>
jilles, Jilles Tjoelker <jilles -at- stack.nl>
nenolod, William Pitcock <nenolod -at- nenolod.net>
AndroSyn, Aaron Sethman <androsyn -at- ratbox.org>
anfl, Lee Hardy <lee -at- leeh.co.uk>
beu, Elfyn McBratney <elfyn.mcbratney -at- gmail.com>
Entrope, Michael Poole <mdpoole -at- trolius.org>
ThaPrince, Jon Christopherson <jon -at- vile.com>
twincest, River Tarnell <river -at- attenuate.org>
w00t, Robin Burchell <surreal.w00t -at- gmail.com>

And for leading the development efforts of ircd-seven, for putting up with my many quirky and often unreasonable requests:
spb, Stephen Bennett <stephen -at- freenode.net>

I’d also like to express my gratitude to the following freenode volunteers for the hard work they’ve put in to make the migration go as smoothly as possible. I’ve been amazed at the initiative and responsibility shown in this last phase. Your help has been invaluable and I feel privileged to work with you:

kloeri, Bryan Østergaard
Lorez, Mike Mattice
Martinp23, Martin Peeks
Md, Marco D’Itri

With the exception of port(s) 7000 and 7070 which are now being used for SSL, all other ports and DNS stay the same as it did prior to migration.

If you are a regular freenode user you will most likely be aware that there’s some user facing changes with the move to ircd-seven (and likely to have been annoyed by my global notices on the subject), you may wish to familiarise yourself with the updated FAQ and glance at some of these earlier ircd-seven related blog posts:

http://blog.freenode.net/2010/01/connecting-to-freenode-using-tor-sasl/

http://blog.freenode.net/2008/11/help-us-test-ircd-seven/

http://blog.freenode.net/2010/01/migration-to-new-ircd/

http://blog.freenode.net/2010/01/ircd-migration…-jan-30th-2010/

Again, thank you for helping out, however small or large your contribution may have been. We are celebrating the migration to ircd-seven with a special fundraiser “Give £7 for seven”. This campaign will end on February 7th 2010, until such time you may read more and donate here. Any donation of £21 or any multiple of £7 over £21 will receive a freenode t-shirt.

To all our users, thank you for using the network, and welcome to seven!

Happy New Year 2010

The New Year is arriving in various parts of the world, and we’d like to take this opportunity to thank the people who continue making freenode possible.

Our very dedicated and generous hardware and bandwidth sponsors, for whom the tail end of 2009 have been a particularly challenging time, we’re very grateful for the extra manpower you’ve all put in to help with the recent DDoS attacks the network has been experiencing. While we’ve lost some sponsors due to the costs involved over the attacks, we’d like to thank those for the time they were able to continue supporting our services and express our complete understanding for the decisions they’ve had to make in choosing to discontinue the support. For those of our sponsors who have been able to continue providing hardware and bandwidth we’d like to thank you for your generousity and for the patience while the attacks have been ongoing.

We’d like to thank all the PDPC supporters for their donations, Canonical Ltd and the Gallery project for their generous donations, as well as those donations from indiviual users which in 2009 enabled us to purchase some additional hardware and bandwidth and we hope that the support continues throughout 2010 and that we’ll be able to start making some progress with the work on our upcoming freenode live conference. Your support is invaluable to us and we’re grateful for the continued support. Should you wish to become a donor; you may make a donation here.

We’d also like to thank the freenode staff volunteers, past and present, for administering the network and putting in a lot of time to help both projects and end users with their freenode experience.

And finally, we’d like to thank the most important people of all — the many projects and users who make freenode what it is. 2009 saw us passing the 60,000 concurrent users mark and it’s fantastic to see that so many people use and contribute to the various FOSS projects on the network. Thank you all for using freenode.

We’d like to wish you all 12 months of happiness, 52 weeks of fun, 365 days of success, 8760 hours of good health, 52600 minutes of good luck and 3153600 seconds of joy! Have a very happy New Year!

When bots go bad..

First off, allow me to apologise to all users affected by the recent “client killing” rampage of our utility bot; Syn. She appears to have gotten into the Halloween spirit a bit too much!

You may have noticed a large number of people disconnecting from freenode with the reason ‘Nick collision from syn.’ We feel we should explain what happened.

For those of you not already familiar with her, syn is a utility bot that, amongst other duties, regulates gateway access to the network. This could be web gateways such as CGI:IRC or our own webchat, NAT gateways, or some conferences and shell services. One of the things that she does, for web gateways in particular, is to match the reported IP address (hex-encoded in the ident field) against network bans, and deny the connection if a match is found.

It was this particular part that had an unfortunate pair of bugs resulting in the incident you observed. Firstly, in using sscanf() to detect a hex-encoded IP address in the ident field, the validation was not quite strict enough — any ident that *began* with a series of valid hexadecimal characters (the digits 0-9 and letters a-f) would result in a number being decoded. In normal circumstances, this would be relatively harmless as the resulting IP is clearly invalid — in most cases, it would begin 0.0.0., and not match any network bans.

Unfortunately, there was a second bug introduced more recently as part of a performance fix. This meant that in certain cases, a K:line whose host part contained wildcards would incorrectly match against these invalid IP addresses.

Each of these, taken in isolation, would be relatively innocuous, and so they slipped under the radar and made it into production. The combination of the two, however, had rather disastrous results.

We apologise, and welcome you to castigate our developers and staff for our incompetence and for allowing these bugs to make it into production.

We want YOU!

We are currently looking to expand the freenode volunteer staff team, seeking people involved with the target communities we currently serve.

The freenode network has seen substantial growth in recent times and as such we are looking to add to our existing team of volunteers. freenode volunteers hail from a variety of backgrounds and come in all ages, it’s a diverse group of people and we all share a passion for Free and Open Source Software, Free Culture and Peer-Directed Project Communities. We are looking for peoplewho would complement the current set-up and make a refreshing addition to the team.

Each freenode volunteer has a individual role within the project; utilizing their strengths, experience and interests. No minimum level of privileges are guaranteed. The corner-stones of network operation is that of the role of ‘levelone support volunteers,’ who build the foundation upon which the rest of the network is based. All volunteers, regardless of seniority are encouraged to spend as much time as possible on levelone duties.

We are currently looking for:

  • Perl Developers — familiarity with catalyst, sql and git would be a bonus.
  • Support Volunteers — the first point of contact for users and groups looking for assistance with the network.
  • Community co-ordinators — to work closely with groups and projects to ensure they best utilize what we have to offer. This role will involve doing research for the podcast and fair bit of blogging, wordsmiths encouraged to apply!

Perl Developers — are wanted for work on freenodes’ Group Management System (GMS), if you want further information or would like to see how you can help, please join us in #freenode-gms or drop a line to code AT freenode DOT net.

Support Volunteers — if you want further information prior to applying, please come speak with us in #freenode, all current staff are voiced and we would all be happy to answer any questions you may have. Prior experience is not necessary, but familiarity could be a bonus. In particular we would love to hear from you if you are in a time-zone where we are currently short. This link gives an indication of grey areas which need covering, 2AM UTC to 6/7AM UTC in particular.

Community co-ordinators — if you want further information about this role, please get in touch with christel, JonathanD or Martinp23 in #freenode or private message.

Prior to applying

Please familiarise yourself with our guidelines, policies, procedures and philosophies as per our website: http://freenode.net and assess whether you feel you agree with what we are all about. We’re looking for someone who is able to work as a team but also happy to take initiative. We also ask that you read the section on our website relating to volunteering.

How to apply

If you’ve done the above and you are happy to continue, we ask that in the first instance you drop us an e-mail to volunteering AT freenode DOT net containing the following information:

  • Your Nick (Nickserv Accountname)
  • Your Name
  • Your Age
  • Why you would like to volunteer
  • Why do you currently use freenode (eg. projects involved with, user/contributor etc)
  • An indication of time committment you would be able to make
  • Optional: Make us laugh! Tell us a joke

Applications for Support Volunteers and Community Co-ordinators will CLOSE at midnight (UTC) on June 30th 2009 and we ask that you e-mail us prior to this deadline if you are interested in our current recruitment drive. We regret that we may not be able to accept all applicants, but ask that if unsuccessful in this round you consider re-applying in the future. Upon successful completion of “stage one,” you will be asked to partake in “stage two”, where you will be expected to fill out our volunteer questionaire upon which the final decision will be made. Thank you for your consideration, and for using freenode.