Database prune

Every couple of years, freenode likes to get out the shears and prune the services database. Recently we broke the 80,000 usercount barrier, but the services stats are way ahead:

Sat 13:35:46 -OperServ(OperServ@services.)- Registered accounts: 446777
Sat 13:35:46 -OperServ(OperServ@services.)- Registered nicknames: 557497
Sat 13:35:47 -OperServ(OperServ@services.)- Registered channels: 141373

We’ve noticed that nearly half of the accounts shown there haven’t been used in the past 6 months! More importantly, over the past few months many people have noticed significant waits when issuing certain services commands – and we’d like to fix that.

Hopefully, the services upgrade should help with this, but we’re going to coincide this with a database prune.

As of the services upgrade date, any nicks unused for > 150 days are at risk of being dropped. This includes grouped nicks. The easy way to avoid this happening is to use each of your grouped nicks (while identified to the appropriate account) within the next few weeks – and to drop those that you don’t need anymore!

The testnet (testnet.freenode.net, port 9002. 9003 for SSL) is running a database snapshot from mid-March and will be periodically updated from the production network. This database instance is being regularly pruned – so check there to see how your account will be affected (use /msg nickserv info on both the production and test networks to see the differences).

Remember that testnet isn’t running a real-time duplicate of the production network, so when you use nicks which would be expired on the production network, they will still appear expired on testnet until the next database snapshot is migrated. Don’t worry though – the actual pruning will only occur on the current database at the time of upgrade.

On which note.. an upgrade date hasn’t been formally fixed but we’re aiming for mid-May.

Thanks, and don’t forget to test the testnet!

Help us test our services upgrade!

Very soon we will be upgrading your favourite network helpers… (no not erry…): NickServ, ChanServ, Alis etc. They’re currently connected to our testnet and we need your help with testing, looking for any issues which may affect the production network.

You can connect to our testnet at testnet.freenode.net port 9002 (or 9003 for SSL)

The full changelog is rather long and not all of the features offered by atheme are loaded on freenode. So to help you out, we’ve pulled out the highlights which we think deserve attention:

  • NickServ’s certfp module. (see /msg nickserv help cert and this link.)
  • NickServ will now notify you in real time of failed logins.
  • NickServ’s previous limit on password lengths has been increased.
  • ChanServ will still hand over single-# channels to freenode-staff on expiration of the channel founders, but the method has changed.
  • NickServ & ChanServ’s ‘set’ commands have had a general reorganisation behind the scenes. Nothing should be visibly different but it won’t hurt to check them!

Please note that the services database on the testnet is probably more than a few days old. Don’t be surprised if recent changes you have made on the production network aren’t replicated there.

We’re all in in #freenode on the testnet so please come find us there if you have any questions or bugs.

Finally, look out for a followup blogpost (hopefully quite soon) with some important information on the upgrade itself and our planned database cleanup!

Thanks for using freenode!

P.s. a full list of changes from atheme ~5.1 to ~6 can be found here

Java webclient decommissioning

Following our successful switch of cloaking on our web gateway (http://webchat.freenode.net) to show the full IP address of connecting users (see this blog post), we have decided to transition our old and relatively unused Java client (pjIRC) to our webchat service. This will be done via a HTTP redirect.

Only around 30 users at a time can be found from the java client, hence as time goes on it makes less and less sense to continue to support this platform. We’ll be decommissioning the Java client on Sun 8th August.

Other pjIRC instances which connect to freenode will be unaffected. We are simply removing our version of the program.

If you’ve any concerns, queries or comments we’d love to hear from you either in #freenode or via support at freenode.net.

freenode is dead, long live freenode

After much time in development and testing, the move to ircd-seven is finally complete. The migration took place in the early hours of today, Saturday January 30th 2010.

I would like to express thanks to everyone who has helped us get here — those staff and users who have helped find and squash bugs, those who have done extensive load testing and those who have helped finalising documentation in preparation for the migration earlier today.

In particular I would like to thank the Charybdis development team and the ratbox contributors whose work left us with a brilliant ircd platform to build upon to create the more freenode specific ircd-seven. In no particular order my thanks go to:

dwr, Valery Yatsko <dwr -at- shadowircd.net>
gxti, Michael Tharp <gxti -at- partiallystapled.com>
jilles, Jilles Tjoelker <jilles -at- stack.nl>
nenolod, William Pitcock <nenolod -at- nenolod.net>
AndroSyn, Aaron Sethman <androsyn -at- ratbox.org>
anfl, Lee Hardy <lee -at- leeh.co.uk>
beu, Elfyn McBratney <elfyn.mcbratney -at- gmail.com>
Entrope, Michael Poole <mdpoole -at- trolius.org>
ThaPrince, Jon Christopherson <jon -at- vile.com>
twincest, River Tarnell <river -at- attenuate.org>
w00t, Robin Burchell <surreal.w00t -at- gmail.com>

And for leading the development efforts of ircd-seven, for putting up with my many quirky and often unreasonable requests:
spb, Stephen Bennett <stephen -at- freenode.net>

I’d also like to express my gratitude to the following freenode volunteers for the hard work they’ve put in to make the migration go as smoothly as possible. I’ve been amazed at the initiative and responsibility shown in this last phase. Your help has been invaluable and I feel privileged to work with you:

kloeri, Bryan Østergaard
Lorez, Mike Mattice
Martinp23, Martin Peeks
Md, Marco D’Itri

With the exception of port(s) 7000 and 7070 which are now being used for SSL, all other ports and DNS stay the same as it did prior to migration.

If you are a regular freenode user you will most likely be aware that there’s some user facing changes with the move to ircd-seven (and likely to have been annoyed by my global notices on the subject), you may wish to familiarise yourself with the updated FAQ and glance at some of these earlier ircd-seven related blog posts:

http://blog.freenode.net/2010/01/connecting-to-freenode-using-tor-sasl/

http://blog.freenode.net/2008/11/help-us-test-ircd-seven/

http://blog.freenode.net/2010/01/migration-to-new-ircd/

http://blog.freenode.net/2010/01/ircd-migration…-jan-30th-2010/

Again, thank you for helping out, however small or large your contribution may have been. We are celebrating the migration to ircd-seven with a special fundraiser “Give £7 for seven”. This campaign will end on February 7th 2010, until such time you may read more and donate here. Any donation of £21 or any multiple of £7 over £21 will receive a freenode t-shirt.

To all our users, thank you for using the network, and welcome to seven!

Connecting to freenode using Tor: SASL

With our change of ircd to the all new ircd-seven, we are trialling a new method of allowing users to connect to the network via Tor. This method brings a number of changes:

  • The only Tor hidden service is: the new p4fsi4ockecnea7l.onion.
  • You will need to have a registered and verified NickServ account to connect using Tor. Beyond this, no further steps are necessary.
  • You will need to use a SASL mechanism to identify to the server.

We have collected together scripts for irssi and mirc, while Conspire supports SASL natively. Scripts may be available for other clients in addition.

irssi

Download and install this script (cap_sasl.pl) and, after loading it, configure it using

/sasl set <network> <username> <password> <mechanism>

Supported mechanisms are PLAIN and DH-BLOWFISH.

mirc

A mirc script is available, taken from a forum post by Kyle Travaglini. You can retrieve the source here.

Instructions (adapted from that forum):

  • Place SASL.dll and sasl.mrc into your $mircdir.
  • Load sasl.mrc into your remotes.
  • Press F2 and configure the network, before connecting as usual.

If you have any problems, either pop into #freenode from a non-torified connection or drop an email to support AT freenode.net.

This method of connecting to freenode using Tor supersedes all previous methods, including Tor-GPG. We hope that this method of connecting via Tor will help to make it somewhat more accessible to you!

ircd Migration Sat Jan 30th 2010

In the coming weeks, we will be migrating freenode to our new ircd, ircd-seven.  Presently, freenode uses hyperion and efforts have been underway for some time move us off this platform for reasons of stability and functionality.  We are now almost there.

As users please be aware that during the migration all clients will be temporarily disconnected and will need to reconnect in order to move over to the new servers. For most of you this will happen as the old servers are shut down.

Please Note: While we will copy over channel modes and topics for registered channels (there will be no changes to the services database, all nick and channel settings with services will stay the same) we are unable to do so for channels NOT registered with ChanServ. If your project utilises non-registered channels for whatever reason, please make note of the topics and modes so you can make a manual transfer of these yourselves. For more information on registering a channel, see this post.

If you operate a channel on freenode and have any concerns, feel free to stop by #freenode to discuss any issues you might have.  If you run any channel utility bots, you may want to test them on the current testnet.  More information can be found here.

Important Changes

There are several significant changes users should be aware of in ircd-seven:

Channel quiets are no longer a modified version of bans but are now on their own list, queried with “mode #channel q”, and as such do not appear on the normal banlist.

After the migration, we will have ssl access available on the production network.

Identifying upon connection works as before but there are two new ways to do so: specifying username:password in the server password field will allow you to login to a specific account, and SASL authentication is also available.  Using SASL varies by client and is not supported in all clients.

The CAP command:

A brief summary:

  • The CAP LS command will list all client capabilities that are available to the client.
  • The CAP REQ :<cap1> <cap2> <...> command can be used to request one or more capabilities. The response to this will be either CAP ACK :<cap> <...>, or CAP NAK :<cap> <...>, depending on whether the request was successful.
  • A CAP name token can be prefixed by - to disable that capability. This was not available with hyperion’s CAPAB command.
  • CAP negotiation can take place either during connection and registration (as is required for SASL), or afterwards, to enable identify-msg.

For those implementing support for it, a full specification is at http://www.leeh.co.uk/draft-mitchell-irc-capabilities-02.html.

The IDENTIFY-MSG capability still exists but there is a new way to activate it.  It is now part of the CAP mechanism.   A script for irssi that understands both hyperion’s and seven’s identify-msg capability is available at http://adipose.attenuate.org/~stephen/ircd-seven/format_identify.pl.

The n= and i= prefixes are not used, instead ~ is prefixed to a non-identd username as is common in most other ircds.

For further information on changes that might impact you please visit http://freenode.net/seven-for-hyperion-users.html

As always, thank you for using freenode, and see you on the other side!

Web chat updates

Over the last few  weeks we have had quite a bit of feedback from our new web chat client.  As a result of this we’ve been able to feed back requests to the qwebirc developers who have been able to add many requested features:

  • Optional Nick colour support
  • Optional join, part and quit message hiding
  • Optional last position indicator to track which content is new since you last focused on IRC
  • CSS changes to highlight messages from yourself
  • https support
  • NickServ authentication

Some of the optional features are disabled by default, but can be enabled in the option pane, accessible from the menu (top left).

Help us test ircd-seven!

As many of you will have noticed, our current IRC server software, hyperion, has been showing its age for some time now. Expectations for its eventual replacement are nothing if not high — hyperion contains a great many features not found elsewhere, most of which are fairly unique to the way in which freenode operates, so anything that wants to take over from it must provide all of these, in a more robust, maintainable and future-proof package.

Charybdis looks like a good start — it’s a modern, modular IRC daemon supporting many of hyperion’s strange features, and built on top of ircd-ratbox, which gives it a good heritage of stability and scalability. ircd-ratbox is perhaps best known for powering the majority of EFNet, which seems to make it an excellent base on which to build.

However, neither ratbox nor Charybdis implements freenode’s more unique features, such as ban-forwarding or hidden IRC operators. So, some work is needed.

Enter ircd-seven. Seven is based on Charybdis, with the features freenode needs added in. Channel operators and network operators alike should recognise most of the useful, and heretofore unique, features of hyperion, without many of the bugs and oddities that have become an unfortunate fact of life.

Development and internal testing of seven has been ongoing for some time, and we’re now ready to open up testing to a wider audience. The test network is currently running on testnet.freenode.net, port 9002 for normal connections or 9003 for SSL connections. This is a new server, sharing no code with the current software, so all aspects of it need thorough testing, both that it works, and behaves in a way consistent with how most people want to use it — this last aspect is particularly difficult to do in small-scale private testing.

ircd-seven is designed to be capable of everything hyperion is, but not necessarily as a drop-in replacement. Some functionality is still available in a different form, or with a different interface. The most notable differences for users are summarised below:

SSL support
seven supports SSL, for client and server connections. Users connecting via SSL will get user mode +Z to denote this.

Channel bans and quiets
Channel mode +q (quiet) is now sent as a separate mode — hyperion’s translation of +q foo to +b %foo is gone. Extended ban types are supported for all ban-like modes (+bqeI). These extended masks begin with $, followed by an optional ~, to negate the match, and a single letter denoting the type of match to do. For example:

  • +b $r:Lee* will ban any client whose realname (gecos) field begins ‘Lee’. This is equivalent to hyperion’s +d mode.
  • +I $a:spb will set an invite exception for any client logged in to services as spb.
  • +q $~a will prevent any user not logged in to services from speaking. This is roughly equivalent to hyperion’s mode +R.

Forward channels for bans are now delimited with $ instead of hyperion’s !, and can be used with extended ban masks as well. Setting and unsetting of bans via the hyperion syntax (nick!user@host!#channel) is supported — it will be translated to nick!user@host$#channel.

Identified status
There is no user mode +e. The IRCd keeps track of the account name of every user who is identified to services, and uses this to determine whether a user is identified or not. The ‘is identified to services’ line in WHOIS output is no longer present; there is, however, a line containing the account name if the user is logged in.

Identifying on connect
Using a NickServ password as a server password still works as it does in hyperion. However, there are two new mechanisms:

  • You can specify : in the server password field, to log in to a specific account. This removes the requirement to connect using a nickname that is grouped to your services account.
  • seven supports SASL authentication, to log in to services during the connection process. This requires client support; a script for Irssi to do so is located here. Conspire supports this natively. Other clients, as far as I’m aware, do not.

Username prefixes
The n= and i= prefixes are not used; instead ~ is prefixed to a non-identd username, as in most other daemons.

IDENTIFY-MSG
The identify-msg capability is still present, but the way to enable it has changed — it is now part of the same CAP mechanism that is used to control SASL and multi-prefix capabilities. A script for irssi that understands both hyperion’s and seven’s identify-msg capability is available here. Conspire will also support this natively once w00t remembers to apply the patch.

[Announcement] Group registration and fundraising.

A quick note on a few current topics. First, new manual group registrations have been, if anything, slower than before. We apologize for this; we’re working on ramping up the freenode-registry project and concentrating on coding for the moment. The code is in the early prototyping stage and is going fairly slowly, though it’s beginning to pick up speed. However, the current manual group registration process is suffering while we work on the code which will eventually automate our processes. We’ll try to pick off some of the more active projects in the group registration queue and get them moving again. Thanks in advance for your patience.

A new social channel, #freenode-groups, has been set up for current holders of primary channel group cloaks. If you’re not in that category, the channel will forward you to #freenode, the old network staff social channel, which is actually currently forwarded, in turn, to ##bookmark (on a temporary basis) while we sort out which channels the network project is going to use.

Also, you may have noticed that the 2006-2007 PDPC fundraiser has not yet begun. Preliminary budgeting suggests that we’re very much on track for 2006-2007. We’ll likely have one or two small fundraisers during this fiscal year rather than a single large one. We’ll keep you posted, and you can help us stay on track by making a donation now instead of later.

Finally, the staff of freenode and PDPC want to thank you for using the network. We’re grateful that you continue to use the network in record numbers, and we’ll keep doing our best to help things run smoothly. Have an excellent week!