Network Status Update

Good afternoon to all. The network is currently experiencing extreme stability problems. Our lead coder is currently looking over core files is gdb and working on resolving the issue. However, expect the network to be extremely volatile until he has a fix in hand. Apologies for this inconvenience. A week after the upgrade, while we’ve collected a number of noncritical bugs, today’s critical problem is our first of the new release. We’ll get the problem resolved as quickly as possible.

Thank you again for your patience, and thank you for using freenode!

Edit:

NETWORK STATUS UPDATELatest news: We’ve got a working upgrade for the server code, dancer-ircd 1.0.31+pre24 and we’re preparing to upgrade. We’ve installed the new code on the primary hubs to filter out problem state as it travels the network. The next step will be a global restart of the main rotation servers. We’ll follow that with the smaller regional servers and the IPv6 servers. Please bear with us as we restore the network to more reliable operation.

 Edit:

NETWORK STATUS UPDATE

With what we hope will be the final status update of the evening, the network appears restored and is functioning nominally. In our haste to bring out the new code, we believe that some of our small regional servers may have inadvertently been compiled with unnecessarily small user limits; but this should not affect the majority of users and we’ll handle any problems on a case-by-case basis.

We anticipate at least one more upgrade, scheduled with as much advanced notice as possible. We’ll keep you posted.

Thanks again for your understanding and patience and, as always, we appreciate your using the network.

[Scheduled Maintenance] IRCD Upgrade

Hi, all. Yesterday at about 10pm UTC we posted a global notice to let people know that we were considering an upgrade this weekend to pre-1.0.32 dancer-ircd. We continue to experience clonebot attacks, and the additional features in the new release will be helpful. We directed people to channel #upgrade and asked for questions and feedback. After answering questions and taking comments for about an hour, it seemed that the response to the idea of trying the upgrade was pretty favorable.

The release candidate code can be found here, and we provided information on new server features in a previous news item. It seems from our experience in this and other development cycles that the only real way to get production-load testing in dancer-ircd 1.0 is to go ahead and try the conversion, once we’re pretty certain we have reliable code. The code looks fine in the test environment, and delaying will not help us with kiddie attacks.

The plan is to try today (Sunday in most venues), converting the small European servers and Australia and a couple of small servers in the US, and then the larger servers in the main rotation. The old protocol is not fully-compatible with the new protocol, so servers which have been converted will be connected to other converted servers, but we’ll have two networks until the conversion is complete. It could take a couple of hours if all goes well.

Obviously, there is no guarantee we’ll be successful. If need be, we’ll back up to the 1.0 environment, which should take about a half hour. If we’re able to stay up using pre-1.0.32, we’ll just do a bit of cleanup next weekend, which may involve restarting the servers one at a time, but should not require a partition of the network.

It’s suggested that you not change nickname and channel configuration in NickServ and ChanServ while the conversion is going on. Please stay tuned for global and server notices, and we’ll probably provide more detail still on wallops (/mode yournick +w). Thanks, and thank you for using freenode!

Edit:

Hi all. Update was successful. One known bug; +q quietly supersedes +b. We’ll get a fix out for that as soon as possible, please bear with us. Thanks to everyone for their patience and understanding.

With the usual abominable timing exhibited by the real world, one of our US hubs will need to reboot this morning to patch a security hole. We believe this will occur at about 07:00 UTC this morning. Apologies for the inconvenience, and thank you again for your patience.