denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_news2012-01-06 07:10 am

Dreamwidth Update: 6 January 2012

Hello, Dreamwidth! We're ringing in 2012 with a collection of useful information and news from the DW head office (aka my house) and auxiliary offices (aka Mark's house and Fu's house). Behind the cut you will find information on:

* Development, Coding, and Styles
* Community and Icon Importing
* Site Growth and Stability
* Invite Codes
* SOPA
* The Ada Initiative
* Paid Accounts, PayPal, and DW Business
* The Great Community Rec-O-Matic



Development, Coding, and Styles



We start this week's news by bringing you the 29 Dec 11 - 5 Jan 12 code tour, detailing the bugs that were resolved this week.

Most of this week's development effort went to more long-running projects (and thus the bugs weren't completely resolved yet and don't appear on this list), but the biggest highlight this week includes finally having resolved that really annoying bug where interest counts weren't being updated properly, leaving us with a number of interests being listed as having 16,777,215 accounts having that interest listed. It was a bear to track down, but thanks to some amazing detective work from a team of people, with heavy lifting done by [personal profile] exor674 on the code end and [personal profile] alierak on the sysadmin end, interest counts should now be a lot closer to reality.

For those who are just joining us: yes, you too can come hack on the DW code! We offer hosted development environments so you don't have to faff around with installing the code on your own development machine (or have your own development machine), and our Bugzilla install is public for those who are looking for a project to pick up. The Getting Started and Development categories on the DW volunteer wiki have useful information on the development process, and [site community profile] dw_dev and [site community profile] dw_dev_training are the communities to go to on DW itself. Or, if you'd just like to browse the code, you can read through the Mercurial repositories; the two repositories that most development happens in are dw-free (for the freely-available, open-source-licensed code that primarily runs the service) and dw-nonfree (for dreamwidth.org-specific code, or things that are part of our site branding or identity).

DW backend development is mostly done in Perl, with a little bit of Javascript on the frontend here and there. If your coding skills lean more towards HTML and CSS (or S2, the custom layout language shared by DW and LJ), you may be interested in creating or converting themes and styles. If you've created a journal layout that you'd like to offer as a system style, you can submit it to [site community profile] dreamscapes. Or, if you'd like to wrangle others' submissions into a form that can be used on the site, you can take the layers and styles that people submit and convert them into a patch that can be applied to the codebase. (There's a guide for people patching styles, which assumes a level of familiarity with both S2 and the process of working with the code and creating patches -- we'll work on getting the process explained in a more beginner-friendly fashion, but you can always ask in [site community profile] dw_dev_training for help.)


Community and Icon Importing



After last week's announcement that community importing was available, [staff profile] mark spent a few days coding his fingers off to improve and optimize the importer to better handle larger import jobs. Thanks to his tireless work, we've been able to lift one of the restrictions we had to place on community import: communities that have over 100,000 comments can now be imported.

If your community has more than a few hundred thousand comments, the comment import may still fail. We're working with the admins of a very large community to work out the last of the problems. We're asking for now that if your community has more than 500,000 comments, please hold off on doing an import for another week or so -- the code won't enforce the limits anymore, but it will definitely make everybody's lives easier.

We also should also be able to lift the restriction that only paid accounts or paid communities can use the community importer in another week or so, once we're fully satisfied with the optimizations we've made -- stay tuned there. Mark's also actively working on the ability to "claim" your posts and comments that were imported and attributed to your OpenID account, so that should be available soon as well.

Meanwhile, we've made a change to the way icon imports are handled. Previously, if you had more icons available on the remote site than you had icon slots on DW, the site would only import your default icon. We ran into some problems this week with LJ reporting the wrong number of icons if you deleted some icons recently (so if you had 102 icons on LJ, then deleted 3 to bring it down under the 100 icon slots available to a paid account on DW, the system was still reporting you had 102 icons and thus the import was erroring). Rather than spend hours and hours of our time troubleshooting, we just lifted the restriction, and now import all the icons you have on the remote site.

As many people have noticed, this is allowing some people to circumvent the restriction on icon slots and use more icons than their account type allows. The reason we don't run the process to deactivate icons immediately upon import is because at that point, we don't have enough data about the imported posts and comments to tell which icons are the most-used icons in the journal. (When deactivating icons because an account expires, the least-used icons are deactivated first.) Rather than just guessing randomly at which icons should be deactivated, we made the deliberate decision to let it slide for now, then clean up in a week or two once the rush to import has died down a bit. (The extra icons won't be deleted; they'll be deactivated, precisely like what happens when a paid account expires.)

If you import your icons and it'll bother you to have icons available now and have them be deactivated later when we do the cleanup, or if you'd like to help us out a bit, you can force a deactivation by visiting the Edit Icons page while logged into your account. This page loads the routine that checks your current icons against your total number of icon slots, so will deactivate (or activate, if for some reason an account upgrade doesn't reactivate inactive icons) icons until you have the right number.

We know this will let some people have access to more icons than their account level allows for a bit, and thank you to those who reported the situation! It really did just work out to save us a lot of time to do it this way, since we had a lot of people who were having import problems and troubleshooting the issue would've taken valuable time away from other projects.


Site Growth and Stability



[staff profile] mark made an excellent post about DW's load vs capacity, and how we're doing in terms of being able to support all the new traffic we've seen this month. The great news is, we're handling the increased traffic wonderfully: we've seen a few bumps along the past few weeks (everybody's watching [site community profile] dw_maintenance for details, right?) but on the whole, things have been ticking along nicely.

In the past few weeks, we've added two new webservers and two new database servers, bringing the total number of machines up to 12: four database machines (since we add dbs in pairs), four machines that work as webservers only, two load balancers (the load balancer + a backup load balancer in case of failure that also functions as a webserver), the search machine (which also functions as a webserver), and the admin machine.

We constantly keep an eye on site performance to see if and when we need more hardware, and since we still rent our dedicated servers rather than colocate our own hardware with a hosting facility, we can get our hosting provider to add more hardware when we need it -- it's more complex than it would be for you to rent webhosting for a small project, but it's not as complex as it would be to buy additional hardware ourselves and have to set it up.

We'd like to move to owning servers and colocating in the future, but for now, dedicated hosting is more cost-effective for us: the way the numbers work out at the moment, it'd take us nearly three years to "earn out" the initial hardware purchase, in terms of amount saved monthly, and there are a lot of practical benefits to dedicated hosting. (Mind you, there are a lot of practical benefits to colocation, as well, but those don't start to show up until you're much bigger than we are right now.)

So, in other words: we're really happy with the site performance right now, and we have a clear plan on how to increase our capacity as our traffic and load increases. [staff profile] mark also has a lot of experience with handling high-availability web applications, and he's very comfortable with how things are running. If you've been worried about our ability to handle this month's increased traffic, you can rest easy.

(Also, for the record, since we name our database clusters after stars and constellations, the new cluster, which is on the new database pair, is the Draconis cluster, joining Alpha Centauri, Betelgeuse, and Capella. If you're curious, you can ask the site Where am I? This has no real bearing on anything -- all of the clusters save for Draconis are on the same physical machine -- but dividing accounts by database cluster even if they're on the same machine makes certain maintenance tasks easier.)


Invite Codes



Speaking of site growth and how happy we are with site performance, many people have noticed that we haven't yet re-enabled the need to have an invite code to create a DW account. We'd intended to leave invite codes off for the second half of December as one of our regular no-accounts-needed account creation phases. After adding the additional servers necessary to support the traffic bump, our performance and capacity was large enough that we decided we could support open account creation for longer.

So, invite codes will definitely be off for a little while longer -- this means that anyone can create a Dreamwidth account without needing an invite code for now. Rather than specify an exact time period for how long invite codes will be off, we've decided to just let open account creation run for an unspecified time, as long as this increased demand keeps up and as long as we're happy with the site performance. (And, of course, as long as the [site community profile] dw_antispam team is able to keep up with the spammers that open account creation inevitably attracts -- all praise be to our dedicated team of spamwhackers!) We'll let you know when we turn invite codes back on.


SOPA



SOPA (the Stop Online Piracy Act) and PIPA (the PROTECT IP Act) are two bills pending in the US legislature to force online service providers (such as Dreamwidth) to remove enormous amounts of content from, and block access to enormous amounts of, the internet, based on the demand of intellectual property rightsholders in an attempt to combat digital intellectual property infringement.

The Electronic Frontier Foundation has a more detailed explanation about why these bills would make bad law: Internet Blacklist Legislation. Dreamwidth joins hundreds of other companies and organizations, both inside the tech industry and out, in standing opposed to these attempts to censor the free and open internet.

We'd like to take a moment to ask you, if you live in the US, to write to your legislators and ask them to oppose the bills.


The Ada Initiative



We said a few weeks ago that we'd be donating 10% of December's gross income to the Ada Initiative, a new nonprofit corporation devoted to supporting women in open technology and culture. (Disclaimer: I volunteer on the board of directors, but I receive no compensation for doing so.)

We're pleased to announce that thanks to a record-breaking month of sales (which we totally did not expect, omg!) we are donating $4625 to help fund the Ada Initiative's 2012 programs. This makes us incredibly happy! Thank you to everyone who bought paid time during December and helped increase that number. Meanwhile, if you'd like to contribute more directly, you can still donate to TAI.


Paid accounts, PayPal, and DW Business



We're obviously very pleased about the aforementioned record-breaking month of sales, and I wanted to say thank you to those who have bought paid time -- it was a fabulous way to close out the year, and looking at the figures on our payment backend has been very, very encouraging lately.

For those who have been wondering what we do with the money from paid accounts: there are two major categories of expenses we have on a recurring basis. The first is professional services; this covers things like processing fees imposed by our payment processor, as well as hosting and backup services. (The processing fees work out to be around 7-9% of all payments, between the flat fees and percentages from our merchant bank account and the flat fees and percentages from our payment gateway; the hosting and backup services run around $4500 a month with the latest upgrades.) The second category is salary: [personal profile] fu is employed by DW fulltime, plus we pay our backup sysadmin and one of our Terms of Service people on a retainer basis as contractors.

We also pay me a very small amount monthly to be working on things full-time; previously [staff profile] mark was also working full-time on DW, but he needed to take a non-DW job because at the time we couldn't pay him what he needed to be earning to support his family. (I'm lucky; my wife has a very good, very stable government job, and we live in a much less expensive area of the country.)

We post an annual end-of-year analysis of our financial position to [site community profile] dw_biz every year, which I will start working on as soon as the end-of-year bookkeeping is finished. Before this month's payments explosion, and aside from the one-time income from our May '11 Seed Account sale (which we did in order to replenish our "war chest", the money we keep in reserve against future disaster, after nearly three months of being unable to accept payments in early 2011), we were already on target to take in slightly more than we spent in 2011. (Which was incredibly exciting, let me tell you!)

This past month's increase in income has allowed us to increase our server power, continue to restock the war chest, and invest in some infrastructure upgrades. (Plus, we bought [staff profile] mark a pizza while he was hacking on community import.) Beyond that, we don't want to make long-term plans counting on this level of income as a given, but we're really hoping that if this level of support continues we'll be able to get [staff profile] mark back on DW work full-time and be able to pay me market rate for my full-time work on DW.

Meanwhile, many people have asked us about why we don't accept PayPal for payments, and I've written up a longer post about the answer to that question: Why you can't pay for your account with PayPal or Google Checkout. There's some workarounds for people without credit cards in that post.


The Great Community Rec-O-Matic



We've done this before, but given how many people are just joining us, it's time to run another round of the Great Community Rec-O-Matic.

Here's how it works:

* If you're looking for new communities to join, comment here with a list of some of your interests, and the kind of community you enjoy reading and participating in.

* Or, read through the other comments and see what things other people are listing. If you see someone you think would be a perfect match for a community you admin or participate in, comment back to them and point them at the community!


*

That's it from us for another update! As always, if you're having problems with Dreamwidth, Support can help you; for notices of site problems and downtime, check the Twitter status page; if you've got an idea to make the site better, you can make a suggestion.

(Also, I am very close to having followed the hobbit tradition of giving presents to other people on my birthday -- if I'd started this a day earlier I could've posted it on the 5th. Thanks to everybody who sent me birthday wishes!)

We (and my runaway abuse of parentheses) will see you next week for our next update!
0jack: Primitive webcomic depiction of a wide-eyed girl in a pink dress against a rainbow backdrop. She looks ecstatic. (JOY.)

[personal profile] 0jack 2012-01-06 12:57 pm (UTC)(link)
These posts make me happy. That is all. <3

Happy birthday and Happy New Year!
john: Kosh the Vorlon from Babylon 5. Caption: YES. (YES)

[personal profile] john 2012-01-06 01:01 pm (UTC)(link)
Hurrah! Although Ambassador Kosh wishes me to point out that you have typoed "Centuri" above, and Ambassador Mollari is making rather a noise in the diplomatic sector of the station.

[personal profile] ex_witchiest447 2012-01-06 01:01 pm (UTC)(link)
You are all so amazing ♥
ilyena_sylph: snowflake on blue background. no, not a special one. (Art: snowflake)

[personal profile] ilyena_sylph 2012-01-06 01:04 pm (UTC)(link)
I love y'all.

Yay for all of this, and all the thought you put into this place!
haruka: (zz - gerbil girls)

[personal profile] haruka 2012-01-06 01:06 pm (UTC)(link)
I always get excited when I see your updates in my inbox. ^_^ Thanks for all your hard work!

If anyone doesn't get an answer to their community inquiries here, they're welcome to pop over to [community profile] create_my_comm and ask someone to make their desired comm for them. :)
Edited 2012-01-06 13:07 (UTC)
jhumor: (Default)

[personal profile] jhumor 2012-01-06 01:07 pm (UTC)(link)
(Also, I am very close to having followed the hobbit tradition of giving presents to other people on my birthday -- if I'd started this a day earlier I could've posted it on the 5th. Thanks to everybody who sent me birthday wishes!)

Happy Birthday! Also? It can still count for Orthodox Christmas XD
divinesquishy: (Default)

[personal profile] divinesquishy 2012-01-06 01:11 pm (UTC)(link)
I love this place ♥♥

Right, shall try to make use of the Rec-O-Matic 8D
I'm looking for
- Kaizoku Sentai Gokaiger
- Prince of Tennis
- Super Junior
- miss A
So if anyone has fanworks and/or discussion places for these at hand it'd be awesome if you could point me there :o
psyco_chick32: (MMPR - OT3)

[personal profile] psyco_chick32 2012-01-06 01:14 pm (UTC)(link)
Mark's also actively working on the ability to "claim" your posts and comments that were imported and attributed to your OpenID account, so that should be available soon as well.

This is awesome <3

Happy belated birthday!
psyco_chick32: (Gokaiger - Luka/Ahim)

[personal profile] psyco_chick32 2012-01-06 01:17 pm (UTC)(link)
While it appears the Gokaiger fan comm is still getting up and running, [community profile] finalauraburst is the Sentai Fandom Newsletter (fic, discussions and subs links)
divinesquishy: (Default)

[personal profile] divinesquishy 2012-01-06 01:20 pm (UTC)(link)
Awesomeness, thank you! :D I'm fairly new to Super Sentai in general, so that looks like a good place to get more into it :3
bread: vuvuzela (Flora)

[personal profile] bread 2012-01-06 01:21 pm (UTC)(link)
Wow normally I ignore the newsletter emails that come to me, but that was oddly fun to read.
haruka: (luka-happy)

[personal profile] haruka 2012-01-06 01:22 pm (UTC)(link)
I know of:

[community profile] princeoftennis
[community profile] sentai
[community profile] letsgokaiger

and I'm not sure if you're looking for individual fanworks to read, I've got a lot of PoT and Gokaiger on my fic journal: [personal profile] harukafics
hellkitty: (rainbowrrose)

[personal profile] hellkitty 2012-01-06 01:30 pm (UTC)(link)
I can't express enough how impressed I am at the yeoman's work y'all have been doing. Despite what must be a MASSIVE load of imported stuff, DW has never been inaccessible to me the entire time (oh it's loaded slowly, but that's different!)

I also want to thank you for making a stand against SOPA/PIPA. Is there a DW comm that has taken up analyzing it? If so, let me know, because the implications are terrifying.
camwyn: Me in a bomber jacket and jeans standing next to a green two-man North Andover Flight Academy helicopter. (Default)

[personal profile] camwyn 2012-01-06 01:30 pm (UTC)(link)
The large RP community I mod for at LJ is pleased to hear about the restriction being lifted. Thank you!
ursamajor: Remus with a book (just trying to read)

Betelgeuse, Betelgeuse, Betelgeuse!

[personal profile] ursamajor 2012-01-06 01:32 pm (UTC)(link)
Am totally looking for communities of people talking about library and information science issues - not necessarily limited to traditional libraries! Basically people who geek out about things like information organization and presentation, knowledge management, taxonomies and ontologies, the user experience, etc. I do know about [community profile] librarians, though it's been a bit quiet there - if there aren't other library-related comms out there, perhaps we can rejuvenate this one?

In return, I offer my fave comms:

- [community profile] white_lotus: all fics Avatar (the REAL Avatar, the one with Airbending ;) )
- [community profile] fresh_haul: farmer's market pr0n. <3
- [community profile] dreamwidth_haikai: meta about DW in the form of poetry

And for residents (past, present and future) of Cambridge and Somerville (MA), I offer up [community profile] camberville. Let your neighbors know about fun things going on! Reminisce about Bartley's Burgers and how much better it is to watch the fireworks on the north side of the river! Kvetch about the dratted Red Line shuttles! Plot in-person Dreamwidth meetups! (My own pledge to contribute more frequently: I will regularly round up all of the new hyperlocal restaurant and food truck openings that I hear of. :) )
Edited (srsly, the universal bonding experience of Cambervillains is Dealing With The Em Bee Tee Ay) 2012-01-06 13:38 (UTC)
divinesquishy: (Default)

[personal profile] divinesquishy 2012-01-06 01:33 pm (UTC)(link)
Ooh, I think I've read some of your Gokaiger fic on AO3 :D
Thank you ♥
haruka: (basco-smile)

[personal profile] haruka 2012-01-06 01:35 pm (UTC)(link)
You're welcome, and thanks for reading! I hope you'll enjoy Dreamwidth!
moem: A computer drawing that looks like me. (Default)

[personal profile] moem 2012-01-06 01:37 pm (UTC)(link)
Welcome to Dreamwidth, where the news posts are oddly fun!
ashen_key: ([QoS] Camille is going 'ah-huh')

[personal profile] ashen_key 2012-01-06 01:39 pm (UTC)(link)
Now, hopefully we don't crash DW whenever we get around to importing it... *wry grin*
azurelunatic: Vivid pink Alaskan wild rose. (Default)

Communities!

[personal profile] azurelunatic 2012-01-06 01:39 pm (UTC)(link)
These are my general recommendations for anybody who's looking for things to read and do and thinks the topic sounds interesting:

[community profile] followfriday, which is the DW version of the Twitter phenomenon, and sometimes has awesome recommendations. (Seems to have slowed down some.)
[community profile] omnomnom, for delicious cooking and baking.
[community profile] bitesizedcleaning, for tackling large and recurring housecleaning adventures in steps much easier to sustain than "Clean ALL the things!"
[community profile] bestthingever, for pan-fannish reccing.
[community profile] sauce, for general reccing of awesome things.
[community profile] capslock_dreamwidth, FOR ALL-CAPS HILARITY AND YELLING.
[community profile] fandomcalendar, for every fannish and vaguely related event members can think of.
[community profile] metaquotes, for funny things across the site (a little slow, but good).
hara: (Gundamphile)

[personal profile] hara 2012-01-06 01:40 pm (UTC)(link)
Thanks for all the hard work you guys do, and your wonderful communication. I've been so consistently happy and impressed with everything you post!
karanseraph: (ramirez will bust your pinata)

Rec-O-Matic

[personal profile] karanseraph 2012-01-06 01:40 pm (UTC)(link)
Any recommendations for fandom-related communities for:

1980's cartoons, in general
Thundercats
Transformers
Jem & the Holograms
G.I. Joe
My Little Pony

Comms for old + new series within these franchises would be cool, too.

Also anything related to the DCAU or DCnU would be interesting. Especially Bat family or the female heroes.

Page 1 of 27