Entry tags:
Dreamwidth Update: 6 January 2012
Hello, Dreamwidth! We're ringing in 2012 with a collection of useful information and news from the DW head office (aka my house) and auxiliary offices (aka Mark's house and Fu's house). Behind the cut you will find information on:
* Development, Coding, and Styles
* Community and Icon Importing
* Site Growth and Stability
* Invite Codes
* SOPA
* The Ada Initiative
* Paid Accounts, PayPal, and DW Business
* The Great Community Rec-O-Matic
We start this week's news by bringing you the 29 Dec 11 - 5 Jan 12 code tour, detailing the bugs that were resolved this week.
Most of this week's development effort went to more long-running projects (and thus the bugs weren't completely resolved yet and don't appear on this list), but the biggest highlight this week includes finally having resolved that really annoying bug where interest counts weren't being updated properly, leaving us with a number of interests being listed as having 16,777,215 accounts having that interest listed. It was a bear to track down, but thanks to some amazing detective work from a team of people, with heavy lifting done by![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
exor674 on the code end and alierak on the sysadmin end, interest counts should now be a lot closer to reality.
For those who are just joining us: yes, you too can come hack on the DW code! We offer hosted development environments so you don't have to faff around with installing the code on your own development machine (or have your own development machine), and our Bugzilla install is public for those who are looking for a project to pick up. The Getting Started and Development categories on the DW volunteer wiki have useful information on the development process, and![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png)
dw_dev and dw_dev_training are the communities to go to on DW itself. Or, if you'd just like to browse the code, you can read through the Mercurial repositories; the two repositories that most development happens in are dw-free (for the freely-available, open-source-licensed code that primarily runs the service) and dw-nonfree (for dreamwidth.org-specific code, or things that are part of our site branding or identity).
DW backend development is mostly done in Perl, with a little bit of Javascript on the frontend here and there. If your coding skills lean more towards HTML and CSS (or S2, the custom layout language shared by DW and LJ), you may be interested in creating or converting themes and styles. If you've created a journal layout that you'd like to offer as a system style, you can submit it todreamscapes. Or, if you'd like to wrangle others' submissions into a form that can be used on the site, you can take the layers and styles that people submit and convert them into a patch that can be applied to the codebase. (There's a guide for people patching styles, which assumes a level of familiarity with both S2 and the process of working with the code and creating patches -- we'll work on getting the process explained in a more beginner-friendly fashion, but you can always ask in dw_dev_training for help.)
After last week's announcement that community importing was available,![[staff profile]](https://www.dreamwidth.org/img/silk/identity/user_staff.png)
mark spent a few days coding his fingers off to improve and optimize the importer to better handle larger import jobs. Thanks to his tireless work, we've been able to lift one of the restrictions we had to place on community import: communities that have over 100,000 comments can now be imported.
If your community has more than a few hundred thousand comments, the comment import may still fail. We're working with the admins of a very large community to work out the last of the problems. We're asking for now that if your community has more than 500,000 comments, please hold off on doing an import for another week or so -- the code won't enforce the limits anymore, but it will definitely make everybody's lives easier.
We also should also be able to lift the restriction that only paid accounts or paid communities can use the community importer in another week or so, once we're fully satisfied with the optimizations we've made -- stay tuned there. Mark's also actively working on the ability to "claim" your posts and comments that were imported and attributed to your OpenID account, so that should be available soon as well.
Meanwhile, we've made a change to the way icon imports are handled. Previously, if you had more icons available on the remote site than you had icon slots on DW, the site would only import your default icon. We ran into some problems this week with LJ reporting the wrong number of icons if you deleted some icons recently (so if you had 102 icons on LJ, then deleted 3 to bring it down under the 100 icon slots available to a paid account on DW, the system was still reporting you had 102 icons and thus the import was erroring). Rather than spend hours and hours of our time troubleshooting, we just lifted the restriction, and now import all the icons you have on the remote site.
As many people have noticed, this is allowing some people to circumvent the restriction on icon slots and use more icons than their account type allows. The reason we don't run the process to deactivate icons immediately upon import is because at that point, we don't have enough data about the imported posts and comments to tell which icons are the most-used icons in the journal. (When deactivating icons because an account expires, the least-used icons are deactivated first.) Rather than just guessing randomly at which icons should be deactivated, we made the deliberate decision to let it slide for now, then clean up in a week or two once the rush to import has died down a bit. (The extra icons won't be deleted; they'll be deactivated, precisely like what happens when a paid account expires.)
If you import your icons and it'll bother you to have icons available now and have them be deactivated later when we do the cleanup, or if you'd like to help us out a bit, you can force a deactivation by visiting the Edit Icons page while logged into your account. This page loads the routine that checks your current icons against your total number of icon slots, so will deactivate (or activate, if for some reason an account upgrade doesn't reactivate inactive icons) icons until you have the right number.
We know this will let some people have access to more icons than their account level allows for a bit, and thank you to those who reported the situation! It really did just work out to save us a lot of time to do it this way, since we had a lot of people who were having import problems and troubleshooting the issue would've taken valuable time away from other projects.
mark made an excellent post about DW's load vs capacity, and how we're doing in terms of being able to support all the new traffic we've seen this month. The great news is, we're handling the increased traffic wonderfully: we've seen a few bumps along the past few weeks (everybody's watching dw_maintenance for details, right?) but on the whole, things have been ticking along nicely.
In the past few weeks, we've added two new webservers and two new database servers, bringing the total number of machines up to 12: four database machines (since we add dbs in pairs), four machines that work as webservers only, two load balancers (the load balancer + a backup load balancer in case of failure that also functions as a webserver), the search machine (which also functions as a webserver), and the admin machine.
We constantly keep an eye on site performance to see if and when we need more hardware, and since we still rent our dedicated servers rather than colocate our own hardware with a hosting facility, we can get our hosting provider to add more hardware when we need it -- it's more complex than it would be for you to rent webhosting for a small project, but it's not as complex as it would be to buy additional hardware ourselves and have to set it up.
We'd like to move to owning servers and colocating in the future, but for now, dedicated hosting is more cost-effective for us: the way the numbers work out at the moment, it'd take us nearly three years to "earn out" the initial hardware purchase, in terms of amount saved monthly, and there are a lot of practical benefits to dedicated hosting. (Mind you, there are a lot of practical benefits to colocation, as well, but those don't start to show up until you're much bigger than we are right now.)
So, in other words: we're really happy with the site performance right now, and we have a clear plan on how to increase our capacity as our traffic and load increases.mark also has a lot of experience with handling high-availability web applications, and he's very comfortable with how things are running. If you've been worried about our ability to handle this month's increased traffic, you can rest easy.
(Also, for the record, since we name our database clusters after stars and constellations, the new cluster, which is on the new database pair, is the Draconis cluster, joining Alpha Centauri, Betelgeuse, and Capella. If you're curious, you can ask the site Where am I? This has no real bearing on anything -- all of the clusters save for Draconis are on the same physical machine -- but dividing accounts by database cluster even if they're on the same machine makes certain maintenance tasks easier.)
Speaking of site growth and how happy we are with site performance, many people have noticed that we haven't yet re-enabled the need to have an invite code to create a DW account. We'd intended to leave invite codes off for the second half of December as one of our regular no-accounts-needed account creation phases. After adding the additional servers necessary to support the traffic bump, our performance and capacity was large enough that we decided we could support open account creation for longer.
So, invite codes will definitely be off for a little while longer -- this means that anyone can create a Dreamwidth account without needing an invite code for now. Rather than specify an exact time period for how long invite codes will be off, we've decided to just let open account creation run for an unspecified time, as long as this increased demand keeps up and as long as we're happy with the site performance. (And, of course, as long as thedw_antispam team is able to keep up with the spammers that open account creation inevitably attracts -- all praise be to our dedicated team of spamwhackers!) We'll let you know when we turn invite codes back on.
SOPA (the Stop Online Piracy Act) and PIPA (the PROTECT IP Act) are two bills pending in the US legislature to force online service providers (such as Dreamwidth) to remove enormous amounts of content from, and block access to enormous amounts of, the internet, based on the demand of intellectual property rightsholders in an attempt to combat digital intellectual property infringement.
The Electronic Frontier Foundation has a more detailed explanation about why these bills would make bad law: Internet Blacklist Legislation. Dreamwidth joins hundreds of other companies and organizations, both inside the tech industry and out, in standing opposed to these attempts to censor the free and open internet.
We'd like to take a moment to ask you, if you live in the US, to write to your legislators and ask them to oppose the bills.
We said a few weeks ago that we'd be donating 10% of December's gross income to the Ada Initiative, a new nonprofit corporation devoted to supporting women in open technology and culture. (Disclaimer: I volunteer on the board of directors, but I receive no compensation for doing so.)
We're pleased to announce that thanks to a record-breaking month of sales (which we totally did not expect, omg!) we are donating $4625 to help fund the Ada Initiative's 2012 programs. This makes us incredibly happy! Thank you to everyone who bought paid time during December and helped increase that number. Meanwhile, if you'd like to contribute more directly, you can still donate to TAI.
We're obviously very pleased about the aforementioned record-breaking month of sales, and I wanted to say thank you to those who have bought paid time -- it was a fabulous way to close out the year, and looking at the figures on our payment backend has been very, very encouraging lately.
For those who have been wondering what we do with the money from paid accounts: there are two major categories of expenses we have on a recurring basis. The first is professional services; this covers things like processing fees imposed by our payment processor, as well as hosting and backup services. (The processing fees work out to be around 7-9% of all payments, between the flat fees and percentages from our merchant bank account and the flat fees and percentages from our payment gateway; the hosting and backup services run around $4500 a month with the latest upgrades.) The second category is salary:fu is employed by DW fulltime, plus we pay our backup sysadmin and one of our Terms of Service people on a retainer basis as contractors.
We also pay me a very small amount monthly to be working on things full-time; previouslymark was also working full-time on DW, but he needed to take a non-DW job because at the time we couldn't pay him what he needed to be earning to support his family. (I'm lucky; my wife has a very good, very stable government job, and we live in a much less expensive area of the country.)
We post an annual end-of-year analysis of our financial position todw_biz every year, which I will start working on as soon as the end-of-year bookkeeping is finished. Before this month's payments explosion, and aside from the one-time income from our May '11 Seed Account sale (which we did in order to replenish our "war chest", the money we keep in reserve against future disaster, after nearly three months of being unable to accept payments in early 2011), we were already on target to take in slightly more than we spent in 2011. (Which was incredibly exciting, let me tell you!)
This past month's increase in income has allowed us to increase our server power, continue to restock the war chest, and invest in some infrastructure upgrades. (Plus, we boughtmark a pizza while he was hacking on community import.) Beyond that, we don't want to make long-term plans counting on this level of income as a given, but we're really hoping that if this level of support continues we'll be able to get mark back on DW work full-time and be able to pay me market rate for my full-time work on DW.
Meanwhile, many people have asked us about why we don't accept PayPal for payments, and I've written up a longer post about the answer to that question: Why you can't pay for your account with PayPal or Google Checkout. There's some workarounds for people without credit cards in that post.
We've done this before, but given how many people are just joining us, it's time to run another round of the Great Community Rec-O-Matic.
Here's how it works:
* If you're looking for new communities to join, comment here with a list of some of your interests, and the kind of community you enjoy reading and participating in.
* Or, read through the other comments and see what things other people are listing. If you see someone you think would be a perfect match for a community you admin or participate in, comment back to them and point them at the community!
*
That's it from us for another update! As always, if you're having problems with Dreamwidth, Support can help you; for notices of site problems and downtime, check the Twitter status page; if you've got an idea to make the site better, you can make a suggestion.
(Also, I am very close to having followed the hobbit tradition of giving presents to other people on my birthday -- if I'd started this a day earlier I could've posted it on the 5th. Thanks to everybody who sent me birthday wishes!)
We (and my runaway abuse of parentheses) will see you next week for our next update!
* Development, Coding, and Styles
* Community and Icon Importing
* Site Growth and Stability
* Invite Codes
* SOPA
* The Ada Initiative
* Paid Accounts, PayPal, and DW Business
* The Great Community Rec-O-Matic
Development, Coding, and Styles
We start this week's news by bringing you the 29 Dec 11 - 5 Jan 12 code tour, detailing the bugs that were resolved this week.
Most of this week's development effort went to more long-running projects (and thus the bugs weren't completely resolved yet and don't appear on this list), but the biggest highlight this week includes finally having resolved that really annoying bug where interest counts weren't being updated properly, leaving us with a number of interests being listed as having 16,777,215 accounts having that interest listed. It was a bear to track down, but thanks to some amazing detective work from a team of people, with heavy lifting done by
exor674 on the code end and alierak on the sysadmin end, interest counts should now be a lot closer to reality.For those who are just joining us: yes, you too can come hack on the DW code! We offer hosted development environments so you don't have to faff around with installing the code on your own development machine (or have your own development machine), and our Bugzilla install is public for those who are looking for a project to pick up. The Getting Started and Development categories on the DW volunteer wiki have useful information on the development process, and
dw_dev and dw_dev_training are the communities to go to on DW itself. Or, if you'd just like to browse the code, you can read through the Mercurial repositories; the two repositories that most development happens in are dw-free (for the freely-available, open-source-licensed code that primarily runs the service) and dw-nonfree (for dreamwidth.org-specific code, or things that are part of our site branding or identity).DW backend development is mostly done in Perl, with a little bit of Javascript on the frontend here and there. If your coding skills lean more towards HTML and CSS (or S2, the custom layout language shared by DW and LJ), you may be interested in creating or converting themes and styles. If you've created a journal layout that you'd like to offer as a system style, you can submit it to
dreamscapes. Or, if you'd like to wrangle others' submissions into a form that can be used on the site, you can take the layers and styles that people submit and convert them into a patch that can be applied to the codebase. (There's a guide for people patching styles, which assumes a level of familiarity with both S2 and the process of working with the code and creating patches -- we'll work on getting the process explained in a more beginner-friendly fashion, but you can always ask in dw_dev_training for help.)Community and Icon Importing
After last week's announcement that community importing was available,
mark spent a few days coding his fingers off to improve and optimize the importer to better handle larger import jobs. Thanks to his tireless work, we've been able to lift one of the restrictions we had to place on community import: communities that have over 100,000 comments can now be imported.If your community has more than a few hundred thousand comments, the comment import may still fail. We're working with the admins of a very large community to work out the last of the problems. We're asking for now that if your community has more than 500,000 comments, please hold off on doing an import for another week or so -- the code won't enforce the limits anymore, but it will definitely make everybody's lives easier.
We also should also be able to lift the restriction that only paid accounts or paid communities can use the community importer in another week or so, once we're fully satisfied with the optimizations we've made -- stay tuned there. Mark's also actively working on the ability to "claim" your posts and comments that were imported and attributed to your OpenID account, so that should be available soon as well.
Meanwhile, we've made a change to the way icon imports are handled. Previously, if you had more icons available on the remote site than you had icon slots on DW, the site would only import your default icon. We ran into some problems this week with LJ reporting the wrong number of icons if you deleted some icons recently (so if you had 102 icons on LJ, then deleted 3 to bring it down under the 100 icon slots available to a paid account on DW, the system was still reporting you had 102 icons and thus the import was erroring). Rather than spend hours and hours of our time troubleshooting, we just lifted the restriction, and now import all the icons you have on the remote site.
As many people have noticed, this is allowing some people to circumvent the restriction on icon slots and use more icons than their account type allows. The reason we don't run the process to deactivate icons immediately upon import is because at that point, we don't have enough data about the imported posts and comments to tell which icons are the most-used icons in the journal. (When deactivating icons because an account expires, the least-used icons are deactivated first.) Rather than just guessing randomly at which icons should be deactivated, we made the deliberate decision to let it slide for now, then clean up in a week or two once the rush to import has died down a bit. (The extra icons won't be deleted; they'll be deactivated, precisely like what happens when a paid account expires.)
If you import your icons and it'll bother you to have icons available now and have them be deactivated later when we do the cleanup, or if you'd like to help us out a bit, you can force a deactivation by visiting the Edit Icons page while logged into your account. This page loads the routine that checks your current icons against your total number of icon slots, so will deactivate (or activate, if for some reason an account upgrade doesn't reactivate inactive icons) icons until you have the right number.
We know this will let some people have access to more icons than their account level allows for a bit, and thank you to those who reported the situation! It really did just work out to save us a lot of time to do it this way, since we had a lot of people who were having import problems and troubleshooting the issue would've taken valuable time away from other projects.
Site Growth and Stability
mark made an excellent post about DW's load vs capacity, and how we're doing in terms of being able to support all the new traffic we've seen this month. The great news is, we're handling the increased traffic wonderfully: we've seen a few bumps along the past few weeks (everybody's watching dw_maintenance for details, right?) but on the whole, things have been ticking along nicely.In the past few weeks, we've added two new webservers and two new database servers, bringing the total number of machines up to 12: four database machines (since we add dbs in pairs), four machines that work as webservers only, two load balancers (the load balancer + a backup load balancer in case of failure that also functions as a webserver), the search machine (which also functions as a webserver), and the admin machine.
We constantly keep an eye on site performance to see if and when we need more hardware, and since we still rent our dedicated servers rather than colocate our own hardware with a hosting facility, we can get our hosting provider to add more hardware when we need it -- it's more complex than it would be for you to rent webhosting for a small project, but it's not as complex as it would be to buy additional hardware ourselves and have to set it up.
We'd like to move to owning servers and colocating in the future, but for now, dedicated hosting is more cost-effective for us: the way the numbers work out at the moment, it'd take us nearly three years to "earn out" the initial hardware purchase, in terms of amount saved monthly, and there are a lot of practical benefits to dedicated hosting. (Mind you, there are a lot of practical benefits to colocation, as well, but those don't start to show up until you're much bigger than we are right now.)
So, in other words: we're really happy with the site performance right now, and we have a clear plan on how to increase our capacity as our traffic and load increases.
mark also has a lot of experience with handling high-availability web applications, and he's very comfortable with how things are running. If you've been worried about our ability to handle this month's increased traffic, you can rest easy.(Also, for the record, since we name our database clusters after stars and constellations, the new cluster, which is on the new database pair, is the Draconis cluster, joining Alpha Centauri, Betelgeuse, and Capella. If you're curious, you can ask the site Where am I? This has no real bearing on anything -- all of the clusters save for Draconis are on the same physical machine -- but dividing accounts by database cluster even if they're on the same machine makes certain maintenance tasks easier.)
Invite Codes
Speaking of site growth and how happy we are with site performance, many people have noticed that we haven't yet re-enabled the need to have an invite code to create a DW account. We'd intended to leave invite codes off for the second half of December as one of our regular no-accounts-needed account creation phases. After adding the additional servers necessary to support the traffic bump, our performance and capacity was large enough that we decided we could support open account creation for longer.
So, invite codes will definitely be off for a little while longer -- this means that anyone can create a Dreamwidth account without needing an invite code for now. Rather than specify an exact time period for how long invite codes will be off, we've decided to just let open account creation run for an unspecified time, as long as this increased demand keeps up and as long as we're happy with the site performance. (And, of course, as long as the
dw_antispam team is able to keep up with the spammers that open account creation inevitably attracts -- all praise be to our dedicated team of spamwhackers!) We'll let you know when we turn invite codes back on.SOPA
SOPA (the Stop Online Piracy Act) and PIPA (the PROTECT IP Act) are two bills pending in the US legislature to force online service providers (such as Dreamwidth) to remove enormous amounts of content from, and block access to enormous amounts of, the internet, based on the demand of intellectual property rightsholders in an attempt to combat digital intellectual property infringement.
The Electronic Frontier Foundation has a more detailed explanation about why these bills would make bad law: Internet Blacklist Legislation. Dreamwidth joins hundreds of other companies and organizations, both inside the tech industry and out, in standing opposed to these attempts to censor the free and open internet.
We'd like to take a moment to ask you, if you live in the US, to write to your legislators and ask them to oppose the bills.
The Ada Initiative
We said a few weeks ago that we'd be donating 10% of December's gross income to the Ada Initiative, a new nonprofit corporation devoted to supporting women in open technology and culture. (Disclaimer: I volunteer on the board of directors, but I receive no compensation for doing so.)
We're pleased to announce that thanks to a record-breaking month of sales (which we totally did not expect, omg!) we are donating $4625 to help fund the Ada Initiative's 2012 programs. This makes us incredibly happy! Thank you to everyone who bought paid time during December and helped increase that number. Meanwhile, if you'd like to contribute more directly, you can still donate to TAI.
Paid accounts, PayPal, and DW Business
We're obviously very pleased about the aforementioned record-breaking month of sales, and I wanted to say thank you to those who have bought paid time -- it was a fabulous way to close out the year, and looking at the figures on our payment backend has been very, very encouraging lately.
For those who have been wondering what we do with the money from paid accounts: there are two major categories of expenses we have on a recurring basis. The first is professional services; this covers things like processing fees imposed by our payment processor, as well as hosting and backup services. (The processing fees work out to be around 7-9% of all payments, between the flat fees and percentages from our merchant bank account and the flat fees and percentages from our payment gateway; the hosting and backup services run around $4500 a month with the latest upgrades.) The second category is salary:
fu is employed by DW fulltime, plus we pay our backup sysadmin and one of our Terms of Service people on a retainer basis as contractors. We also pay me a very small amount monthly to be working on things full-time; previously
mark was also working full-time on DW, but he needed to take a non-DW job because at the time we couldn't pay him what he needed to be earning to support his family. (I'm lucky; my wife has a very good, very stable government job, and we live in a much less expensive area of the country.)We post an annual end-of-year analysis of our financial position to
dw_biz every year, which I will start working on as soon as the end-of-year bookkeeping is finished. Before this month's payments explosion, and aside from the one-time income from our May '11 Seed Account sale (which we did in order to replenish our "war chest", the money we keep in reserve against future disaster, after nearly three months of being unable to accept payments in early 2011), we were already on target to take in slightly more than we spent in 2011. (Which was incredibly exciting, let me tell you!)This past month's increase in income has allowed us to increase our server power, continue to restock the war chest, and invest in some infrastructure upgrades. (Plus, we bought
mark a pizza while he was hacking on community import.) Beyond that, we don't want to make long-term plans counting on this level of income as a given, but we're really hoping that if this level of support continues we'll be able to get mark back on DW work full-time and be able to pay me market rate for my full-time work on DW.Meanwhile, many people have asked us about why we don't accept PayPal for payments, and I've written up a longer post about the answer to that question: Why you can't pay for your account with PayPal or Google Checkout. There's some workarounds for people without credit cards in that post.
The Great Community Rec-O-Matic
We've done this before, but given how many people are just joining us, it's time to run another round of the Great Community Rec-O-Matic.
Here's how it works:
* If you're looking for new communities to join, comment here with a list of some of your interests, and the kind of community you enjoy reading and participating in.
* Or, read through the other comments and see what things other people are listing. If you see someone you think would be a perfect match for a community you admin or participate in, comment back to them and point them at the community!
*
That's it from us for another update! As always, if you're having problems with Dreamwidth, Support can help you; for notices of site problems and downtime, check the Twitter status page; if you've got an idea to make the site better, you can make a suggestion.
(Also, I am very close to having followed the hobbit tradition of giving presents to other people on my birthday -- if I'd started this a day earlier I could've posted it on the 5th. Thanks to everybody who sent me birthday wishes!)
We (and my runaway abuse of parentheses) will see you next week for our next update!
![[community profile]](https://www.dreamwidth.org/img/silk/identity/community.png){kind=small}
Page 1 of 27