denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
Denise ([staff profile] denise) wrote in [site community profile] dw_news2012-01-06 07:10 am

Dreamwidth Update: 6 January 2012

Hello, Dreamwidth! We're ringing in 2012 with a collection of useful information and news from the DW head office (aka my house) and auxiliary offices (aka Mark's house and Fu's house). Behind the cut you will find information on:

* Development, Coding, and Styles
* Community and Icon Importing
* Site Growth and Stability
* Invite Codes
* SOPA
* The Ada Initiative
* Paid Accounts, PayPal, and DW Business
* The Great Community Rec-O-Matic



Development, Coding, and Styles



We start this week's news by bringing you the 29 Dec 11 - 5 Jan 12 code tour, detailing the bugs that were resolved this week.

Most of this week's development effort went to more long-running projects (and thus the bugs weren't completely resolved yet and don't appear on this list), but the biggest highlight this week includes finally having resolved that really annoying bug where interest counts weren't being updated properly, leaving us with a number of interests being listed as having 16,777,215 accounts having that interest listed. It was a bear to track down, but thanks to some amazing detective work from a team of people, with heavy lifting done by [personal profile] exor674 on the code end and [personal profile] alierak on the sysadmin end, interest counts should now be a lot closer to reality.

For those who are just joining us: yes, you too can come hack on the DW code! We offer hosted development environments so you don't have to faff around with installing the code on your own development machine (or have your own development machine), and our Bugzilla install is public for those who are looking for a project to pick up. The Getting Started and Development categories on the DW volunteer wiki have useful information on the development process, and [site community profile] dw_dev and [site community profile] dw_dev_training are the communities to go to on DW itself. Or, if you'd just like to browse the code, you can read through the Mercurial repositories; the two repositories that most development happens in are dw-free (for the freely-available, open-source-licensed code that primarily runs the service) and dw-nonfree (for dreamwidth.org-specific code, or things that are part of our site branding or identity).

DW backend development is mostly done in Perl, with a little bit of Javascript on the frontend here and there. If your coding skills lean more towards HTML and CSS (or S2, the custom layout language shared by DW and LJ), you may be interested in creating or converting themes and styles. If you've created a journal layout that you'd like to offer as a system style, you can submit it to [site community profile] dreamscapes. Or, if you'd like to wrangle others' submissions into a form that can be used on the site, you can take the layers and styles that people submit and convert them into a patch that can be applied to the codebase. (There's a guide for people patching styles, which assumes a level of familiarity with both S2 and the process of working with the code and creating patches -- we'll work on getting the process explained in a more beginner-friendly fashion, but you can always ask in [site community profile] dw_dev_training for help.)


Community and Icon Importing



After last week's announcement that community importing was available, [staff profile] mark spent a few days coding his fingers off to improve and optimize the importer to better handle larger import jobs. Thanks to his tireless work, we've been able to lift one of the restrictions we had to place on community import: communities that have over 100,000 comments can now be imported.

If your community has more than a few hundred thousand comments, the comment import may still fail. We're working with the admins of a very large community to work out the last of the problems. We're asking for now that if your community has more than 500,000 comments, please hold off on doing an import for another week or so -- the code won't enforce the limits anymore, but it will definitely make everybody's lives easier.

We also should also be able to lift the restriction that only paid accounts or paid communities can use the community importer in another week or so, once we're fully satisfied with the optimizations we've made -- stay tuned there. Mark's also actively working on the ability to "claim" your posts and comments that were imported and attributed to your OpenID account, so that should be available soon as well.

Meanwhile, we've made a change to the way icon imports are handled. Previously, if you had more icons available on the remote site than you had icon slots on DW, the site would only import your default icon. We ran into some problems this week with LJ reporting the wrong number of icons if you deleted some icons recently (so if you had 102 icons on LJ, then deleted 3 to bring it down under the 100 icon slots available to a paid account on DW, the system was still reporting you had 102 icons and thus the import was erroring). Rather than spend hours and hours of our time troubleshooting, we just lifted the restriction, and now import all the icons you have on the remote site.

As many people have noticed, this is allowing some people to circumvent the restriction on icon slots and use more icons than their account type allows. The reason we don't run the process to deactivate icons immediately upon import is because at that point, we don't have enough data about the imported posts and comments to tell which icons are the most-used icons in the journal. (When deactivating icons because an account expires, the least-used icons are deactivated first.) Rather than just guessing randomly at which icons should be deactivated, we made the deliberate decision to let it slide for now, then clean up in a week or two once the rush to import has died down a bit. (The extra icons won't be deleted; they'll be deactivated, precisely like what happens when a paid account expires.)

If you import your icons and it'll bother you to have icons available now and have them be deactivated later when we do the cleanup, or if you'd like to help us out a bit, you can force a deactivation by visiting the Edit Icons page while logged into your account. This page loads the routine that checks your current icons against your total number of icon slots, so will deactivate (or activate, if for some reason an account upgrade doesn't reactivate inactive icons) icons until you have the right number.

We know this will let some people have access to more icons than their account level allows for a bit, and thank you to those who reported the situation! It really did just work out to save us a lot of time to do it this way, since we had a lot of people who were having import problems and troubleshooting the issue would've taken valuable time away from other projects.


Site Growth and Stability



[staff profile] mark made an excellent post about DW's load vs capacity, and how we're doing in terms of being able to support all the new traffic we've seen this month. The great news is, we're handling the increased traffic wonderfully: we've seen a few bumps along the past few weeks (everybody's watching [site community profile] dw_maintenance for details, right?) but on the whole, things have been ticking along nicely.

In the past few weeks, we've added two new webservers and two new database servers, bringing the total number of machines up to 12: four database machines (since we add dbs in pairs), four machines that work as webservers only, two load balancers (the load balancer + a backup load balancer in case of failure that also functions as a webserver), the search machine (which also functions as a webserver), and the admin machine.

We constantly keep an eye on site performance to see if and when we need more hardware, and since we still rent our dedicated servers rather than colocate our own hardware with a hosting facility, we can get our hosting provider to add more hardware when we need it -- it's more complex than it would be for you to rent webhosting for a small project, but it's not as complex as it would be to buy additional hardware ourselves and have to set it up.

We'd like to move to owning servers and colocating in the future, but for now, dedicated hosting is more cost-effective for us: the way the numbers work out at the moment, it'd take us nearly three years to "earn out" the initial hardware purchase, in terms of amount saved monthly, and there are a lot of practical benefits to dedicated hosting. (Mind you, there are a lot of practical benefits to colocation, as well, but those don't start to show up until you're much bigger than we are right now.)

So, in other words: we're really happy with the site performance right now, and we have a clear plan on how to increase our capacity as our traffic and load increases. [staff profile] mark also has a lot of experience with handling high-availability web applications, and he's very comfortable with how things are running. If you've been worried about our ability to handle this month's increased traffic, you can rest easy.

(Also, for the record, since we name our database clusters after stars and constellations, the new cluster, which is on the new database pair, is the Draconis cluster, joining Alpha Centauri, Betelgeuse, and Capella. If you're curious, you can ask the site Where am I? This has no real bearing on anything -- all of the clusters save for Draconis are on the same physical machine -- but dividing accounts by database cluster even if they're on the same machine makes certain maintenance tasks easier.)


Invite Codes



Speaking of site growth and how happy we are with site performance, many people have noticed that we haven't yet re-enabled the need to have an invite code to create a DW account. We'd intended to leave invite codes off for the second half of December as one of our regular no-accounts-needed account creation phases. After adding the additional servers necessary to support the traffic bump, our performance and capacity was large enough that we decided we could support open account creation for longer.

So, invite codes will definitely be off for a little while longer -- this means that anyone can create a Dreamwidth account without needing an invite code for now. Rather than specify an exact time period for how long invite codes will be off, we've decided to just let open account creation run for an unspecified time, as long as this increased demand keeps up and as long as we're happy with the site performance. (And, of course, as long as the [site community profile] dw_antispam team is able to keep up with the spammers that open account creation inevitably attracts -- all praise be to our dedicated team of spamwhackers!) We'll let you know when we turn invite codes back on.


SOPA



SOPA (the Stop Online Piracy Act) and PIPA (the PROTECT IP Act) are two bills pending in the US legislature to force online service providers (such as Dreamwidth) to remove enormous amounts of content from, and block access to enormous amounts of, the internet, based on the demand of intellectual property rightsholders in an attempt to combat digital intellectual property infringement.

The Electronic Frontier Foundation has a more detailed explanation about why these bills would make bad law: Internet Blacklist Legislation. Dreamwidth joins hundreds of other companies and organizations, both inside the tech industry and out, in standing opposed to these attempts to censor the free and open internet.

We'd like to take a moment to ask you, if you live in the US, to write to your legislators and ask them to oppose the bills.


The Ada Initiative



We said a few weeks ago that we'd be donating 10% of December's gross income to the Ada Initiative, a new nonprofit corporation devoted to supporting women in open technology and culture. (Disclaimer: I volunteer on the board of directors, but I receive no compensation for doing so.)

We're pleased to announce that thanks to a record-breaking month of sales (which we totally did not expect, omg!) we are donating $4625 to help fund the Ada Initiative's 2012 programs. This makes us incredibly happy! Thank you to everyone who bought paid time during December and helped increase that number. Meanwhile, if you'd like to contribute more directly, you can still donate to TAI.


Paid accounts, PayPal, and DW Business



We're obviously very pleased about the aforementioned record-breaking month of sales, and I wanted to say thank you to those who have bought paid time -- it was a fabulous way to close out the year, and looking at the figures on our payment backend has been very, very encouraging lately.

For those who have been wondering what we do with the money from paid accounts: there are two major categories of expenses we have on a recurring basis. The first is professional services; this covers things like processing fees imposed by our payment processor, as well as hosting and backup services. (The processing fees work out to be around 7-9% of all payments, between the flat fees and percentages from our merchant bank account and the flat fees and percentages from our payment gateway; the hosting and backup services run around $4500 a month with the latest upgrades.) The second category is salary: [personal profile] fu is employed by DW fulltime, plus we pay our backup sysadmin and one of our Terms of Service people on a retainer basis as contractors.

We also pay me a very small amount monthly to be working on things full-time; previously [staff profile] mark was also working full-time on DW, but he needed to take a non-DW job because at the time we couldn't pay him what he needed to be earning to support his family. (I'm lucky; my wife has a very good, very stable government job, and we live in a much less expensive area of the country.)

We post an annual end-of-year analysis of our financial position to [site community profile] dw_biz every year, which I will start working on as soon as the end-of-year bookkeeping is finished. Before this month's payments explosion, and aside from the one-time income from our May '11 Seed Account sale (which we did in order to replenish our "war chest", the money we keep in reserve against future disaster, after nearly three months of being unable to accept payments in early 2011), we were already on target to take in slightly more than we spent in 2011. (Which was incredibly exciting, let me tell you!)

This past month's increase in income has allowed us to increase our server power, continue to restock the war chest, and invest in some infrastructure upgrades. (Plus, we bought [staff profile] mark a pizza while he was hacking on community import.) Beyond that, we don't want to make long-term plans counting on this level of income as a given, but we're really hoping that if this level of support continues we'll be able to get [staff profile] mark back on DW work full-time and be able to pay me market rate for my full-time work on DW.

Meanwhile, many people have asked us about why we don't accept PayPal for payments, and I've written up a longer post about the answer to that question: Why you can't pay for your account with PayPal or Google Checkout. There's some workarounds for people without credit cards in that post.


The Great Community Rec-O-Matic



We've done this before, but given how many people are just joining us, it's time to run another round of the Great Community Rec-O-Matic.

Here's how it works:

* If you're looking for new communities to join, comment here with a list of some of your interests, and the kind of community you enjoy reading and participating in.

* Or, read through the other comments and see what things other people are listing. If you see someone you think would be a perfect match for a community you admin or participate in, comment back to them and point them at the community!


*

That's it from us for another update! As always, if you're having problems with Dreamwidth, Support can help you; for notices of site problems and downtime, check the Twitter status page; if you've got an idea to make the site better, you can make a suggestion.

(Also, I am very close to having followed the hobbit tradition of giving presents to other people on my birthday -- if I'd started this a day earlier I could've posted it on the 5th. Thanks to everybody who sent me birthday wishes!)

We (and my runaway abuse of parentheses) will see you next week for our next update!
ursamajor: people on the beach watching the ocean (Default)

Re: Betelgeuse, Betelgeuse, Betelgeuse!

[personal profile] ursamajor 2012-01-07 05:06 am (UTC)(link)
I hear ya - it's really only in the last couple of years that we've gotten actual winter farmers' markets to take off. We are limited for the fresh plant life in that it's primarily root veg and similar fruit that store well in root cellars, along with some more distant but apparently traditional citrus imports from Florida, but the meat and recent wine additions help make up for it :)
ursamajor: devil uses the internet (do i look like an information booth?)

[personal profile] ursamajor 2012-01-07 05:08 am (UTC)(link)
You're welcome! Now if I can think of things to contribute to it. Maybe I should actually do something with all those IS-related links I keep collecting >_>
torine: (Default)

[personal profile] torine 2012-01-07 05:23 am (UTC)(link)
I'm looking for an 18+ panfandom RP along the lines of Dirty Vegas. Those in the recs on the first page of comments didn't suit what I'm looking for.
jazz: (shinee: jonghyun qtest smile)

[personal profile] jazz 2012-01-07 05:28 am (UTC)(link)
anybody know if there's an active vancouver (bc, canada) comm? i found [community profile] vancouver, but it's never been updated. perhaps i'll go make the first post.

also, if you're interested in SHINee-related news come check out [community profile] shinee, which i am newly co-modding. i'd love the comm to develop further!
dragonfly: The dreamwidth sheep with a green esperanto star on the body (esperanto)

[personal profile] dragonfly 2012-01-07 05:34 am (UTC)(link)
I know! That's what I thought, too. I've started painstakingly translating the shortest of all my fanfics. *g*
schwertlilie: Watanuki with pipe fox spirit. (watanuki: hearts)

Re: Eternal Law, anyone?

[personal profile] schwertlilie 2012-01-07 05:35 am (UTC)(link)
+1 to tracking this down. *_* I've never heard of this show before now, but angels + that OP + wingfic opportunities = win.
staringiscaring: (Default)

[personal profile] staringiscaring 2012-01-07 05:37 am (UTC)(link)
I just have to figured out a way to post them without spamming. lol.
staringiscaring: (Default)

[personal profile] staringiscaring 2012-01-07 05:38 am (UTC)(link)
Yay, awesome! I'm a big fan of moonlight, blade and trueblood. Thanks for the link.
dodificus: (Default)

1000 tag limit

[personal profile] dodificus 2012-01-07 06:37 am (UTC)(link)
Off topic question, not sure if this is the right place to ask but anyway:

If a community reaches the 1000 tag limit, is it possible to get the limit increased? Or is that the end?
dodificus: (Default)

Re: 1000 tag limit

[personal profile] dodificus 2012-01-07 06:41 am (UTC)(link)
Thank you for your super speedy and very helpful reply:)
terabient: Anime-styled profile pic that is kinda, sorta like me (DC: Superman & Batman - picnic)

Re: Rec-O-Matic

[personal profile] terabient 2012-01-07 06:53 am (UTC)(link)
[community profile] scans_daily is pretty much my favorite comics place on the whole internet. :D
terabient: Chell jumping out of a portal (Portal: The Girl Who Lept Through Space)

Valve time!

[personal profile] terabient 2012-01-07 06:58 am (UTC)(link)
Here are some Valve-related communities if anyone's interested~

[community profile] thinkwithportals A place to share Portal fanworks of all kinds
[community profile] capslock_l4d CAPSLOCK + LEFT 4 DEAD = GOOD TIMES, DEAD ZOMBIES, ETC

I've been pretty poor about keeping them active but I am going to try harder to keep things active this year. ^^;

[personal profile] tallblue 2012-01-07 07:02 am (UTC)(link)
Recs, I'm looking for:

-Anything about the topic of paranormal (not the movie(s))
-Horror Movies
-Horror Books
-Mystery book like Dennis Lehanne
-poetry - such as John Donne, I am more of a reader than a writer.
-Healthy living
-Aquariums
-Terrariums
-Photography (DSLR)or just digital photography
-graphic communities such as paint shop pro
-graphic tuturals types
-Some great icon comms!



yabamena: (Default)

Re: Rec-O-Matic

[personal profile] yabamena 2012-01-07 07:03 am (UTC)(link)
Ooh, I haven't poked around there in a bit. Thanks!
basilmemories: (Default)

Re: SOPA/PROTECT IP

[personal profile] basilmemories 2012-01-07 07:23 am (UTC)(link)
As an artist, I have a problem with this statement, and the general idea that I would want or need to be "protected" from this kind of thing. Most art these days, when examined at it's closest portions, involves theft. Getting inspiration from something, taking an old idea and making something new out of it, deconstructing an existing work or theme, or even remaking the work itself, these are all acts of infringing on someone's intellectual property. This is a current system where a game company can go after an indie studio because of the word "Scrolls". This is a system where one guy who shoves dead animals into tanks of formaldehyde can sue someone who uses aspects of his work in what amounts to a photoshop mashup.

The artists are not protected. The only people who will benefit are the larger corporations who reap from those artists in the first place. This was part of the reason cd prices shot up during the napster days, not piracy. Piracy was only spurred on by the inflated prices. When the artists income was measured to overhead costs and managerial fees, the result was staggering: an artist, unless amazingly successful, made roughly a dollar per cd sold.

With current legal issues we couldn't really have another wave of dadaists, or readymades. The people who can make that work have copyrighted it. The music profession, which for ages has thrived on people incorporating and improving from other work, now faces themselves in a legal quagmire when it comes to new artists trying to be noticed. It is only through sites that may allow for some of this "piracy" to be shared (like youtube, starting grounds of. yes. even Justin bieber)that a number of otherwise unknown animators, directors, singers, and the like have been discovered.

SOPA doesn't help me, SOPA doesn't protect the normal sort of people, it only helps the companies. And in my opinion? they don't need any more help, they already have enough connections in Washington.

[personal profile] tallblue 2012-01-07 07:23 am (UTC)(link)
Thank you I joined!
krait: a sea snake (krait) swimming (Default)

[personal profile] krait 2012-01-07 07:33 am (UTC)(link)
Ladies and Star Trek? Beam me onboard! *subscribes*
krait: a sea snake (krait) swimming (Default)

[personal profile] krait 2012-01-07 07:36 am (UTC)(link)
I just joined [community profile] 1fish_2fish, a fish/aquarium comm; haven't had a chance to really check it out yet (I found it upthread about two pages ago!), but the more, the merrier, right? :D
krait: a sea snake (krait) swimming (Default)

[personal profile] krait 2012-01-07 07:46 am (UTC)(link)
Latecomer is late, but has comm recs and requests!

For anything related to Dragcave.net that is not daily clicking, please come join [Bad username or unknown identity: dragoncavers"]! Tell us about the lineage you're working with, post a wishlist or what you have to trade, discuss spriting, or just come chat about your favourite dragon breed!

Do you like to read with other people? Come to [community profile] readingtogether and set up a group-read for your favourite series or something you've always meant to get round to but might need some accountability to keep you going with!

Everyone has a specialty; every writer sometimes finds they need details about a specialty they're unfamiliar with. Come over to [community profile] factfinding or [community profile] little_details and ask a question, then see if you can answer someone else's inquiry.

----------

Comms I would be interested in finding:

1. Dragaera fandom/fanworks
2. Hurog fandom/fanworks; barring that, an all-works-of-Patricia-Briggs comm
3. Original slash fiction
4. A comm for finding betas in small/rare fandoms
5. A comm for tea enthusiasts
6. ?? You tell me! :D
dancing_serpent: (Default)

[personal profile] dancing_serpent 2012-01-07 08:50 am (UTC)(link)
*g* I'm stalking DW for material for my newsletter and pick things up on the way. And also, there's [personal profile] old_man_of_hoy's Masterlist of Masterlists of Communities on Dreamwidth - my best resource.
dancing_serpent: (Default)

[personal profile] dancing_serpent 2012-01-07 08:58 am (UTC)(link)
Also, in addition to those already named:

[community profile] avatar
[community profile] atla_contest
[community profile] donebykorra rewatching A:TLA

newly migrated to DW:
[community profile] pokefics
[community profile] pokeprompts
dancing_serpent: (Default)

[personal profile] dancing_serpent 2012-01-07 09:01 am (UTC)(link)
[personal profile] strina has compiled a list of DW and LJ Big Bangs here.

Also, new on DW: [community profile] betaplease
dancing_serpent: (Default)

[personal profile] dancing_serpent 2012-01-07 09:07 am (UTC)(link)
For Fate/stay night etc maybe you're interested in reading/subscribing to [personal profile] issei, who writes a lot of "Fate" fanfiction.

Page 18 of 27