Dreamwidth news: 10 June 2014

[denise]

Hello, Dreamwidth! I bring you news and glad tidings of the new stuff that came out of last night's code push.

(I also bring yet another apology for the length of time in between this news post and the last; my ongoing RSI issues have been so bad that I've had to continue sharply limiting my typing time. The "good" news is that I'll be going in for surgery, or at least a first round, next month, so with a little bit of luck I'll be able to get back in the game a bit better after some time for recovery.)

Behind the cut:

* Development
* First steps towards our new responsive design
* Community admin redesign: admins take note
* Bugfix: deeply nested comments
* Apologies about thread tracking
* Advance warning: date/time checks at posting going away in future
* Embedding sources
* Spam (spam spam spam)


A reminder: Whenever a news post is posted, all notifications are delayed for a little while as the mail system sends out notifications of the announcement. Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated after an update is posted to dw_news. This was posted slightly before 0800 EST (see in your time zone). Please don't worry about missing notifications until at least 1000 EST.

Development

First off, some bad news: the period covered by this update saw a technical problem that, unfortunately, means we lost the contents of our bug tracking database. It was one of those things where one mistake compounded a bunch of times, and when the dust cleared, we realized that a misconfiguration meant the server backups had been wiped along with the server itself.

This does not mean that the same thing could happen to your data -- the bug tracking database was on an entirely separate server with an entirely different configuration than any user data. We also were able to reconstruct much of the data from other sources. We haven't, however, imported the old bug data into our new bug tracking setup. Instead, we're using this as an opportunity to start fresh: a lot of the items in the database had been in there for a while, and although we tried to stay on top of closing bugs that were no longer relevant or were reporting things that had been fixed by another change, there were still a lot of things in the database that weren't relevant to the state of Dreamwidth today.

We've done our best to make sure that actual bugs that are still happening and still interfering with people's use of the site have all been filed in our new bug tracking system -- you can look over the list, to check whether one that you've reported has been re-opened. If there's a bug you're still experiencing, and it's not on the list, you can report it to Support and they'll doublecheck for you, then file the bug if it needs to be filed. The big one we know about is the problem with icons that weren't given keywords at the time of upload, and were automatically given the system-generated 'pic###' pseudo-keyword before being renamed later, are reverting to the 'pic###' pseudo-keyword in some unknown circumstances. We're still trying to figure that one out.

(User suggestions from dw_suggestions haven't been moved back into the bug tracker yet, but they will be once I'm able to clean up the suggestions queue a little! No need to re-report those.)

Anyway, moving on to happier news: this time around, we welcome new contributors darael and fhocutt. The code tours for the changes that just went live were a little more difficult to pull together, thanks to the problems with the bug tracking database, but we're pretty sure we've found most of it:

24 Nov - 10 Mar
11 Mar - 10 Jun

First steps towards our new responsive design

This code push brought with it a redesign of the community administration pages. This was done for a few reasons, both as a "test run" of some of our new responsive design elements and to improve the workflow for community admins.

I'll talk about the changes in workflow in the next section; this one is about the purposes behind the project, and what kind of feedback we're looking for!

There are several purposes of this redesign. Aside from the boring technical ones (making it easier for us to maintain and troubleshoot the code, reducing browser incompatability, giving us more options in terms of designing, etc), the two big ones are:

* Improve the way the site looks and feels on small screens (such as on tablets and mobile devices) without worsening the experience for people using large screens (desktops and laptops).

* Improve the accessibility of the site: make it easier to access the site with screenreaders, interact with the site using alternate input devices, interact with the site using only the mouse or only the keyboard, etc.

This is a "first draft"-type implementation, and we're sure that people are going to have a lot of feedback! Usually we would have run this through a dw_beta-style beta test that you could activate for your account, but Boring Technical Reasons means that doing a beta-style test for this would have been difficult if not impossible. So, we've converted the community admin area, including the site skins as displayed on those pages, as a working example. This will let you get a sense of what the site skins will look like, and the sort of "visual vocabulary" we'll be using.

We want to hear two things from you: if there are any display bugs or glitches while viewing these pages in any of your devices (on your computer vs on your phone, for instance), particularly any problems involving your use of assistive technology, and feedback on the aesthetics of both the site-skin elements of the pages and the page-contents elements of the pages.

A note on feedback: "how the site looks" is probably the one thing that people feel most strongly about, and discussions about changes to the layout and appearance of the site can get very passionated and heated, very quickly. We'd like to ask you to make sure your feedback is specific, concrete, and actionable, even if -- especially if -- you dislike it. "It sucks, I hate it" doesn't give us anything to go on, while a list of things you think are in the wrong place/too big/too small/too cramped/too spread out/not enough contrast/too much contrast/etc, and why, is something we can evaluate and work from.

We particularly want to hear about the process of using the redesign in your assistive technology setup, whether good or bad: do you find it easy to navigate with the keyboard only? does your screenreader have trouble identifying the links? That kind of thing would be very helpful.

Likewise, if you really like the design, we'd love to hear that. (And not just because it's nice to hear! When it comes to redesigns, it's pretty common for only the people who hate it to speak up, which makes it hard to evaluate the overall reaction.) But if you have a few moments, we'd especially love to hear the specifics of what you like about it, both so we can keep those elements in second and subsequent drafts and so we can make sure we do more of those kinds of things as we work through other pages on the site.

Finally, this is a first draft, and those of you who have been through this process with us before know that things can change considerably between first draft and final draft. If you're newer to the Dreamwidth way of doing things, rest assured that if you really do hate something, we'll do our best to figure out why and what can be done about it. We can't always please everybody, and there are times when y'all disagree with each other or when we have to do something in a particular way because of external reasoning, but we do our best to find something that works for as many people as possible and to explain why we're doing what we're doing if we can't.

Please do not leave your feedback here -- it'll just get lost among the usual responses to announcements. We want to be able to find all the feedback together at once. Go to the dw_beta post "New responsive-design site skins and community admin pages" and leave your feedback there, please! We'll take it into account and use it to make improvements.

If you aren't a community admin, don't feel like you have to make a community just to see what it looks like. You can still see the site skin on those pages if you want to get a preview, and we'll be doing another round of feedback when we convert the next set of pages. (We chose the community admin area to start with because it's a smaller subset of people who use it regularly, and it's always helpful to limit the participants in a first-draft beta to just a subset of y'all so the feedback is less overwhelming.)

Community admin redesign: admins take note!

Community admins: if you only read one section of this update, read this one! We've changed a bunch of stuff about how you handle the day-to-day work of adminnning your communities.

We chose the community administration area as the "proving ground" for our new responsive design because it was a self-contained area and a smaller subset of y'all who use it, but while we were in there, we made a number of small workflow changes and one large change to how "posting access" works.

The workflow changes are mostly merging pages and views together so you no longer have to remember where exactly things are hidden. For instance, communities used to have two places to change their settings: the Manage Settings page, and a separate settings page where you'd pick the settings that only applied to communities and didn't apply to personal accounts. These have been merged, and now even the community-specific settings are part of the Manage Settings page. If you pick a community that you're an admin of from the "Work as" dropdown, or follow the 'Settings' link from your Manage Communities list, a new 'Community' tab will show up. That's now where you change membership and posting access settings, set the community to moderated or unmoderated posting, and designate the links to the community rules that will be shown to members when they join.

The "Edit Community Members" workflow and the community member list has had a few new filters added, making it easier to handle communities with a high number of members: you can filter only to people who have unmoderated posting, for instance, or only people who are moderators, or only people who are admins. "Check-all" checkboxes have also been added at the top of the grid to make it easier to perform actions in bulk.

The processes of inviting a new member and the process of viewing/managing outstanding invites have been merged onto a single "Invitations" page. There, you can issue invitations for people to join your community (as well as set permissions for their accounts, so you can invite someone specifically as an admin rather than waiting for them to join and then adding the admin flag to their membership). That's also where you check for outstanding invites, and cancel them if you've changed your mind.

Finally, we've changed the posting-access settings and what they do, to be a bit more intuitive for new members and first-time community admins:

* Previously, if you had your community posting set to 'select members', gave some people posting access but not others, then switched the community posting setting to 'all members', people who joined the community after that point would have posting access but people who'd already been members of the comm, and didn't have posting access before the switch, would not. Now, when you switch from 'select members' to 'all members', all members of the community can post immediately: they don't need to be granted posting access. If you want to selectively grant posting access, choose 'select members'.

* When you choose 'select members' as the posting setting, there's now a new option that will help make letting people join easier: you can choose to allow new members to have posting access by default, or keep new members from having posting access by default. (The default right now is 'don't allow new members to have posting access', since that most closely replicates the previous behavior.)

A few examples of when you might want to use each of these settings:

* A community that's open for discussion to anyone: set posting to 'all members'.

* A community that posts entirely members-locked to keep casual passersby from reading it, but wants to allow anyone who joins the community to post: set posting to 'all members'. (And set minimum entry security for the community to 'Members'.)

* A community that has open membership, but is intended for announcements by a limited set of people (for instance, dw_news!): Set posting to 'select members', set new-member posting to 'new members cannot post'.

* The scenario that has changed slightly: A community that has open membership and wants to allow most people to post, but has one or two members: people who disrupt the community enough that the admin doesn't want to allow them to create new posts, but not enough to warrant banning them from the community entirely. Previously, you could leave posting set on 'all members', but uncheck that single person's posting ability. Now, you should set posting to 'select members', set new-member posting to 'new members can post', and then uncheck that person's posting ability.

Bugfix: Deeply nested comments

We have good news!

Previously, when a particular comment thread starts getting up into the hundreds of replies (...or sometimes the thousands -- I've definitely seen it) and most or many of those comments are in a single thread replying to each other, the system would start to error out once you got down about two or three hundred replies into the thread. This always used to happen in journal-skinned pages. You used to be able to avoid it by switching to site-skinned pages, but once we switched over everything to use the same backend, that wouldn't work anymore either.

Thanks to some deep dive code guru work by mark with assists by fu, that shouldn't oughta happen anymore. (Why we love mark: he basically looked at the problem and said, right, we've got to add a whole new function to the S2 programming language that runs the customization system in order to fix it right instead of hackishly.)

We tested it pretty thoroughly (and then of course found a bug that snuck through testing as soon as we introduced it to the realities of how y'all use the site) but there's always a chance there might be more issues cropping up in the edge cases. If you run into problems with comments not displaying when they should, let us know on the post code-push announcement and we'll get right on it.

Apologies about thread tracking

Because we've been getting a lot of people asking about this: I wanted to apologize for a bug we accidentally introduced, but not exactly the bug people were reporting.

With our last code push, when we rewrote the subscription/notification system, we accidentally allowed the "track this thread" subscription option to display to everyone when they were viewing a comment thread, even though tracking individual threads (as opposed to tracking the whole post) was, and always has been, a paid user feature. So, free users would see the option to track the thread, select it, and then wonder why they weren't receiving any emails for that subscription: they'd report to us that they were missing emails or that their subscription wasn't working, when the issue was that they shouldn't have had access to that subscription type in the first place. (There are a few subscriptions that are only available to paid users, either as a thank-you for people who support the site or because those subscriptions tend to generate a lot of email and thus we have to limit them because of server load.)

I'm very sorry for the confusion, and I'm very sorry for us having accidentally given the impression that individual thread tracking was an option available to everyone. We've fixed things now so the option to track individual threads no longer shows to account types that don't have the access to use that subscription.

Advance warning: date/time checks on posting going away in future

Okay, hands up, how many people have ever opened a window to start writing a long entry, remembered something else you wanted to post about, opened a second tab and dashed that one off quickly, then gotten an error message when you try to post the second one: "You have an entry which was posted at [time], but you're trying to post an entry before this. Please check the date and time of both entries..." etc, etc.

This check was added way back in the LJ days to prevent a very common problem with people's computer clocks being set wrong. It was a horrible support burden (leading to dozens if not more support requests per day): someone's computer battery would be dying and their clock was set wrong because of it, or their clock would just be set a year or two off. Because entries in personal journals are displayed on the Recent Entries page by the time they're dated, not by the time the server received the post, a post dated, for instance, 1970-01-01 would disappear completely: the person would post it, it would display on their Recent Entries behind every other post they'd ever made, and they wouldn't be able to find it when they loaded their journal to see it so they would assume it hadn't been posted at all.

(This is not a problem in communities: to avoid the problem with having posters from many timezones, communities show all entries ordered by server time, not by user-supplied time.)

So, to solve that problem, somebody -- I think it was far enough back that it was when it was pretty much just Brad working on LJ! -- added the date/time check on post. It definitely fixed "help, I lost this entry", but introduced the opposite problem: in addition to the scenario above, someone who posts once with an accidental date of, say, 2025 then has to keep remembering to post in 2025 or do a lot of mucking around with the "date out of order" flag. (Which is a whole 'nother complex thing to explain.)

We've decided that check was pretty useful in 2002 or whatever when fewer people used network-based time correction and might have very mis-set computer clocks, but it's 2014 now, and that's pretty much not an issue anymore. So, sometime in the near future -- not yet, but soon -- we'll be removing the check and no longer trying to detect very mis-dated posts. We wanted to give you all a heads up well in advance of the change.

This will not change the order that things sort on your Recent Entries page. If you post something dated 1970-01-01, it will still show up all the way at the very very end of your Recent Entries. If you post something dated 2020-01-01, it will still show up at the top of your Recent Entries page. (But if you're posting something dated 2020-01-01 to make it show on the top of your Recent Entries page, you should probably use a sticky entry instead -- sticky entries were designed to work specifically for that purpose.)

(Also: every time this comes up, someone suggests doing away with the date/time box entirely and simply assigning the entry the server time at the point the entry was received by the server, adjusted for the user's time zone, rather than letting users specify their own times or using the date/time that the update page was opened. This is one of the Perpetual Suggestions: every time it comes up, it becomes clear that opinions are split pretty evenly as to which would be better, default-to-time-you-hit-post or default-to-time-you-open-update-window, and that the people who care about it really care about it. When opinions are that split, and people hold their opinions that vehemently, we generally keep things the way they are. You can still check the 'use time when entry is posted' checkbox while posting if you want the timestamp on the entry to be the time you hit 'post' and not the time you opened the tab.)

Embedding sources

A reminder to all that embedding media, with HTML tags such as <object> and <embed>, is restricted for security purposes. (Someone with malicious intent can use those tags to embed malware in their journal and then trick other people into visiting it.)

We have a very wide range of "whitelisted" sites that we allow embedding media from, and we're always willing to add more. If there's a media site you want to embed media from, and it hasn't been whitelisted yet, open a support request with a sample of the code that the site provides you for embedding, and we'll add it to the whitelist.

Spam (spam spam spam)

And another reminder: we hate spammers! We hate them so much we have an entire project team dedicated to getting rid of them. But we're always happy to have more help from our users.

If you receive a spam comment: Delete the comment, choosing the "mark as spam" option. The dw_antispam team will take it from there.

If you come across a journal (through Latest Things or site search) that looks like it exists only to be spammy -- posting bland 'articles' only for the purpose of linking to other sites and boosting their search engine ranking, for instance: Open a support request in the Anti-Spam category, with a link to the journal. The dw_antispam team will nuke it from orbit. (It's the only way to be sure.)

*

That's it from us for another update! As always, if you're having problems with Dreamwidth, Support can help you; for notices of site problems and downtime, check the Twitter status page; if you've got an idea to make the site better, you can make a suggestion. (I'm a lot behind on the suggestions queue, though, just as a warning.)

Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated after an update is posted to dw_news. This was posted slightly before 0800 EST (see in your time zone). Please don't worry about missing notifications until at least 1000 EST.

Comments

[madfilkentist]

Good luck with the surgery.

[denise]

Thanks! We're doing awful things to my elbow in hopes it will fix everything downstream and I won't have to have the carpal tunnel release surgery done again. Fingers crossed.

[cheyinka]

My only concern with the date/time thing is crossposts failing because of it, though I imagine that'll happen infrequently enough that Support can handle it when it does. (The "you can't crosspost this because there is an entry newer than it already crossposted" error might need to mention that this might be a reason why this happened, if it doesn't already; I haven't seen the error in a couple months so I don't remember if it does.)

Heading off to the redesign post to comment there about that :)

[denise]

Yes, I'm hoping that will be rare enough that it won't be too much of a problem!

I don't actually think it's possible to change the error message you get when an xpost fails because of date/time issues -- that message is a direct passthrough from LJ, and I don't think we get any status code other than "nope, didn't work" that would allow us to parse out those failures from more general failures.

[ilyena_sylph]

*wanders through, drops a bucket of ♥*

DW: the only place where I totally love the news posts.

[havocthecat]

I think I got lucky and saw this early. Thanks again for the update! I appreciate the thoughtfulness you put into them.

[alierak]

Did you just call me hackish? :) IIRC I didn't own the comment nesting bug so I just gave a proof of concept that it could be done iteratively.

[denise]

no no no! That's not what I meant at all (I'd forgotten you had even gotten near that bug, actually) -- I meant that there were lots of ways we could have solved it, many of them hackish, or we could have killed ourselves trying to solve it an iterative-but-non-hackish way, but then Mark came in and went for the slightly-overkill-but-effective solution. I'm sorry I phrased it badly. :)

[filialucis]

On embedding sources: will you be adding support for embedding *in comments* at some point? I miss the ability to embed youtube vids in comments rather than just in the body of an entry and would love to see it added.

[denise]

There is a bug already reopened for that, yes! (We have to be stricter with comments for a few Boring Security Technical Reasons, but I'm pretty sure we'll be able to wrangle it.)

the very model of a modern mom and pop

[jjhunter]

'responsive' applies
to both design and admins?
the future is now

if no cyborg bodies yet
we embody our ideals

Re: the very model of a modern mom and pop

[dungeonmarm]

I thought, Is that a tanka? and counted the syllables.

Not only is it a tanka, it's a lovely one.

[squeaky.insanejournal.com]

Hey guys, I wrote some code a few months back when I was doing a username purge that uses the same email system, but drops these kind of messages into a separate queue with separate runners so they don't interfere with standard notifications.

Semes like something you can use to help with stuff like this. While it doesn't directly apply to this use case, what I did can easily be integrated (assuming you are still using gearman for notifications).

[denise]

I want to say that the Gearman queue already tries to process news posts separately, but honestly, I'm not 100% sure that's correct -- I view Gearman much akin to the depths of Ry'leh, etc. If you have the patch to email, that would be wonderful, thank you!

[red_lasbelin]

Just wanted to say that I enjoy your news updates. I always read them, and that's one of the reasons I noticed it had been quiet for a while. I hope your surgery goes well and the recovery too.

You and the rest of the DW team are awesome! You care about the DW community and listen to us, and you guys do your best to make everyone as happy as possible.

[denise]

Thanks for the good wishes! My surgeon is pretty confident, so I have my fingers crossed.

[passionfruit]

That exact scenario with the date/time problem happened to me earlier in the week. I'll be glad to finally see it on its way out. The community admin changes sound great too.

I'll definitely be reporting any spammers I come across. Ugh, I hate seeing spam cluttering up the latest things page.

[denise]

The minute we work out the last details on virtual gifts and set the system up, I am sending you a special, custom v-gift that consists of a Golden Hammer of Spammer Whacking. Because you are ace at finding them. :)

[musyc]

Woot, code push! News post! Huzzah! XD

[lavendertook]

Thank you for all the work and then explaining it to us.

Hoping that surgery will do the trick for you. <3

[holyschist]

Hmm, apologies if this is a stupid question, but if timestamps become a server thing and maybe the custom time box goes away entirely (what if people want to manually backdate specific entries?), what implication does that have for the long-so promised feature of scheduled entries?

[denise]

Timestamps aren't becoming a server thing -- we'll still be using the date and time from your computer (either the timestamp when you loaded the update page, or the timestamp when you posted the entry, depending on whether you have the 'use time when entry was posted' checkbox checked or not). We'll just be removing the "you can't post this entry, you have an entry timestamped after it" error that stops you from dating things out of order.

(We haven't decided yet what the UI is going to look like for scheduled entry scheduling -- we're still working on solving backend stuff, and because of how we're implementing things, we have to get server-side draft posts implemented before we can do scheduled posts; we've got a, well, draft of some of the draft post backend complete, but we're still trying to solve a few things there, too. The whole thing turned out to be way, way more complicated than we thought it would be originally. Still, I doubt our implementation of scheduled posts will look like LJ's, with the timestamp on the update page used as a "you should post it at this time" trigger; it doesn't fit in with some of the features I really want in the implementation.)

[owl]

Yay for responsive design! I've been doing that for about 18 months now in work, it's fun.

Is it going to be done dynamically to "flow" with the browser's viewport size, or is it keyed off the useragent string?

[denise]

fu may correct me, but I'm almost positive it's dynamic by viewport -- the page does re-flow as you resize your browser.

[azurelunatic]

Yay code push! Yay news!

I explained to my partner-in-crime that I was ducking out of work "early" (by 8pm*) last night and why, and that there was a certain amount of superstition around a fully uneventful code push which terrified everyone about the possibility of That Horrible Grinding Noise Six Weeks Later. I began singing selections from RENT. He began singing Blue Oyster Cult, on the grounds that "one teeny, tiny spark" was the best case, and destroying cities was the worst case.

Special <3 <3 <3 to Mark & Fu for fixing stuff on the fly!


* This is not as terribly late for me as it sounds, as I'm generally showing up in the neighborhood of noon.

surgery

[equinoxe]

good luck. all thoughts are with you.

[thistleingrey]

Glad to have an update! Best wishes for the surgery and swift recovery.

[retsuko]

Thanks so much for writing this up in a clear, helpful way. I wish you all the best with the surgery!

[nenya_kanadka]

Good luck with the surgery, and thanks so much for looking into and fixing (and so fast!) the nested commenting bug I stumbled on last night. Y'all are great at this.

<3

[fallen]

As someone responsible for one of those comment edge cases, thank you guys for the hard work and the bug fix! Honestly I always figured this was unintended/not officially supported behavior anyway, so we've been contenting ourself with flatview in our stupidly long comment thread. The DW team really does carefully think about everything and I appreciate that a lot.

[gloriouscurse]

You guys are the best, and good luck with surgery!

[ceded]

Thanks for the post, Denise. Good luck with your surgery and all the best.

[princessofgeeks]

thank you and best wishes.

[prototypical]

I think my right arm's as bad as your bum one since right now I can't use a mouse for more than ~5 minutes at a time or hold anything without pain in my hand, wrist, elbow, and shoulder in some combination. I hope everything goes well because RSIs are hell.

[misskat]

Co-signed. RSI are the worst.

[juliet316]

Great update and best of luck with the surgery.