mark: A photo of Mark kneeling on top of the Taal Volcano in the Philippines. It was a long hike. (Default)
Mark Smith ([staff profile] mark) wrote in [site community profile] dw_news2009-10-13 12:26 am

Weekly Post: October 12th, 2009

Monday, the 12th of October, and time for another weekly update about what's going on over here in the land of Dreamwidth.

This will be a little short and one-sided, mostly owing to the fact that I spent the whole day coding and forgot it's Monday, but I'll try to cover the highlights. And without further ado...

1. Code Push



Early last week (shortly after the update) we did a code push to bring live many of the exciting new things we'd been working on. I didn't call them out in advance, so, I'd like to do so now.

First up, we pushed a change that makes all posts start to crosspost automatically. This means that those of you who use services that post to Dreamwidth from another web site or client (Twitter repeaters, Semagic, other clients) will see your posts automatically start crossposting. If you don't want this behavior you can turn it off in the Manage Account page on the Other Sites tab. I know this came as a surprise for some people, so I apologize that this wasn't as well messaged as it should have been.

Second, we have a neat new feature that has long been asked for but just got implemented by [personal profile] yvi. If you have been maintaining an entry that shows up on top of all of your other entries by dating it far in the future, you don't have to do that anymore. We added the ability to specify a "sticky entry" that will always show up at the top of your journal, no matter what style you use. You can set this on the Manage Account page, Display tab.

Third, our styles team has been working really hard at building new styles. Sometimes, we find things that have to change in the core or in some of the layouts that power everybody's journals. If you ever find that your journal has suddenly gone weird (like a tribble took up residence in the corner or maybe even worse!) please feel free to let us know. If you file a support request or comment in the latest [site community profile] dw_maintenance post, we'll do our best to figure out what we changed and how to fix your style. We can't promise that we will always be able to, but we'll try our best.


2. Development



This week we closed 28 bugs (resolved!) and would like to give a special hand to [personal profile] yati who is a first time contributor this week. Thank you so much for helping make the site a better place. :)

Our code tour this week was put together by the tireless [personal profile] foxfirefey:

http://dw-dev.dreamwidth.org/26557.html

We made good progress on the review queue this week, too. I sat down and got the queue down to some 40 patches still needing review and the oldest being ~2 weeks old. This is still far too old, but the week was spent reviewing some of the bigger, more gnarly patches. The remaining few dozen should go faster. (Famous last words...)


3. Site Search



I promised before that this week we'd work on site search: and we have! Today I committed the bulk of the new code and I just have a few more tweaks to do before it can be pushed out with the next code push. (Sometime within the next week?)

Search is one of the things that we really want to do, and do well. But a lot of people expressed concern over privacy and making things too easy to find. We had a really good discussion on this in a [site community profile] dw_news post a few months ago and from that Denise and I put together a plan to address the privacy concerns in a way we felt was the best compromise between providing maximum security out of the box and still giving you a feature that works great.

There are two facets to the new things we're doing to search, so I'll break it down into the two components. Site Search lets you search for new content somewhere on Dreamwidth, and Journal Search is an expanded form of what we have today: finding content in a specific journal.

Site Search will let you search public posts (and only public posts!) on Dreamwidth and will be available to all users: even free users. I feel that this is a great community tool and will really help people to connect and find interesting content, so we're going to try to make this available to all users. (If the load demands turn out different than we expect, we MAY have to change this. Fair warning.)

Now, security and privacy concerns. Any search system is only as worthwhile as the usefulness of the data it indexes. We want to make sure that having site search is a useful thing, so we're making the system opt-out. However! We are going to automatically opt-out everybody who has chosen the "minimize my journal's inclusion in search engines" option. In other words, if you already opt-out of search inclusion (for external search engines) we will automatically opt you out of Dreamwidth's new Site Search.

Once the feature goes live, you are welcome to go turn it on, of course. I'd encourage you to do so: having content available to be found is one thing that will really help Dreamwidth to grow and users to connect, so while you may not want your content appearing on other search engines, I hope you will consider letting it be found from within the site.

Journal Search is the other side of search: you know the content is in a certain journal but you just don't remember which entry it's in exactly. Right now, we already have the ability for you to search your own journal and communities. We are extending this option to allow you to search other people's journals -- but of course, with their consent.

A new option is being added to the Privacy tab of the Manage Account page: "allow search by". You can specify that you want to allow everybody to search your journal, only people on your access list (default), or only you. I believe that limiting it to your access list is the best compromise of utility and privacy.

If you have any comments or thoughts, I'd love to hear them. I expect both Site Search and Journal Search to be released in the coming week. It will take some time to actually port the search index over to the new format, too, so when this is going out there may be a 6-12 hour downtime of the search system. I'll try to keep it short and will keep [site community profile] dw_maintenance and Twitter updated.


4. Styles Development Opening Up



If you have been interested in contributing to the overall styles experience on Dreamwidth -- from the S2 system itself to the pages used to make styles to the styles themselves -- please see this post in [site community profile] dw_styles:

http://dw-styles.dreamwidth.org/11790.html

We're not just looking for people to make styles; we're very much looking for people who want to improve the experience of building a style, selecting them, organizing or customizing or whatever. Even the language itself, if you really want to get down and mess with it.


5. Signing Off...for now!



Okay; short update this week. I have been asked to do one community promo:

* Shuttercookie is a new magazine for people who like photography and freedom and open source, but in order to publish anything, it needs contributors! If you're interested, take a look at the community: [community profile] shuttercookie.

We'll see you next week, when hopefully I will be telling you that we've successfully launched search and other new features and fixes for the week and have gotten our review queue down to 72-hours-max!
yvi: woman showing her biceps, text: "We can DW it" (Dreamwidth)

[personal profile] yvi 2009-10-13 07:39 am (UTC)(link)
I would like (well, not exactly 'like) to add that there's a bug in the sticky entry thing where old entries won't have their subject and entry text. However, this was fixed by an unrelated patch (yay code-cleanup!) and will be fixed in the next code-push. I own my bugs, they do not own me ;)

And shiny, shiny search things. I think the privacy defaults are great the way you chose them.

elizaria: generation kill text foxtrot.uniform.charlie.kilo (kings- David and icon butterfly)

[personal profile] elizaria 2009-10-13 09:38 am (UTC)(link)
Thanks for mentioning this - I just noticed this and went wtf? Now I know it's just to await the next push.

And this was a great option, before I forget to say thanks :)
yvi: Kaylee half-smiling, looking very pretty (Default)

[personal profile] yvi 2009-10-13 10:03 am (UTC)(link)
As a work-around, you can repost the post and then use that as a sticky entry. It just needs to be one of the last 200 posts in your journal at the moment. But yeah, I am hoping for the next code-push :)
elizaria: generation kill text foxtrot.uniform.charlie.kilo (rammstein- Till; let me light up the sky)

[personal profile] elizaria 2009-10-13 10:05 am (UTC)(link)
Oh, cool. Thanks for letting me know (I just edited thepost back to 2020 again), that way I can use it without having to wait the code-push :D
cesy: Home is where the <3 is (Dreamwidth) (Dreamwidth)

[personal profile] cesy 2009-10-13 07:53 am (UTC)(link)
Yay for site search, and it looks like you've done it very sensibly, so thanks.
downtide: shore birds (Default)

[personal profile] downtide 2009-10-13 08:22 am (UTC)(link)
Once the feature goes live, you are welcome to go turn it on, of course. I'd encourage you to do so: having content available to be found is one thing that will really help Dreamwidth to grow and users to connect, so while you may not want your content appearing on other search engines, I hope you will consider letting it be found from within the site.

Two questions about this -
1) will Site Search be available to only logged-in users or is it available to anyone with or without a Dreamwidth account? Will it be available to bots, for instance? I don't mind my journal being searchable by real Dreamwidth users, but I would hate to think that some automated system or spammer is searching for personal data.

If I have "minimise inclusion in search engines" selected, you opt me out of Site Search, which is fine. But if I then turn ON Site Search, does this also automatically make my journal vulnerable to search engines?

2) If I have Site Seach ON, will it be possible to exclude certain posts on a post-by-post basis without having to turn them private? or is it an all-or-nothing thing?
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2009-10-13 12:05 pm (UTC)(link)
Based on my reading of the code (and if I'm wrong I hope Mark will correct me):

1a). Logged-in users only; if someone who's not logged in hits the search page, it returns a "you need to be logged in to use this", just like it does now. We so don't have the resources to allow anyone to search any journal, not to mention it being something I know a lot of people would find skeevy!

1b). They're actually separate preferences entirely; once this feature goes live, there'll be a separate setting on the Privacy tab, and the two settings won't be connected. (We just didn't want to delay releasing this through another code push in order to push the preference first, make everyone pick their search settings, and then push the code that actually does the search in another two weeks or so, once everyone had a chance to make a choice.) We're just pre-populating the "can other DW users search my journal?" setting with roughly the same restriction level people have already chosen -- it's not permanently tied together.

2). We don't currently have plans to opt single posts out of site search, and probably won't ever -- not only would it require a major redesign of the update page (the number of things you can set on an entry update is already cluttered and hard for new people to really get if they've never used a LJ-based system before, so introducing a new per-entry setting is something we'll have to do very carefully) but it's also something that can already be handled by just setting an entry's security level to something more restrictive if you don't want it searchable.
downtide: shore birds (Default)

[personal profile] downtide 2009-10-13 05:51 pm (UTC)(link)
This is great, thanks. :)
noxie: friendly girl smiling (!friendly)

[personal profile] noxie 2009-10-13 09:48 am (UTC)(link)
I love the idea of the sticky entry, but it doesn't seem to work. If I define one, my journal content is gone and only an error message is displayed. So I guess I won't be using this. But thanks anyway for implementing this, it's a great idea!
elizaria: cat laying on a pillow (cat- lazy)

[personal profile] elizaria 2009-10-13 10:07 am (UTC)(link)
Could the discussion above be why yours didn't work? My post got empty too, but if I clicked on the Sticky it brought me back to the original post.

http://dw-news.dreamwidth.org/12509.html?thread=1317597#cmt1317597
noxie: friendly girl smiling (Default)

[personal profile] noxie 2009-10-13 10:44 am (UTC)(link)
I'm not sure. It's probably something related to my custom layout? But thanks for the input!
Edited 2009-10-13 10:47 (UTC)
yvi: Kaylee half-smiling, looking very pretty (Default)

[personal profile] yvi 2009-10-13 10:13 am (UTC)(link)
It probably (note the probably) breaks because you are using a custom layout(?) I suppose someone on [community profile] style_system or [site community profile] dw_styles might be able to help, as I am not very good with style modifications and such.
Edited 2009-10-13 10:14 (UTC)
noxie: friendly girl smiling (!friendly)

[personal profile] noxie 2009-10-13 10:47 am (UTC)(link)
Yes, I thought that might be the problem.
afuna: Cat under a blanket. Text: "Cats are just little people with Fur and Fangs" (Default)

[personal profile] afuna 2009-10-13 10:48 am (UTC)(link)
Hmm. Oh! i bet it's because core1 layouts don't have the StickyEntry class defined.

so I guess we need to add the StickyEntry class, and then put up instructions somewhere, explaining how to add the StickyEntry to the top of their entry list.
yvi: Kaylee half-smiling, looking very pretty (Default)

[personal profile] yvi 2009-10-13 10:50 am (UTC)(link)
Oh, that makes sense!

so I guess we need to add the StickyEntry class, and then put up instructions somewhere, explaining how to add the StickyEntry to the top of their entry list.

If someone could do that, that would be wonderful :)
afuna: Cat under a blanket. Text: "Cats are just little people with Fur and Fangs" (Default)

[personal profile] afuna 2009-10-13 10:51 am (UTC)(link)
Hehehe, if I remember, I'll try!
juliet: (dreamwidth dimension)

[personal profile] juliet 2009-10-13 10:29 am (UTC)(link)
Re search: am I correct to assume that if your access list or People In General are allowed to run Journal Search on your journal, it won't show up with links to any entries that that person doesn't have access to?
yvi: Kaylee half-smiling, looking very pretty (Default)

[personal profile] yvi 2009-10-13 10:32 am (UTC)(link)
I suppose Mark is in bed, but from his comment to the bug open for site-wide and journal search:

By default, it is set to "people I give access to only". This seemed to be the sane option: you allow these people some level of access, so we allow them to search the public content of your journal. (But only the public content.)

So it seems that locked entries don't show up for other people currently.

FUTURE IMPROVEMENTS --

I did a lot of the legwork to allow for journal search to show all posts you can see: not just public posts. I will try to implement that before this feature goes live, but it does depend on how well Sphinx handles many constraints on searches.
juliet: (dreamwidth dimension)

[personal profile] juliet 2009-10-13 03:16 pm (UTC)(link)
cool, thanks for that!
sophie: A cartoon-like representation of a girl standing on a hill, with brown hair, blue eyes, a flowery top, and blue skirt. ☀ (Default)

[personal profile] sophie 2009-10-14 04:58 am (UTC)(link)
Just to let you know, the only reason it was 'public only' posts at that time was because it hadn't yet been programmed that it could do only posts that the searcher could see, so it was
public-only until it could be programmed. It's been programmed in now, so now it'll show posts that the searcher would normally be able to see, including locked posts - but *only* if they could see them otherwise.

[edit: Yes, this is before the feature has even gone live. Mark was betting he'd be able to do this before it went live, which is why the public-only-ness of it wasn't mentioned in this post.]
Edited 2009-10-14 05:00 (UTC)
juliet: (dreamwidth dimension)

[personal profile] juliet 2009-10-14 12:21 pm (UTC)(link)
Ooh, shiny! Sorry to jump the gun there :)
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2009-10-13 11:58 am (UTC)(link)
We will never ever ever show anyone a post they're not authorized to see in search results, don't worry!
juliet: (dreamwidth dimension)

[personal profile] juliet 2009-10-13 03:16 pm (UTC)(link)
I figured this would be the case - was just clarifying :)
janinedog: (Default)

[personal profile] janinedog 2009-10-13 03:22 pm (UTC)(link)
What [personal profile] yvi said is correct. Right now, search only returns public posts (unless you search your own journal directly, in which case it returns all posts). In the future, it's hoped that it'll be able to search all entries you have access to see.
princessofgeeks: (Default)

[personal profile] princessofgeeks 2009-10-13 11:36 am (UTC)(link)
the search function sounds amazingly wonderful.

thank you for all your hard work.
cypher: (:D)

[personal profile] cypher 2009-10-13 01:33 pm (UTC)(link)
Thank you for all your hard work, and especially for thinking about the ways that people would want to control their search participation -- it's awesome to see how committed you are to making the site work for all of us. :3
inarticulate: Ginshu from Amatsuki smiling. (we're on a bridge atsuki!11!)

[personal profile] inarticulate 2009-10-13 02:52 pm (UTC)(link)
Thank you so much for addressing privacy concerns there! I'm curious to see how it'll work out. :D
janinedog: (Default)

[personal profile] janinedog 2009-10-13 03:24 pm (UTC)(link)
One thing I noticed you didn't explicitly mention...Journal Search still requires that either the searcher or the journal being searched is a paid account. So if you want to search your friend's journal, either you or your friend must be a paid account.
helens78: A man in a leather jacket, seated on the ground, looks up hopefully. (Default)

[personal profile] helens78 2009-10-13 03:33 pm (UTC)(link)
The search function sounds fantastic, and it looks like you've made great choices that allow people to preserve privacy while still being accessible in search. Kudos!
shyfoxling: Ravenclaw crest (confused (severus ?))

[personal profile] shyfoxling 2009-10-13 07:39 pm (UTC)(link)
"Crosspost by default" with Semagic worked like a charm, but I'm getting an error when I subsequently try to edit the post (which is also how I would delete it without having to go delete each one individually):

Server reply is broken and cannot be processed

events_1_anum 234 events_1_event This+is+testing+the+crosspost-by-default+now+suppsoedly+being+enabled+in+client+as+well+as+posting+from+web+via+Dreamwidth. events_1_eventtime 2009-10-13 12:33:00 events_1_itemid 776 events_1_url http://elethian.dreamwidth.org/198890.html events_count 1 prop_1_itemid 776 prop_1_name xpost prop_1_value

[and then three Unicode characters which the comment box won't let me reproduce here]

I can view the journal history list but if I try to actually open the crossposted post, I get that error.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

[staff profile] denise 2009-10-13 10:52 pm (UTC)(link)
Yeah, unfortunately editing (crossposted) posts in certain clients is broken, because the clients don't know what to do with extra entry properties (like the properties that govern crossposting). We do hope to get our 'own' clients as soon as we find someone who's got desktop experience (none of us do!), but until then, you may want to suggest to the client author that he change how extra properties are handled!
ranunculus: (Default)

[personal profile] ranunculus 2009-10-14 04:03 pm (UTC)(link)
Really wonderful things going on. I love the "Sticky Note" idea, and search-able journals is awesome. Makes me wish I knew how to write code and could help, but you really, really don't want this dyslexic/mirror image brain helping!!!
emory: (Default)

[personal profile] emory 2009-10-15 06:27 am (UTC)(link)
I <3 you guys SO HARD.
elf: Dreamwidth D with flowers (Dreamwidth)

Site search

[personal profile] elf 2009-10-15 02:42 pm (UTC)(link)
I love this. Love, love LOVE this.

I've opted out of botsearches, but would very much like to be included in site searches. I'm one of the advocates for a security level of "logged-in users only" (and yeah, I know all the problems with people thinking that's got the same kind of security as a f'lock, and so on; I more-or-less understand why it was decided not to have it).

OMG, it's like ljseek done right. And the opt-out approach is perfect--you've got info about people who want privacy for some searches, and using that data as a starting point is terrific.
moondancer: (Default)

Client crossposting

[personal profile] moondancer 2009-10-15 04:04 pm (UTC)(link)
Uh, really don't like the auto crosspost from these (Semagic, Twitter, etc.) as for the most part, folks who want to cross post those entries have already set it up in their clients, and having them repost something that's already posted is bad, like spamish, almost.

Otherwise, great work as usual!
cesy: "Cesy" - An old-fashioned quill and ink (Default)

Re: Client crossposting

[personal profile] cesy 2009-10-16 09:41 am (UTC)(link)
You can turn off auto-crossposting in your settings. Then if you ever update from the web interface, you can turn on crossposting just for that entry.
trinity_clare: (breaking news)

[personal profile] trinity_clare 2009-10-16 04:15 am (UTC)(link)
Question: I went and set the sticky post up, but the future-dated one is still there. Is that expected behavior? Should I go give it some random date in the past so it goes away?
cesy: "Cesy" - An old-fashioned quill and ink (Default)

[personal profile] cesy 2009-10-16 09:32 am (UTC)(link)
Yes, that is expected behaviour, so you can now set it to any date you like.
sinaienantia: (Default)

[personal profile] sinaienantia 2009-10-17 02:43 am (UTC)(link)
I love the sticky feature!

But........... question: do you think it will be possible in the future to sticky more than one post? Not something I need currently, but I could definitely see making two or three organizational posts that I might want stickied.

Either way, thanks for this! And for search - I love the idea of being able to search mine or other journals for older stuff (makes it a lot less frustrating when there's some old information I vaguely remember and want to refresh my memory).
cesy: "Cesy" - An old-fashioned quill and ink (Default)

[personal profile] cesy 2009-10-17 05:23 am (UTC)(link)
For the moment, I have one sticky post which includes links to others, which is a reasonable workaround. To be able to sticky more than one post, the normal process would be to suggest it.
sinaienantia: (Default)

[personal profile] sinaienantia 2009-10-17 07:36 pm (UTC)(link)
Point. Thank you!
starlady: (moon dream)

[personal profile] starlady 2009-12-21 02:27 am (UTC)(link)
We added the ability to specify a "sticky entry" that will always show up at the top of your journal, no matter what style you use. You can set this on the Manage Account page, Display tab.

Sorry to comment on such an old entry, but can this possibly be added to the FAQs, or if it's there, could it be put somewhere more prominent and/or set to the keyword "sticky"? I just spent 10 minutes searching, and had to use Google, to find this information. Thanks!
cesy: Unofficial Dreamwidth volunteer (Dreamwidth volunteer)

[personal profile] cesy 2009-12-21 10:21 am (UTC)(link)
I believe there's a project currently underway to update the FAQ for this. Thanks for pointing it out.
starlady: (moon dream)

[personal profile] starlady 2009-12-21 08:54 pm (UTC)(link)
Nice. Thanks for your reply!