Weekly Update: February 16th, 2010

[mark]

Time for the weekly update! It's late this week because I was out with some friends yesterday and didn't get home until late, sorry about that. This week we bring you more information on cross-site reading and some updates on what's been going on around here in Dreamwidth land.

1. Cross-Site Reading

I'll cover this first since it seems to be a well received topic when mentioned last week. This is sort of scattershot facts, feedback is welcome.

* The feature is going to be a paid user feature, because of how much data we have to store and the load I expect it to place on our servers.

* We will only download new entries every 60 minutes (this may change). This means that, when you view your reading page on Dreamwidth, we only contact remote sites every ~60 minutes to ask for new entries.

(If you typically view your reading page and refresh it every 30 seconds to see if there are new entries, it won't work that way with cross-site reading. This 60 minutes might change when we see how many people are using the feature, but this is the main way that we are able to control how often we hit remote sites. If we hit them too often, then they'll ban us for being bots/bad/broken/etc. If we wait too long, we could miss entries.)

* Technically, we can only load the most recent 100 entries from the remote site. This is a limitation of the protocol, it only allows us to get recent items. Skipping back indefinitely won't work. However, if there never get to skip=100 on your reading usually, this won't affect you.

* Comment counts will be updated every time we download entries, but it could be up to an hour out of date. The same goes with edited entries; if we download entries, we won't be able to tell that they were edited until the next time we pull (next hour).

* We only download entries while you're browsing the site. In short, whenever you view a page on Dreamwidth (any page), we will update your reading page with remote entries if it's been more than an hour.

In effect, this feature will work best for average users. If you only watch 200-300 accounts and get fewer than 200-300 new entries a day on your remote reading/friends pages, you will find that this feature works pretty well for you. On the other hand, if you've maxed out your subscriptions and the people you read post hundreds of posts every few hours, you will probably not find this feature too useful.

Okay, with the technicals out of the way, let's discuss privacy. Since we're downloading entries for people that don't use Dreamwidth, privacy is a really tough question here and one we finally think we have a good answer for.

Downloaded entries are only viewable to the person who is using the cross-site reading feature.

That's the rule, and it is pretty simple. We will not show entries to anybody except the person who we downloaded them for. Even if it's a public entry; you will never be able to see it on the site unless you are logged in to the account that has cross-site reading turned on and subscribes to that account on the remote site.

Further, we will not allow any interaction with the entry on Dreamwidth. They can be viewed, but that's about it -- you can't add them to memories, bookmark, comment on, edit, or anything. It's a read-only view.

Lastly, entries are only stored for two weeks. At that point we remove them from our servers completely. (Similar to syndicated feeds.)

2. Weekly Community Participation Meme

I've seen a meme floating about Dreamwidth since yesterday that I really like. It's about community participation and was originated by rydra_wong:

http://rydra-wong.dreamwidth.org/197790.html

In short, people who say 'DW is too quiet' can solve the problem by actually making it louder! If we all take a bit of time out of our week to post to a community about something we find interesting, we'll build them up and work towards being the kind of place people want to gather.

For this week, denise recommends poetry for all of your poetical needs. zarhooie further mentioned selfportraitweekly as a new community that is just getting off the ground and could use some participation, too.

If you have your own suggestions, leave them in the comments!

3. Community Imports

A lot of people have asked us about importing communities from remote sites. We've been doing this manually until now, but we are at the point where we want to start opening this up to everybody.

A bug has been filed to make the importer work for communities for everybody. There will be some guidelines (you must be a maintainer, you should get permission of your community first), but we will no longer be manually vetting every community that gets imported.

This is made possible because of all the OpenID changes we made to allow users of other services to actually maintain their entries in communities: delete them, moderate comments, etc. Now that these users have the tools they need to actually maintain their entries, we feel confident opening up the community import tools.

Please keep an eye on the weekly updates for more information about when this comes to fruition.

4. Latest Posts - Olympics!

The 2010 Winter Olympics are going! We've added a feed to the Latest Things tracker that will let you find posts related to the Olympics:

http://www.dreamwidth.org/latest?feed=olympics

Keep in mind this feed only updates when someone makes a post, so it looks a little barren right now (I just made it!) but should fill up as people post things.

5. Suggestions Update

denise wanted me to mention that our suggestions process has reached another milestone. We just passed 400 suggestions that have gone through the process, and the breakdown of results is:

* 86 (21%) suggestions rejected due to impracticality, community disapproval, or not being what we want for Dreamwidth
* 24 (6%) deferred for later review (read, denise and I sitting down and hashing them out)
* 221 (54.7%) in Bugzilla waiting for implementation
* 73 (18%) have been implemented

So, ~40% of suggestions have been resolved one way or another, and ~60% are still waiting for resolution. I feel this is a good ratio! I don't expect us to ever implement every single good idea people have (if only we had the developers to do that!) but to know that we're making progress on them and actually getting things done that users suggest is a good sign.

6. Update Page Redesign

This is a big issue and we want to start getting everybody thinking about it now. denise has spent a lot of time (a LOT of time) on working on the update page. She's been putting together wireframes, researching other blogging tools, showing designs to people, and iterating. She's getting to the point where the wireframes can move to the next step, so let me take a minute to tell you what's going on.

In short, we're going to change the "metaphor" that represents posting to your account. Right now, the post interaction metaphor is akin to a "neverending scroll". You write an entry, post it, and that's that. If you want to edit it, you edit the "last entry" or you have to look around the scroll to find what you want to edit.

Similarly, if you want to create a draft, it only stores what you're writing at the newest end of said scroll. One item, there until you either rip it off or commit it. This metaphor has worked for a while, but it's not going to support the new functionality that we want to add to the system.

The new metaphor denise has been working on is in the wireframe stage right now. What we're going to do next is create a mockup (a small, non-functional preview) that you will be able to interact with. You will be able to get a feel for how it looks, how we want it to work, and then provide feedback.

Once feedback is received, we'll update the mockup and do another round of review on it. "Okay, this is what you told us about the first mockup, how do you like the second?" That process can repeat as many times as necessary, but at some point it will be considered done and we'll move to a full implementation of it.

One thing to note now: we will not be supporting the 'old update page' AND a 'new update page'. Once we've gone through the community review and gotten the mockup to a good place and implemented it, that will be the only update page.

Hopefully by telling you about this in the weekly update and in advance, we can make this process relatively painless. I know it will be a big change for everybody, and there will certainly be some people who prefer the old page, but I hope we can convince you with the great functionality the new metaphor will provide.

Stay tuned!

7. Long Update is Long

This is probably the longest update I've written in quite some time, but we've now reached the end. Yes, I didn't do a development update section this week, focusing instead on bigger items. Next week we'll dive into the development of the past two weeks and see how things are going.

Thanks to all of you for being part of the Dreamwidth community. See you next week!

Comments

[matgb]

Posting that material over here will be a disincentive to that dialogue

Corrected for you there.

This will allow those that predominantly or exclusively use DW to read what you've posted on LJ. If they like or want to respond, they're more likely to do so using this feature.

REgardless, your journal (assuming the same username on LJ) has the RSS feed active.

That means anyone can take your content and read it using their feedreader (off LJ), and users of sites or services that allow for authenticated feeds can also read your freinds only content off site.

If you want only people that use their LJ friends page to know you've updated, you can turn off your feed, and remove those that've switched to DW from your access filters.

But your objection is based on a false assumption; this will, genuinely, encourage more interaction with your journal.

In addition, by ensuring only the person subscribed can access the entry, even if you've posted it publicly, there is no infringement. Regardless of which, you release an unlicenced RSS feed, which has far fewer restrictions.

[kishenehn]

Thank you for noting that I'd forgotten to tighten the RSS restrictions on that journal ... I've corrected that now.

And I'll say that I still strongly disagree with your other assertions, but that's what I expected and that's the way it goes. I'll leave it all at that, but I'll reiterate once more: if you honestly respected the users of other blogging platforms, you wouldn't import our material without first obtaining our permission. If its truly as positive a thing as you all claim it to be, then what would be the big deal in asking first? The fact that DW doesn't plan to do that seems like an implication that my assumption is correct, rather than yours.

[ree]

Does Bloglines ask you? Does Google Reader?

How would you expect to be contacted about this feature?

[kishenehn]

All it would take is a single, one-or-two sentence e-mail the first time the DW server wants to download material from a journal on another platform, with a link that could be licked on to accept. That wouldn't be too difficult, at all, and would certainly be a sign of good faith from DW.

[flick]

To what email address? If it's openly listed, most people will spam filter it to death; if not, it won't get there.

[kishenehn]

I don't know about other platforms, but the vast majority of DW's requests would go to LJ, and [journalname] at livejournal.com should work for everyone over there.

[flick]

As I said, if it's openly listed, it will get spammed to death.

You can't assume that people are going to monitor an address that's known to be compromised.

[janinedog]

The @livejournal.com email is a paid only feature, so it only works for paid users (and only then if the user hasn't disabled it).

[amadi]

Right now, the set up goes something like this...

Your Entry -> Aggregated on LJ Friend's List Page -> Reader

The Dreamwidth set up will make it something like this...

Your Entry -> Aggregated on Dreamwidth Reading Page -> Reader

You've already given permission for Reader to see Your Entry by giving Reader access to your entries. When they read, they can choose to click through to comment, on LJ, just as normal.

The server processes behind the scenes are really irrelevant to the Reader, their concern is getting the content they want in a manner that's convenient, whether that's an LJ friend's page, a Dreamwidth Remote Sites page or an RSS reader. Opposing this before it's even implemented is placing a higher value on the URL displayed in the address bar of the page where your friends read your entries than on actually having your entries read, on the unproven assumption that people will be disinclined to click a link to comment.

[flick]

(I, personally, really look forward to when this happens. However.)

I can see where you're coming from, but would it be better or worse if they instead did the existing (fiddly) work-around so that they can read your LJ via an RSS reader that doesn't give them handy comment links? Because that is possible, even for locked posts.

[helens78]

First off, just want to let you know that this is not meant as a criticism of you or your desire to keep your blogging encapsulated entirely within LJ. If that's your choice, you are welcome to do it! No harm done to anybody.

But!

if you honestly respected the users of other blogging platforms, you wouldn't import our material without first obtaining our permission

I do disagree with this assertion; an RSS feed is by its very nature not just providing that permission, but inviting users to experience content on the platform that most suits their needs.

I recognize that many users are not familiar with the fact that LJ (and other blogging platforms) by-default offer an open RSS feed of all public content, but that does not change the fact that RSS feeds and readers are an integral part of the blogosphere as a whole. The most popular blogging platforms all offer RSS feeds by default: Blogger, WordPress, TypePad, MovableType, LJ and all LJ-clone sites, and now DW and its fork. Even podcasts are generally available not just to iTunes users, but by RSS feed.

Simply put, people use RSS and other blog aggregators to create a community experience tailored to their needs as a reader -- sometimes because it's easier, sometimes because it's more accessible, sometimes because it suits their needs better in ways I'm sure I haven't thought about yet.

Certainly if a blog wants to turn that ability off, that's fine (although there are many blogging platforms that don't allow users to disable RSS, and I think people should make sure they research their platform to ensure that it doesn't offer the public any more data than they personally want to see offered; it seems like every week we're finding another service that isn't as private as we'd like). But to suggest that more permission needs to be asked for and received before using a feature built into the code of the vast majority of the blogosphere... well, I have to admit, it does not make a whole lot of sense to me! If I'm missing some nuance in your argument, please feel free to correct me!

[ladysunflow]

"if you honestly respected the users of other blogging platforms, you wouldn't import our material without first obtaining our permission"

I do disagree with this assertion

When it comes to RRS feeds, you're right. But you can't explain the import feature in the same way. The fact is, people who do not want their LJ content (comments to someone LJ or LJ community posts) to be available on DW have no say in the matter. The most they can do is try to hunt it down and delete it piece by tiny piece.

[mark]

Some people do not want, some people do. It's impossible to actually do what everybody wants, so you have to make a decision about what you believe is the right thing to do.

That's sort of the entire point of Dreamwidth for me: we make a stand for what we believe, and we're honest about it.

I believe in a site that doesn't pay lip service to users while simultaneously doing what the advertisers or investors say to do.

I believe in a site that actually listens to its users and tries to find good solutions to the problems they have.

I believe in not bowing to pressure from lawyers to remove content that is legal; similarly, I'd rather not use PayPal than accede to their demands to run our service the way they want.

But all of these issues have two sides: obviously, we could make more money if we wanted to skip that principle. Or we could be more popular by ignoring our users and trying to duplicate Facebook. Similarly, we had to make it harder to pay for some users because we no longer support PayPal.

The import system is the same. We have chosen what is right for our users (allow imports) even though it means that some people will not like the decision.

Nothing is perfect. We make choices on what we believe is right, and we are always happy to explain and talk about the choices. In the end, the community can choose whether or not our choices are correct by choosing to support us or not.

[jumpuphigh]

Nothing is perfect. We make choices on what we believe is right, and we are always happy to explain and talk about the choices. In the end, the community can choose whether or not our choices are correct by choosing to support us or not.

Bravo!

[ladysunflow]

Therefore I do not support you and your very relative definition of right. And your very US-ian "speech talk" does not work on me. Works on others. Oh well.

However would it be so hard to build in an option to allow Livejournal openIDs to delete witha single click all of their content that's been downloaded? Or to check a box that says "in future imports /dev/null all the content with this username"? Or at least a page that would point to specific DW usernames where one might find that content (not necessarily single posts). People and communities change names when they import. When I delete something off LJ, I want to really be able to delete it on DW, and not just in theory.

[spudtater]

I'm pretty sure that when you delete an entry off LJ, it'll disappear from DW within an hour. Plus, everything will disappear within at most two weeks.

Your problem appears to be with LJ, not DW

[matgb]

I think, from what you've said in these two comments, that your objection is to comments you've made on other journals being imported to DW so that they show on the DW entries of users that are now here, correct?

If so, that's not a problem with DW or the way they've implemented the import. My journal is backed up in a number of places, and all of the comments made to it on LJ up to a certain point exist here on DW and on a backup file on my harddrive and on an import I have made of my complete journal to wordpress.com.

Of all the backups I have, the only one that commenters have any control of whatsoever is DW, where they maintain control through their OpenIDs.

When you make a comment to a post controlled by another, or make a post to a comm maintained by another, you are putting control of your content in the hands of someone else. They are then free to take your content and do things with it--you, for the most part, maintain copyright, but not control (and the copyright maintenence is unusual in LJ based sites, most other platforms assume a comment made is assigned to the site you're making it on).

If I have understood your objection correctly, then your problem is with the LJ export API, which allows all of us to extract our journals in a number of ways. DW is merely taking advantage of that API in a more popular way (wordpress have upgraded their LJ import feature a lot of late, in response to demand from others seeking to leave LJ).

If you don't want your comments to be exported elsewhere, you need to lobby LJ to allow an opt out from that API, as otherwise you'll need to persuade every other blogging platform that knows how to use that API to block you as an individual, and frankly, that isn't going to happen.

If I've misunderstood your objection, then apologies, but if I'm right, then DW are not in the wrong on this, it is LJ that allows them access to the content, and the people you've commented on that have chosen to export that content.

[kishenehn]

Well, thank you for at least confirming the inherent hypocrisy in the professed philosophy of your operation. It's troubling, but at least in retrospect not all that unexpected to find that you're only truly interested in respecting those writers who've bought into your project.

[mark]

Life is hypocrisy, by your standards. Let's all sit around and not actually do anything!

You can't do what everybody wants at the same time. If you believe that is possible, please tell me how! I'd love to make every single person in the world happy concurrently.

[medie]

I definitely fall into the "omg, want!" category. This is the sort of feature that'll enable me to more fully move into DW as my home site as opposed to LJ as there are still so many people and communities on LJ that I don't want to miss reading/commenting to.


Thank you! This is great news!

[ninetydegrees]

Nothing is perfect. We make choices on what we believe is right, and we are always happy to explain and talk about the choices. In the end, the community can choose whether or not our choices are correct by choosing to support us or not.

You know what? I wish *feeds* were done the way you're planning on doing cross-site reading. I would never have restricted them to titles only otherwise. :)

[anatsuno]

What you're basically objecting to is not the actions of Dreamwidth, but the limitations of protocols that are completely ubiquitous today. You control the fact that you put out an RSS feed or not; other services drink from that hose if that hose exists. If you don't want to have a hose, either you're using the wrong service or you're using it badly - it's not the protocol's fault. So of course you can blame Dth for doing something you find incorrect/unethical (and googleReader, and Netvibes, and any number of RSS readers either on the web or desktop clients), but you have, to be fair, to place equal blame on Lj for fitting a hose to your journal in the first place. Or... you can consider that picking LJ as a service was not what suited all your needs... But then, most of us compromise when using a service or another (I haven't found any that do exactly what I want them to so far, I don't think :/).

[stormy]

If you'd like people to ask you - perhaps an announcement on your journal to your friends list stating your opinions on it will be helpful. Since this really only applies to those who have you friended, it's best to be up front about your feelings.

[msilverstar]

I agree, and will respect the wishes of any of my ljfriends who prefer not to be read here.

[blktauna]

If LJ honestly respected their users wishes, they wouldn't be spewing your content all over in the hopes of making $$ from it. If you want your stuff to stay on LJ, take it up with them. They are the ones who are offering up the material.