twitter – D’Arcy Norman dot net

Community Detection on Twitter

I’ve been thinking a lot about how to visualize online presence and community. There are lots of great tools to do post-hoc analysis, but I’m thinking about something more realtime. It doesn’t exist yet, though. In the meantime, I’m playing around with the current tools to get a feel for what stories they can pull from the social graph data.

Yesterday, I followed the howto from Caleb Jones, to pull the social graph data from my Twitter account. The process took about 15 hours, because of Twitter’s helpful throttling of API calls. Thankfully, the twecoll python tool takes that into account and gracefully pauses when Twitter API tells it to cool it.

Once twecoll pulled out the raw data, I fed it into Gephi, and then followed Caleb’s howto for community detection.

I tweaked the layout a bit, played with the rendering settings, and came up with this:

dlnorman Twitter Graph 2015-01-05 v2 — Community detection on @dlnorman, showing clusters of “edtech”, “LMS”, UCalgary” and “Calgary” network members.

There are a few main concentrations of people. The blue-ish one on the right is loosely “edtech folks” – but it’s strongly biased by “BC Edtech Folks”. The red patch at the top is “LMS-ish folks”, strongly represented by D2Lers. The far left is “UCalgary” – and it was able to pull out a cluster of official-ish accounts, student union accounts, and various other subclusters from UCalgary. The bottom left is loosely “Calgary” – and includes subclusters for politics, media, design, and cycling. Lots of overlap between design and cycling subclusters. Go figure.

Lessons learned from this exercise:

It takes waaaay too long to do anything with this kind of community analysis on the fly. Post-hoc after-the-fact analysis is where things are now.
Even with super-helpful scripts, the process is not something most people will do. And the new Gephi 0.9 is fantastic – but, again,It’s an excellent tool for researchers, and most people aren’t going to use it. The user experience for personal-social-network-analysis needs to come a long way before it can be used by everyone.
Even with the pretty picture and community detection – so what? What can you actually do with this information. I have some ideas about that, but need to do some exploration first.

Update: I tweaked the layout. Here’s a better version of my twitter network graph with community detection:

Tent – distributed social networking

Via a post by John Gruber, Tent:

Tent is a protocol for open, decentralized social networking. Tent users share content with apps and each other. Anyone can run a Tent server, or write an app or alternative server implementation that uses the Tent protocol. Users can take their content and relationships with them when they change or move servers. Tent supports extensible data types so developers can create new kinds of interaction.

Oh, hell yes. Sign me up. I’ll be installing the reference implementation ASAP. This distributed approach makes much more sense than the app.net model of replacing one central silo with a shiny new one…

Update: Timmmmmmmyboy reminded me about Diaspora, and asked how is Tent different? I don’t know if it is (or will be) different. It comes down to a few things:

how easy it is for people to install. if it’s hard1, it will die on the vine, like diaspora did (or is doing. or whatever).
how well self-hosted and service-hosted instances interoperate. if they don’t interoperate, it won’t have enough activity to make it worth using. death spiral ensues.
and how well it plays with other platforms. can it post to twitter? facebook? vice versa? pinning our hopes on the chance that everyone on the planet just moves to the new platform is just not reasonable or desirable. heterogeneity. competition. etc… platforms need to play well with each other.
apps that can talk to it. twitter took off because there were apps for every device that made it easy to do stuff with it2. will there be an iPhone/Android/Win8 app for this? A desktop app for win/mac? a decent web interface? etc…

whereby hard is defined as “non-trivial” rather than “requires a PhD”. if an application is harder to install than, say, WordPress, it just won’t get installed and maintained by even geeky humans… [↩]
and then TwitterCo™ decided to start killing third party apps… interesting… [↩]

on commercial silo-ification of online discourse

I complain about twitter, facebook, and other corporate silos as much as the next person. If only there was some alternative… Something that didn’t mine everything and everyone I know to sell that data to the highest bidder(s).

One response to this has been the development of new private commercial silos, with barriers to entry (subscription fees, or invitation requirements) that are intended to keep out the riff-raff while letting the cool and worthy folks into the conversation.

This is the wrong direction.

I am not even remotely interested in participating in the shiny new (and newly funded) app.net twitter clone/replacement, nor the online-discussion-blogosphere-replacement Branch.com. These solve the noise problem, but don’t solve the mining-my-everything-for-monetizing-synergy-blech problem1.

Then, there’s Diaspora. This project sounded extremely promising. An installable application to replace social silos like Facebook. Very interesting software, but it’ll never take off as a high-volume application run by humans, because most humans won’t be installing Ruby on Rails apps.2 Maybe, if it was offered as a one-click-installer on common web hosting providers? It’s not.

In a perfect world, I’d post my stuff to a place3 I own. Like, say, my blog4. And people who choose to follow, would see the updates alongside the rest of the folks they follow. On something they own. And vice versa. This isn’t new. RSS and PubSubHubbub, with a decent installable interface. Done.

We don’t need more silos. We need to be able to extract ourselves from them, and still be able to connect with each other. There has to be a better solution to that problem, before we continue to fragment and isolate our online interactions.

I’ll be staying with Twitter, as much as I hate it, because that’s where the people I follow are. But, I’m only staying for as long as I have to. As soon as something comes along that lets me own my stuff and follow who I want to, without exposing everything to monitoring/monetizing by central third parties, I’m gone.

I’m not saying that app.net or branch.com are currently mining everything. but what’s to stop them? if I let someone else control my interactions with others, I have to give them a pretty high level of trust. I have no reason to do that with either of those services… [↩]
yes, I get that it’s not too hard, and it’s something I could do, but then what? I’d get to connect with the other 36 geeks that installed it? yeah… [↩]
simple, lightweight, easily installable by humans [↩]
I’d experimented with “asides” here awhile back – but without the notifications layer, it was pretty much useless… [↩]

biotwitter

published

so… this happened…

If I had to guess, it’d likely be related to this. Or maybe this.

photo by Boone, who surprisingly didn’t host the image himself, so I reclaimed it from YFrog 😉

indicating category in blog post autotweet

I’ve been using the Twitter Tools plugin to automatically post tweets whenever I publish a post here. Scott mentioned that with my recent consolidation, there are a fair number of tweets coming through the blog, and it’d be handy to have some indicator if it’s a tweet about a blog post (which he might bother to read) or a bit of ephemeral media (which he likely wouldn’t).

I couldn’t see a way to do that within Twitter Tools directly, so I wrote a VERY simple (and horribly ugly hackish) plugin to add that functionality. If it works (and I’m not anywhere near 100% confident that it will), then this post should show up on Twitter with a ” (blog)” appended to the tweet. Ephemeral media posts have ” (media)” appended. The ephemeral-appending-bit worked, but I’m not sure if it’s doing that for all posts, or behaving as intended. Only one way to find out…

*Update*: Well, whaddayaknow? It worked. Actual PHP code that executes without barfing out errors, and does what I intended. The wonders never cease…

historical tweets

A [blog of funny faux-historical tweets](http://historicaltweets.com/) published a book a couple of years ago, and used [one of my photos](http://www.flickr.com/photos/dnorman/867989188/) as the background for a twitter account of a caveman. Pretty cool. They’re going to print again, and the photo’s still in there.

The [book’s available on Amazon.com](http://www.amazon.com/Historical-Tweets-Completely-Unabridged-Ridiculously/dp/034552263X/). I don’t see a penny of sales, but I wouldn’t have seen a penny for that photo anyway…

global twittererings

[frog design](http://frogdesign.com) [built a really cool realtime twitter geolocation visualizing app](http://aworldoftweets.frogdesign.com/), that builds a heatmap of locations for tweets posted as it monitors. Here’s the heatmap after letting it run for an hour, on a Wednesday, around noon MST:

I’d expected Canada to be much more highly represented – my perception of using Twitter is that it’s largely US/Canada/UK centric. Turns out, the US is responsible for about a third of all tweets, followed by the UK, Indonesia (? REALLY! Indonesia. I know!), Netherlands and Mexico. Canada is down in 9th place. Surprising.