Editing Google Reader subscriptions with a text editor

Having tried many RSS readers I finally stayed with Google Reader. It's not perfect, but works - and let me track my feeds from many computers.

Still, while I am happy with Google Reader as a reader, I am by far less excited with it's management functions. Renaming feeds, reorganizing folders, disabling feeds, replacing them with alternative versions, etc - all those functions are difficult, or not present.

So I wrote a small script which let me export my subscriptions as a simple text file, manage them in the usual text editor and save edits back to Google Reader.

What it does

greader2org exports current subscriptions from Google Reader as a simple (and - contrary to OPML - human readable) text file. It is also able to save the changes made in this file back.

Here is how this file looks like:

* Apps I use

** Linode Blog

   feed: http://blog.linode.com/feed/

** Nozbe Blog

   feed: http://feeds.hitrss.com/nozbe

** Redmine: News

   feed: http://www.redmine.org/projects/redmine/news?format=atom

* Web HowTos

** Lifehacker: Top

   feed: http://lifehacker.com/tag/top/index.xml

** MakeUseOf.com

   feed: http://feeds.feedburner.com/Makeuseof

** Mashable!

   feed: http://feeds.feedburner.com/Mashable

More about this format below. As you can guess, main chapters respond to folders, sections to feeds.

What one can do with this file? Rename folders, rename feeds, reorganize the hierarchy (move feeds to other folders, create new folders etc), mark selected feeds as disabled (temporarily unsubscribe), also filter selected feeds via PostRank.

All those operations can be done directly inside Google Reader. But editing it as text let you perform search&replace, cut&paste, undo, version control etc.

Why those stars?

While the text as above can be easily edited in any text editor, it is tuned towards editing inside Emacs, using org-mode.

It's a very ergonomic tool for any kind of outline editing. In this case it helps by allowing me to collapse/expand folders and feeds, supporting keyboard shortcuts for structure reorganization, and offering the syntax highlighting.

In case you use Emacs, but never tried org-mode, I strongly recommend watching this video (available in better resolution on YouTube):

Ahd here are the org-mode docs.

Going back to greader2org.

Installation

You need Python together with lxml and simplejson libraries. Debian/Ubuntu way to install them:

$ sudo apt-get install python-lxml python-simplejson

My script should work on Windows too, although I haven't tried it there. Just install Python, SetupTools, lxml and simplejson (there are binary distributions for both those libraries).

Then run:

$ sudo easy_install \
    http://mekk.waw.pl/download/python/mekk.feeds-1.1.0-py2.5.egg

(to avoid polluting system directories you can consider virtualenv).

Configuration

Run

$ greader2org

You should see sth like:

Program not yet configured! Edit the file

   /home/marcink/.feeds/feeds.ini

using some text editor

That means that the program just created draft version of the configuration file (~/.feeds/feeds.ini on Linux, similarly named file on Windows). Spawn some text editor and save your Google username and password there. Also, consider setting permissions on this file so it is readable only for you:

$ chmod go-rwx ~/.feeds/feeds.ini

(and remove this file after using, or move it to encrypted partition and symlink from there, or send me a sensible suggestion which alternative way of saving password should I use)

First run

Call:

$ greader2org init

This command will create the file ~/.feeds/feeds.txt with info about all your current subscriptions.

Note also that greader2org run without parameters prints minimal help.

Regular usage

Usual session consists of downloading updates from Google, editing them in the text editor, then submitting the changes back. I do this from time to time, whenever I feel I need some major reorganization in my RSS subscriptions.

Grabbing updates from Google Reader

The command:

$ greader2org get

grabs all updates from Google and saves them to feeds.txt. In detail, it:

• adds to feeds.txt all feeds which are subscribed in Google Reader but are not present in this file (those are the feeds I subscribed since the previous update)

• detects feeds which are no longer subscribed in Reader (likely manually unsubscribed) and labels them as :disabled:

There is a difference between greader2org get and deleting feeds.txt and creating it from scratch by greader2org init: the former preserves comments, disabled feeds, filtering info etc.

Editing

Edit feeds.txt in your text editor of choice. Emacs with org-mode is strongly recommended, but not necessary.

The following edits are supported:

• changing folder names,
• changing feed titles,
• moving feeds between folders (or to new folders),
• marking selected feeds as :disabled: (to unsubscribe them),
• removing :disabled: mark (to subscribe such feeds back)
• changing level: (see section about PostRank filtering below)
• adding comment: comments where useful.

Saving changes back to Google Reader

Save the file and run:

$ greader2org put

This commands analyses the feeds.txt file, checks it against current Google Reader subscriptions, and performs the following edits:

Subscription changes:

• feeds marked as :disabled: are unsubscribed (if present)

• feeds present in feeds.txt and not marked as :disabled:, but not present in Google Reader are subscribed (any feed for which :disabled: was cleared is subscribed, just like any feed manually added to feeds.txt)

• feeds for which postrank filtering level (level:) has changed are resubsribed using another feed url (this means a loss of read/unread state), see below for more information about PostRank filtering

Name changes:

• feeds for which the title changed are renamed on Reader (without the loss of read/unread state)

Folder changes:

• feeds moved to another folder are re-tagged (old tag is removed, new tag is added, read/unread state is preserved),

• similarly feeds moved to another folder have their tags changed.

feeds.txt file structure

In it's simplest form the feeds.txt file has the following structure:

* Devel: Web

** SitePoint.com

   feed: http://www.sitepoint.com/recent.rdf

** Smashing Magazine

   feed: http://feeds.feedburner.com/SmashingMagazine

* Apps: Emacs

** emacs-fu

   feed: http://emacs-fu.blogspot.com/feeds/posts/default

** M-x all-things-emacs

   feed: http://www.emacsblog.org/feed/

Here chapter titles denote folder names, section titles denote feed names and section bodies contain feed URLs.

A few other marks can be added, here is the more sophisticated example:

* Chess: Forum

** Chess-Server.net Forum - All Forums              :private:

   feed: http://forum.chess-server.net/syndication.php?type=atom1.0

** ICCF Forum                              :private:disabled:

   feed: http://iccf.com/forum/external.php?type=RSS2
   comment: ICCF important notes

* Devel: Web

** SitePoint.com

   feed: http://www.sitepoint.com/recent.rdf
   postrank_feed: http://feeds.postrank.com/d6e48cfa50dc7f1cb13e8df0ad93b4a6
   level: good

Special markers right to the title (org-mode tags) have the following meaning:

• :disabled: disabled feed (I use this mainly to temporarily switch some feed off), such a feed will be unsubscribed, but when I remove this mark, will be subscribed back,

• :private: personal feed (at the moment it just means that the feed should never be passed to PostRank for scoring)

Actual content consists of the following fields:

• feed: - url (address) of the feed
• postrank_feed: - url (address) of the PostRank-filtered feed (generated automatically, see below)
• level: - PostRank filtering level (best, great, good, all), when this field is added, the original feed is replaced with PostRank-filtered version,
• comment: - any comment/remark (not used, but preserved on file rewrites).

Note: feeds.txt is being rewritten during "greader2org get". Everything except folder titles, feed titles, and the fields described above is lost.

PostRank

PostRank service allows one to subscribe to the quality-filtered version of a given feed (so, for example, instead of reading whole feed, you read only best posts from it). See their website for more details.

Personally I use it in two cases: for hyperactive feeds which generate many posts a day (using Great or Good to reduce the pace a bit), and for sites which are close to being unsubscribed (filtering by Best let me spot if some valuable post appears).

greader2org makes it easy to switch between normal subscription and PostRank filtered subscription, or to change the level of filtering. To use it:

• perform normal greader2org get as described above

• review the file and mark any feed you would not like to publish to PostRank as :private: (in particular do it for personal/secret feeds in case you use them)

• run the command

$ greader2org postrank

which will add postrank_feed: parameter to all feeds for which PostRank feed is available

• the command above may report some feeds as unavailable on PostRank, if so, register on PostRank and paste those feeds for analysis (and rerun greader2org postrank on the next day)

• edit the feeds.txt file and set

  level: best

or

 level: great

or

 level: good

for any feed you would like to subscribe filtered (best picks only the best articles, good all articles which caused some reader reaction, great lies in between)

• execute normal greader2org put, which will perform all normal changes but also for any feed for which the level was set (or changed) will unsubscribe the original feed and subscribe the filtered feed instead of it

To stop filtering, just remove level: from given feed and run greader2org put, the filtered feed will be unsubscribed, and the original feed subscribed.

Known problems

Password in open text

Surely I could keep the Google password in a safer way. Suggestions welcome.

feeds.txt is rewritten

Any loose comments/notes saved in feeds.txt are lost during greader2org get and greader2org postrank.

I never really needed them, so I don't feel it makes sense to spend time on patching this.

Empty folders cause parsing error

Empty folder, for example:

* Folder1

* Folder2

** some feed

raises the parsing exception (here because Folder1 has no content).

Illegal characters cause Reader to croak

There are some characters (for example slash / and colon ,), which - if used in a folder name or in a channel name - cause the following error:

HTTP Error 400: Bad Request <class 'urllib2.HTTPError'>
Traceback (most recent call last):
(...)

If you face this, just patch the corresponding name and reissue greader2org put. I could validate them, but I don't know what is and what is not allowed for sure, so it would be incomplete.

Source code

To see the script source code, just download the egg (mekk.feeds-1.1.0-py2.5.egg file) and unpack it (it's usual .zip).

I wrote it using this description of Reader API (and referring to firebug when in doubt). Postrank API is officially documented (surely I could use it better, for example avoiding the need to copy&paste the feeds to subscribe - but I was too lazy to work on OAuth implementation).