Forum OpenACS Development: News aggregator: Purging brainstorming

Currently the news aggregator (NA) package (which I'm working on in HEAD) is a bit of a mess, but I'm working to clean it up. I would like some feedback from the community on one important aspect of current NA design, purging.

Current Design

The current purging mechanism is inspired by Radio UserLand and Python Desktop Server (PyDS). All items are displayed on one page in blog format. (NA expands on this by allowing the user to have multiple aggregators, though the current template doesn't display the additional aggregators. I have an old master template around that does this, but I will integrate it into the package aggregator soon.) When you're done reading all the items on the page, simply click the button Purge this page of news, and the items are no longer visible.

There is a separate table for saved items, na_saved_items. Here is a copy of the central NA query along with its execution plan. When the user has 5 purges for a given aggregator, those 5 purges are deleted and the aggregator_bottom field is updated. No items are displayed (by default) that have an item_id value below the user's aggregator_bottom.

Problems

This design was thought to be scalable, has some problems (including scalability!). Imagine that Alice uses NA a lot and has a lot of purges. The last time Alice used NA and purged her aggregator was 9:00, so all items that were scanned before 9:00 are not visible.

Now Alice creates a new subscription for New York Times. Lots of other users are subscribed to New York Times's RSS feed, so there's no new scan of that feed. All the items in the NYT were scanned before 9:00. As a result, it appears to Alice that nothing has happened — no items appear unless Alice clicks the "Purge Off" link.

Solutions

How should this problem be fixed? One solution would be to give up saving space and simply mapping every single item to each aggregator instance. When Alice has purged a page, a row is inserted into the table for each item that was visible.

Another solution could be to implement have purges at the subscription level (the mapping between aggregators and sources). Would this solve all the problems?

I'd like some ideas!

NA does support folders in the form of "aggregators". Each user can create multiple aggregators. You can visit my old 4.6.3 dev site (screenshot) to see a template that actually displays them 😉

In any case, as long as we retain the view where views from all sources are gathered on one page (unlike Bloglines, which displays each source on separate pages), I'd like to retain explicit purging.

/Guan

In the same vein, you can arrange the feeds into subfolders, and view all of a particular subfolder at ones (i.e. News, blogs, geek).

This is a nice compromise between the two main feedreader styles, methinks, and the purge-after-read behavior in this way is very useful.

Hm, keeping track of your own internal feeds, so to speak ...

We just need to make rss-feeds for all the important packages and perhaps make one or several "unified" feeds. It might also be a good idea to implement a mechanism which lets a forums instance (say) subscribe automatically to its own feed (via news-aggergator) and pop up a link within the forum instance.

Should it also be possible to purge news items based on the time you last visited the "source" of a feed (a forum say)?

Talk about eating your own dog food 😉

When will this end up in the toolkit? 6.0 or even earlier?

I take back my previous question. The answer should be obvious considering what you said in your first response ...

(Feeds for free... Geez, talk about increased incentive to make your (non CR) package searchable.)

It could also allow for the unified access url we always talk about (/o?object_id=123).

You have to do it for performance for things like that. I ended up with subsite/o/OBJID (since for users for example you want to link to display in the context of the subsite).