This just came across my reader. A new version of feedjack has just been released. In particular, I’m interested in the multi-threaded feed fetching update. I’m hoping that this will reduce the length of the runs of feedjack, which in the case of Planet NEO can take up to 30 minutes. That’s rare, but I’ve seen it consistently take 15 minutes to update.
I will be updating the planets to this version, so you may experience some turbulence while traveling to my various planets.
Update: As far as I can tell, the updated feedjack_update.py is substantially faster. Looks good!
Planet NEO is killin’ me. Particularly on this low-memory vps.
Planet NEO has close to 500 feeds now. I have the update running hourly still. As I said in this post I dumped all the archived articles from before April 1, 2008, but I’m still sitting at 16,000 posts. So I have two problems.
Feed Updates
The first is when the feed updates. Due to the number of feeds being pulled, it’s taking a long time. And when the cron job runs, it just eats memory of which there is not much of to begin with. Almost invariably, the site doesn’t respond or you get a server error when the feed update is running.
Number of Posts and Tags
I think the other thing is probably just the sheer number of posts, and probably more importantly, the tags. When I run planet neo (it’s run as a standalone django app, powered by Feedjack), it eats up almost half of the total memory available. I’m not sure there is anything I can do about it other than eliminating the tag portion of the Planet.
GAE to the Rescue?
One of the feeds I am now following is called “GAE Planet“. Not surprisingly, it’s a “planet” running on top of gae, or google app engine. Perhaps, moving Planet NEO to use the GAE planet may be the solution. It’ll get it off of this box which will speed up my other planets. I am a little concerned that it will easily outstrip the “free” usage limitations of gae though. Then again, it may not be a huge deal. May be worth trying on small scale basis to start with…
I’ll need to check whether the gae planet code is available…
Two of my planets appear to be down — Planet NEO and Neohawk IT. They are not actually down, but for some reason not responding. Even the two that are responding take an awfully long time to respond. I haven’t figured out why, nor do I particularly have time right now.
I’ll look at it this weekend. My apologies to to the folks using the Neohawk.Info Planets.
Update The planets are back up. I had to restart services, as it looks like I was running out of memory.
