FeedWordPress 0.97: one small increment for the revision number, one giant leap for the software

Posted on by

Update 2007-11-21: FeedWordPress 0.97 is now out of date. You can download the latest release — 0.991 at the time of this writing — from the project homepage.

Huzzah! 0.97 is now available for download[1] (also in format[2], if you’re in to that kind of thing). Enjoy, and please feel free to report any problems you have with the software or the documentation to me in the comments section below or e-mail(http://radgeek.com/contact).

[3]: http://projects.radgeek.com/download/feedwordpress-0.97.tar.gz
[4]: http://projects.radgeek.com/download/feedwordpress-0.97.zip

Version 0.97 represents *major* changes, I hope for the better, on several fronts in FeedWordPress. (If I weren’t running out of revision numbers between the current version and 1.0, I would have added a lot more than 0.01 to the version. Stupid decimal notation.) This release will address several long-standing bugs, and hopefully also make other bugs easier to diagnose and to fix. First, the minor inconvenience that this update will impose on you if you are upgrading:

* **INSTALLATION PROCEDURE:** Some of the changes between 0.96 and 0.97
require upgrades to the meta-data stored by FeedWordPress to work
properly. Thus, if you are upgrading from 0.96 or earlier to 0.97, most
FeedWordPress operations (including updates and template functions)
WILL BE DISABLED until you run the upgrade procedure. Fortunately,
running the upgrade procedure is easy: just go to either Options –>
Syndication or Links –> Syndicated in the WordPress Dashboard and press
the button.

Now, the fun stuff. The most important are the new, humane interface for setting feed-specific settings (check it out under Links –> Syndicated –> Edit … no more nerding around with key-value pairs!), Atom 1.0 support, and the overhaul of `update-feeds.php`. Details follow:

* **FEED FORMAT SUPPORT:** Support has been added for the Atom 1.0 IETF
standard. Several other elements are also newly supported
(dcterms:created, dcterms:issued, dcterms:modified, dc:identifier,
proper support for the RSS 2.0 guid element, the RSS 2.0 author element,
the use of Atom author or Dublin Core dc:creator constructs at the feed
level to identify the author of individual items, etc.)

N.B.: full support of several Atom 1.0 features, such as categories
and enclosures, requires you to install the optional rss-functions.php
upgrade in your wp-includes directory.

* **BUG FIX:** Running `update-feeds.php` from command line or crontab
returned “I don’t syndicate…” errors. It turns out that WordPress
sometimes tramples on the internal PHP superglobals that I depended on
to determine whether or not the script was being invoked from the
command line. This has been fixed (the variables are now checked
*before* WordPress can trample them). Note that `update-feeds.php` has
been thoroughly overhauled anyway; see below for details.

* **BUG FIX:** Duplicate categories or author names. Fixed two bugs that could
create duplicate author and/or category names when the name contained
either (a) certain international characters (causing a mismatch between
MySQL and PHP’s handling of lowercasing text), or (b) characters that
have a special meaning in regular expressions (causing MySQL errors when
looking for the author or category due to regexp syntax errors). These
should now be fixed thanks to careful escaping of names that go into
regular expressions and careful matching of lowercasing functions
(comparing results from PHP only to other results from PHP, and results
from MySQL only to other results from MySQL).

* **BUG FIX:** Items dated Decembr 31, 1969 should appear less often. The
function for parsing W3C date-time format dates that ships with
MagpieRSS can only correctly parse fully-specified dates with a
fully-specified time, but valid W3C date-time format dates may omit the
time, the day of the month, or even the month. Some feeds in the wild
date their items with coarse-grained dates, so the optional
`rss-functions.php` upgrade now includes a more flexible parse_w3cdtf()
function that will work with both coarse-grained and fully-specified
dates. (If parts of the date or the time are omitted, they are filled in
with values based on the current time, so ‘2005-09-10’ will be dated to
the current time on that day; ‘2004’ will be dated to this day and time
one year ago.

N.B.: This fix is only available in the optional `rss-functions.php`
upgrade.

* **BUG FIX:** Evil use of HTTP GET has been undone. The WordPress interface
is riddled with inappropriate (non-idempotent) uses of HTTP GET queries
(ordinary links that make the server do something with significant
side-effects, such as deleting a post or a link from the database).
FeedWordPress did some of this too, especially in places where it aped
the WordPress interface (e.g. the “Delete” links in Links –>
Syndicated). That’s bad business, though. I’ve changed the interface so
that all the examples of improper side-effects that I can find now
require an HTTP POST to take effect. I think I got pretty much
everything; if there’s anything that I missed, let me know.

Further reading: Ruby 2005-05-06: This Stuff Matters (http://www.intertwingly.net/blog/2005/05/06/This-Stuff-Matters)

* **BUG FIX:** Categories applied by `cats` setting should no longer prevent
category-based filtering from working. In FeedWordPress, you can (1)
apply certain categories to all syndicated posts, or all posts from
a particular feed; and (2) filter out all posts that don’t match one
of the categories that are already in the WordPress database (allowing
for simple category-based filtering; just load up WordPress with the
categories you want to accept, and then tell FeedWordPress not to create
new ones). However, the way that (1) and (2) were implemented meant that
you couldn’t effectively use them together; once you applied a known
category to all syndicated posts from a particular feed, it meant that
they’d have at least one familiar category (the category or categories
you were applying), and that would get all posts past the filter no
matter what categories they were originally from.

Well, no longer. You can still apply categories to all syndicated posts
(using either Syndication –> Options, or the feed-level settings under
Links –> Syndicated). But these categories are not applied to the post
until *after* it has already passed by the “familiar categories” filter.
So now, if you want, you can do category filtering and *then* apply as
many categories as you please to all and only posts that pass the filter.

* **BUG FIX:** Other minor typos and HTML gaffes were fixed along the way.

* **PERFORMANCE:** get_feed_meta() no longer hits the database for information
on every call; it now caches link data in memory, so FeedWordPress only
goes to the database once for each syndicated link. This may
substantially improve performance if your database server resources
are tight and your templates make a lot of use of custom settings from
get_feed_meta().

* **API CHANGE:** Link ID numbers, rather than RSS URIs, are now used to
identify the feed from which a post is syndicated when you use template
functions such as get_feed_meta(). The practical upshot of this is you
can switch feeds, or change the feed address for a particular syndicated
site, without breaking your templates for all the posts that were
syndicated from the earlier URI.

* **API CHANGE:** if you have plugins or templates that make use of the
get_feed_meta() function or the $fwp_feedmeta global, note that the
data formerly located under the `uri` and `name` fields is now located
under the `link/uri` field and the `link/name` field, respectively. Note
also that you can access the link ID number for any given feed under the
global $fwp_feedmeta[5] (in plugins) or
get_feed_meta(‘link/id’) (in a template in post contexts).

* **FEATURE:** the settings for individual feeds can now be edited using a
humane interface (where formerly you had to tweak key-value pairs in the
Link Notes section). To edit settings for a feed, pick the feed that you
want under Links –> Syndicated and click the Edit link.

* **FEATURE:** The “Unsubscribe” button (formerly “Delete”) in Links –>
Syndicated now offers three options for unsubscribing from a feed: (1)
turning off the subscription without deleting the feed data or affecting
posts that were syndicated from the feed (this works by setting the Link
for the feed as “invisible”); (2) deleting the feed data and all of the
posts that were syndicated from the feed; or (3) deleting the feed data
and *keeping* the posts that were syndicated from the feed
setting the Link to “Invisible” (meaning that it will not be displayed
in lists of the site links on the front page, and it won’t be checked
for updates; (2) deleting the Link and all of the posts that were
syndicated from its feed; or (3) deleting the feed data but keeping the
posts that were syndicated (which will henceforward be treated as if
they were local rather than syndicated posts). (Note that (1) is usually
the best option for aggregator sites, unless you want to clean up the
results of an error or a test.)

* **FEATURE / BUG FIX:** If you have been receiving mysterious “I don’t
syndicate…”, or “(local) HTTP status code was not 200”, or “(local)
transport error – could not open socket”, or “parse error – not well
formed” errors, then this update may solve your problems, and if it does
*not* solve them, it will at least make the reasons for the problems
easier to understand. That’s because I’ve overhauled the way that
FeedWordPress goes about updating feeds.

If you use the command-line PHP scripting method to run scheduled
updates, then not much should change for you, except for fewer
mysterious errors. If you have done updates by sending periodic HTTP
requests to ,
then the details have changed somewhat; mostly in such a way as to make
things easier on you. See the README file or online documentation on
Staying Current for the details.

* **FEATURE:** FeedWordPress now features a more sophisticated system for
timed updates. Instead of polling *every* subscribed feed for updates
*each* time `update-feeds.php` is run, FeedWordPress now keeps track of
the last time it polled each feed, and only polls them again after a
certain period of time has passed. The amount of time is normally set
randomly for each feed, in a period between 30 minutes and 2 hours (so
as to stagger updates over time rather than polling all of the feeds at once. However, the length of time between updates can also be set
directly by the feed, which brings us to …

* **FEATURE:** FeedWordPress now respects the settings in the `ttl` and
Syndication Module RSS elements. Feeds with these elements set will not
be polled any more frequently than they indicate with these feeds unless
the user manually forces FeedWordPress to poll the feed (see Links –>
Syndicated –> Edit settings).

76 thoughts on “FeedWordPress 0.97: one small increment for the revision number, one giant leap for the software”

  1. I could access to the shell prompt, and into the editor.

    Now for the next 2 questions, and sorry if I sound so dumb.

    The user and pass are the ones you use to get to the server or to the blog?

    If the user is “hipo” and pass “me”, the line must be:

    */30 * * * * curl –silent –user hipo:me http://xyz.com/wp-content/update-feeds.php –data update=quiet

    if that´s correct?

    Thanks

  2. **hipocratico:**

    Don’t worry about asking — like most Unix things, cron is very useful but also pointlessly arcane.

    The username and password that you should use are a username and password that will log you in to the WordPress Dashboard. *Not* the username and password you use to log in to the shell prompt. The line you have above is basically correct; but you should use *two* dashes in front of the `silent`, `user`, and `data` options. (Sorry about that! I hadn’t noticed that WordPress was converting double dashes into a single em dash until after I saw your comment. So the line should be:

    */30 * * * * curl –silent –user hipo:me http://xyz.com/wp-content/update-feeds.php –data update=quiet

    Hope this helps.

  4. I just can´t get to update. I used “crontab -e”, the text editor appeared, copy-paste the line you wrote, with the correct user-pass. I set it to 10 (10 mins). Nothing happened.

    I save the line after I exit the editor. I think there´s no problem there.

    Do I have to CHMOD the archives?

    Is there hope for me?

  5. Excellent plugin. Had it up and working in a few minutes.

    I’d like to make a suggestion. When syndicating we have the permalink option set to our website as I want to encourage comments about the feed on our site as well as maintain the visitor longer.

    However I would also like to show the link that the RSS article originally linked to. So can this be added as an option to show within the post body the original feed link.

  6. Dear Charles,

    I must be doing something wrong. I’ve been trying to figure this out for a few days but I cannot.

    After the fresh installation of feedwordpress 0.97, I am unable to view the “Syndication” tab under option. Can you please tell me what I should look into?
    (I’m using wpmu.)

    Thank you alot!
    Calnet

  8. Hi Rad,

    Is there a way in which I can limit the amount of post that feedwordpress post at any one time?

    For example, my sports blog pulls in a lot of posts, sometimes 20 odd. I would really like to adjust the amount of posts that are posted or set a maximum limit.

    thanks as always john

  10. Hi Rad,

    I have a problem with the timestamp of the feeds. It seems like it’s odd by 6 hours. But it is actually the local servertime, so I wonder if there’s something wrong with the feeds. I am using both rss2 and blogspots atom-feed.

    Still I guess it should be possible to configure the “fallback-offset” to some other value than the local servertime, but how/where ???

  11. I am having consistent problems with the crontab and getting the job to go thru.

    With the original setting you provide,

    8 * * * * cd $HOME/http://yoursite.com/etc

    the CRON mailer sends me a message stating that the PHP command does not work.

    with the other cron jobs you suggest,

    curl –user login:pass http://mysite.com/blog/wp-content/update-feeds.php -d update=quiet

    and

    /usr/bin/curl http://xyz.com/blog/wp-content/update-feeds.php -d update=quiet -d log=peter -d log=ab (or pwd=ab, either one)

    I am getting the following from the mailer when it attempts:

    curl: option -d: requires parameter
    curl: try ‘curl –help’ or ‘curl –manual’ for more information

    All of the necessary CHMOD stuff is done. I have also tried using the secretword to no avail. The commands work from within the bash shell if I run them from a command line, but the cron job will not complete.

    Any suggestions?

  13. This is a great plugin! I have one problem and one request:

    – I set some feeds to a manual Link Name, so “Joe’s Really Cool Blog” displays only as “Joe’s Blog”, using this markup:
    ” rel=”bookmark” title=”Link to “>

    However, when I run cron and it updates, it ignores my manual setting. Am I doing something wrong?

    – Speaking of cron, is there a way to to display, outside the loop, the timestamp of the cron? I would like to show at the top of my site “Updated 1/2/2005 at 9:00 EST” or words to that effect.

  15. One other thing (sorry!).
    I am also having the time offset problem. The timestamp on my feed posts are +6hrs, which is my server’s time (UTC). Is there a way to offset this so it matches my timezone?

  17. OK, the trick is done in feedwordpress.php , approx. line 2070 and 2071, where you add an offset. (I and the majority of the users live in Denmark, which is GMT+1 and the server is somewhere in USA)

    I replaced

    $post[‘post_date’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘issued’]);
    $post[‘post_modified’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘modified’]);

    with

    $post[‘post_date’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘issued’]+6*60*60);
    $post[‘post_modified’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘modified’]+6*60*60);

    Not being a php-programmer, it took me a while..

    Next version could have define-statement with SERVER_OFFSET set to 0 , and the lines:

    $post[‘post_date’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘issued’]+SERVER_OFFSET);
    $post[‘post_modified’] = date(‘Y-m-d H:i:s’, $post[‘epoch’][‘modified’]+SERVER_OFFSET);

    Then it should be easy to make the adjustments.

    Hope it works..

  19. Well, it is hard to write messages in this place, I mean: 6 times 60 times 60 aka 6x60x60 with the x’s replaced with *’s

  21. Absolutely great plugin!

    I’ve been doing some hacking around and finnaly got it working with WordPress 2.0 RC1.
    I would like to suggest a change to the code though: Users are currently identified by their name which means if two users, from two different feeds, happen to have the same username they will be taken for the same user when aggregated into wordpress. My suggestion would be to compose the username created by FWP for each user with the author’s name and author’s url, so that they would look something like this:

    jhondoe.somedomain.com/weblog

    For a user who’s original username is “jhondoe” and who’s weblog URL is actually htp://somedomain.com/weblog

    I’m not sure if I’m missing something though, it’s quite possible this is just silly 🙂

    any toughts?

    Cheers

  22. I’m having a new problem with the plugin..it seems to be not updating all the feeds it used to..it seems to randomly not update them, even when I manually hit update-feeds.php

  23. Steve, what happens if you schedule the “Next update” for “ASAP” before hitting update-feeds.php?

  24. There’s definitely a problem..when I do the manual update, it lists three or four sites only, and I know that lots more sites in my list have udpated feeds. It’s as if update-feeds.php times out or something. If update each one manually, it works..update All is where it hangs up.

Comments are closed.