FeedWordPress 2011.0512: better performance, better troubleshooting, and no more “There may be a bug” white-screens-of-death

FeedWordPress 2011.0512 is now available for download. Hooray!

This announcement will actually cover details for a couple of maintenance releases that I’ve pushed out since 2010.0905. This is mainly a maintenance release, with a number of small tweaks around the edges, most of them for better internal performance or organization, but — if you’re upgrading from 2010.0905 — it includes one important, major performance improvement. And it includes a number of changes which should make it much quicker and much less painful to troubleshoot if you encounter any problems with your feeds. Besides having brought the ChangeLog up to date with the most recent version of the code, here are some of the main things that have changed:

Changes since Version 2010.0905

  • BUGFIX: AVOID DUPLICATE POSTS WHEN GUIDS ARE TOO LONG: When feeds included
    exceptionally long GUIDs, FeedWordPress could occasionally get into
    a situation where posts with the long GUIDs would be duplicated over
    and over again with each update (because FWP failed to store the full
    GUID, due to length constraints in the relevant database tables).
    Without the full GUID, FWP would not know that the post had already
    been syndicated once. This bug has been fixed, and should no longer
    produce duplicate posts.

  • HTTP TIMEOUT SETTING: If you are frequently running into timeout
    problems with one or more of the feeds you syndicate, FWP now allows you
    to adjust the timeout for HTTP requests using a global or feed-by-feed
    setting.

  • HTTP GET PARAMETERS: You can now temporarily or permanently add HTTP
    GET parameters to a subscription using an interface in Syndication –>
    Feeds & Updates. This is especially helpful for making quick, short-term
    changes to a subscription (for example, to pull in all the previous
    items from a web service, before settling down to pulling in only newly
    updated items). You can also use it to include private parameters with values that you do not wish to make publicly visible when a syndicated post displays the URL of its source feed — such as API keys or passwords.

  • DIAGNOSTICS SYSTEM: Added several new diagnostics which are useful in
    troubleshooting, and established a framework for add-on modules to hook
    in with their own diagnostic messages.

  • UI: Adjusted some internal coding, which should allow for settings
    pages and add-ons to properly display multiple category pickers on a
    single settings page.

  • PHP4 COMPATIBILITY TWEAKS: This version makes some tweaks to the handling
    of object references which should improve compatibility with older
    versions of PHP. (Although, I should note, web hosts that still force
    you to run under PHP 4 — in 2011! — are bad web hosts.)

  • IMPROVED PERFORMANCE: This version eliminates a major performance drag
    that shows up on sites with large numbers of users (due to some poor
    decisions about where to place a user query, which caused the user table
    to be scanned frequently when it did not need to be). If you experienced
    serious problems with CPU load or slow database performance under
    2010.0905, which kicked in immediately when FWP was loaded and tended
    to disappear immediately if FWP was de-activated, it is likely that
    upgrading away from 2010.0905 to the most recent version will resolve
    your problem.

Changes since Version 2011.0211.2

  • DIAGNOSTICS IMPROVEMENTS; “THERE MAY BE A BUG IN FEEDWORDPRESS” CRITICAL ERROR NOTICES ELIMINATED: This version includes some major improvements
    to the Syndication –> Diagnostics section, which should aid in
    troubleshooting difficulties with items failing to be imported, posts
    failing to be properly inserted into the database, or updates failing to
    be recorded. If you have been encountering critical error / bug notices
    with a white screen and the message “THERE MAY BE A BUG IN
    FEEDWORDPRESS,” followed by an extraordinarily long dump of mostly
    incomprehensible diagnostic information, you’ll be happy to know that
    the condition causing these notices has been eliminated. In the few
    cases where errors may still crop up with database insertions,
    FeedWordPress will now produce a significantly more manageable and more
    useful diagnostic message.

  • BUGFIX: NEW POSTS FAILING TO APPEAR IN A CLEANLY-INSTALLED FEEDWORDPRESS SYSTEM. If you encountered a recurring problem with FeedWordPress
    failing to import new posts, after a clean install of FeedWordPress
    (i.e., not an upgrade from a previous version), this problem may have
    been the result of a bug with author-handling which has now been fixed
    in the 2011.0512 release. (If the problem does not go away with the
    upgrade, this version also includes significant improvements to the
    Diagnostics system, which will help track down what is causing it
    in your particular case.)

  • PERFORMANCE: New handling of update hashes allows FeedWordPress to avoid
    a certain kind of infinite loop, caused when two more more different
    syndicated feeds each carried a version of the same item (for example,
    because it appeared on two different aggregator feeds that you’re
    syndicating). In previous versions, when this kind of loop cropped up,
    syndicated posts could pile up an indefinitely large number of revisions
    — each revision alternating between the version from each of the two
    feeds where it appeared — which would, over time, dramatically inflate
    the size of the database, and kill the performance of queries on the
    post table. This issue has been resolved: revisions of the post that
    have been syndicated once will not be re-syndicated over and over again.

  • AUTHOR LISTS: Lists of authors presented on the Author settings pages
    should now be easier to scan through, with author names arranged in
    alphabetical order.

  • FEED ITEM DATE PARSING: More tweaks to make date-time handling more
    resilient when feeds provide broken or weird values for the timestamps
    on syndicated items. FWP will now attempt to work around unparseable
    timezone values.

  • AUTHOR MATCHING: Now attempts to match author names against the WP login
    name in addition to display_name; when creating user record, also fills
    in some best-guess values for nickname, firstname and lastname. Also
    properly picks up Atom 1.0 author/uri data from feed.

  • COMPATIBILITY: FeedWordPress has been successfully tested for
    compatibility with recent releases of WordPress, up to version 3.1.2.

As always…

As always, if you have any issues with the release, or any questions I can help answer, or if there is anything that you would like to see included in a future release, please use the comments form or drop me a line to let me know about it. If you have an issue to report, please be sure to tell me what version of FeedWordPress you’re using, what version of WordPress you’re using it with, which web browser you are using to view the FeedWordPress user interface, and try to tell me, as clearly as possible, (1) what you were trying to do, (2) what the circumstances were, (3) what you expected to see, and (4) what you ended up seeing instead.

Please remember that your generous gifts to the project tip jar make ongoing development and support for FeedWordPress possible.

Now get on out there, download and enjoy!

FeedWordPress 2010.0905: bug fixed; Categories and Tags now correctly assigned

Note. As with the previous release, before you download or install, please note that this version of FeedWordPress requires WordPress 3 or higher to operate correctly. Do not install the upgrade unless or until you have made the upgrade to WP3.

FeedWordPress 2010.0905 is now available for download.

I’ve pushed out this release quickly because it contains fixes to an important issue that y’all helped me discover in early user reports back about FeedWordPress 2010.0903. As I mentioned in the release notes, the new release of FeedWordPress has some extensive internal changes to allow for a great deal more flexibility in how it handles Categories and Tags (up to and including support for custom taxonomies provided by plugins). Unfortunately, the changes introduced a bug, due to the fact that the new method used to insert Categories and Tags in WordPress 3 screens the terms to be inserted by the current logged-in user’s capabilities. Which is fine for normal operations within the WordPress administrative interface; but since automatic updates and updates from FeedWordPress cron jobs are typically carried out without any user logged in whose credentials could be checked, this meant that several users noticed posts being inserted with only the category Uncategorized, and with no Tags. Which presents a problem.

Fortunately, the problem has been fixed in today’s quick upgrade:

  • BUGFIX: CATEGORIES AND TAGS CORRECTLY ASSIGNED IN AUTOMATIC UPDATES.
    Version 2010.0903 switched over to a new way of assigning categories and
    tags as part of its support for handling custom taxonomies.
    Unfortunately, the method that it uses is subjected to some checks of
    the current user’s capabilities, which creates problems for posts that
    are being inserted into the WordPress database when there is no
    current user logged in (as, for example, when an update is being carried
    out from a cron job or automatic update). The result was that posts
    from cron jobs and automatic updates ended up with no Categories and no
    Tags being assigned. This bug has now been fixed: in 2010.0905, Tags and
    Categories should be correctly assigned to all posts, regardless of
    whether they were added from manual updates, cron jobs, or automatic
    updates.

All you need to do to apply the fix is download the newest release and drop it into place in your WordPress plugins directory. As always, if you have any issues with the release, or any questions I can help answer, or if there is anything that you would like to see included in a future release, please use the comments form or drop me a line to let me know about it. If you have an issue to report, please be sure to tell me what version of FeedWordPress you’re using, what version of WordPress you’re using it with, which web browser you are using to view the FeedWordPress user interface, and try to tell me, as clearly as possible, (1) what you were trying to do, (2) what the circumstances were, (3) what you expected to see, and (4) what you ended up seeing instead.

Download and enjoy!.

FeedWordPress 2010.0903: bug fixes, interface improvements, lower memory load & big new features for convenience & uber-geekery

I am happy to announce that FeedWordPress 2010.0903 is now available for download..

Now stop just a minute. Before you proceed, be advised: FeedWordPress 2010.0903 has been thoroughly tested for compatibility with WordPress 3.0. It also requires WordPress 3.0. If you do not have WordPress 3.0 installed, and aren’t yet ready to make the upgrade, you should not attempt to upgrade to this version of FeedWordPress, until you are ready to make the upgrade to WP3.

This release includes a big passle of improvements, fixes, and new features. Some of them have been a long time coming; others are new and fancy. For the fixes, you may be most interested in a fix which apparently solves a really vexing vanishing act that Syndicated Sources performed for some users on various versions of Internet Explorer — where a big chunk of the user interface would suddenly disappear as soon as you added your first feed. You may also notice some of the new features, like subscribing to multiple feeds or the reductions in memory consumption. Many of the best changes, though you may not even notice — a great deal of the improvements are in low-level interface work, and if I’ve done my job right it will make your life a few minutes more pleasant every day, without you hardly even noticing it. Many of them are improvements under the hood, which are mainly intended for uber-geeks, hotrodders, or — and here is where it matters for the rest of you — Add-On and Filter Developers, who will have that much more of a chance to enrich the FeedWordPress ecosystem. But there’s also plenty that should be right out on the surface, for anyone to see, and I hope you enjoy it.

Here’s a breakdown of some of the major changes since the previous release:

  • WORDPRESS 3 REQUIRED: Please note that this release of FeedWordPress
    requires WordPress 3.0 or later. If you are currently using a 2.x
    branch of WordPress, you will need to upgrade to WordPress 3 before you
    can successfully upgrade FeedWordPress.

  • BUGFIX: NO MORE DISAPPEARING “SYNDICATED SOURCES” PANEL; INTERNET
    EXPLORER UI GLITCH APPARENTLY FIXED:
    Several users independently
    reported a problem with FWP 2010.0623 and various versions of IE. A
    problem with the HTML markup caused IE (but not Firefox or
    Chrome) to completely hide the Syndicated Sources administration panel
    (the main list of currently-syndicated sources, and the main location
    for adding new sources, under the Syndication menu item) when a user
    added their first syndicated feed. Maddeningly, the glitch seemed to
    affect some IE users and not others: I was never able to reproduce the
    problem for myself on my own machines. However, the markup of Syndicated
    Sources has undergone significant changes and corrections since
    2010.0623, and two independent sources who had been having this problem
    confirm that they no longer encounter it with the updated version. For
    the time being, I am going to declare this bug squashed.

  • BUGFIX: MORE PROTECTION AGAINST FATAL ERRORS FROM PLUGGABLE VERSIONS OF
    SimplePie:
    FeedWordPress now takes some precautions that should help to
    better avoid conflicts for users who have installed pluggable versions
    of SimplePie for another plugin or theme. (You may not know that you have
    done this; but if you’ve been encountering fatal errors indicating that
    you cannot redeclare class SimplePie, or something along those lines,
    there is now a better chance that those fatal errors will be eliminated.

  • PERFORMANCE: SIGNIFICANTLY REDUCED MEMORY CONSUMPTION FOR LARGE UPDATES:
    FeedWordPress is still a memory-hungry little module, especially when
    you are dealing with very large feeds. However, users should notice a
    significant reduction in memory overloads, especially if they update a
    large number of feeds at once.

  • USER INTERFACE IMPROVEMENTS: Nothing is radically different, but there’s
    been a fair amount of extra spit and polish added, including a convenient
    new Dashboard widget that may save you a trip to the Syndication menu,
    a lot of effort to make the relationship between global and feed-by-feed
    settings more obvious to the user and more easily controllable, to make
    navigation between settings pages easier, to sand off a few rough edges,
    and to make other improvements on the margins. I hope you’ll like how
    it looks.

  • ADDING MULTIPLE FEEDS: FeedWordPress now provides a convenient mode for
    adding multiple feeds at once, using either a copy-and-pasted list, or
    else an OPML file. Go to Syndication –> Syndicated Sources and check
    out the two new buttons underneath the New Source input box. When you
    have to add a number of feeds at once, this can save you considerable
    time and trouble.

  • IMPROVED HANDLING OF AUTHORS WITH DUPLICATE E-MAIL ADDRESS AND AUTHORS
    WITH NAMES WRITTEN IN FOREIGN SCRIPTS:
    WordPress 3 is increasingly picky
    about what it will accept for new author accounts, and some of the
    conditions it imposes can cause error conditions that prevent posts from
    being properly syndicated, or properly attributed, if authors happen to
    have identical e-mail addresses, or if users are given usernames that are
    written in non-Western scripts. FeedWordPress now handles these much
    better, and systematically works to avoid clashes between syndicated
    authors’ account names or in their e-mail addresses, which should result
    in significantly better results in mapping author names to WordPress
    user accounts.

  • MAPPING CATEGORIES ON SYNDICATED POSTS TO TAGS NOW BETTER SUPPORTED:
    In previous versions, the only way for the Categories provided by a
    syndicated feed to be mapped into Post Tags was to instruct FWP to
    create new tags, rather than new categories, for unfamiliar categories
    from the feed. This works fine if you want tags to be the default; but
    if you want only a specific set of tags, there was no way to get them
    without getting most or all other categories imported as tags. You can
    now do this by creating a tag (under Posts ==> Post Tags) before
    importing the post; when the syndicated category matches a pre-existing
    tag, the incoming post will be tagged with that tag, without creating
    a local Post Category.

  • REL-TAG MICROFORMAT SUPPORT FOR INLINE TAGS: Syndicated posts that
    contain inline tags, marked up using the Rel-Tag microformat
    http://microformats.org/wiki/rel-tag, are now tagged with the tags
    provided by Rel-Tag format links.

  • MUCH GREATER CONTROL OVER CATEGORY AND TAG MAPPING: This is partly the
    result of building in support for a potentially endless set of custom
    taxonomies (see below), but in general there has been a great deal of
    effort towards giving you more control over how categories and tags
    provided by the feed are mapped into terms on the local blog. In
    particular, you can now force FeedWordPress to create only categories
    from categories and tags provided by the feed; or to create only tags;
    or to search both categories and tags for a match; or you can simply
    force it to drop all of the categories provided by the feed and use
    only categories or tags that you explicitly provide. In addition, you
    can now also choose whether to override global categories settings with
    a local, feed-specific setting; or whether to add together both the
    global categories and the local feed-specific categories — depending
    on whatever your use-case may demand.

  • CUSTOM POST TYPES AND TAXONOMY SUPPORTS: This is mainly for the
    super-geeky, but if you use other plugins or themes that make
    significant use of WordPress’s support for custom post types and custom
    taxonomies, you may be pleased to find that FeedWordPress now allows you
    to feed incoming posts into any custom feed type that you wish, and to
    map categories and tags from the feed to custom taxonomies as well as
    to the standard Category and Tag taxonomies.

  • STORING NAMESPACED CUSTOM FEED ELEMENTS IN POST CUSTOM FIELDS: If you
    would like to use FeedWordPress’s support for storing custom meta-data
    from feed elements in the custom fields for a post (for example, to
    store geolocation data or iTunes media meta-data), you’ll find that it’s
    now much easier for you to access these namespaced elements. You always
    could access them, but in previous versions you might have to write
    something ugly like $(/{http://www.w3.org/2003/01/geo/wgs84_pos#}lat)
    just to get at the value of a <geo:lat> tag. Now, as long as you use
    the same mnemonic codes that the feed producer used, you should always
    be able to write a nice, simple expression like $(/geo:lat) to get the
    value of a <geo:lat> tag. Huzzah!

  • CUSTOM DIRECTORY STRUCTURE SUPPORT: if you poke at it enough, WordPress
    is relatively flexible about where it should store admin interface code,
    uploaded content, plugins, and a number of other things that occupy an
    important place in the WordPress directory structure. Previous versions
    of FeedWordPress encountered serious errors or broke entirely when used
    with directory structures other than the default. This should now be
    fixed: FWP now supports custom directory structures wherever WordPress
    allows them to be customized, rather than depending on the default
    locations. Enjoy your freedom!

  • MANY NEW FILTERS AND API UTILITY FUNCTIONS FOR ADD-ON PROGRAMMERS: There
    have been too many improvements to list them all in this ChangeLog, but
    it means that much more power and ease for folks who are customizing
    FeedWordPress through PHP filters or add-on modules. Fuller
    documentation will be put up at the Wiki at feedwordpress.radgeek.org
    soon.

Download and enjoy! If you have any issues with the release, or any questions I can help answer, or if there is anything that you would like to see included in a future release, please use the comments form or drop me a line to let me know about it. If you have an issue to report, please be sure to tell me what version of FeedWordPress you’re using, what version of WordPress you’re using it with, which web browser you are using to view the FeedWordPress user interface, and try to tell me, as clearly as possible,

One of the remarkable fact about this release is the number of the new features in this release — including improved namespace support, subscription to multiple feeds, and custom post type and taxonomy support — that I was able to make due to the generous support of FeedWordPress users, whose gifts made it possible for me to devote a several solid days to some major new conveniences and some significant internal overhauls that have dramatically improved FeedWordPres’s flexibility and power. I’d like to thank all the FWP users who have helped make ongoing development possible. It’s really flattering, y’all, and I’m incredibly grateful that you’ve made it possible for me to devote the needed time to this project.

As always, please remember that your generous gifts to the project tip jar make ongoing development, quick fixes and timely support for FeedWordPress possible.

Now get out there and enjoy FeedWordPress 2010.0903!

suggestion for the next update.

Hi!
I always wondered if there existed any easier method to display images with except/thumbnail with the aggregated content using feedwordpress. Unfortunately it has been talk of the web and still no perfect solution. The use of string “snap” is compromising solution.

I would like to suggest and this idea is withdrawn from the pligg module. they had a code that automatically fetches the website thumbnail in rss import module. If a add on as such could be built or could be integrated in core , to automatically add the website snap from snapping services and add that with the excerpt, it would be rocking.
Hope there would be plausible discussion in this regard.

Thanks and Regards.

Custom Post Settings Not Pulling GEORSS Elements from Atom Feed

I can not get FWP to pull a GEORSS element from a atom feed and set a Custom Post Field…

My incoming atom feed looks like this:






33.831032 -117.893439


FWP should allow me to set “Custom Post Settings” which sets “Custom fields can be used to add extra metadata to a post that you can use in themes.”
Syndication -> Post & Links -> Custom Post Settings

My key/value looks like this:
Key: wpgeorsspt
Value: $(georss:point)

But the field never gets created or filled in my posts.

Is this because FWP does not support GEORSS elements?

FeedWordPress 0.993: WordPress 2.5.1 compatibility and a couple new features

Update 2008-11-06: FeedWordPress 0.993 is now out of date. You can download the latest release — 2008.1105 at the time of this writing — from the project homepage.

FeedWordPress version 0.993 is now available for download.

There are a few new features that I am in the midst of working on for an upcoming release of FeedWordPress, but I have released version 0.993 now in order to resolve the critical compatability issue with WordPress 2.5.1. I am still doing compatibility testing to see whether there are any kinks in compatibility with WordPress 2.5.x, but upgrading to this release should eliminate the fatal error that prevented 2.5.1 users from accessing the Syndication Options and the feed settings pages from within the WordPress pages. There are some small bug fixes and the beginning groundwork for some features that will become more fleshed out in the upcoming, more feature-rich release, which aren’t worth going into in detail; besides those, here is what’s new since FeedWordPress 0.992:

  • WORDPRESS 2.5.1 COMPATIBILITY: FeedWordPress should now be compatible
    with WordPress 2.5.1.

  • WORDPRESS 2.5 INTERFACE IMPROVEMENTS: FeedWordPress’s Dashboard
    interface has undergone several cosmetic changes that should help it
    integrate better with the WordPress Dashboard interface in WordPress
    version 2.5.x.

  • SYNDICATED POSTS CAN BE MARKED AS “PENDING REVIEW”: WordPress 2.3 users
    can now take advantage of WordPress’s new “Pending Review” features for
    incoming syndicated posts. Posts marked as “Pending Review” are not
    published immediately, but are marked as ready to be reviewed by an
    Administrator or Editor, who can then choose to publish the post or
    hold it back. If you want to review syndicated posts from a particular
    feed, or from all feeds, before they are posted, then use
    Syndication –> Syndicated Sites –> Edit or Syndication –> Options to
    change the settings for handling new posts.

  • AWARE OF NEW URI FOR del.icio.us FEEDS: Previous releases of
    FeedWordPress already automatically split del.icio.us tags up
    appropriately appropriately when generating categories. (del.icio.us
    feeds smoosh all the tags into a single <dc:subject> element,
    separated by spaces; FeedWordPress un-smooshes them into multiple
    categories by separating them at whitespace.) Unfortunately, del.icio.us
    recently broke the existing behavior by changing host names for their
    feeds from del.icio.us to feeds.delicious.com. Version 0.993 accounts
    for the new host name and un-breaks the tag splitting.

If you have put off upgrading to WordPress 2.5.1 due to this bug, and plan to upgrade after installing FeedWordPress 0.993, please remember that after you upgrade WordPress, you will need to reinstall the FeedWordPress MagpieRSS upgrades in order to keep your feed parsing from getting broken.

Enjoy! As I mentioned, I’m actively working on a release, probably due sometime before the end of the month, including bug fixes and a few significant new features, so let me know about any ongoing issues that you may still have.

FeedWordPress and WordPress 2.5.1 compatibility issue

I wanted to post a quick note because I have received several reports of a serious compatability issue between FeedWordPress and the most recent public release of WordPress, version 2.5.1. Although FWP is broadly compatible with the initial release of WordPress 2.5, the recent update apparently eliminates an interface function that FeedWordPress used to display a box for selecting categories in the Syndication options and the feed settings pages. Since the function no longer exists in WordPress 2.5.1, it means that a fatal PHP error (shown either by a prematurely cut-off display, or in the form of a printed PHP error message) will be triggered if you attempt to use either of these pages in the WordPress Dashboard. (As far as I know, syndication will continue to work fine with WordPress 2.5.1; the error will prevent you from changing configuration settings from within the WordPress Dashboard.)

This problem should, hopefully, be something that it won’t be too hard to fix once I am able to sit down and work on it. Unfortunately, the release came after I had left to visit family in Alabama, so I can’t get to making the fix until I return home early in the upcoming week. Once I do, I’ll release an immediate compatability fix, and then return to work on a more thorough update, which I hope to release sometime during the month of May.

I’ll check back in with y’all once I’ve gotten home.

Upgrade Downgrade: Compatibility bugs and possible quick fixes for issues with FeedWordPress after upgrading to WordPress 2.5

WordPress 2.5 was recently released, and as a result many FeedWordPress users have upgraded their blogs to the latest version of WordPress. I am currently in the process of testing for any compatability issues between WordPress 2.5 and the development version of FeedWordPress (0.993a); if I notice any definite problems, then I will make them high-priority bug fixes and try to push out the release of 0.993 as quickly as possible. (That probably means either tonight, or some time around the end of the month, depending on when I find any problems that I may find.) If you have tried using FeedWordPress with WordPress 2.5, either in version 0.992 or in the current trunk development version, and have noticed any problems since the upgrade that aren’t fixed by what I’m about to suggest, then please feel free to report them in the comments here or to me by e-mail, as you prefer. The most helpful bug reports are those that state, in as much detail as possible, (1) what precisely is going wrong, (2) under what conditions, (3) with what version of FeedWordPress, (4) under what version of PHP, and, if the problem is with syndicating posts, then (5) with which feeds at which specific URIs. If you are getting symptoms of a fatal error (either a printed error message or a blank screen where a page should be), then you can also help me out a lot by copying and pasting the contents of the error message into your report, or, if you have a blank screen, checking the bottom of your web server’s error logs to see if there is a PHP error report down there, and, if so, copying and pasting that.

That said, one of the most common sources of error reports when new versions of WordPress are released come not from a real compatability issue, but rather from the fact that, if you’re not careful, upgrading your copy of WordPress will downgrade your copy of MagpieRSS from the newer version shipped with FWP to the very old and busted version that WordPress continues, for whatever reason, to ship with new releases of WordPress.

Diagnosis

Here are the most common symptoms of this problem:

  • Some feeds (notably, feeds produced by Blogger and other Atom 1.0 feeds) stop syndicating post contents. You get the headline of the post and nothing else.
  • Some feeds (notably, those produced by blogs hosted at WordPress.com (!)) start appearing with just the capital letter A as the content of the post.
  • Categories stop being properly syndicated. Everything is placed in Uncategorized or in bizarre, mashed-up categories (only one per post) that seem to contain several category names.
  • Podcast attachments are no longer syndicated.

And so on, and so forth. If you notice these problems with your feeds just after you’ve upgraded your copy of WordPress, it’s probably because you need to re-install the MagpieRSS upgrade.

Cure

Here’s how you do that. In the FeedWordPress plugin directory (wp-content/plugins/feedwordpress/, relative to your WordPress installation), there is a directory called MagpieRSS-upgrade, which contains, or at least at one point contained, two files, rss.php and rss-functions.php. If you still have these files, you need to copy them to your WordPress wp-includes directory, where they will overwrite the older version of MagpieRSS that ships with WordPress. If you do not (because, for example, you moved them rather than copying them when you first installed FeedWordPress), then you can get new copies of these files by downloading the latest version of FeedWordPress, and extracting these two files from the archive.

Etiology and pognosis for the patient

The reason that this happens is that every installation of WordPress includes a very old version of MagpieRSS, the library that FeedWordPress uses to parse the feeds that it syndicates. As of the 2.5 release, WordPress still ships with a package derived from MagpieRSS 0.51, which is the same version it shipped with when I started work on FeedWordPress three years ago. This version of MagpieRSS is adequate for what WordPress needs it to do (basically, fetch headlines for the Dashboard from a select few feeds), but it was already outdated three years ago, and it is especially outdated now–it could not handle multiple categories; it could not handle enclosures, it could not translate feeds in alternate encodings; and, importantly, it cannot correctly handle Atom 1.0 feeds (now the default for Blogger feeds) or feeds with MediaRSS extensions (now the default for WordPress.com feeds). Unfortunately, since there is a version of MagpieRSS, which is loaded every time you load WordPress, it is hard to drop in a newer version which can do these things without causing errors from the collision in function and class names.

The solution I settled on was the bundled MagpieRSS upgrades, which in the past I sometimes described as optional, but which now really are mandatory if you hope to do any serious syndication in the modern environment. Users can avoid collisions by copying the upgrade so that it just overwrites the older version in wp-includes. Problem solved for the time being.

But the downside of this solution is that every time an upgrade of WordPress comes out, it comes out with older versions of the MagpieRSS package included, and when you overwrite all the files in wp-includes with the newer files from the WordPress release, one of the things you overwrite is your upgraded copy of rss.php. Meaning that, unless you remember to re-upgrade MagpieRSS every time you upgrade WordPress (something which is easy enough to forget), it breaks your syndication until you remember, or I remind you, to re-do the upgrade.

I frankly consider this a design flaw in FeedWordPress, but it’s not a flaw that is easy for me to fix. I am considering different ways of getting around it, and honestly the most likely solution at this point is probably simply to abandon MagpieRSS and package another feed parsing package (such as SimplePie) in the feedwordpress plugin directory, where upgrades to the WordPress core code cannot interfere with it. But doing that will involve pretty dramatically refactoring some of FeedWordPress’s internal workings, and that may take a while. In the meantime, if you have a working aggregator, you should probably apply this quick fix and see how many of your problems it solves.

FeedWordPress 0.992: author remapping and URI bug fix

Update 2008-11-06: FeedWordPress 0.992 is now out of date. You can download the latest release — 2008.1105 at the time of this writing — from the project homepage.

FeedWordPress 0.992 is now available for download.

Since I’ve had to spend time either traveling or working on other projects, it’s been longer than I would have liked since the last update of FeedWordPress. This release is a rather limited one: it fixes one outstanding bug and adds one major new feature:

  1. BUG RELATED TO URIS CONTAINING AMPERSAND CHARACTERS FIXED: A bug in
    WordPress 2.x’s handling of URIs in Blogroll links created problems for
    updating any feeds whose URIs included an ampersand character, such as
    Google News RSS feeds and other feeds that have multiple parameters
    passed through HTTP GET. If you experienced this bug, the most likely
    effect was that FeedWordPress simply would not import new posts from a
    feed when instructred to do so, returning a “0 new posts” response. In
    other cases, it might lead to unpredictable results from feed updates,
    such as importing posts which were not contained in the feed being
    syndicated, but which did appear elsewhere on the same website. This bug
    has, hopefully, been resolved, by correcting for the bug in WordPress.

  2. NEW FEATURE – AUTHOR RE-MAPPING: I’ve been promising this one to my e-mail
    correspondents for a couple of months now; and I’m happy to announce that I’ve been able to polish
    up the preliminary implementation that I was working on late last year and make it ready for general
    consumption. FeedWordPress now offers a new feature in the site-wide Syndication Options
    (Syndication –> Options), and in the settings for each syndicated feed (Syndication –> Syndicated
    Sites –> Edit). You can now create re-mapping rules to determine how author names in a feed are
    translated into usernames within the WordPress database.

    Traditionally, what FeedWordPress has done, when adding a post, is to take the author information
    from the feed and search the WordPress database to determine whether there is a username with
    the same name or e-mail address. If it found a match, then the new post would be assigned to that
    user. If there was no match, then FeedWordPress would consult the settings for the feed, or the
    global default settings, for what to do with an unfamiliar author. These could be set by the
    administrator from within the WordPress Dashbard, with three options: (1) create a new user
    account with the same name as the author, and assign the new post to that new user; (2) filter out
    the post rather than adding it to the database; or (3) assign the new post to a default user (user #1
    in the WordPress database, i.e. the site administrator).

    With the new re-mapping feature, you have considerably more control over what FeedWordPress
    does with post authorship data from the feed. In the settings for each syndicated feed (Syndication
    –> Syndicated Sites –> Edit), you can now define rules that tell FeedWordPress what to do with
    each particular author name on that feed — to filter out all posts by that author, or to assign posts
    by that author to any username that you like. You can also tell FeedWordPress what to do with
    usernames that aren’t listed in your rules, and which don’t match any of the users already in the
    WordPress database — to filter out the posts, or to assign them to any username you like, or to
    create a new username to match the new author, or to follow the default setting for new author
    names on all feeds. The site-wide default setting can be changed using Options –> Syndication.

    So, for example, one of my friends runs a FeedWordPress aggregator site that syndicates
    http://praxeology.net/blog/feed/. The problem is that, like an increasing number of WordPress
    blogs out there, all the posts on this blog are attributed to Administrator. For those visiting
    the blog directly, it’s clear that Administrator is the blogger; but when someone aggregates
    the blog on another website, the naming now mistakenly implies that the Administrator of the
    aggregator website is the author of the post. (It also means that if two or more blogs both
    attribute their posts to Administrator, the aggregator site will mistakenly treat all of the
    posts from all of those blogs as being by the same author.) With the new re-mapping feature, you
    could now define a rule that would attribute posts by Administrator from the praxeology.net
    feed to a different username — in this case, Roderick Long. Problem solved. Huzzah!

One more note before I go. I regret that I haven’t been able to develop FeedWordPress more actively than I currently am developing it, or to spend as much time handling and responding to bug reports as I would like. I originally created FeedWordPress for my own use, and made it available to others in the hope that it would be useful, so the best guarantee of getting a feature added or a bug identified and quickly fixed has always been whether it’s one that I personally encountered, or one of my friends encountered, in the course of using it. But I’m really very pleased with how much uptake and interest there has been for FeedWordPress. Ideally I would like to devote much more time to FeedWordPress development and support than I currently do. But I earn my living freelance, by the hour or by the project, and I do have to pay my rent every month, so the only way that I can keep up with this on more than a limited and casual basis is if the donations from FeedWordPress users allow me to free up the time needed to work on FeedWordPress, rather than spending the same time looking for paying gigs.

So, if you would like to see more regular upgrades and bugfixes, and more rapid replies to your support questions, I’d urge you to consider how much ongoing development and support for FWP is worth to you, and consider making a contribution through the project donation jar at http://projects.radgeek.com/feedwordpress/. It’s in your hands, and anything you can offer will help. (If I had $5 for everyone who ever sent me a FeedWordPress tech support question, I’d be flush for the next several years….)

Thanks,
-C

How to deal with dates and timestamps in WordPress

If I hadn’t moved across the country recently, I probably never would have discovered a bug in how FeedWordPress handles the timestamps of syndicated posts. Fortunately, I did, and this bug is fixed in the recently-released version 0.99.

To explain what I mean, I’ll have to back up a bit, first.

There are a couple things that FeedWordPress uses date and time information for:

  1. Dating new posts: When a new post comes in over a feed, FeedWordPress uses the date and time information reported by feeds to date posts appropriately in the WordPress database. This means parsing date and time information and putting the resulting timestamps into the database.

  2. Checking for updates to existing posts: After a post has been imported into the WordPress database, FeedWordPress will keep checking to see whether the feed reports any updates to that post. This means comparing the last-updated timestamp on the feed to the last-updated timestamp in the database to see which version is newer.

The problem that I noticed came about because FeedWordPress was trying to do something sensible and easy to handle time zones when it was doing all this. Date/time handling in WordPress is fairly easy, once you understand what to do, but it is certainly anything but sensible, and therein lies the problem.

Part of the problem is, as usual, the stack of programs on top of which FeedWordPress has to sit. In order to get dates from the feed, FeedWordPress has to process two different human-readable date formats. RSS feeds use RFC 822 (or something like it), which FeedWordPress can convert to a Unix timestamp using the PHP strtotime() function. Atom feeds use the W3C DateTime Format, which FeedWordPress can convert to a Unix timestamp using a custom function provided by MagpieRSS, called parse_w3cdtf(). Unix timestamps supposedly do not vary by time zone: a Unix timestamp for a particular time is defined as the number of seconds between that time and 12:00 midnight on January 1, 1970 Greenwich Mean Time. So both strtotime() and parse_w3cdtf() make use of the time zone data provided by the format that they handle, and use it to convert the date-time information to a timestamp based on GMT.

So far so good. Now, when converting a syndicated item into a post for the WordPress database, FeedWordPress has to generate four different timestamps: post_date, which gives the date and time that the post was first published in the local time zone; post_date_gmt, which gives the date and time that the post was first published in Greenwich Mean Time; post_modified, which gives the local date and time that the post was last modified; and post_modified_gmt, which gives the GMT date and time that the post was last modified. These fields are stored in the WordPress database as MySQL DATETIME objects. WordPress later converts them back into Unix timestamps whenever it is necessary for formatting purposes. Here is how I did this in version 0.981:

$post[1] = date('Y-m-d H:i:s',
    (!is_null($post[2][3])
    ? $post[4][5]
    : $post[6][7]));
$post[8] = date('Y-m-d H:i:s',
    $post[9][10]);
$post[11] = gmdate('Y-m-d H:i:s',
    (!is_null($post[12][13])
    ? $post[14][15]
    : $post[16][17]));
$post[18] = gmdate('Y-m-d H:i:s',
    $post[19][20]);

The $post[21] array contains the Unix timestamps that FeedWordPress got from the feed. The PHP date() function formats a Unix timestamp relative to the local time zone. The gmdate() function formats it relative to Greenwich Mean Time. Given that I need local and Greenwich representations of the date and time that the post was first published, and of the date and time that the post was last updated, this seems like the sensible thing to do. But in spite of (or because of) its seeming so sensible, this way of generating post timestamps introduces a subtle bug. More on that later.

When determining whether or not a post has been updated, FeedWordPress uses two items of information: the last-updated date and time provided by the feed, and the last-updated date and time stored in the database. In order to get these two dates into a common format so that they can be compared, FeedWordPress uses either strtotime() or parse_w3cdtf() to convert the feed’s last-updated date and time to a Unix timestamp, and it uses the MySQL function UNIX_TIMESTAMP() to get a Unix timestamp from the last-updated date and time in the database. So here is how I did this in version 0.981:

    $guid = $post[22];
    $result = $wpdb->get_row("
    SELECT id, guid, UNIX_TIMESTAMP(post_modified) AS modified
    FROM $wpdb->posts WHERE guid='$guid'
    ");


if (!$result) : $freshness = 2; // New content elseif ($post[23][24] > $result->modified) : $freshness = 1; // Updated content else : $freshness = 0; endif;

strtotime() and parse_w3cdtf() take account of time zone data; UNIX_TIMESTAMP() presumes that the time is in the local time zone. So, I just need to feed it the DATETIME object that’s in the local time zone — post_modified instead of post_modified_gmt, right? Wrong. Again, in spite of (or because of) its seeming so sensible, this way of comparing timestamps introduces a subtle bug.

Here’s what was wrong with both of these sensible steps: in any given WordPress installation, there are three potentially distinct local time zones that you have to consider:

  1. The local time zone of the web server on which WordPress is running (usually set by the web server’s administrator);

  2. The local time zone of the MySQL server that provides the WordPress database (usually set by the MySQL server’s administrator);

  3. The local time zone set by the user in WordPress’s General Options page (set by the blog owner);

If any of these differ from each other, then the mismatch could cause problems for the way that older versions of FeedWordPress handled dates.

When the PHP date and time functions convert back and forth between human-readable formats and Unix time stamps, they do so relative to the web server’s local time zone. When MySQL converts a DATETIME object to a Unix timestamp, it does so relative to the MySQL server’s local time zone. When WordPress prepares dates and times for storage in the database, or processes them for sorting and displaying posts, it does so relative to WordPress’s local time zone, as set under General Options.

So, when FeedWordPress used the PHP date and time functions to generate post_date and post_modified, it used the wrong time zone — WordPress expects these to be local times in the time zone set under General Options, but the PHP functions use the web server’s local time zone. If the user has set a different time zone from the web server’s default time zone, then this date and time information will be incorrect. In order to get the correct time, we need to get the time zone information from the WordPress database, and then manually apply that offset, rather than leaning on PHP’s date and time functions. So here is how we do this task in FeedWordPress 0.99:

// Dealing with timestamps in WordPress is so fucking fucked.
$offset = (int) get_option('gmt_offset') * 60 * 60;
$this->post[25] =
    gmdate('Y-m-d H:i:s', $this->published() + $offset);
$this->post[26] =
    gmdate('Y-m-d H:i:s', $this->updated() + $offset);
$this->post[27] =
    gmdate('Y-m-d H:i:s', $this->published());
$this->post[28] =
    gmdate('Y-m-d H:i:s', $this->updated());

Note that we use gmdate() in all cases, because we are doing the time zone offset manually rather than letting PHP do it for us.

On the other hand, when FeedWordPress used the MySQL date and time functions to convert MySQL DATETIME objects to Unix timestamps, it used the wrong time zone again — since it was using post_modified, and in the older versions of FeedWordPress post_modified was generated using PHP date and time functions, the time was a local time relative to the web server’s time zone. But UNIX_TIMESTAMP() presumes that the time it is given is a local time relative to the MySQL server’s time zone. Usually this difference should cause no problem, since the web server and the MySQL server are the same machine, or if not the same machine, at least machines that are located in the same place as one another. But if that assumption ever fails, UNIX_TIMESTAMP() will return the wrong time–a time that is either a few hours before, or a few hours after, the real last-updated time. Which will mean that posts either get updated when there’s nothing new to update, or don’t get updated even when there is something new to include.

So we need to convert the MySQL DATETIME value to a Unix timestamp in a context where we know, and can set, the right time zone for the conversion. In this case, the best thing to do is to get the GMT date and time of the last update (in order to avoid issues that might arise from changes in the local timezone setting in WordPress), and then manually convert that into a Unix timestamp using a function that works relative to GMT. So here’s how FeedWordPress 0.99 checks the update times against each other:

$guid = $wpdb->escape($this->guid());


$result = $wpdb->get_row(" SELECT id, guid, post_modified_gmt FROM $wpdb->posts WHERE guid='$guid' ");

preg_match('/([29]+)-([30]+)-([31]+) ([32]+):([33]+):([34]+)/', $result->post_modified_gmt, $backref); $updated = gmmktime($backref[35], $backref[36], $backref[37], $backref[38], $backref[39], $backref[40]); if (!$result) : $this->_freshness = 2; // New content elseif ($this->updated() > $updated) : $this->_freshness = 1; // Updated content $this->_wp_id = $result->id; else : $this->_freshness = 0; // Same old, same old $this->_wp_id = $result->id; endif;

post_modified_gmt always returns the string representation of a MySQL DATETIME object; the regular expression breaks that representation down into its component parts; and the PHP function gmmktime() reassembles those parts into a Unix timestamp. (You might worry that using the PHP date and time functions might reintroduce the first problem, since it doesn’t account for the local time zone set in WordPress. But since everything is guaranteed to be in GMT, this problem doesn’t arise.)

Strictly speaking, it would probably be better to use MySQL functions, instead of a regular expression, to extract the parts of the MySQL DATETIME object, since ostensibly MySQL knows more about its internal formats than PHP does. In practice this is not likely to make a difference, but it’s likely that in future releases of FeedWordPress I’ll change the section to something more like this:

$guid = $wpdb->escape($this->guid());


$result = $wpdb->get_row(" SELECT id, guid, YEAR(post_modified_gmt) AS year, MONTH(post_modified_gmt) AS month, DAYOFMONTH(post_modified_gmt) AS day, HOUR(post_modified_gmt) AS hour, MINUTE(post_modified_gmt) AS minute, SECOND(post_modified_gmt) AS second FROM $wpdb->posts WHERE guid='$guid' ");

if (!$result) : $this->_freshness = 2; // New content else: $updated = gmmktime( $result->hour, $result->minute, $result->second, $result->month, $result->day, $result->year ); if ($this->updated() > $updated) : $this->_freshness = 1; // Updated content $this->_wp_id = $result->id; else : $this->_freshness = 0; // Same old, same old $this->_wp_id = $result->id; endif; endif;

Oh, and in case you were wondering, the reason that moving across the country helped me find this out is that I moved from Eastern Time to Pacific Time, and I changed the default time zone on one of my web servers before I changed the default time zone on my MySQL server. The mismatch exposed this bug while I was doing testing for the most recent release of FeedWordPress. Not particularly interesting, but it did expose a bug to fix, and I hope the guide to time zone issues that resulted may be of some interest.