NITF/Atom/NewsML extensions for FeedAPI

It should be straightforward but there are some twists.

The NITF spec is huge and overly ambitious, while the implementations tend to be invalid in one way or another. Some of the legacy newsroom systems emit real trash. In the AP Exchange and NewsML examples, NITF is entangled with yet another layer of XML.

So you have to accommodate error gracefully and provide an open interface for the Atom and NewsML integrations down the road if anybody is going to be able to use this.

Take the AP for example.

AP produces a general news service that's targeted for immediate online publication, called AP Online. It's minimally marked up, but generally clean. It's not hard to build a FeedAPI loader that works with it.

But a lot of us (particularly newspapers) don't want general newswires; everybody has them and there's no real value to being one of a thousand sites carrying AP Online. What we want is deep information about specific subjects of high interest to our geographic or other niche. Much of that information doesn't go on the AP Online wire and isn't available to competitors such as Google and Yahoo.

AP's proposed solution is AP Exchange, which gives editors global access to a huge AP database. But the output implementation is frustrating.

The NITF spec has a tag structure for compound headlines. AP sticks non-headline information into it.

The proper headline in this example is "Martians land at White House" and everything else is metadata that might or might not be there and doesn't belong there. This is basically a 1970s ANPA-1392 slugline stuck into the headline field, probably for the benefit of some luddite newspaper that hasn't discovered the 21st century and is still using an editing system that runs on a PDP-8.

I've been bitching about this a great deal, and the AP has agreed to provide a clean headline -- but external to NITF, in the Atom layer. So in that example the process might work like this:

• Editor defines a persistent search in AP Exchange, creating in effect a custom newswire.
• The search is exported as an Atom feed. AP has extended the Atom namespace with custom tags and AP can embed the full NITF for a story inside the Atom feed.
• Drupal process fetches the Atom feed. For each valid item, it gets the headline from the Atom layer, then hands the embedded NITF to the NITF parser, getting back an object. We then can ignore the NITF object's headline and prefer the Atom version, but we use the body and perhaps other components of the NITF object.

Along the way you have to make some decisions about how much information to keep and how much to discard. Presumably this ought to be part of the setup process for each feed, allowing the Drupal admin to map entities to CCK fields,

Now, it's not necessarily for all of this to be accomplished in the SoC project. I'm trying to paint the larger picture because if all we get is a parser that only works on clean, sane, proper NITF and doesn't accommodate the real world, we won't be laying the right foundation.

Is this a good example feed for NITF? http://www.nitf.org/IPTC/NITF/3.2/examples/nitf-fishing.xml are there others?

I'm not crazy about that example. I know what Alan Karben was trying to demonstrate but it would be better if IPTC also had some examples that were structured as common news stories.

Next week I'll see if I can get AP's permission to post some actual example files from its feeds, and also try to dig up some NITF output from a DTI system. If anyone else could provide some example data from other sources (hello, Reuters?) it would be helpful.

I think there's an assumption that NITF is a single-document text formatting standard, although there are accommodations for embedding references to other media types and related items.

The support in NITF for packaging multiple related items is weak, and that's one of the reasons Reuters pushed for NewsML, which conceptually is a wrapper that can contain a mix of items.

Some background on NITF:

NITF began its life as SGML markup intended to replace the ANPA-1392 standard for wire service transmissions, which dated back to the era when newspapers set type in molten lead. The NITF standard was unfinished when the Web came along, so many of its tags were adjusted to accommodate HTML conventions. Then it was still unfinished when XML became the hot item, so it was adjusted some more to make it conform with XML conventions. Unfortunately after that it was still unfinished and delivering no value to wire service subscribers.

There is a division in the newspaper business between slow-moving "big iron" print-centric and "web guys." NITF was in the hands of the former group.

Back around 1997 or so a bunch of us "web guys" got together at an American Press Institute workshop. Ffrustrated at the lack of progress on NITF, began working on our own quick-and-dirty XML standard designed to meet Web needs.

Dan Froomkin, who now writes about politics for washingtonpost.com, made the original pitch and it was called "Danco Web-O-Matic," quickly renamed NML as we brought into the conversation a number of serious players from Atex, WSJ.com (Alan Karben, who had implemented a proprietary XML standard for the Journal), some news librarians, and academicians. We were trying to be thorough and introduced geocoding, the Dublin Core and other concepts into the conversation. We discovered that Dave Megginson was working in the same area and swallowed as much of his work as we could.

This alarmed the NITF folks. They asked us to join with them, so we came together in a daylong summit in Dallas. We did a quick gap analysis, agreed to add some NML items to the NITF mix (notably the "abstract" tag, which is the equivalent of Drupal's "teaser" in a display context), and shoved it out the door. About a month later NITF 1.0 was published and the way was cleared for wire service adoption.

I pretty much walked away from the conversation at that point, having accomplished my immediate goal. Now, a decade later, I'm looking at what some of the implementors have done (or not done) with the standard and frankly I'm appalled. But that's the world we live in and that's the world any Drupal project has to accommodate.

Sorry, I have been traveling (Macau, Bangkok, Phuket, Kuala Lumpur) for a couple of weeks and almost completely offline.

Yes, I can mentor. I have a fairly good grasp of NITF and the (awful, in my opinion) implementations. The problem is not accommodating the standard, but rather dealing with real-world data.