Posts tagged as: opml

What problems are RSS libraries trying to solve?

I spent the day reading Python books, and now I'm starting to look at the many libraries that are available for reading and writing RSS in Python. There are certainly more libraries available than in Ruby, and that makes Python seem like a place I should spend some time, but all of the libraries seem to have the same perspective, which is to treat RSS as a specific variety of XML. What if I don't care about DOM and SAX and stepping through a tree of XML? I keep thinking about what happened when dBASE first appeared in 1981. At that time the standard way to do database programming was to use CBasic or Microsoft Basic to construct a record in memory using string variables. It was up to the programmer to do what was called record blocking, where the strings were padded out to fixed lengths and then written out to a text file. If you wanted an index as well, you had to find a separate library and manage insertion and management of nodes in a B-tree. Then dBASE came along, and all you had to do was say APPEND or EDIT in a program. dBASE wasn't about database details, it was about application building. If you wanted an index, you just said "INDEX ON <field>," and the index was created and automatically maintained. Instead of being a database programmer, you could be an inventory programmer, or a doctor's office programmer. In fact, doctors could become programmers and create their own applications. That is what I want with RSS and OPML. I don't care whether a feed is RSS or RDF, or what the specific XML tags are. I want to be able to say "Give me the posts from this list of feeds that are less than 1 week old. Now combine them all into a new feed." That should be 3 or 4 lines of code. There has to be a reason that RSS programming is still about creating generic aggregators instead of RSS applications. It is a combination of tools and coders who are thinking about RSS as streams of XML data, and not application specific data that should be represented as vertical applications that have nothing to do with reading blogs.

Exploring the tacit knowledge between RSS and the Semantic Web

I started reading about the Semantic Web again last week, and my immediate reaction was the same as the first time I tried a few months ago. This is such a perfectly specified, intellectually rigorous collection of standards and practices that it seems almost impossible to find an entry point. If it is so hard to get started, then how does anyone work with it? The answer is that the people who understand it now are the same people who helped to build it. Each of the many sub-standards and protocols were introduced in reaction to a specific problem discovered during the creation of some other portion of this edifice. An analogy is trying to understand how an immense cathedral could possibly have been built by walking around the finished building. Once the scaffolding and the masses of workers are long gone, it seems like every part fits seamlessly into every other, and the thousands of decisions that were made during its construction are erased.

I decided to pull back a step and look at the areas of namespaces in RSS and the many competing standards for structured microcontent on the Web. This is much messier and clearly a work in progress, but once again as with the Semantic Web, the same individuals keep popping up in these many projects. The problem with the social nature of the construction of microformats, structured blogging, RSS, Atom, etc., is the unspoken, or at least underdocumented, aspects of the decision process. Why are there two competing sets of blog microcontent formats? Why are there apparently dozens of overlapping collections of RSS namespaces? The answers are lost in the maze of blog posts and standards announcements made over the last few years. Why isn't everyone involved with this area terminally confused? Because they lived through the process and understand the political, social, commercial aspects of each of these multiple body collisions.

What we now have is a continuum from the ultra-simplistic, under specified formats of RSS and OPML to the ultra-rigid, crystalline perfection of the Semantic Web. In between is a rabbit warren of partially completed, interconnected attempts to add more structure and functionality to RSS and HTML.

So what is the solution? I'm not conceited enough to believe that I can unravel the current mess lying between RSS and the Semantic Web, and I'm also not smart enough to try to storm the castle of the Semantic Web by brute intellectual force. What my past history has shown me is that I am capable of helping people build tools and writing documentation that can help bridge this gap. The process I'm going to follow is to start studying and coding with the RSS namespaces and microcontent formats until they gradually make sense, and then try to get tools built by others that will provide a more accessible conceptual model. In other words, I'm going to live there until I grok the neighborhood.

I went through the same process when I moved to Boston. The classic line when trying to explain how to navigate witihin Boston is "I can't tell you how to get there, but I can take you once and show you." This is a perfect example of tacit knowledge. It is something you and your community knows, but which can't be explained in words. It may be an urban legend, but there are many stories of truck drivers paying taxis to lead them through Boston's streets to a specific location. The only way to deal with Boston's streets is to carry a map for the first few weeks until your brain somehow builds the tacit knowledge you need to feel comfortable.

Grazr update

Grazr has now been in an Alpha test for a little under a month, and the response has been overwhelming. I believe that there is an intrinsic demand for any piece of software, and you never know how big that will be until the product is offered to the public. In the case of Grazr, the possible uses have completely outstripped the original conception of a simple OPML viewer. One big area that wasn't anticipated is the use of MP3 files within OPML. Dan MacTough supported this first in his Optimal Browser, and Mike Kowalchik quickly followed suit and added an MP3 player to Grazr. James Corbett came up with the term Podgrazing to describe this type of application, and within a week or so one of James' countrymen launched Podgrazr.eu to showcase Irish podcasters. Another Irishman, Robin Blandford, also launched CommentCasting using Grazr as the user interface.

Another great idea from James Corbett has been the reinvention of the old webring idea in the form of OPML rings. James calls this grazer blades, but thankfully everyone is ignoring that particular pun. You can see this demonstrated in the Grazr box on the left side of any page of James' blog. Click on the link for "My Grazer Blades," then click on the link for "Pieter Overbeeke." At the bottom of this list of links you'll find a link for "James Corbett" that brings you right back to where you started. James is also making a good start on a world wide outline through the rapid growth of his Open Irish Directory, which is in the same Grazr box on his navbar. This makes use of the include capability of OPML, which allows the growth of distributed outlines with many sources and independent editors.

Ireland is a real center of Grazr and OPML activity, thanks to James, but the overall global appeal of Grazr has been a real shock. The Grazr below shows the results of a Google blogsearch for "grazr," and the number of links in languages other than English is impressive. Mike had to spend some time in Unicode Hell to get all the languages working well, but that has really paid off in international adoption.



Another area of interest that wasn't anticipated, but should have been, is the use of Grazr on mobile devices. Russell Beattie wants this so badly that he even created a mockup of it on his blog to show everyone what it would look like.

Grazr will be entering Beta test in a few weeks, and the new features that will be added then should really open some eyes. Right now Mike is working on making the backend as stable and scalable as possible, but once that is under control, the real fun can begin. People assume that the current interface is the only one available, but that is just the starting point. Grazr is built on a Javascript library that makes all types of interfaces possible, such as an expand/collapse outline, and an automatically scrolling "river of feeds." Different types of transitions, similar to what are used in video editing are also in the works.

Once again, I'll make my disclosure that I have no financial connection with Grazr, and hold no equity position. I just love the idea of seeing good software get built and helping to grow a community around OPML.

The second coming of the Web

I've been watching Danny Ayers' attempts to have Semantic Web people consider outputting RSS and OPML data or using OPML tools to visualize Semantic Web data. I respect and applaud his efforts, but I wasn't surprised by the universally negative reactions. I know that users of RDF based formats have tremendous disdain for RSS and OPML as being poorly defined, which they admittedly are. What I was shocked by was the tone and terms used in the responses. There is an almost religious sense of RSS and OPML as evil, and a possible source of spiritual contamination. Now Semantic Web people are extremely intelligent, as they'll be quick to admit, so what could have happened to them to cause such an adverse reaction to what is simply a set of formats for text files? It is easy to point to the creator of RSS and OPML as the root of this negative feeling, he certainly is mentioned often in the response to Danny's pleas. But that is just scapegoating. I think the visceral emotion exhibited, almost a form of terror, at the idea of having to co-exist with RSS and OPML, has a deeper cause that fits into the religious fervor with which it is voiced.

When Tim Berners-Lee first gave mankind the Web, he made a tragic mistake. He granted us free will to use less than perfect HTML. His tools, and the tools of those to follow him, allowed users to develop sinful habits based on ignorance and sloth. The result was a Web of corrupt data, in which misformed tags abounded. This great fall from grace by the users of the Web prevented it from ever attaining the state of perfection desired by all computer scientists, a completely machine readable database. So the disciples of Berners-Lee, with his blessing, developed XML as a way of wiping the Web clean of the sinful and broken HTML, and replacing it with perfectly specified and implemented data. Now, just as the second coming of the Web is in sight in the form of the Sematic Web (well, its been in sight for years, but we'll put that aside), here comes a poorly specified corruption of XML, what Danny jokingly calls "quasi-XML", that threatens to again lead mankind astray. Is it any wonder that Semantic Web devotees are reacting as if RSS and OPML are the work of Satan?

Do you find all of this over the top? Good. That is the point of satire. I find the reactions of Semantic Web people over the top as well. It's just data. Converting from one format to another is so trivial that even I can write the code to do it. Surely anyone who can code for RDF could import or export RSS and OPML. Why should anyone do it? As Danny keeps pointing out, there are millions of RSS users. In time many of them are likely to use OPML as a container for RSS. There is no reason why OPML can't be viewed as a bridge between these two sides of the Web. But then if I was in league with the devil, I would say something like that, wouldn't I? After all, my namesake was led astray by the devil once before.

Why OPML?

"OK," the answer comes back, "we can now see what you are doing with OPML, but why bother? OPML is poorly specified, it isn't nearly as complete as an RDF based standard like the Semantic Web, and it's inevitably going to be the center of political firestorms because of who created it." Let me present the basic arguments that persuaded me to spend so much time supporting the format:

• My first blog post to get any links pointed out that RSS was helping to explode the Web's architecture. What I meant by that is the growing trend to make all Web content available via RSS. This effectively everts the traditional website, putting the content outside in a machine readable form. If you accept that RSS will be a major architectural component of the future Web, whether or not the users know they are using technology based on RSS, then OPML as a container for multiple RSS text streams deserves attention. OPML allows us to easily create and consume reading lists of multiple RSS feeds, pushing back the limits of infoglut by at least an order of magnitude. If you can read 10-20 blogs on their websites, and 100-200 RSS feeds in an aggregator, then reading lists each containing 100-200 feeds allow us to juggle over a 1,000 feeds. Not easily, but it is at least possible. It isn't the final solution, but the fallacy is believing there is an ultimate solution in technology. It is a journey, not a destination. RSS and OPML are just steps. RDF is another step, admittedly a big one. The Semantic Web, if we reach it, will just be a temporary resting place.
• OPML is not just a container for RSS. It is a general purpose outline structure (hence the name Outline Processing Markup Language), which allows the construction of hierarchies based on any type of XML data. The new OPML 2.0 specification will make that possible through the use of namespaces. This means that any XML formatted data can be incorporated into an OPML outline. There are two big areas of Web data that fit into this model: microcontent and API results. If microcontent, meaning individual molecules of data floating free in the bloodstream of the web, is to become a viable delivery mechanism for information, then a structural equivalent of a protein is necessary to package these molecules in a consumable form. OPML is a first step towards that structure. As for API results, I've already performed a simple experiment demonstrating the use of OPML as a container for this type of data. I plan on doing a lot more work in this area. OPML data combined with a good viewer makes the construction and delivery of mashup data a trivial task.
• If we need a building block for the next generation of the Web, why even stop at OPML? Why not go to something perfect like RDF, asks the RDF crowd. OK, maybe I'm mischaracterizing some of them, maybe they just think it is many orders of magnitude better. I tried reading about RDF and the Semantic Web, and I had to stop because I was afraid I was coming down with narcolepsy. I don't think I'm smart enough to grok the Semantic Web yet, and anyone who reads this blog knows I think I'm pretty smart. I need to work my way up to that level of complexity, and the way I do that is by blogging, and writing code, and helping to design tools with a simpler, more accessible format like OPML. I'm conceited enough to believe that I'm at least as smart as the average computer user, so if I need to work myself up to the Semantic Web one step at a time, they probably do also.
• Does this mean that all of the work on OPML will eventually be wasted. Will it all have to be thrown away? First of all, after working in the software industry for 26 years, I know that all software is eventually thrown away. When I moved out of my last house, my wife made me throw away an entire dumpster full of software packages. At least now it can all be done by just wiping a hard disk. But that doesn't mean the present OPML development work is a waste. OPML tools are built to work with XML data, and despite its flaws, that is what OPML is inside. Converting from OPML 2.0 to OPML with namespace extensions to RDF is an evolutionary process, which is the way I believe that all software is created in practice. As long as I've slipped into Darwin territory, let me repeat one of his favorite mottos: "Natura non facit saltum." Nature does nothing in jumps. I believe that since software is a product of human nature, it also moves in slow, often inefficient, and gradual steps. I am fully convinced that the virtual product line I am helping to construct around OPML will make the transition to a fully XML based Web more smoothly and with more users than waiting several years until the computer scientists perfect the Semantic Web.
• Finally, we get to the political issue. Sure there are firestorms around RSS and OPML. Are they more vicious than the ones around RDF? I have no idea, but if RDF is being created by humans, then there are fights, and cliques, and petty jealousies in the RDF world also. If you want to see how someone is able to overcome the name calling surrounding the battle between OPML and RDF, read Danny Ayers' blog. He's been doing an amazing job of trying to get RDF people to output OPML and OPML people to see how there is a better world on his side of this debate. I am learning a lot from Danny, and if I can't work out a way to get him to Boston for OPML Camp, I plan on flying over to Italy to see if he can teach me about the Semantic Web without me falling asleep.

Adding a Bitty browser to your browser

I talked to Scott Matthews of Bitty yesterday by phone, and learned a lot about what he wants to accomplish with the product. Some bloggers have been describing Bitty as an OPML viewer, like Optimal or Grazr, but it really does more than that. Bitty is a full web browser than can display HTML web pages as well as RSS and OPML files. Scott's goal is to allow people to create a "picture in picture" experience within their web pages. For example, I have added a Bitty browser below to display the contents of my mashup blog:



Admittedly this is a silly example, but there are many great uses for Bitty. Imagine a site for a fan of a sports team using multiple Bitty browsers on the page. There could be a window for browsing the official team site, and other windows showing the sites of the top players. Bitty would also have been useful when people were demonstrating the differences between the US version of Google and their censored China search site. Both Google sites could be put side by side on the same web page. The key to this idea is that the user of a page with a Bitty browser can navigate a different site or even the entire web without leaving the original page. I know lots of people who dislike reading blogs, because they keep getting sent off to other pages to follow the bloggers train of thought. Within a few links, they lose track of where they started. Creating a browsable container within the constant context of a web page would be very useful as a way of solving this problem.

Bitty can also be used as viewer for OPML and RSS files. Here is an example with this blog's feed:



Whether you prefer Bitty's RSS display over Grazr or Optimal is a matter of personal taste. I like Optimal's expand and collapse outline, and Grazr's compactness. On the other hand, when the RSS item contains a link to a web page, Bitty can display the page without launching a new window. One thing to be aware of when displaying RSS or OPML with Bitty is the fact that it adds a set of Yahoo advertisements at the end of the RSS or OPML content. Bitty is the only one of these products to display ads. User reaction to advertising within one of these web page widgets is still something that has to be worked out in the marketplace.

One feature that Grazr should definitely adopt from Bitty is the ability to launch itself into a separate, smaller window. This is done by clicking the small launch button on the top right corner of the title bar. This window can then remain open after you have left the page where you found the Bitty browser. I'm sure Scott can also get ideas for Bitty by studying Grazr and Optimal as well.

In a field that is this new and this active, we are sure to see lots of cross pollination, which is great for users. I'm working on getting Scott and Optimal's Dan Mactough to come up to Boston for OPML Camp, so they can sit down with Gazr's Mike Kowalchik. I love it when things are at a stage where authors can just talk to other authors, instead of company reps talking to each other.

scrAPI for OPML

Now there's a title to conjure with. John Musser has an interesting ProgrammableWeb post on the use of screenscraping as a poor man's API. The idea is to use a script to parse a web page, and then return some specific set of data in an XML format. John credits the idea to Thor Muller, who provides some excellent details on the pros and cons of a scrAPI vs. an official API. Thor in turn recognizes Paul Bausch for coining the term SCRAPI in 2002. John notes that the result of the scrAPI can be returned "in some cleaner XML format." Yeah, like ... uh ... OPML?

A scrAPI is clearly your only alternative for a site that doesn't offer an API, but surely once an API is available you wouldn't need to adopt the scrAPI approach. Except when the API provider has a limit on the number of times you can call the API each day, and refuses to respond to email requests on how to get beyond the limit. I ran into exactly this problem with Technorati when I built my Tech Memeorandum - Technorati mashup. I hit an undocumented limit on daily Technorati API calls, and despite repeated emails to the company, including directly to Dave Sifry, I never got a response. I've been reading up on PHP the last few days, and this sounds like exactly the type of example project I should try out. Yes, it is a terrible kludge, but then what isn't on the Web? I think scrAPIs that return OPML can become a useful way of building a mashup, and I'll experiment with it until a better coder can implement the idea more cleanly.

Update: I guess it isn't too surprising that Dave Sifry would have a vanity feed at Technorati. At least I assume he has, because soon after this post pinged Technorati, I got an email response from him to my earlier messages. If they actually do give me a larger daily allotment of API calls, I'll be able to start posting some Technorati API scripts on my programming blog. If not, I'll just have to start scraping. Getting something done because of a blog post is fun, but it encourages bad behavior. This is probably how Scoble turned into the spoiled kid he can sometimes be. I love it when he starts yelling on his blog, "This blog is broken, and somebody better fix it right now!" I wonder if his wife lets him get away with that stuff at home, "I'm hungry, and someone better feed me right now!"

Update: Sean O'Hagan emailed me to point out the scrape command he wrote for YubNub. I guess this lazyweb thing really works.

What the heck am I doing with OPML?

It's now getting more common for people who read my blog to say something along the lines of "I don't know what you are trying to accomplish with OPML, but it seems pretty cool." I've had a pretty clear goal for a while now, but I haven't written about it here, because it seemed too unlikely to describe before it started to happen. Now that Grazr has been opened up to the public for testing, and a few more OPML tools are getting closer to the same state, I guess it is safe to talk about. My basic plan is to help create an OPML product line. (Note to rapid responders: Before you fall off your chair rushing to reply, take a breath and reread the previous sentence. See the word "help." It means aid, assist, be a part of. It does not mean control, dictate, dominate.) I hesitate to say this, because of my experience with the blogosphere and trigger words implying control. I'm trying to act as a virtual product manager, and do my best to help the growth of a marketplace or ecosystem around OPML. I'm not planning to own anything, or be the "boss-man" of anyone.

What do I mean by a product line? What I don't mean is a set of centrally designed and controlled products that neatly interconnect. I'm thinking more loosely of a collection of websites and products from multiple vendors that share a common data format of RSS and OPML files. Even the word vendor is loose, since I'm assuming these products will be largely free. Unlike many in the Open Source world, I like competition, so it would be great if there were multiple OPML viewers, editors, and code libraries. They should also overlap in various areas of functionality. Competition for users, even among free products, gives everyone the best set of tools. It's also a lot more fun, even for the people creating the products. Often a rival is the only other person who knows exactly what you are going through.

What do I mean by virtual product manager? What I don't mean is that I want to tell anyone what to do. I would like to be able to make suggestions of what I think would make a more useful and more successful product, and see some of these ideas carried out. I think of this as vicarious product design. (Sort of like being a grandparent, an experience I look forward to. You get to play with the kids whenever you want, but don't have to change their diapers or pay for their college tuition.) I get a huge kick out of sitting down with a bright programmer, laying out a possible spec and set of interface features, and then seeing it in use by real people. Grazr is the first chance I've had to do that in seven years, and it proved to me that I still get just as much of a thrill when it works out well.

I have no financial connection with Mike Kowalchik of Grazr or anyone else making any OPML products now, and don't intend to have any for quite a while. That means no consulting fees, paid advisory boards, a little "taste" of equity, or investing in anyone. Luckily I saved enough from the dotcom era that I don't have to make money for a while, although I will need to eventually. How long is eventually? I don't know, but at least a year, probably longer. I can promise you that when I do decide that some idea is so compelling that I just have to take a financial stake, I will announce it here, and significantly change how I blog, or even stop blogging. By the time this happens, I'll probably have moved on to something else, so there is no way of knowing if I'll ever make money from OPML.

What role do I actually take with an OPML product? It depends on the product and its creator. In some cases I may help to spec out a completely new product or website based on an idea that I have or that someone brings to me. In other cases, it may be a product that already exists that I think can be improved or "productized" in a better way. I like the process of design, promotion (although I'm sure many would call it hype), teaching, and even introducing the right people to each other. I also like to help people find jobs. In fact, building a product as resumeware to get industry recognition is the most powerful reason for coders to build many of these products. I'll also be working to interconnect the OPML community through projects like OPML Camp.

Why am I doing this? What do I get out of it? It isn't power, although I'm sure that will come up in criticisms of this post. Well, maybe power is part of it. But it is the power to see cool ideas get an audience, and to help smart people earn a living doing what they love. The real answer is that I love software and the people who make it. In the Eighties I played a similar role in helping to build a marketplace around dBASE, and it was a great time to be in the software industry. The parties and being able to hang out with people like Tom Rettig were worth being a part of it. If I can help recreate just a small part of this, it will have been worth it for me. For anyone concerned about my being on a power-trip or trying to assert control, here is the basic test. If you ever see me trying to stop anyone from doing anything, then you will be right. As long as I am helping anyone who wants assistance in making a better product, or in helping people understand how to use it, you will be wrong.

Should I wait longer before launching this master plan or even talking about it? Perhaps, but I've always believed in the just do it philosophy. When I bought a TRS-80 in early 1980, I quit my job as a chemist so I could teach myself how to program. When I first grokked the Web in early 1995, I ran around the software industry screaming that this was the coolest thing I'd ever seen and that I was going to be a web master, even though I had no idea what that meant. Is OPML as important as the first micros or the early Web? Of course not, but RSS is going to be huge, and OPML is the container for RSS and will have a big impact on how data is architected on the Internet. So I'm willing to bet a lot of my time on helping it grow.

Grazr works directly on RSS files too

The response to the public alpha test of Grazr has been tremendous, but one thing that seems to have escaped most people's notice is the fact that it can display the contents of an RSS feed, even if there is no OPML file. Here is an example of Grazr displaying the RSS file for this blog. Note that the URL it is using is my RSS feed, not an OPML file. One great application for this is a listing on every blog's page of its most recent posts. The visitor can read past blog posts without leaving the current page. You can use this capability to display the result of searching for tags on sites like Del.icio.us, or Digg. The alpha version of Grazr still needs work on the display format for posts, but that will be cleared up soon. Since Grazr can display photos without having to open a new browser window, you can also use it to display images from RSS feeds delivered by Flickr, or NASA.

Grazr OPML viewer begins public alpha test

I've been hinting about Grazr for a while now, but I didn't post anything explicit because it wasn't open for public testing. Mike Kowalchik, Grazr's author, has now opened up his website to allow anyone to try an Alpha version of the product with any OPML file, and to put a copy on their own web site. Here is an example of Grazr with a sample OPML file:



Grazr uses Javascript, so if you don't see a really cool OPML viewer here, your web browser or aggregator isn't displaying Javascript output. You may have to change your browser preferences. You can also experiment with different OPML files on the Grazr site, as well as create a script to run your own copy of Grazr. If you want to see the raw OPML that is being displayed above, it is here.

The dimensions of Grazr are flexible, and you can shrink the text size, so you can also put one on your side navbar. James Corbett has been using it on his Eirepreneur blog for a while to display his Open Irish Directory. This form factor makes a great reading list widget, because your visitors can actually do feed grazing right on your web page. The term grazing may still be foreign to many people, but basically Grazr is a very capable RSS aggregator. There are still a number of cosmetic issues to improve, but that is why this is still an Alpha version. If you want to make suggestions for improvements, I'm sure Mike will be glad to get the feedback. Here's an example of Grazr with my Tech.Memeorandum dynamic reading list:

I've been helping Mike with the design of Grazr for a while now, but I have no financial relationship with the product or any of Mike's efforts. This is true of all the OPML products I'm working with, and actually all other products and websites. I don't believe in blogging as an independent observer and taking money or equity positions at the same time. There is nothing wrong with writing a company blog, however, such as Mike's own blog, or the blog of any other company employee, as long as that is the clear position of the blogger. For now, I'd rather remain independent, so I'll avoid any financial relationships. When I decide to join a startup or become an investor in one, I'll probably stop this blog and start a new one. At the very least, I'll make my new situation known and change the focus of my writing.

Any product based on the idea of feed grazing owes a debt of gratitude to James Corbett for inventing the term and promoting the concept. James has been a big supporter of Grazr and the rest of the OPML community.

Rethinking my programming strategy

Ever since I did my last burst of Ruby programming a week ago, I've been trying to figure out where I want to go with Ruby and Web programming in general. The truth is that I'm hitting a few roadblocks that are rather frustrating. I still find the Ruby syntax cleaner than any language I've used since dBASE. On the other hand, the available Ruby libraries are not as robust or mature as Python or Perl. This is a particular problem with XML parsing. I didn't expect this when I started programming again last fall, but since almost all of my programming has been with RSS, OPML, API calls, I'm extremely dependent on REXML, the XML parser built into Ruby. I've been hitting bugs and weirdnesses with REXML, and after emailing the author 3 times over the last 2 months, I've still not been able to get a response. On the other hand, I'm now getting to know a lot of Python programmers. There is even a Python programmer in Seattle who keeps emailing me, offering to build RSS and OPML libraries to do whatever I want.

I knew Ruby had fewer libraries than older languages when I started using it, but I assumed I would just write my own libraries to do whatever I wanted. That's what I've always done in the past. The other thing I've discovered in the last few months is that I'm going to have less time to code than I originally expected, so building libraries for things like XML parsing are not likely to happen. Actually my whole approach to programming is going to be different than I had expected. Instead of getting more and more proficient in one language and tackling all kinds of tasks, I now see that I will be focusing on RSS, OPML and API calls for possibly a few years. It might make more sense to see how this is done with many of the available languages, such as Perl, Python, PHP, and Javascript. Instead of becoming a language expert, I'd now rather become a domain expert.

This decision is also dependent on ideas I might have for writing books. I really enjoy that process, and know that a good book can be a valuable aid to developing my new career as a tech blogger. Instead of writing a Ruby book someday, it now seems more likely that I'll want to write an RSS and OPML book, which means I may want to be able to discuss techniques in multiple languages.

I haven't made up my mind yet. I'm going to spend a few days playing with Python and looking at other XML solutions for Ruby. Who knows, maybe someone will tell the author of REXML to answer my email and tell me how to keep it from choking on ampersands.

Kosso has an interesting idea about extending RSS and OPML

Kosso sends his regrets for having to work last night, and not being able to join the OPML podcast, but he also drops an interesting hint on what he has been working on:

One way of implementing this would be to use a namespace to add an audio timestamp to RSS, and then create an RSS file with one item for each timestamped portion of a .mp3 file. A playlist linking to multiple annotated RSS files could then be constructed in OPML. Apple already has an iTunes namespace for RSS 2.0, but this doesn't allow for timestamping sections of the .mp3. Something tells me that namespaces for RSS and OPML are going to explode this year. Things may get messier than some people would like.

OPML Podcast with John Tropea, Josh Porter and Alex Barnett

I did another podcast with Josh Porter and Alex Barnett last night, and this time John Tropea joined us. John is a corporate librarian in Perth, Australia, who has been been cataloging all of the new OPML tools on his Library clips blog. We discussed the OPML 2.0 draft, namespaces, tools, integrating structured blogging with OPML, and OPML Camp. You can find the podcast on Alex's blog.

RSS Advisory board finds a new way to extend RSS

You have to give Rogers Cadenhead credit. After enduring several weeks of blistering attacks by Dave Winer over attempts to create a new version of RSS 2.0, he is once again trying to improve RSS. This time, Rogers and his fellow members of the RSS Advisory Board are proposing a new RSS namespace called XRSS. He evidently came to the same decision as I did with OPML. If the only way Dave will allow RSS or OPML to be extended is by the use of a new namespace, then that is what has to be done.

The funny thing is that I just tried to write an explanation of the difference between using a namespace and changing the core format, and I can't find a way to do it that shows why one is better than another. In the end, an application that uses these extensions still has to know what to do with them. The only benefit seems to be that a namespace gives the application permission to ignore the new tag. I can hear the crickets chirping right now. This is not a subject that will hold an audience. I'm going to start working with namespaces in OPML, and perhaps then I'll find a way of explaning them that won't make normal humans run screaming out of the room.

Creating an OPML 2.0 namespace

Matt Terenzio, one of the OPML Camp attendees, has suggested that we hold a session on OPML namespaces. This is permitted in the OPML 2.0 spec: "An OPML file may contain elements and attributes not described on this page, only if those elements are defined in a namespace, as specified by the W3C." We should definitely do this, but maybe we could take this one step further and create an OPML Camp namespace that we can all experiment with. If an OPML Adv... oops ... Stan... oops ... if a publicly recognized group discussing OPML is formed, this namespace can be given over to them to hold and maintain.

Apple is trying to patent RSS autodiscovery, aggregation, and reading lists

I'm not a lawyer, but these two patents by Apple sure look like they are claiming ownership of RSS autodiscovery, aggregating multiple RSS feeds into one, and automatically updated reading lists. Time for some real lawyers to step in and comment on this. John Palfrey, as the representative of Harvard's RSS copyright, what do you think?

(via Dave Winer)

LinkBlox: OPML based blogroll viewer

Laszlo Systems now has a cool OPML blogroll widget called LinkBlox. If your browser/aggregator supports Javascript and Flash, you'll see it below with the default OPML blogroll in use:



LinkBlox is free for use on your own pages, and Laszlo has a number of nice ideas for creating a Javascript widget. When you register for LinkBlox, you are given a form that allows you to specify all the parameters, such as the OPML file to use and the colors of the widget. You are then given a single line of Javascript that you can embed in any web page. Since the widget displays the contents of an OPML file, every time the contents of that file changes the blogroll will automatically change. This is a great way for a site like Technorati or Top10Sources to deliver fresh reading lists to people's pages.

LinkBlox currently has some limitations, which should be addressed if Laszlo wants to be competitive in this rapidly growing space. The OPML structure accepted is limited, and can't work with an outline more than one level deep. More importantly, LinkBlox doesn't know about RSS links, so you can't use it for feed grazing. Sorry, James.

(via John Tropea)

Annotated Reading Lists

Pito Salas, Project Leader of BlogBridge, and I just had a very productive lunch. One of the things we discussed was the Technorati-Memeorandum mashup I built into an OPML file. We decided that this process should be called annotating a reading list. If the River of Feeds built on a source like Tech Memeorandum is a useful way of sampling new feeds, then an annotated version is even better, since it provides more context for each blog. Pito explained that BlogBridge is already calling the Technorati API to gather background data on each feed it displays, so I suggested that he add automatic annotation as well.

Here's how this would work. When you open a reading list in BlogBridge, or any other aggregator that wants to use this technqiue, it would allow you to select a set of possible annotations, such as Technorati tags, Delicious tags, Flickr photos, etc. These annotations would be used to create a new OPML file, such as I just created with this new mashup. You could then host the annotated reading list on BlogBridge or post it on your own site. The file could be viewed with Optimal, or any of the other OPML browsers that will show up shortly.

Annotation would enhance the process of finding new, interesting feeds to read, and could also become the basis for automated blog recommendation systems. Imagine submitting a blog URL to a site, and having it return a list of blogs with a similar tag profile.

Technorati - Memeorandum mashup

I finally got a chance to explore the idea of storing API results in an OPML file. The window below shows the results in the Optimal Browser. You can also retrieve the raw OPML file here. To create this OPML file I took the list of blogs that my Ruby script has been collecting from Tech Memeorandum and searched the Technorati API with each blog's URL. I then combined the blog's rank, number of blogs linking in, and the most recent blog posts linking to the URL into a single OPML outline. I still have a number of programming details to work out, so I won't be pubishing the source code for this for another few days, but I will start describing all the details of the API calls and how to construct the OPML file on my mashup blog. One problem I discovered is that Technorati apparently limits the number of API calls per day, a fact that doesn't seem to be mentioned anywhere on their website. Until I can get someone there to raise this limit, I will have to leave this OPML file as it is. With a higher limit I hope to have it refreshing every hour.

This is either really cool, or really depressing

There are apparently over 900 sites that qualify as Web 2.0. If they put it all into an OPML file, it would be cool.

(via Steve Rubel)

Group grokking of OPML

I don't know if anyone has ever used the term "blogosphere time," but I'm definitely experiencing it in terms of the growing excitement and quickening pace of innovation and conceptualization in the OPML world. Blogosphere time is several orders of magnitude faster than the Internet time of the Nineties, and I think we have aggregators and memetrackers to thank for that. The group consciousness surrounding OPML is evolving at a pace that can be measured in weeks and days, where changes in Internet time merely compressed years into months. I woke up this morning and found John Tropea almost frantically trying to catalog all of the OPML tools, like the librarian he is. Do I dare tell John that being able to write down the names of all the products related to a hot area of technology is a sign of a very early stage, and rapidly becomes a sisyphean task? I don't think librarians want to think about information expanding faster than their catalogs. Next there was Steven Cohen's post on Optimal, showing the same level of excitement about a single product that John was about all the OPML tools. Steven is also a librarian, see the connection with OPML there? Finally, I turned to James Corbett, our poet, to see where his mind is getting ready to take us. James was trying to work out his theory of the collective intelligence of social bookmarking, making him a sort of Jung of tags. I've gradually been able to get into James' head over the last month or two, and I can tell you that it is a fun place to be. Although his latest post isn't explicitly about OPML, I can see that he is building up to a new stage in his unified theory of information grazing.

OPML Camp is almost two months away, and I'm afraid to plan any sessions, because it is hard to imagine where the collective thinking will be by then. The tricky part is figuring out how to get James here to Boston from Ireland, and John from Australia. Steven has already agreed to drive up from New Jersey.

New version of Optimal Browser

Dan MacTough just emailed me about his new version of the Optimal OPML Browser, and he certainly has been busy. You can find the complete list of changes here, displayed in an OPML file of course. My favorite improvement is that links to web pages now break out of the IFRAME window and take over the whole page. I also love the new docs. Yes, I said docs. If you are a Ruby programmer, you may not have seen what these actually look like.

Since I'm sure you're sick of seeing me drag out the same old Tech Memeorandum reading list, I present for your reading pleasure Optimal version 0.3's rendering of the Top10Sources guide to Bollywood. Enjoy.

Is OPML 2.0 ready to be announced?

Yesterday Dave Winer teased an announcement due today for a 2.0 of something. He did it again today on the comments for a Flickr photo of Marc Canter (WTF?). David Sifry of Technorati replied on the same comment thread "Hmmm, OPML 2.0? :-)"

Update: The announcement of OPML 2.0 is now up on Dave's site.

State of OPML software

John Tropea has a great round-up of all the OPML capable software and online tools. I just discovered his blog yesterday, but it is must reading for anyone interested in OPML.

Saving API mashup data in OPML files

Now that I can use the Optimal browser to display OPML files easily on a web page, I can go ahead with an idea I've had for creating mashups of API results. I wrote about this on my mashup blog 2 weeks ago, so you can find the details there. I'm going to start coding now, and as usual the details will appear on the mashup blog as I work them out, and the source code will appear on my Ruby blog. I'll report back here when anything interesting is ready for viewing.

Feed grazing with the Optimal Browser

Feed grazing just got a lot easier, because Dan MacTough has released an open source OPML browser called Optimal. It is still an early version, but it is already good enough to allow people to view OPML reading lists without having to install a desktop aggregator. Optimal is written in PHP, so it shows up right in your browser. I was able to install it by simply copying the files into a new subdirectory on my server. I also made a few simple changes: I modified the title of the page, added a link that will load the dynamic Tech Memeorandum reading list I've been working with for the last few weeks, and added code to refresh the page every hour. Now I can just keep this page open in a browser window and have a true river of interesting feeds.

Optimal doesn't have all the bells and whistles of a full aggregator. Most importantly, it doesn't keep track of the posts you have already read. But it does satisfy the current demand to finally see what an OPML reading list looks like. You can enter the URL of any OPML file and start grazing right away.

(via Eirepreneur)

Planning begins for OPML Camp

Now that I've had a chance to see how Mashup Camp went, I'm going to begin the planning on my proposed OPML Camp. You can follow the process at the camp blog.

Is there an optimal size for a reading list?

I was reading Jim Moore's blog for the first time, and I came across this recommendation in his instructions for new users of OPML reading lists:

I've seen this idea before and it's always puzzled me. Amy Bellinger makes the same point: And so does Dave Winer:One issue may be the technical problem of an aggregator not being able to load too many feeds at once. The solution to that problem is to try a different aggregator. I just tested Anne Zelenka's BlogHer reading list in BlogBridge, and it took under 60 seconds to open 129 blogs and read 1,525 posts. Waiting a minute doesn't sound excessive, and posts were visually loading the whole time, so it was clear that the program was still working. I don't think that experience would scare any user away.

I'm sure Jim, Amy and Dave have the user's best interests in mind, but saying people will be intimidated by more than a dozen feeds is rather patronizing. It's like telling students "now don't read too many books on a subject, or you'll get overwhelmed." That is a very individual preference. What is too many books to buy when you want to learn about all the new Web 2.0 technologies? I don't know, but I can count over 50 on my bookshelves. How many books should I buy to learn about philosophy? 100? 200? I'm sure I bought more than that while I was studying history of science. (You probably thought I was going to drop the H-bomb again, didn't you?)

Maybe a better solution would be to signal the user on what to expect before they open an OPML file. I support the tradition of listing the file size next to links for big files, like videos or .mp3s. It could be a good idea if links to reading lists were followed with something like: "(100 feeds)." Then the users could decide for themselves if 10 feeds was optimal.

Update: Amyloo clarifies her comments: it's not my aggregator that stumbles when it gets fed a list yielding 1,500 posts; it's my brain." Her aggregator may not choke when it sees that many feeds, but it clearly isn't giving her a presentation that augments her brain. I still think it is a problem that can be solved with software, perhaps a good feed grazing tool, when it is available. The goal should be to allow users to comprehend and manipulate more information, not to restrict the available information to our inherent limits. I couldn't write papers of any serious length during college, so I switched my major from English to Chemistry. After college there were word processors, and I started writing books.

Welcome to RSS Alley

There is so much activity based on RSS and OPML in the Boston area, that it seems appropriate to start calling it RSS Alley. I've even created an RSS Alley map. Along with locations for companies and bloggers, there are many points of historical interest, including the site where Dave Winer lived while working on RSS and blogs at Harvard. Please email me (adam AT darwinianweb DOT com) if you have any additonal locations for the map.

Update: The details on how this map was constructed are available on my mashup blog.

OPML Camp is coming to Boston

One idea that came out of last night's Geek Dinner was to do an OPML Camp in the Boston area. I'd like to wait to announce this until there is more (some) planning done, but what the hell. This is another one of those Web 2.0 practices, announce and then plan. I've registered opmlcamp.com, and am working on setting up a venue. There should be a date, wiki, and signup page ready in about a week. Stay tuned...

(Credit goes to Brian Del Vecchio for suggesting an OPML Camp)

Feed Grazing Workgroup

I've been using this term as a joke for the last week or so, but today it really seems to be coming true. Things have grown from the "isn't this cool" stage to what does it all mean, how can it work better, and what should it look like.

API programming is this week's priority

I let myself get a little distracted with reading lists and blogosphere politics over the last week, but now I have to get to some serious coding to prepare for Mashup Camp next week. That means blogging will be light over here. You can follow my progress on my mashup blog, and I'll post the source for anything I write in Ruby on my Ruby blog. My focus will be on using my Tech Memeorandum XML and OPML files as sources for calling various APIs.

Reading list podcast

I did a podcast today with Josh Porter, Alex Barnett, and Danny Ayers. It was a lot of fun, especially because of the great mixture of accents. We discussed RSS, OPML, reading lists and feed grazing. We also explored the idea of OPML as a transition to the Semantic Web. It got pretty geeky, but I guess that was to be expected. All of the details are on Alex's blog.

Danny talks quite a bit about the benefits of RDF over OPML, and if you want to know more about this format after listening to the podcast, you should get the book he wrote with Andrew Watt. It's the best explanation available of the history of RSS related formats, and goes into great detail on programming with these formats in several common languages.

Grazing ON feeds versus grazing IN feeds

I can now see where the distinction lies between my original idea of a river of feeds and the notion of feed grazing by James Corbett. The difference is the target of consumption. I saw dynamic reading lists as a way of finding new feeds. Instead of searching them out, I could let a dynamic reading list bring them to me like a river flowing past my house. When a feed looked interesting, I could read the most recent posts without having to subscribe. If I wanted to subscribe to it, I could just drag and drop it into my permament subscription list. My model was using a dynamic reading list as a sampling method.

James was more interested in using reading lists as a way of finding interesting posts. The feeds in the list were branches on the tree that allowed him to reach tasty new acorns. He was thinking of a reading list as a browsing method. Joshua Porter was making the same point as James when he wrote that he was more interested in the post than the actual feed.

I visualize this difference as a matter of topology. I wanted to find new containers (feeds) of information that I could continue to look in for a changing set of information. James and Josh wanted to look inside the containers to find interesting items (posts) without having to worry about the original location (feed) in which they were found.

Extending feed grazing beyond simple reading lists

This evening I decided to spend some more time on my Tech Memeorandum mashup. The original goal of this project was to use the list of people and blogs cited by TM as the starting point for experimenting with multiple APIs, such as Technorati and Del.icio.us. I was sidetracked for a while following the path of dynamic OPML reading lists, but now I was again ready to tackle the API issues. I had been stuck on the problem of creating a data structure that could hold the TM blog citations and all of the API results that were based on this list. Only this time I was approaching the problem with a better understanding of OPML, and an appreciation of James Corbett's thinking on this issue.

James coined the term feed grazing, and has been blogging for a while about the ways OPML could be used to tie together disparate information. I decided to reread his blog on this subject, and discovered that he had already solved my problem in a post he wrote today:

This was my solution. Instead of some complex data structure, I could create a simple collection of inter-related OPML files. The result of each API query could go into its own OPML file, and I could modify my current TM reading list to point to all of these lower branches on the tree. I knew that James had written a post based on the awful pun of OakPML that addressed this idea, so I went back and found this rather colorful, but highly useful metaphor: Once I had finished groaning, I could see that feed grazing was more than just a way of reading a list of feeds, it was a metaphor for traversing a potentially complex outline based on OPML. The squirrel doesn't understand the whole tree, he just knows how to find his next acorn. I still have to create a user interface for this collection of OPML files, but I'll leave that for the future. Hopefully, James will come up with the answer when I need it.

Dynamic reading lists just got easier to read

BlogBridge version 2.13 has just been released, making it a lot easier to view dynamic reading lists in OPML. It is free and supports Windows, Mac, and Linux. There are plenty of RSS aggregators that allow you to import OPML files as a quick way of subscribing to a large number of feeds, but these are basically a static form of subscription. BlogBridge, on the other hand, is able to stay in synch with the original OPML. If you subscribe to an OPML file on a server, and the contents of the file changes, then the set of feeds that show up in BlogBridge also changes. This is what separates a reading list from a regular OPML file. The internal format isn't different in a reading list, it is the fact that the contents of the file changes over time that makes an OPML file into a dynamic reading list.

It isn't completely obvious how to open an OPML file as a reading list in BlogBridge, as opposed to just subscribing to all of the file's feeds, so here are the basic steps:

1. Select 'Add Guide' from the Guides menu.
2. Enter a title for the new Guide.
3. Select the 'Reading List' tab.
4. Click the '+' button.
5. Enter the URL of a reading list. If you can't find one to try, you can get started with the one I have created based on Tech Memorandum.
6. Click 'Check and Add'.
7. Click 'Add'.
8. When the new Guide appears, all of the feeds listed in the reading list will be read, and the feed items will then appear.

By default, BlogBridge only checks for new contents in the reading list when it is first run. This is fine if you tend to start the program, read some feeds, and then close it. If you keep the program open, as I do, you will probably want to tell it to recheck the contents of the OPML regularly and resynch to match. This is done by:

1. Selecting 'Preferences' from the Tools menu.
2. Clicking the 'Reading Lists' tab.
3. Changing the 'Check for changed Reading Lists' setting to 'Once per Day' or 'Once per Hour.'

My enthusiasm on this subject has prompted some emails asking if I have a financial interest in promoting BlogBridge and reading lists, and the answer is no, although I am friends with Pito Salas, BlogBridge's project leader. I am actively looking for other aggregators that support OPML reading lists, so if you know of one, let me know about it and I'll be glad to write it up. I've said before that I believe RSS is a key component of the Web's future growth, and OPML reading lists are a great way of delivering RSS.

Reading list icon

There is already an OPML icon that can be used for links to reading lists, but just as with the RSS icon, I'd prefer something with a little more meaning and a lot less geek. The Blogbridge.com site uses this icon, and Pito Salas has kindly agreed to put it into the public domain. One confusion is that there are static reading lists, which are just a list of RSS feeds that doesn't change, and also dynamic reading lists, which are generally the result of automatic scripts. We eventually need icons that differentiate the two. For now I'll use this icon solely for dynamic reading lists, since those are the type I'm most interested in. One example is Danny Ayers' Del.icio.us reading list. As I find lists like this on the Web, I'll build a link list on my navbar.

Danny Ayers builds a reading list about reading lists

Danny has created an automatically updating reading list based on the Del.icio.us tags "readinglist+tech". I wonder if any of the reading list posts in this reading list reference reading lists? If automatic inclusion gets turned on for these lists dangerous things could happen. Of course, he has to make the obligatory "RSS sucks but I'll use it anyway" comment. Yes, RSS sucks and OPML sucks even more, but reading lists are cool. I'm glad Danny doesn't let the suckage of the spec get in the way of the coolness of the app.

Meanwhile, James Corbett has thrown out a little snark bait by saying that feed grazing is really Web 3.0. Come on now, Jimmy boy, that's just asking for trouble.

Giving James Corbett proper credit

I have a feeling that 'feed grazing' is going to catch on, both the term and the underlying idea of reading feeds based on continually changing OPML files. Since the origin of common phrases is often a source of controversy, I wanted to nail this one down before everyone is using it. Danny Ayers' reported it as appearing in a comment on his blog from James Corbett on February 6th, which is true, but the better and earlier source is a post James made on his own EirePreneur blog on January 31st. I also discovered that James coined the term "river of feeds" on his blog two days before I used it on my mashup blog. Damn. I was feeling so good about coming up with that. Anyway, it is good to set the record straight. Now everyone will know where to look when the Wikipedia article is written.

As long as I'm pointing to James' blog, let me link to one more great post in which he compares Steve Gillmor to James Joyce.

I've got a fever, and the only prescription is more feeds!

I've been accused of being obsessive, but I can't help it. I gotta have more feeds. This whole subject of real-time feed aggregation, or feed grazing as it's now being called, has really caught my imagination. So I've been looking for something that will cure my fever. I haven't found an aggregator that will satisfy my craving completely, but there are a number of websites that demonstrate the type of interface I need. I'll list them in the hopes that someone will build an OPML capable aggregator with this type of presentation:

• AliveNews has a cool realtime display, but it must be a proof of concept rather than a real site, because it doesn't have any options for expanding the list of pre-defined feeds. Still, the fade-in of feed excerpts is sweet.
• Digg spy is really compelling for the ADD set, but it isn't a true feed aggregator, and it makes me twitch if I watch it for too long.
• LiveMarks by Alex Bosworth applies a different take on this type of presentation to Delicious bookmarks. (via ProgrammableWeb)

Grazing, Feed Jockey, Fire Hose: Is a new RSS paradigm forming?

Anne Zelenka has once again managed to consolidate a disparate group of phenomena into a coherent pattern. This time it is the idea of automated OPML reading lists like my little Memeorandum experiment. What I like is the new terms she notes that are being used to describe this practice, such as Danny Ayers' grazing, Pito Salas' feed jockey, Amy Bellinger's churning urns of burning funk and Dave Winer's virtual publications. All of this new slang indicates a new paradigm is forming around automated lists of RSS feeds.

Update: 'Grazing' is from a comment James Corbett placed on Danny Ayers' blog: "I'm actually coming to the conclusion that the whole subscriptions mindset is a problem and that in future we'll 'graze' for the most part instead of subscribing."

The ultimate fire hose

I've been having a lot of fun today watching random feeds pass through my Tech Memeorandum reading list. Wait a minute, that is kind of sad isn't it? Anyway, this type of automated aggregation from a meme hunter is the ultimate fire hose. There is something almost illicit about having all these feeds pass by. It is qualitatively different from choosing to subscribe to feeds. I never know what I'll find. Shelley Power's comment feed just passed by for some reason.

I already have over 150 feeds that I'm subscribed to, and I'd pretty much stopped adding to them, because I feel guilty when I have to flush a whole bunch of unread posts away. Using an automated reading list and an aggregator like BlogBridge allows me to read feeds I've never seen before with no guilt. I don't care if they disappear on the next hourly refresh, because new ones will take their place. The Greek philosopher Heraclitus is famous for saying that you can never step into the same river twice. I think Heraclitus would have liked this new model of a river of feeds. See, I knew my History of Science education would be useful someday.

I finally grok OPML reading lists

The best way for me to understand a new software technology is to start writing code that supports it. I finally did that with OPML reading lists, and as Marc Canter would say, it is coolio! You can find all the details on my mashup blog. The short version of the story is that an OPML file based on all of the blogs cited on Tech Memeorandum is generated every hour and placed here where you can grab it and use it as a reading list. I poke fun at Dave Winer from time to time, but I can see that OPML reading lists really do take RSS to the next level. Good work Dave, and this time I'm around to see that your role doesn't get erased from history.

OPML is popping up everywhere

Now that I've started looking at OPML, I'm discovering a lot more activity all around me. When talking to Pito this morning about the next Geek Dinner, he pointed out that he's just added the ability to publish OPML reading lists to his BlogBridge aggregator. I've been using BlogBridge as my RSS aggregator for a couple of weeks, and its great for managing lots of feeds. I'll have to try out his new OPML features and report back. By the time we get to February 15th, we may have to call it an OPML Dinner.

Time to start working with OPML

Anyone reading Scripting News can see that Dave has been working overtime lately to generate momentum behind reading lists and OPML. I've been holding back until the conventions solidify around something closer to a standard, but Anne Zelenka's description of her OPML project makes me realize that the train is leaving the station. Time to get aboard everyone.

The world already is an outline, man

To be honest, Dave didn't actually mention OPML in his list of future goals. He dreamed about a "world outline." This is something he has written about often. It is a much grander vision than OPML and will require greater changes by users. The key is the concept of "including" an outline at any point from a different server. This would be an alternative, or at least a structural framework for the World Wide Web. I have to experiment with the existing tools, and try to build some of my own to know more, but the World Outline does have two things going for it: it is viral by nature, and it leaves a skeletonized framework behind of people's actions. It is as much a virtual building material as HTML and hyperlinks. If you give people hammers they see everything as nails. Dave keeps trying to give people outline processors.

It always comes back to outlines

The last time I wrote publicly about a product by Dave Winer it was also used for outlining. That was in the mid-1980s and the product was called Ready. It was a really sweet outliner that was able to run memory resident. This was before Windows, so being able to run in memory along with other apps was a cool thing. Anyway, Dave and I both thought Ready had a chance to be a breakthrough product. I decided to help out and wrote the user's manual that Dave's company, Living Videotext, shipped with Ready. I later wrote a book about Framework, a great integrated product from Ashton-Tate that was built around outlines. Neither product was a success. Dave did make lots of money not long after, but it was with a later product called More, which was about presentation more than outlines.

So I approach the study of OPML with some trepidation. I really like outlines, and we know how Dave feels about outlines. But how much of the public cares about outlines? My guess, less than 20%, maybe 15%. Luckily for OPML that doesn't matter. I'll be shocked if vast numbers of users decide to "outline" as a verb, but they will surely consume masses of data in an "outline" as a noun.

The problem with XML is that it is so flexible it will allow an infinite number of structures. We just need a few. We already have RSS, which describes a list of data items. These items can be blog posts in WordPress or news items on CNN and Google. Because OPML already has the majority of outline fanatics on the Internet involved, and because Dave REALLY wants the XML outline standard to be derived from his work, I can safely say that OPML will be a major outline standard in the future.

But OPML won't be the only outline standard. In fact, there is a guaranteed product position as the outline markup standard that isn't OPML. Not because OPML isn't good enough (I have no idea yet), but there will always be room for at least one well respected alternative to any standard.

Evaluating Dave's next goals

So I asked Dave Winer what he wanted to do next, realizing that I would most likely have to adopt the same thing. That sounds absurd, but for good or bad it is largely true. Dave has adopted a strategy of setting clear goals and then coding, nagging, insulting, and herding everyone else into just doing it the way he wants. In the end people accept his standard or some extended version of it, because his is one that has enough backers to become defacto. That is his strategy: nag until defacto, move on. He knows it works, and increasingly others know it works. I was talking to a big Web 2.0 guy at the SSA conference last week, and he convinced me that OPML was worth using, if only because it got everyone one step closer. Dave has now listed his future obsessions. I'm going to think about them and reply over the coming week.

Dave Winer has the best week ever

Ever since I became aware of the Web 2.0 meme I've been telling people that Dave Winer was one of the pivotal forces behind this new wave, maybe the central force. Everyone would have to admit that with GoogleBase turning out to be the world's biggest RSS database, and Ray Ozzie announcing Microsoft's synchronization and replication protocol based on RSS, Dave Winer is having the best week ever! Ozzie's announcement letter can only be described as effusive in his praise of Winer's role:

Microsoft and Google are being maneuvered into a massive game of chicken. I'll show everyone my Office data if you'll show your search data, and Dave is instigating it. My question is what comes next Dave? What are you working on for the wave after this, because I think this one is going to be pretty condensed.

Lest anyone reading this get the wrong idea, I should also make it clear that Dave and I haven't spoken in a couple of years and I'm hardly a sycophant, but that doesn't diminish my estimate of his influence on where the computer industry is headed. For right or wrong, we're riding the RSS train now.

Architecture for tags

I've been thinking about adding tags to this site, which stimulated some thinking about site architecture. I wrote my own blogging code to manage this site, so I can have maximum flexibility in areas like this. I've decided to walk the walk by building out this site with Web 2.0 architectures. That means I'm going to create my own API that returns XML as either RSS or OPML, and then have other parts of the site deliver page content based on this API. I'll then use that functionality to build a tag viewing interface similar to Delicious for my own posts here.

It sounds like overkill, but look at it this way. The content of this site is in a MySQL database on a server, which may not always be on the same physical machine as the site's Apache web server. As long as I have to adopt a client-server architecture, I can just as easily go around the outside through API calls over HTTP. It may be slower than making database calls directly to MySQL, but it will be a relative issue. If the performance slows down, I can just speed up the hardware or get someone to optimize the code . It is a totally scalable architecture. Of course, I won't try and deliver the entire site this way. The vast majority of the content is generated as a static html file. Just the controlling bits, and results of searches have to pass through the API/XML processing.

I'll write about the coding details on the Ruby site and post here when I have something you can try out.