XODP Blog

The Ongoing Viability of Wikia and Wikimedia

Internet Esquire — Wed, 26 Mar 2008 18:19:00 +0000

For reasons that defy rational explanation, the powers that be at Wikia dispute the claim that Wikia is the for-profit counterpart of the non-profit Wikimedia Foundation. No doubt important distinctions can be made between these two business entities, but the one that stands out to most people is that the for-profit Wikia's primary source of funding is venture capital and banner advertising, whereas the non-profit Wikimedia relies entirely upon charitable donations, and that distinction does not change the fact that both entities sprang from common roots, much like modern primates and humans evolved from a common ancestor. In sum, Wikia and Wikimedia are both free software/free content success stories that involve many of the same characters and that are still being written.

In a blog entry that I posted over a year ago, I asserted that rumors of Wikipedia's imminent demise were greatly exaggerated, Wikipedia being the original free content/free software project from which the Wikimedia Foundation was born. Indeed, even if both Wikia and Wikimedia were to go out of business tomorrow for some reason, the software and content that they've generated up until now would persist and remain viable for the foreseeable future in some other form. Only a far-reaching, cataclysmic event could change that because both the data and software for sites like Wikipedia are free to anyone who wants them, and a truly unique Web 2.0 informational resource like Wikipedia puts Google and all other search engines like Google to shame in certain ways. Most notably, Wikipedia does a much better job than Google does when it comes to disambiguation of keywords.

The key to Wikipedia's success is the more or less selfless contributors who can and often do come and go at will and may number from the hundreds to the hundreds of thousands, depending on whose being counted and whose doing the counting. And while Wikia's various communities are diminutive by comparison to Wikipedia, they operate on the same principles of openness and freedom, and they are just as capable of generating quality content on a wide variety of topics. And while both Wikia and Wikimedia sustain equally viable and "free" communities, they are very different business entities when it comes to their core philosophies in re commercial advertising. To wit, Wikimedia goes out of its way to shelter its users from commercial advertising, whereas Wikia supports its 5,500 or so "free" online communities with above the fold banner advertisements. Not that there's anything wrong with that. As the old GNU saying goes, " [T]hink of free as in free speech not as in free beer."

As of this writing, Wikipedia dominates Google's search results and Wikia's Alexa Ranking is a respectable 339 and rising, with both funding and actual revenue continuing to increase, thereby giving one every reason to believe that the thousands of online communities associated with Wikia and Wikimedia will continue to grow and thrive. At the same time, I would be remiss if I did not comment on the inadequacies of Wikia Search, a Wikia product that went live in January of 2008. Don't get me wrong: I think it's a great idea to have human editors rank and rate search results. That's why I started compiling the XODP Search Results Guides, to work in conjunction with the brute force of existing search engine algorithms, providing added value after said algorithms had done virtually all of the heavy lifting. That having been said, there is a general consensus that Wikia Search truly sucks, and I'm quite mystified as to what Jimbo Wales and his cohorts are trying to accomplish.

Like the late great Open Directory Project (aka ODP, aka dMOZ), Wikia Search hopes to employ an army of volunteers to do . . . well, after reading through the Wikia Search Mailing List Archives, that's not exactly clear. There doesn't seem to be any sort of workable theme behind Wikia Search, and the idea of "trusted user feedback" doesn't seem to have any context or relevance to a wiki-based search engine. What wikis do quite well is allow an exceptionally large group of users to collaborate on content generation, but the only things that this seems to bring to the search engine technology table are: (1) disambiguation of keyword-based search queries; (2) trusted sources of URLs; and (3) the possibility of trustworthy URL meta data. (In theory, the late great ODP was supposed to provide some trustworthy meta data, but ODP is now a historical object lesson in large scale and recalcitrant denial of quality control failure.)

The true believers of Wiki Search counter their critics by pointing to the unlikely success of Wikipedia and the aforementioned inaccurate rumors of Wikipedia's imminent demise. I think a better comparison would be the recalcitrant denial of Larry Sanger and company in re the failure of the Nupedia concept by creating Citizendium, a failure that has already been answered by the unqualified success of Wikipedia. To wit, Wikia Search is a vaporware solution in search of a problem that already has an adequate solution. This sad current state of affairs in re Wikia Search doesn't detract from the accomplishments of Wikia and Wikimedia, which by all accounts should remain viable entities for the foreseeable future.

This Just In: ODP Still Sucks

Internet Esquire — Thu, 29 Nov 2007 18:18:00 +0000

While Googling my byline a month or so ago to keep current on any recent gossip about me, I stumbled upon a thread at Digital Point discussing the merits (or lack thereof) of the Dmoz Sucks website. One "Rezo," who is apparently (one of?) the proprietor(s) of the SevenSeek web directory, pointed out Dmoz (aka ODP) sucks (ambiguity intentional) and recommended browsing the eponymous Dmoz Sucks website; one "nebuchadrezzar" followed up with a glowing reference of both me and XODP, closing with, "You guys should make yourselves known to [David Prenatt], he would probably give you a few pointers."

Nebuchadrezzar also pointed out the fact that Jimbo Wales frequented XODP back in the day and opined that XODP may have played a role in helping Jimbo avoid many of the pitfalls of ODP in creating Wikipedia. I'd like to think that were true, but (knowing Jimbo) I doubt that he would acknowledge that XODP had any kind of profound impact on him or on Wikipedia. Even so, Wikipedia is precisely the sort of open content community that I had hoped to inspire and/or create when I founded XODP, and while I have my reservations about Wikipedia, they are minor ones, and I make a point of extolling the virtues of Wikipedia from time to time on XODP and elsewhere.

I first came across Jimbo Wales during my tenure as Chief Evangelist at the-late-and-never-that-great Wherewithal. Much to my surprise, Jimbo had a distant history with one of Wherewithal's founders, and there was some talk of licensing Wherewithal's ad serving technology to Jimbo's company Bomis. At the time, my stock as an Internet celebrity was very high because of XODP, which is how I and Wherewithal first came to Jimbo's attention. At the time, I was also consulting with Project Napa as their Chief Ontologist and Community Architect, and I also tried to sell them on the idea of serving up ads a la Wherewithal as a way of generating revenue for their open content people portal. However, both of these promising business leads (and many others) were burned by Wherewithal's founders who simply did not have the wherewithal (pun intended) to put a profitable business deal together. Consequently, I stopped bringing the leads to Wherewithal and started operating in stealth mode, quietly playing matchmaker with my various business contacts while Wherewithal faded into obscurity.

The dot-com bubble had already begun to burst when I signed on at Wherewithal and Project Napa, but I felt (and still feel) that there was (and still is) quite a bit of promise in the area of open content generation and indexing. Even now I think that the ideas underlying Wherewithal and Project Napa were sound, and I've long toyed with the idea of reincarnating both of them in some form. Instead, I have focused on the needs of my paying clients; occasionally, I have worked on various XODP Web Guides and contributed to Wikipedia to satisfy my innate need to index online content.

In addition to satisfying my innate need to index online content, the XODP Web Guides have allowed me to demonstrate "proof of concept." And but for the fact that I am making a decent living as an Internet consultant for attorneys, work that I truly enjoy, I would embrace the opportunity to make more money by publishing XODP Web Guides while also improving the quality of online content indexing. Indeed, there may yet come a day when an investor throws so much money at me that I will take the necessary time and effort to hire and train enough staff to publish and maintain thousands (or even tens of thousands) of XODP Web Guides instead of the three hundred or so I currently publish. Of course, if making money were my primary motivation for indexing online content, I would probably start up a Web Directory like the aforementioned Seven Seek and charge for website submissions.

While I don't know how Seven Seek is doing financially, charging for website submissions is a proven business model that has worked on a large scale for Yahoo! and LookSmart and on a smaller scale for GoGuides and Best of the Web. Getting back to the subject implied by the title of this post, if there is a failure inherent in the ODP business model, it is the fact that ODP has no business model. Rather, when Netscape acquired ODP for the sum of one million dollars back in 1999, it purchased the good will of the then nascent open content indexing community.

When Netscape acquired ODP, it almost seemed to make sense, as Netscape had (a largely unearned) reputation for being a good 'Netizen. However, when AOL acquired Netscape and gave away ODP's content (albeit with bizarre and probably unenforceable licensing restrictions), it made sense to do so in a different way: Business sense. To wit, because AOL exercised editorial control over ODP content, it was able to offer quid pro quo to professional content providers like AOL and Rolling Stone. It also took the profit out of LookSmart's bottom line as a content provider, thereby appearing to eliminate a serious AOL competitor from growing and prospering. However, when LookSmart turned the tables on AOL and offered to share advertising revenue with its publishers, ODP became totally irrelevant and has been ever since.

My original purpose in founding XODP was to point out the flaws in ODP and foster discussion about the future of the Open Content Movement. Since that time, Wikipedia has deposed ODP and rightfully claimed the mantle of that movement with a viable business model that relies upon charitable donations. However, rather than letting ODP die with dignity, AOL keeps ODP on public display in its critical care unit, a sad state of affairs that will probably last at least until the next ODP system crash occurs. In a perfect world, the powers that be at Wikipedia would acquire ODP and make it a truly open directory, but I wouldn't hold my breath waiting for that to happen. As for the proprietors of the aforementioned Dmoz Sucks website, I wish them well, and they are more than welcome to join XODP and rant. However, I really can't be bothered to review their website any deeper than their Home Page and offer a substantive opinion on it: The fact that ODP sucks ain't news, and it hasn't been for quite some time.

Upgrading to Web 3.0

Internet Esquire — Mon, 19 Mar 2007 08:32:00 +0000

The term "Web 2.0" was coined sometime in 2004 by Dale Dougherty of O'Reilly Media. Far from a new technical standard, Web 2.0 began as a marketing gimmick for the eponymous Web 2.0 Summit, an annual high tech trade show that will be in its fourth year as of October 2007. As Tim O'Reilly wrote in September of 2005:

"The bursting of the dot-com bubble in the fall of 2001 marked a turning point for the web. . . .

" . . . Far from having 'crashed,' the web was more important than ever, with exciting new applications and sites popping up with surprising regularity. What's more, the companies that had survived the collapse seemed to have some things in common. . . ."

Far from a universal or even ubiquitous technical meme, Web 2.0 has nonetheless come to mean many things to many people in the high tech industry while meaning absolutely nothing to others. But this hasn't stopped many people in the know from calling for a Web 3.0 upgrade. The first person worthy of note to call for such an upgrade was Jeffrey Zeldman, who did so tongue-in-cheek as part of a rant against the Web 2.0 meme:

"When I started designing websites, if the guy on the plane next to me asked what I did, I had to say something like 'digital marketing' if I wanted to avoid the uncomprehending stare.

"A few years later, if I told the passenger beside me I was a web designer, he or she would regard me with a reverence typically reserved for Stanley-Cup-winning Nobel Laureate rock stars.

"Then the bubble burst, and the same answer to the same question provoked looks of pity and barely concealed disgust. . . .

"[ . . . ]

"Eventually . . . [t]he web was "back" even though it had never left. . . .

"[ . . . ]

"But how to persuade the other sharks in the tank that this blood feast was different from the previous boom-and-bust? Easy: Dismiss everything that came before as 'Web 1.0.'"

As a technical meme, Web 3.0 assumes the validity of a still emerging Web 2.0 meme, and then asks, "Who and what will be left standing when the next economic bust and boom cycle has finished ravaging the Internet?" I'd like to think that sexy applications like natural language processing, artificial intelligence, and the Semantic Web will rise from the ashes. However, I've been through the cycle of disruptive technologies too many times to expect this sort of purposeful progress.

The Waning Relevance of Search Engines

Internet Esquire — Sat, 10 Mar 2007 23:10:00 +0000

When the Web was young -- i.e., before the advent of Yahoo! and Google -- USENET[sic] was the primary medium for fielding informational queries on the Internet, and USENET FAQs were the primary medium by which such queries were answered. Incidental to such FAQs were pointers to particularly useful URLs that people had discovered and vetted by following trails of informational bread crumbs. Most of these trails started on USENET. However, over time, more and more Home Pages on the World Wide Web became populated with links to interesting and useful websites, and then along came the spiders.

Spiders (now more commonly known as web crawlers) created the first indexes[sic] of the World Wide Web by following links from one URL to another and making a record of what they found along the way. An individual spider or collection of spiders working together can produce a somewhat comprehensive database of URLs on the World Wide Web. However, that database is (at best) a collection of snapshots from some time in the recent past rather than a live feed that includes recent changes, and it will not include URLs found on "The Invisible Web." These distinctions were once lost on most people, who assumed that a "'Net search" would magically provide up to the minute information from the Web; this distinction is still lost on some people who have no idea how a search engine works.

The term search engine predates the Web. The earliest reference to the term that I've found was by Professor Lee A. Hollaar of Utah University back in March of 1985. (The Utah Text Search Engine: Implementation Experiences and Future Plans, Proceedings of the Fourth International Workshop on Database Machines.) But the term really caught on with the Internet community in the early 1990s. By that time, spiders were already cataloging an enormous amount of content found on the Visible Web. Even so, very few people otherwise in the know appreciated just how important search engines would soon become, erroneously assuming that anything more complex than the Unix "grep" query was overkill.

A former colleague of mine was a product manager for Netscape before the company was acquired by America Online, and he once recounted to me how popular the search function was on the Netscape website from the very beginning. Meanwhile, almost everyone at Netscape was trying to figure out how to sell the company's Web browsers and its easily forgotten line of related software products, all of which became completely irrelevant when Microsoft started giving away the Internet Explorer Web browser. Even Yahoo! failed to recognize the importance of search-related products, quickly diversifying from a searchable Web directory into a Web portal and outsourcing its search services to Google until a few years ago.

For all its flaws, the Google algorithm is still the standard by which all other search engine algorithms and post-Google information retrieval mechanisms are judged. The only post-Google innovation in content indexing and retrieval that even comes to close to being such a standard is Wikipedia, and most of those who consider Wikipedia a successful innovation do so because of Wikipedia's prominence and visibility in Google's search results. Even the blogosphere's importance is largely validated by the impact that it has on Google's search results. However, the importance of search engines reached its zenith a while back, and their relevance is slowly waning.

As I reported previously, Wikipedia recently announced that outbound links from Wikipedia to other websites would include a "nofollow" attribute. The rationale for this decision is that it will reduce the incentive for "black hat" search engine optimizers (SEOs) to spam Wikipedia, as a link from an important website like Wikipedia will normally inflate the PageRank that Google assigns to that link's destination URL. However, I think the end result will be something quite different. To wit, to the extent that Google actually ignores outbound links from Wikipedia, Wikipedia will actually offer something unique and different from Google to its end users. I don't think this will put Google out of business, but it will eventually diminish the relevance of search engines as the primary medium for fielding online informational queries.

In a vein similar to Wikipedia, sites like Digg, Fark, Technorati, reddit, and del.icio.us are emerging as places for the purveyors of information to hook up with their audiences with less and less concern for the impact that such sites have upon a(n) URL's Google PageRank. Moreover, like Wikipedia, certain online portals are so prominent that the only reason anyone uses a search engine to find them is because many end users are not in the habit of using bookmarks to pull them up -- e.g., eBay and MySpace. Even people who choose to use a search engine to find websites with the products, services, or information that they want sometimes complain that highly commercialized search results require a serious researcher to dig to the second or third page of search results to find something useful. In sum, while far from obsolete, the relevance of search engines is slowly waning, and commercial interests are slowly turning their attention to other channels of online marketing.

Jimbo Wales Feeds Wikipedians and the Press a Red Herring

Internet Esquire — Fri, 09 Mar 2007 22:29:00 +0000

According to Brian Bergstein of the Associated Press:

"Following revelations that a high-ranking member of Wikipedia's bureaucracy used his cloak of anonymity to lie about being a professor of religion, the free Internet encyclopedia plans to ask contributors who claim such credentials to identify themselves."

Discussions regarding this proposal are still underway at Wikipedia, but when news of the above-quoted article reached Jimbo Wales Wikipedia User Page space, it became quite clear to me that Jimbo had already made up his mind about moving forward with this plan, and that the discussions he had initiated on his user page were simply a way of placating the opposition by giving them an opportunity to speak their mind.

As someone who has repeatedly asserted Jimbo's lack of culpability in the recent Essjay scandal, I am very disappointed to see that Jimbo has offered up the red herring of verified credentials rather than address his [Jimbo's] failure to properly vet a pseudonymous individual whom he appointed to ArbCom, Wikipedia's court of final appeal. Superficially, the Essjay case was about falsified credentials; at its core, it was about an elaborate deception that Essjay rationalized as being necessary because he held positions of trust at Wikipedia.

Verifying credentials will not address the core issue of deception by Wikipedia administrators. All it will do is validate a Larry Sanger-esque type of credentialism at Wikipedia and create an attractive nuisance for credentialists and impostors. Essjay outed himself as an impostor because he felt that he had to so when accepting his paid position at Wikia. As such, the best remedy for the sort of deception that Essjay perpetrated would be a policy that forced people seeking administrator privileges at Wikipedia to provide the powers that be at Wikimedia with their true identity; if Wikipedia administrators wish to remain pseudonymous to the rest of the world, that's their prerogative.

The Digg Effect

Internet Esquire — Fri, 09 Mar 2007 17:20:00 +0000

The Digg website was launched just over two years ago on December 5th, 2004, and now boasts having some one million users. For those of you who have never heard of Digg, it's a community-based website where members can submit news stories for the community's consideration, vote for the stories that they consider newsworthy, and "bury" the stories that they don't like. In close conjunction with the advent of the blogosphere, Digg has emerged as an alternative news source where stories that would otherwise remain obscure become headline news.

While following the trail of bread crumbs that is the blogosphere, I started noticing that many of the stories I was following were becoming very prominent on Digg. I also noticed that many prominent bloggers attributed their blogging success to "being Dugg," and I immediately thought of how quickly a story could rise to prominence on the World Wide Web by being "Slashdotted." The two concepts share much in common, but the initial seed for a Slashdot story must be planted by the powers that be at Slashdot, and there is no way to formally vote on a Slashdot story to make it more popular.

After voting on a number of stories on Digg, I finally decided to post my own. As of this writing, my story has 16 Diggs and is ranked third in popularity out of some 260 "up and coming" stories under the category political opinion, and (based on my observations of the Digg system) it still has over 3 hours to become the most popular up and coming story. Now the question is whether the powers that be at Digg will decide to "make it popular." If they do, then my ISP will have to make sure that I do not become a victim of the Digg effect.

Putting Essjay's Fraud into Perspective

Internet Esquire — Wed, 07 Mar 2007 18:31:00 +0000

I have done my level best to avoid posting yet another entry on the XODP Blog about the Wikipedian impostor known as Essjay, limiting myself to commentary on other blogs and on Wikipedia itself. However, I was very impressed with a post on Miland Brown's World History Blog that seemed to put the larger issue of Wikipedia's reliability into perspective:

". . . [T]his kind of deception is not new and is not limited to Wikipedia. Does anyone remember Jayson Blair? He both plagiarized and fabricated articles at the New York Times for several years. I do not think Blair has proven that the New York Times is a bad resource though despite his fraud.

"And let's not forget about Stephen Glass at the New Republic. 27 of 41 stories written by Glass for the magazine contained fabricated material. He wrote such fake gems as a 15 year old at national hacker convention and a Church of George Herbert Walker Bush, Jesus Christ. I still like the New Republic as a resource despite the Glass incident.

"People fabricating degrees is not new either. The Chronicle of Higher Education frequently exposes people in higher education with diploma mill degrees. For example, history adjunct Fred Ruhlman at the University of Tennessee was reported at Cliopatria, '(His) 'doctorate' is from 'the American University of London,' a notorious diploma mill and his book, Captain Henry Wirz and Andersonville Prison, was withdrawn from publication by the University of Tennessee Press because of its plagiarism from William Marvel's Andersonville: The Last Depot.' While embarrassing for the University of Tennessee, it hardly means UT degrees are now worthless.

"I can find those fake Blair articles in my library right now in the microfilm versions of the New York Times. Those fake Glass articles from the New Republic are still in the bound periodical section of the library too. However, every edit Essjay has made is being examined and altered if it is found to be problematic. Unlike the mainstream press who have their mistakes archived forever in libraries, Wikipedia can be fixed when the fraud is discovered."

In a previous blog post, I qualified my hope that I would like to see Essjay make a fresh start at Wikipedia under a new anonymous pseudonym with the disclaimer that such a fresh start was not possible unless Essjay recants his assertion that he was offered compensation by Stacy Schiff of The New Yorker. However, I am inclined to reconsider that disclaimer in light of new facts brought to light by the Wikipedia Signpost:

"Based on earlier statements he made, it is possible that [Essjay] used calling cards to phone Schiff for the interview, and his claim may have referred to an offer to reimburse the cost of those calls. . . . Schiff's response [i.e., "That is nonsense."] did not address whether this alternate interpretation was correct."

As a (hopefully) final note, I encountered a comment on one blog that explained to me why some people seem to be judging Wikipedia's credibility so harshly in light of Essjay's fraud:

". . . [T]rusting Wikipedia is . . . a greater personal risk than trusting the New Yorker, regardless of whether the New Yorker is actually more trustworthy than Wikipedia. . . [T]he New Yorker’s puffery about its fact-checking effectively indemnifies [someone] against the risk of looking foolish, in a way that Wikipedia does not. . . .

"This sort of indemnification is precisely what professional journalists are defending when they rudely dismiss blogs–or Wikipedia–as inferior. Rather than make serious, careful comparisons between themselves and alternative information sources–comparisons that would inevitably show themselves to be seriously flawed in their own right–they emphasize the process by which they gather and vet their content."

Cooperative Research and Citizen Journalism

Internet Esquire — Mon, 05 Mar 2007 23:33:00 +0000

Viewer-supported Link TV, recently broadcast a program entitled 9/11 Press For Truth as part one of a two part special entitled Truth, Lies, and the Press. This program was based on the work of Internet researcher Paul Thompson, author of The Terror Timeline and the website that inspired the book The Complete 9/11 Timeline, which is now part of the Cooperative Research website. I was very impressed with the content of the Link TV special, and I hope to discuss that at some point on my Internet Esquire Blog; what I hope to cover here on the XODP Blog is the Cooperative Research website, also known as the Center for Cooperative Research.

The Center for Cooperative Research is a non-profit enterprise sponsored by The Global Center, a 501(c)3 non-profit corporation, and its content is openly licensed under the Creative Commons Attribution-NonCommercial-ShareAlike license. As open content licenses go, the Creative Commons is not one of my favorites. However, the larger mission of the Center for Cooperative Research is one that I consider worthwhile. To wit:

"The website is an experiment in open-content civic journalism. It allows people to investigate important issues by providing a space where people can collaborate on the documentation of past and current events, as well as the entities associated with those events. The website can be used to investigate topics at the local, regional, or global level. The data is displayed on the website in the form of dynamic timelines and entity profiles, and is exportable into XML so it can be shared with others for non-commercial purposes."

The site's growth potential is somewhat limited because it relies upon qualified professional volunteers to act as editors for the content submitted by member contributors. However, if the Complete 9/11 Timeline is any indication, what the site lacks in quantity of research, it more than makes up for in terms of quality. To wit, I was quite astonished to see just how many news items that should have been headlines were reported by the mainstream press but buried on page 11. Once these news items are put into a timeline format, they make a compelling case for concluding that Pakistan's ISI Intelligence Service was a sponsor of Al Qaeda and the 9/11 hijackers.

Closure Still Lacking in Essjay Scandal

Internet Esquire — Sun, 04 Mar 2007 07:40:00 +0000

The beleagured Essjay "retired" from Wikipedia today, and I was prepared to let sleeping dogs lie until I read a post at Andrew Lih's blog:

"Paying a source for a story is an absolute no-no in the normal practice of print journalism. And it struck me immediately how incredible it was [Essjay] would accuse Stacy Schiff, a Pulitzer Prize winning author writing for The New Yorker, of this crime. We either have a serious breach of ethics with Ms. Schiff or another dubious statement claim from Essjay (nee Ryan Jordan)."

Notwithstanding the hyperbole of comparing a breach of professional ethics with a crime, a survey of Google search results for ''paying a source' journalism' indicates that paying a source for a story is, indeed, an ethically murky area. As such, there can little doubt that Essjay was lying, once again, adding a truly bizarre spin to an already bizarre story.

In a previous blog post, I stated that I would like to see Essjay make a fresh start at Wikipedia under the protection of a new, anonymous pseudonym. However, that's not possible unless Essjay retracts this rather bizarre claim about Stacy Schiff. And given the fact that a substantial number of Wikipedians remain stuck in denial and steadfast in their support of Essjay, it is highly unlikely that he will ever offer a retraction or an apology.

Credentialists and Impostors

Internet Esquire — Sat, 03 Mar 2007 19:57:00 +0000

The recent intrigue over Wikipedian Essjay's phony credentials has inspired Larry Sanger to reconsider Citizendium's registration policy:

". . . . We are very concerned about the credibility of the Citizendium as a reference work. . . . We simply do not want to wake up in five years, to find that someone has done a study of the Citizendium and demonstrated that in fact 25% of all of our contributors are using neither their real names nor pre-approved pseudonyms. In short, we've reluctantly concluded that the honor principle, even coupled with a willingness instantly to ban people like Essjay who are exposed for using false personas, really isn't due diligence."

Larry goes on to narrate the various methods that Citizendium hopes to use to validate the identities of people who wish to contribute to the project. To the unitiated observer, strong security measures like this probably seem to make sense. However, my experience has been that such security measures serve as an attractive nuisance for both impostors and credentialists, and I'm not sure which is worse.

Conventional wisdom would have us believe that credentials are good and that impostors (in this context, someone who pretends to have credentials that he or she does not actually possess) are bad. However, the underlying issue is expertise, and credentials don't always translate to expertise. As such, when someone without expertise uses credentials to win a debate, this is arguably much worse than someone with expertise pretending to have credentials. The problem with both of these situations is that credentials, or lack thereof, become more important than expertise.

While I'm a big fan of expertise, I'm not a big fan of credentials, and I'm even less fond of credentialists, as my experience has been that someone who truly has expertise will seldom feel the need to fall back on his or her credentials and will generally feel contempt for those who make a habit of doing so. Indeed, those with true expertise are usually most conspicuous to me through their modesty and silence, and what concerns me most about Essjay's misguided actions on and off Wikipedia is that they have provided an opportunity for credentialists without expertise to stifle people without credentials who actually have expertise.

Prior to Jimbo Wales withdrawing his support for Essjay and asking him to resign from his positions of trust at Wikipedia, it was my sincere hope that Essjay would see the light and do this on his own. As it is now stands, Jimbo's statement has only partially resolved the situation. To wit, there are a number of Wikipedians who are urging Essjay's critics to back off, just as there are quite a few people who think that Essjay should still retain his privileges as a Wikipedia administrator. Personally, I would like to see Essjay make a fresh start at Wikipedia, and the only way that he can do that is if he resigns all privileges that he acquired under his false persona and creates a new account at Wikipedia under the protection of a new, anonymous pseudonym.

Correction: In the comments section of the above-cited Citizendium Blog post, Larry Sanger has pointed out my erroneous assertion that Citizendium's recent policy change was (1) inspired by the intrigue at Wikipedia and (2) made by him and him alone. I stand corrected. -- DFP,Jr.

Never Underestimate the Power of Denial

Internet Esquire — Thu, 01 Mar 2007 22:22:00 +0000

In following up on my previous XODP Blog post about the fraud perpetrated by the Wikipedian known pseudonymously as "Essjay," I came upon a post at Kelly Martin's Nonbovine Ruminations blog entitled, Larry Sanger proven right about Wikipedia. I followed a link to the post from Google Blogsearch expecting to find a cheap shot leveled against Wikipedia by a Langer Sanger-esque Citizendium-loving credentialist, but that's not what I found:

"Larry Sanger has long argued that Wikipedia is anti-intellectual. And, sadly, he has been proven right, by Jimbo's recent fiat appointment of Essjay to the English Wikipedia's Arbitration Committee. . . ."

While I agree with most of what Kelly has to say about Essjay, I don't agree with her characterization of Jimbo's misguided loyalty to Essjay as being proof of Wikipedia's alleged anti-intellectual bias. Rather, I see it as being motivated by recalcitrant denial on the part of Jimbo in re Essjay's recalcitrant denial of his [Essjay's] culpability in perpetrating such a fraud. On this note, an update at the end of Kelly's blog post points to a recent "apology" on Essjay's user talk page at enwiki:

"I would like to clear up an oversight on my part. I was, until this morning, under the impression that in my initial post on this subject (in response to a question from Dev920 made some weeks ago) I had made an apology for anyone who felt they were hurt by my decision to use misinformation. In speaking to various different people, including Jimbo, I did make it known that I was sorry that anyone felt hurt by my actions, and I believed I had done so in my initial statement. On re-reading that, I find I did not; it was a rather lengthy statement I had been thinking about for some time, and I seem to have left out a rather critical element of it. So, I rectify that now, with further apologies that it was not included originally, as I pointed people back to that statement in the belief it was complete.

"I *am* sorry if anyone in the Wikipedia community has been hurt by my decision to use disinformation to protect myself. I'm not sorry that I protected myself; I believed, and continue to believe, that I was right to protect myself, in light of the problems encountered on the internet in these trying times. I have spoken to all of my close friends here about this, and have heard resoundingly that they understand my position, and they support me. . . . I'm also sorry the New Yorker chose to print what they did about me; there seems to be a belief that I knew they were going to print it, and that is not the case. . . .

"For two years, I have poured my life into making this site a better place. That many people feel hurt by my decision pains me greatly, and to them I am genuinely sorry. To the stalkers, the trolls, and the vandals, I am not sorry; they are abusive, hateful people, and they have done far worse things than those whole of the Wikipeida Community, myself included, have ever thought about doing. . . . I have no intention of going anywhere, because to do so would be to let the vandals, trolls, and stalkers win."

I wouldn't go so far as to say that Essjay is paranoid and delusional, but this "apology" makes it quite clear that Essjay is unapologetic about perpetrating this fraud. He even goes so far as to blame two of his victims (i.e., Stacy Schiff and her fact checker from The New Yorker) for not being more skeptical about his credentials, credentials to which he frequently averred when attempting to boost his credibility as an expert in theology. Essjay's earlier statements on this matter left some room for reasonable doubt as to whether he was actually bragging about being able to perpetrate this fraud, but what he just doesn't get is that his earlier excuse that such a fraud was necessary, justifiable and/or excusable as a way of protecting himself from online stalkers just does not hold water.

Respected Wikipedian Lies to the Press

Internet Esquire — Thu, 01 Mar 2007 05:31:00 +0000

Stacy Schiff of The New Yorker wrote an article about Wikipedia that was first published online in July of 2006. One of Schiff's sources was a pseudonymous Wikipedian known as "Essjay" who lied to Schiff about virtually all of his biographical details:

" . . . [Essjay] was described in the piece as 'a tenured professor of religion at a private university' with 'a Ph.D. in theology and a degree in canon law.'

"Essjay was recommended to Ms. Schiff as a source by a member of Wikipedia's management team because of his respected position within the Wikipedia community. He was willing to describe his work as a Wikipedia administrator but would not identify himself other than by confirming the biographical details that appeared on his user page. . . . Essjay now says that his real name is Ryan Jordan, that he is twenty-four and holds no advanced degrees, and that he has never taught. He was recently hired by Wikia—a for-profit company affiliated with Wikipedia—as a 'community manager'; he continues to hold his Wikipedia positions. . . ."

While Daniel Brandt first broke this story quite some time ago, it's just now starting to get some traction in the blogosphere, and it took me quite a while to uncover the details. A good place to bring yourself up to speed is on Jimbo Wales Wikipedia User Page. However, the bottom line is that Essjay acknowledges no wrongdoing, claiming to be justified in perpetrating this fraud as a way of protecting himself from online stalkers.

While it's not a crime of moral turpitude, lying to a member of the press when the truth will do just as well is evidence of moral turpitude, not to mention just plain stupid when you lie about falsifiable facts, and Essjay's excuse of necessity just doesn't wash. To wit, in addition to lying to the press, Essjay used his counterfeit credentials to bolster his and Wikipedia's credibility in a letter that he gave to students who wanted to cite Wikipedia as a reference work. He didn't have to give the interview, he didn't need to write the letter, and he didn't need to lie to the press to protect himself from online stalkers.

I am no stranger to online stalkers. Shortly after I founded the XODP Yahoo! eGroup, a (now former) ODP meta editor set up Netesqsucks.com, a hate site dedicated to persecuting me. And while said meta editor set up said hate site under an assumed name and a fake address, an investigative reporter uncovered his true identity by tracking down the credit card that said stalker used to pay for the site's domain registration. Eventually, this meta editor (a private investigator) was terminated for using sock puppets to promote his own websites on ODP and exclude his competitors.

I didn't need to talk to the investigative reporter who uncovered the truth about the person behind the Netesqsucks.com website, but I did, and I took a huge risk in revealing to her in confidence all sorts of personal information about myself on deep background. Because Essjay was referred to The New Yorker by Wikipedia management, he was assumed to be trustworthy without providing that background information. Simply put, Essjay violated that trust.

Vanity. . . Definitely My Favorite Sin

Internet Esquire — Tue, 27 Feb 2007 22:46:00 +0000

While there's still no trackback on the Citizendium Blog for my previous XODP Blog post entitled Larry Sanger Gets It Totally Wrong, Larry has condescended to comment on my more recent post entitled Dealing with Jackasses at Wikipedia and Citizendium:

"What argument or evidence can you offer to support this silly claim that I wish to establish 'centralized content control'?"

That's a good question. But I have a much better question: If it's such a silly claim, why are you bothering to respond to it? Seriously.

Larry continues:

"I'm surprised that you have jumped on the bandwagon of those who say, 'Oh my gosh, if there are experts involved, it must be Nupedia all over again!'"

Hardly a bandwagon, although I would have to agree that many other people worthy of note believe that Citizendium qualifies as a recapitulation of Nupedia. However, I have no problem with having experts involved in wiki projects, and my position on Wikipedia's unfair bias against experts would have been quite apparent to you if you'd carefully read the post to which you responded.

Larry continues:

"The Citizendium has editors who can weigh in as "resident experts" as necessary. . . . To suggest . . . that they simply insist on their views without argument--is in essence to malign a whole bunch of people you don't know at all."

I can't tell you what a joy it is to have you put words in my mouth that I never said. To wit, I did not say that "Citizendium editors simply insist on their views without argument," and suggesting that this was my position is what logicians refer to as a fallacious "straw man" argument. For future reference, if I mean to say that Citizendium editors simply insist on their view without argument, I won't have to suggest it; I'll just say it.

Larry continues:

"[C]redentials are necessary for being an editor. Not for being an author, of course, and most of our registered contributors are authors. But, yes, you have to prove that you're actually an expert."

The need to have a distinction between expert editors and other contributors is somewhat lost on me, but you have answered your own question regarding my characterization of your position on centralized content control.

Larry continues:

"Frankly, we put out on their ear far faster than Wikipedia ever did anyone who actually acts like a jackass. Your own intemperate post, for instance, is the sort of mean-spirited, vicious personal attack that would get you excluded."

Hardly an intemperate post by me, but definitely an intemperate response on your part wherein you stop holding court just long enough to step outside and challenge me to to mutual combat.

Dealing with Jackasses at Wikipedia and Citizendium

Internet Esquire — Tue, 27 Feb 2007 20:54:00 +0000

Following a comment by one Mike on the XODP Blog eventually took me to said Mike's Modern Dragons Blog, which has a very lengthy post about Citizendium. And while Larry Sanger seems to be quite enamored with Mike's post, I think Mike is making extraordinary demands on his potential blogging audience by using over 5,000 words, not including the footnotes, appendices, and comments that follow the main post. Even so, as Mike's post seems to have drawn the attention of the usual suspects, I made a point of reading through the whole thing, and I was rewarded by the following quotable quote cited by Mike:

"Many experts who have left, or otherwise have expressed dissatisfaction with Wikipedia, fall into two categories: Those who have had repeated bad experiences dealing with jackassses, and are frustrated by Wikipedia’s inability to restrain said jackasses; and those who themselves are jackasses."

I have stated before that Wikipedia has an unfair bias against experts, just as I have stated before that project forking at Wikipedia is long overdue. However, as alluded to by the quote above, experts who have left Wikipedia and jackasses who have left Wikipedia are not mutually exclusive categories. Rather, the jackasses who make contributing to Wikipedia an unpleasant experience form a spectrum that runs from the most ignorant and uninformed contributors to the most educated and biased control freaks. I say this as someone who voluntarily limits the vast majority of his Wikipedia contributions to "Talk Pages" rather than getting into "Edit Wars" with said jackasses. Even then, I often find myself blown away by the recalcitrance of the jackasses for which Wikipedia seems to be an attractive nuisance.

On this note, from the comments section of Mike's blog post comes the following gem from Wikipedian Fred Bauder:

"Most Wikipedians support Citizendium, but, having had experience with Larry, most old-timers are somewhat sceptical[sic] about a project controlled by him. Given a choice, he will nearly always choose a solution which involves top-down control. . . . Another problem, he does not so much respect expertise, as credentials. . . ."

Having had direct contact with Fred Bauder during my early editing experiences at Wikipedia, I can honestly say that he was one of the jackasses at Wikipedia that inspired me to give up the proverbial ghost when it came to quality control. I was particularly annoyed at his penchant for moving forward with dramatic changes to an existing Wikipedia article without bothering to form a consensus. And in all fairness to Fred, there are probably just as many people at Wikipedia who think that I'm a jackass.

Ironically enough, the dispute that I had with Fred involved his revisions to the Wikipedia Law article, and Fred is a retired lawyer, which should give him more than a passing familiarity with the topic. And while I do not practice law, I was a Member of Law Review and a Teacher's Assistant for Legal Writing before graduating from UC Davis Law School. As such, from a Larry Sanger-esque credentials standpoint, I think that I too would have been on pretty firm ground. Even so, the most constructive input regarding the Wikipedia Law article that came during my confrontation with Fred came from Wikipedian Slrubenstein (a very modest professor of anthropology), mysterious old-time Wikipedian SJK, and noted Wikipedian Lee Daniel Crocker (a computer programmer by trade who was the primary author of the current MediaWiki software).

By the time I started contributing to Wikipedia in August of 2002, the writing was already on the wall regarding Larry Sanger's once prominent role as Wikipedia's editor in chief. And having successfully cultivated more than one online community of several hundred members (i.e., Wherewithal and Project Napa) and then having had both of those communities taken away from me and destroyed by the people who hired me to cultivate those communities, I was inclined to feel sorry for Larry. However, the more I came to understand Larry's feelings about what Wikipedia should be, the more I came to realize that Wikipedia was much better off without someone like Larry in charge.

While Fred Bauder may or may not speak for the majority of old timers at Wikipedia, (IMHO) he is spot on in his criticism of Larry Sanger. One need not look far for evidence of Larry's failings: Nupedia failed because it favored credentialism and centralized control over content, and once Wikipedia emerged as a viable replacement for Nupedia, Sanger sought to impose credentialism and centralized content control at Wikipedia. Even now, Larry hopes to use Citizendium to reassert the validity of credentialism and centralized content control as a panacea for what ails Wikipedia.

I mentioned above that during my tenure at UC Davis Law School I was a Teacher's Assistant for Legal Writing and a Member of Law Review. These positions gave me an enormous appreciation for just how difficult it is for most people to produce quality writing, just how defensive most people can be about really bad writing, and just how difficult it can be to reach a consensus on what good writing actually is when more than one editor's viewpoint must be appeased. As a result of these harsh realities, the vast majority of promising law review candidates would wash out of law review before the end of their first semester, and the author of an article that qualified him or her for law review membership was seldom willing to claim the final product as his or her own.

Dynamics similar to those that I experienced at UC Davis Law Review are at work at Wikipedia, but with a rather strange twist: Anybody can put in their own .02 in on a Wikipedia article, and quite a few people do. To wit, while most Wikipedia articles end up having a relatively small number of regular and interested contributors, the vast majority of content on Wikipedia is contributed by the rare occasional contributor. What this means is that the occasional contributor is the key ingredient to Wikipedia's exponential growth but is not even a significant part of Wikipedia ongoing quality control problems. Those problems are caused by biased and/or uninformed experts and zealots -- i.e., jackasses.

Rather than encourage the occasional contributor who has helped Wikipedia grow and prosper and discourage jackasses from asserting ownership over particular articles, Larry Sanger hopes to improve upon the Wikipedia model by excluding from Citizendium the people who provide the vast majority of useful content and forming an enclave of like-minded expert jackasses. It's an interesting experiment, but -- given the enormous barriers to entry at Citizendium -- not one that I or any of the experts that I know are interested in joining. And it's probably just as well: Larry Sanger is apparently moderating all comments and trackbacks to the Citizendium blog, as all the comments appearing there harmonize with his point of view and no trackback reference to my previous blog post has yet appeared.

Larry Sanger Gets It Totally Wrong

Internet Esquire — Mon, 26 Feb 2007 05:30:00 +0000

While exploring the blogosphere for commentary about Wikipedia, my attention was drawn to a post by Larry Sanger on the Citizendium Blog:

". . . Wikipedia's reach is now enormous, and if indeed it has gained a reputation, whether deserved or not, as a source of reasonably reliable information, and it defames someone for any significant length of time, such defamation can do very real harm to a person’s reputation. . . .

"[ . . .]

". . . Ethically, and probably legally, Wikipedia’s managers must face up to it, because the injustice the current situation perpetrates is completely and obviously intolerable.

While Wikipedia may or may not have an ethical duty to prevent people from being defamed *on* Wikipedia, an issue on which reasonable minds may differ, the only thing that I find "obvious" on the issue of defamation *by* Wikipedia is that Larry Sanger knows absolutely nothing about defamation law.

BTW, as I am wont to say when commenting on legal issues, I do not practice law, and nothing I have written in this post or elsewhere should be construed as a legal opinion or as legal advice. That having been said, according to Attorney Scott D. Sheftall, the lawyer who is representing Fuzzy Zoeller in the recent defamation suit involving Wikipedia, the law is pretty clear when it comes to the fact that Wikipedia has no legal exposure here:

"Zoeller's attorney, Scott D. Sheftall, said he filed the lawsuit against a Miami firm last week because the law won't allow him to sue St. Petersburg-based Wikipedia."

The irony here is that Sanger's ongoing indictment of Wikipedia is that it allegedly fails to provide accurate information. In Sanger's own words:

"Articles that contain defamatory remarks are not 'objectionable comments' on a 'message board'; they are presented to the world and typically accepted as fact, which is something Jimmy himself encourages by saying, as he does, that Wikipedia is actually pretty reliable."

This is a classic case of the pot calling the kettle black, and a pretty good case could be made that Sanger's comments about Jimbo Wales are defamatory:

"It is reprehensible, moreover, to react to this situation lackadaisically, regarding false claims that are personally and professionally damaging as in effect 'collateral damage' that society must bear as the price of having such a wonderful project as Wikipedia. That reaction is reprehensible because it assumes that Wikipedia's managers cannot improve the way that it deals with this sort of problem. They could, without completely breaking its system; but they choose not to."

According to the guidelines offered by the Electronic Frontier Foundation in re defamation, an attorney could probably make a prima facie case that Sanger is publishing untrue statements about Wales with the intention of impuning Wales' reputation. However, given that Wales would probably qualify as a public figure, said attorney would also have to prove "actual malice" on the part of Sanger. Nonetheless, I think it's incumbent upon Sanger to live up to the standards that he hopes to enforce:

"[Citizendium] will have a zero tolerance policy toward any even possibly defamatory remarks: to say something that might tend to impugn someone’s reputation, even if true, will require extremely good documentation. If no such documentation is offered, or if it does not check out, the person who makes such claims will be "escorted to the door." We simply won't tolerate it."

Powerset Redux

Internet Esquire — Mon, 26 Feb 2007 03:39:00 +0000

In a previous XODP Blog post, I commented on the fact that Powerset had succeeded where Google had failed in wooing natural language processing (NLP) guru Ronald Kaplan to spearhead its efforts at developing a semantic search engine. Unlike most of the commentators that I've read on the blogosphere, I was willing to suspend judgment on Powerset's technology until it came out of stealth mode. However, I remained titillated by the prospect of semantic search finally coming of age, so I did a little more digging, and I found some more information about what Powerset is planning to debut:

"There are two key things here: the use of NLP and the disruption to the search interface. Finally, information retrieval will actually mean information retrieval, not document retrieval. One of the fundamental models of search that may be challenged . . . is . . . that search engines are designed to take people to pages. The more we can understand and summarize the information on those pages, the weaker this model becomes. . . ."

[The above summary comes from the blog of Matthew Hurst, a close friend of Powerset's CEO Barney Pell.]

As promising as this vision of search is, it fails to take into account the cost of search computations:

"The computation for each query has quantifiable costs and benefits. The costs include R&D, computing infrastructure amortization, energy, rent, maintenance. The direct benefit is advertising revenue from the query response; indirect benefits such as market share gained from greater search quality are harder to measure, but can still be estimated. Search engine success is ultimately given by the efficiency with which it delivers advertising revenue given these factors."

This sort of cost/benefit analysis gives rise to an inherent conflict of interest in search engine technology as we know it. To wit, if a search engine's natural search results are too good, there's no incentive to click on paid search results, and the business model breaks down. At the other extreme is an information resource like Wikipedia, with a business model that is based primarily upon a gifting economy. That sort of business model also breaks down when the quality of information it provides gets too good, as it attracts all sorts of people whose primary interest is in getting rich off of those who work for free.

To quote the late Keith Moon, "Sometimes you do alright to steer clear of quality." (A quote that I found misattributed to Pete Townshend.) Nowhere is this more true than in it is when it comes to technology, where a successful disruptive technology (i.e,. blogs) usually starts out as an inferior technology. Assuming, arguendo, that Powerset's technology truly is better than Google's, this is the real challenge that Powerset's evangelists will have to address.

Rumors of Wikipedia's Imminent Demise Are Greatly Exagerrated

Internet Esquire — Sat, 24 Feb 2007 21:48:00 +0000

Following up on some recent XODP Blog posts about Wikipedia, I noticed that it has become somewhat fashionable to predict Wikipedia's imminent demise, the most recent example of this that I found being Peter Da Vanzo's post at the v7n blog, Have DMOZ Taken Over Wikipedia?

"DMOZ and Wikipedia share much in common - other than Wikipedia isn't completely useless. Yet.

"[ . . . ]

. . . You build something that looks open, and appears to be open, but in reality, is locked up tight, and run by a small group of people making ever more insular decisions.

". . . [T]hey're under-resourced for the task, and as the task grows, the more under-resourced they become. In response, they compromise the very thing that made them valuable - accessibility."

I've had recurring issues with Wikipedia since I first started contributing to it back in August of 2002, and I think that some project forking is long overdue. However, I think that Da Vanzo's indictment of Wikipedia is gratuitiously harsh. Ditto for the indictment of Wikipedia proffered by Nicholas Carr back in May of 2006:

"Wikipedia, the encyclopedia that "anyone can edit," was a nice experiment in the "democratization" of publishing, but it didn't quite work out. Wikipedia is dead. It died the way the pure products of idealism always do, slowly and quietly and largely in secret, through the corrosive process of compromise.

". . . A few months ago, in the wake of controversies about the quality and reliability of the free encyclopedia's content, the Wikipedian powers-that-be - . . . tightened the restrictions on editing. In addition to banning some contributors from the site, the administrators adopted an "official policy" of what they called, in good Orwellian fashion, "semi-protection" to prevent "vandals" (also known as people) from messing with their open encyclopedia."

A much more muted prediction of Wikipedia's demise was offered by Eric Goldman on his Technology and Marketing Law Blog entitled Wikipedia Will Fail in Four Years:

". . . I'm . . . basing this prediction on the experiences of ODP. I think it's fair to say that (1) in its heyday, the ODP did an amazing job of aggregating free labor to produce a valuable database, and (2) the ODP is now effectively worthless."

While I agree that Wikipedia has all sorts of limitations and faces all sorts of challenges, I also think that predicting Wikipedia's demise based on comparisons to ODP/dMOZ is very simplistic thinking.

The single biggest problem with the purportedly Open Directory Project was that it was never, ever truly open. A close second was the fact that ODP never had a business plan. Neither of these things can be said about Wikipedia. Rather, the single biggest problem with Wikipedia is quality control, and a close second is the bizarre bureaucracy that has been slowy emerging in response to the problem of quality control. The fallback solution to both of these problems and most of the other problems that open projects encounter is project forking, something that could not happen with ODP because of its corporate ownership, its onerous licensing restrictions, and its failure and refusal to use an open source software platform.

Grokking the Semantic Web

Internet Esquire — Thu, 22 Feb 2007 20:24:00 +0000

Although I've written about what the Semantic Web is previously, I've always felt that there should be a simpler way to explain it. And while exploring the blogosphere, I found a blog post on ....more semantic! that finally did just that by referencing Wikipedia as a global ontology:

"[Y]ou could . . . use a wikipedia reference to indicate the semantic concept that you are writing about. Thus, . . . you could use the link http://en.wikipedia.org/wiki/Rome to indicate that you are refering to the city of Rome, the capital of Italy."

This simple statement sums up just how straightforward it would be for someone to publish a document on the World Wide Web that is compatible with the Semantic Web. It also makes it abundantly clear just how tedious and time consuming such a task would be.

Most people who describe the Semantic Web talk about making web documents easier for machines to process. That's true enough, but that's really not what the Semantic Web is all about. At its essence, the Semantic Web is all about disambiguation, something that Wikipedia does quite well, even breaking through language barriers in the process.

The other day, a friend of mine who is taking a conversational Spanish class asked me if I could help her find something that was written in Spanish to complete an assignment for her class. I immediately thought of Wikipedia and how it links from articles in one language to articles in another language, so I told my friend to find a Wikipedia article in English that she liked and follow the link to the Spanish Wikipedia. I then directed her to use Babelfish for a crude translation of the article on the Spanish Wikipedia.

The Wikipedia article that my friend picked was Cat. However, when I followed the link to the corresponding article on the Spanish Wikipedia, I discovered that the two articles were quite different, so I found the appropriate article on the Spanish Wikipedia and changed the outgoing link on the English Wikipedia. Prior to the change I made, the English Wikipedia considered Cat to be a synonym for House cat, but linked to the more generic Spanish Wikipedia article entitled Felis rather than the more specific article entitled Gato doméstico, which in turn redirected to Felis silvestris catus.

This simple exercise demonstrates how a semantic search engine algorithm could drastically improve the relevancy of keyword-based search results, something that I hinted at in my previous XODP Blog post entitled Wordnet, Disambiguation, and the Semantic Web. And contrary to what some have suggested, a semantic search engine would not need to be intimately familiar with a user and/or the context of a particular user's search. It could default statistically to the most likely meaning of a particular word occurrence and still allow an end user to provide feedback on what he or she really meant.

All of this begs (or rather raises) the question of whether people need or want a user agent that is this sophisticated when it comes to searching the Web. I've done a significant amount of end user training, both for my private clients (most of whom are lawyers) and during presentations that I've made at Mandatory Continuing Legal Education (MCLE) seminars, and few people ever tax my knowledge base with their questions. In fact, I got my best reviews (5 out of a possible 5 for 90 percent of the queries asked of seminar attendees) by taking a full 30 minutes to explain the anatomy of hypertext links, knowledge that most Web-savvy individuals take for granted. Thus, I am inclined to believe that both the Semantic Web and semantic search will remain solutions looking for a problem for the foreseeable future; at best, they may become solutions made by geeks and for geeks.

Linking Wikipedia Articles to ODP/dMOZ Categories

Internet Esquire — Wed, 21 Feb 2007 23:15:00 +0000

While following up on a previous XODP Blog post narrating the death and resurrection of ODP, I found a link in a post on the Text Technologies blog pointing to a post on Joost de Valk's blog entitled DMOZ and Wikipedia: how it should work:

"Wikipedia is an online encyclopedia. . . . An encyclopedia should contain references to other articles in that encyclopedia, but doesn’t nescessarily have to have external links. A directory’s major purpose (at least the ODP’s major purpose) is to contain external links, chosen by editors to be of high quality.

"[ . . . ]

"On some articles, editors have linked to DMOZ, and removed almost all external links. That, to me, looks very good. The links in DMOZ have been checked, and should be of high value, in Wikipedia, this is impossible to do, as anyone can add his [or her] own link."

A truly open Web directory would be a great companion to Wikipedia, but linking a Wikipedia article to ODP and eliminating all other outbound links from that article will create many more problems than it will solve, a fortiori when one considers the uncertain future of ODP.

Astonishingly, no one seems to be pointing out just how bad this idea really is, much less why. When it comes to quality control, ODP's track record is much worse than Wikipedia's, so much so that both MSN and Google now allow webmasters to opt out of using ODP's meta data for their site descriptions, whereas Yahoo! has its own human editors to provide site descriptions. Add to this ODP's historical backlog of hundred of thousands (perhaps millions?) of sites, with a typical wait of some six months or more for a site submission to be reviewed, along with the total lack of transparency in the purportedly Open Directory Project, and Joost's idea makes about as much sense as amending the United States Constitution so that the current President Bush can be elected for a third term and stay the course in Iraq.

Assuming, arguendo, that ODP did not have systemic issues with quality control, was not a black hole for site submissions, and actually was an open directory, the idea of removing relevant outbound links from Wikipedia and replacing them with one link to a purportedly authoritative and comprehensive Web directory would be bad enough all by itself. One of the best things about Wikipedia is that it is a centralized clearinghouse for information on keyword-based topics, and a list of relevant outbound links is a logical component of such an online reference. To this end, many (if not most) Wikipedia articles include easily verifiable citations that link to other websites. Consequently, I find it hard to believe that more than a handful of Wikipedians would take Joost's suggestion seriously, but well over a thousand Wikipedia articles currently use a DMOZ link template, and Wikipedia's external links guideline was revised on November 16, 2006 to make the use of this template the norm. Notwithstanding the six week outage of ODP's website, this recommendation remains the status quo.

ODP/dMOZ - In Memoriam, Once Again

Internet Esquire — Wed, 21 Feb 2007 19:13:00 +0000

While I occasionally mention ODP/dMOZ in a historical context, particularly so in reference to Wikipedia inheriting whatever creative genius ODP once had, I wrote off ODP quite some time ago. Indeed, I had almost written off Web directories altogether until I stumbled upon Robert Barger and Brian Prince at Search Engine Strategies 2005 and discovered that they and others like them had successfully reinvented the concept of indexing communities. Even so, when enough people find some meat on ODP's rotting horse carcass, I write yet another post-mortem on the topic.

Notwithstanding a dismissive post that I wrote on the XODP Blog on June 19, 2006 entitled Will ODP Ever Die?, my last extensive post-mortem on ODP was posted on the XODP Blog on March 23, 2006, just under a year ago, so I guess it's about time for yet another one. To this end, Robozilla posted a brief message on the XODP Yahoo! eGroup pointing to a post on Richard Skrenta's blog from December 16, 2006 entitled, DMOZ had 9 lives. Used up yet?. I'm sorry that I'm so late in getting to the party, but better late than never. In any event, the horse has been dead for quite some time and has been hoisted up on a rope and hit so many times that it's beginning to resemble a piñata, the difference being that there's no candy on the inside of this beast.

For those of you who are unaware of the recent drama over at ODP, here's a pithy excerpt from Skrenta's blog:

"Apparently the machine holding dmoz in AOL ops crashed. Standard backups had been discontinued for some reason; during unsuccessful attempts to restore some of the lost data, ops blew away the rest of the existing data on the system.

"So for the past 6 weeks, a few folks have been trying to patch the system back together again (reverse engineering from the latest RDF dump, I suppose).

"dmoz doesn't exactly operate on a model of transparency, to say the least, so they have been keeping the details of what happened private. . . ."

I first became aware of this drama when the ODP article at Wikipedia (which is still on my Wikipedia watchlist) started to see some really weird edits. Given what I know of the inner workings of AOL, I gave ODP about a 50/50 chance of disappearing from AOL, the way that Disney's Go Directory and Looksmart's Zeal directory disappeared, with the ODP editor community reconstituting itself in one form or another on one or more websites using one of the not-so-recent ODP RDF dumps. I also anticipated that Richard Skrenta and Jimbo Wales would step up to the plate and offer whatever assistance they could. But for the fact that someone over at AOL took notice of Skrenta's blog post, that's what would have happened. As it now stands, ODP's systems have been more or less restored to the status quo ante, for better or for worse.

With the exception of a very small handful of true believers, most of the people who have ever had any dealings with ODP are either very apathetic about it (i.e., me) or have a very negative opinion of it. Many people would like to see ODP completely extinguished, but the ongoing ODP RDF dumps and resulting ODP clones that currently populate the Web make that an impossibility. The best case scenario is one where a white knight like Jimbo Wales would be allowed to give the current ODP a new home.

More than one person with the resources to make it happen (not Jimbo Wales) has asked me if I would be interested in spearheading an effort to reorganize ODP, the last time being about a year and a half ago. And my answer has always been, "Sure, for the right price. However, if you have those sort of financial resources, there's quite a few other projects that I consider more worthwhile." And that really is the problem with ODP. With the rare exception of someone like Jimbo Wales, the only people with the wherewithal to salvage what's left of ODP are either totally disinterested, totally clueless, totally in denial, or totally corrupt. Moreover, while AOL does not seem to be interested in revitalizing ODP, it is dead set on making sure that no one takes it away from them.

WordNet Redux

Internet Esquire — Tue, 20 Feb 2007 18:21:00 +0000

In a recent XODP Blog post, I extolled the virtues of WordNet and declared that it was one of the best kept secrets when it comes to Natural Language Processing (NLP) resources. Following up on this post, I found that few people outside of the NLP arena seem to have even heard of WordNet, and even the experts in this field do not seem to appreciate WordNet's potential. For instance, in a post at the Artificial Artifical Intelligence Blog, Lukas Biewald laments:

"Are concepts really a hierarchy? I’ve heard cognitive scientists think so, but I disagree. And I think that trying to make all the concepts conform to this artificial hierarchal structure has turned WordNet into a much less useful resource.

"[ . . .]

". . . [Some] groups of concepts . . . actually have a hierarchical structure for an unrelated real-world reason. . . .

"[ . . .]

"But this hierarchy completely breaks down for more conceptual things. Is respect in the sense of 'respect for my Father,' a type of 'attitude'' or 'politeness' or 'filial duty' or 'affection?' Clearly it’s all these things. But the guys making WordNet didn’t want to believe that, so they make respect as a type of attitude one semantic category, and respect as a type of politeness another, and so on, until there are ten separate senses for respect."

In their book, Computational Linguistics, which I mentioned in my previous post about WordNet, Igor Bolshakov and Alexander Gelbukh anticipate these sort of objections to creating linear representations (i.e., Text) of non-linear entities (i.e., Meaning):

". . . The human had to be satisfied with the instrument of speech given to him by nature. This is why we use while speaking a linear and rather slow method of acoustic coding of the information we want to communicate to someone else.

"[ . . . ]

"While the information contained in the text can have a very complicated structure, with many relationships between its elements, the text itself has always one-dimensional, linear nature, given letter by letter. . . . [A] text represents non-linear information transformed into linear form. What is more, the human cannot represent in usual texts even the restricted non-linear elements of spoken language, namely, intonation and logical stress. . . .

". . . A text consists of elementary pieces having their own, usually rather elementary, meaning. This meaning is determined by the meaning of each one of their components, though not always in a straightforward way. These structures are organized in even larger structures like sentences, etc. . . . Such organization provides linguistics with the means to develop the methods of intelligent text processing."

Like Wikipedia and its ongoing efforts at disambiguation, WordNet cannot be all things to all people, nor should it try, notwithstanding Lukas Biewald's assertion that WordNet should make allowances for words meaning "all of the above." What WordNet can be, is, and should remain, is a srong foundation for more sophisticated semantic analysis.

WordNet, Disambiguation, and the Semantic Web

Internet Esquire — Tue, 20 Feb 2007 00:28:00 +0000

While following up on a previous XODP Blog post that covered Powerset's attempts to outdo Google by using Natural Language Processing (NLP), I discovered a book entitled Computational Linguistics, available for free in both HTML and PDF format. As someone who has a background in both linguistics and cognitive science, I found the book a fascinating read, and about a third of the way through it, I discovered Princeton's WordNet. While WordNet is anything but obscure, as NLP resources go, I thnk WordNet is one of the world's best kept secrets.

A while back I heaped praised on Wikipedia for its ability to disambiguate keyword-based queries. Without taking anything away from Wikipedia's ongoing efforts at disambiguation, WordNet is already the most comprehensive open source and open content online resource for disambiguation of English nouns, verbs, and adjectives, providing extensive semantic analysis for just under 150,000 unique word strings. Noticeably missing from WordNet's database are articles, prepositions, pronouns, conjunctions, and word particles. Moreover, WordNet does not provide any information about any word's etymology or pronunciation. However, Wordnet does provide operational definitions for wordstrings, information about their common usage, and comprehensive semantic grouping information, including synonyms, antonyms, hyponyms, hypernyms, meronyms, holonyms, and troponyms. And if you want to know what any of those words mean, I suggest that you go on over to WordNet and enter them into the online interface.

WordNet has more or less solved one of the most basic challenges that was facing developers of the Semantic Web, as WordNet provides a relatively comprehensive operational lexical ontology for the English language where the taxonomy is both clear and definite, but still very, very flexible. In lieu of a standard search engine algorithm, which is limited to determining what I call "keyword relevancy," a properly configured semantic web user agent would be able to determine the actual semantic relevancy of online resources. Moreover, the end user would not have to resort to a series of keyword-based searches. Rather, an end user would have an ongoing conversation with his or her user agent, providing more and more feedback on what said end user wanted to know.

The Semantic Web envisioned by Tim Berners-Lee suffers from the perception that it's a high falutin enterprise and is commonly referred to by most web developers as the Pedantic Web. This ivory tower disconnect could easily be remedied by creating user-friendly user agents. In time, casual end users would be willing to use a semantic search agent when they encountered problems finding useful online resources with a standard search engine. Meanwhile, people who want to use the Web for serious research would be able to carve out a semantic niche for themselves.

Powerset Counts Coup with Google

Internet Esquire — Mon, 19 Feb 2007 03:24:00 +0000

A post on John Battelle's Searchblog points to an article on VentureBeat, the latter being journalist Matt Marshall's recently launched online publication that covers developments in the venture capital world.

"Powerset, a San Francisco search engine company, will announce Friday it has won exclusive rights to significant search engine technology it says may help propel it past Google.

"The technology, developed at Palo Alto Research Center (PARC) in Silicon Valley, seeks to understand the meanings between words, akin to the way humans understand language — and is thus called 'natural language. . . .'

"The deal is significant because practical use of linguistic technology has eluded Google. . . .

"[ . . . ]

" . . . Powerset could possibly steal a lead if it improves search results by a significant measure with natural language and simultaneously incorporates a near-equivalent to Google’s existing capabilities. Powerset has been hiring lots of Yahoo search experts and others, to help it do that.

"[ . . . ]

". . . Negotiations on the deal, just completed, were so secretive that Powerset’s executives hid a Xerox PARC scientist, Ron Kaplan, in a back room when VentureBeat stopped by for an interview last year. Kaplan, who has led the 'natural language' group for several years, joined Powerset as chief technology officer in July. This is a coup for Powerset, because Kaplan did not respond to some early probes from Google. In an interview, Kaplan said he didn't believe Google took natural language seriously enough. 'Deep analysis is not what they’re doing. Their orientation is toward shallow relevance, and they do it well.' Powerset, however, 'is much deeper, much more exciting. It really is the whole kit and caboodle.' While natural language has been a vexing problem for decades, Kaplan said he believes it is ready for prime-time."

The fact that a heavy-hitter like Kaplan chose Powerset over Google is quite remarkable, as Google has been responsible for a brain drain in the search engine space for quite some time. Of course, the technology is being hyped, site unseen, and as pointed out by Matt at Venturebeat, it also remains to be seen whether people can be convinced to change their keyword-based search behavior.

Wikipedia's Feckless Attempts to Fight Link Spam

Internet Esquire — Mon, 19 Feb 2007 00:01:00 +0000

Although it's hardly breaking news, I recently discovered that Wikipedia developer Brion Vibber has reinstated nofollow attributes for outbound links on Wikipedia articles. Brion took this action at the request of Wikipedia founder and God-king Jimbo Wales, but Brion also mentioned that he had heard a rumor that a search engine optimization (SEO) contest threatened to overwhelm Wikipedia with spam. According to Brion, the change will be for the indefinite future:

"I'd prefer to see actual improvements (whitelisting, fading, flagging and approval system, etc) rather than just turn it off one day, though."

Hard on the heels of this policy change, most members of the SEO community started crying foul, even as a significant number of Wikipedians started chanting, "Ding, dong, the witch is dead." Both of these responses are what I would have expected. However, when the nofollow feature was first put to a vote on Wikipedia some time ago, a majority of Wikipedians voted to turn off the nofollow attribute on Wikimedia software, returning Wikipedia to the status quo ante.

Even with the nofollow atribute enabled on Wikimedia software, Wikipedia remains an attractive nuisance for link spammers, as Wikipedia is currently ranked 12th on Alexa for overall web traffic. Assuming that the nofollow attribute does what it purports to do, all the current Wikipedia nofollow policy does is make sure that a search engine like Google does not count links on Wikipedia as votes in favor of increasing a particular URL's Google PageRank. While this may have a phenomenal impact on search engine rankings and results over the next month or two, it will have little to no impact on Wikipedia link spam, at least not in the foreseeable future.

Simply put, Wikipedia's nofollow policy makes about as much sense as screen doors on a submarine. While the latter may keep the fish out, the floodgates will remain open, and the shameful joy of the Wikipedians who think that Jimbo has struck a blow for truth, justice, and the American way is sounding more and more like hollow and self-aggrandizing rhetoric:

"We are not a links directory which is one of the problems with various ideas to selectively turn off no follow[sic]. If we do that we are basicaly[sic] admiting[sic] we are a links directory and we would gain very little for doing so."

Notwithstanding claims to the contrary, Wikipedia is a trusted source of URLs, not unlike a links directory. Moreover, it's pretty clear that link spam is already a serious problem for Wikipedia, and that the problem is only going to get much worse for the foreseeable future, prompting me to revise what I asserted in a previous blog post that the first practical limitation that Wikipedia will encounter will be in its ability to attract additional contributors. Indeed, the problem that Wikipedia now has is that it is attracting too many contributors with a hidden agenda.

What Are Wikipedia's Limitations?

Internet Esquire — Sat, 17 Feb 2007 12:49:00 +0000

In a previous XODP Blog post, I stated my belief that whatever creative and progressive genius ODP/dMOZ once had has been inherited by Wikipedia. Following up on this belief, I considered the fact that the number of articles on Wikipedia rivals the number of URLs indexed in the ODP database. And while my cursory review of Wikipedia's statistics failed to determine exactly how many outbound links Wikipedia has, I think it's safe to say that the number of outbound links in Wikipedia articles is already much larger than the number of URLs indexed in the ODP database. This prompts me to ask: What are Wikipedia's limitations?

Given the highly decentralized nature of Wikipedia editing practices and the scalability of Wikipedia's open source platform, there are virtually no internal limitations to Wikipedia's potential for growth, whether it be growth in the sheer amount of content, growth in the number of articles, growth in the number of outbound links, or growth in the number of contributors. Rather, Wikipedia has a demonstrated potential for exponential growth, and its practical limitations are found in the potential number of topics to be covered, the amount of outside content to be indexed, and the number of people who are willing to contribute their time and talents to Wikipedia. A derivative limitation is quality control, as Wikipedia's so-called "conflict of interest" policy scares away many subject matter experts.

Of the many growth metrics one can measure, the only one that indicates Wikipedia's growth is slowing down is its prominence on Alexa, where Wikipedia is currently ranked #12. To a certain degree, this is comparing apples and oranges, as none of the sites that are impeding Wikipedia's growth in this way are reference sites. Rather, Wikipedia is now going head to head with established search engines and community portals and is steadily gaining ground; as I have stated previously, there is every reason to believe that Wikipedia can and will make search engines as we know them obsolete because of Wikipedia's focus on disambiguating keyword-based queries.

Although it's impossible to predict when it will happen, the first practical limitation that Wikipedia will encounter will be in its ability to attract additional contributors. A related problem that I alluded to above is quality control based on Wikipedia's inherent bias against subject matter experts. Absent some incredible breakthroughs in artificial intelligence, these limitations will impede Wikipedia's growth long before any inherent limitations in the number of potential Wikipedia topics to be covered. Meanwhile, based on Wikipedia's success at being the purveyor of general knowledge, I think the various expert-based wikis that are currently emerging will find it easier to attract contributors.