Wednesday, January 21, 2009

More change to believe in

Truthfully, I don't understand the significance of this sort of geekery, but it sure sounds like a new transparency in governance. The robots.txt file from whitehouse.gov yesterday had some 2400 lines of code that looks like this:
User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
The new code today under the Obama administration
User-agent: *
Disallow: /includes/

That's it! BTW, the robots.txt file tells search engines what to include and not include in their indexes.
Supporting material at the link for you geeks. Maybe I'm just overly impressed by the mysteries of coding, but it looks like a huge difference to me.

[More posts daily at The Detroit News.]

Labels:

Bookmark and Share

7 Comments:

Blogger (O)CT(O)PUS said...

Dunno, looks like geek to me.

4:28:00 PM  
Blogger Libby Spencer said...

LOL. Me too.

4:51:00 PM  
Anonymous Anonymous said...

Shorter does not mean better.
Apparently, the old file had lots of point directories that were not to be searched. And some of them, such as cgi or those multiple search and query.html entries, contains programming-related stuff, not a content, and clearly have nothing for indexing by search engines.

The new file disallows indexing for anything under /includes/ directory. Anything! As long as new guys put their stuff there, it is invisible. Who knows what percentage of the stuff will go there. Perhaps, everything.

To sum, one huge prohibition is much worse than a sum of small targeted ones.

Btw, wikipedia explains robots convention.

5:41:00 AM  
Blogger Libby Spencer said...

Anon, this is way over my pay grade but if you read the link where I got it, he posits the exact opposite view you're pushing here. And common sense would suggest that disallow nothing means everything is searchable.

In contrast, you did notice there was 2400 lines of code that disallowed searches under Bush. Your argument doesn't make sense.

Speaking from personal experience, I found trying to search anything on that site under Bush was impossible. I haven't tried it since Obama was sworn in, but I'm willing to bet I'll be able to find what I'm looking for.

1:04:00 PM  
Anonymous Anonymous said...

I've done plenty of websites for years and this doesn't mean anything except a new web person came in and uploaded a bunch of changes to the website, which probably included a new robots.txt file as part of the upgrade.

Using this as some reason to gush over Obama's supposed transparency (when the White House press people can't even politely respond to a reporter)...well, I am continually amazed that people have flip-flopped from hating Bush with an insane passion to jumping with joy every time Obama sneezes. Some middle ground here, please!

10:14:00 PM  
Anonymous Anonymous said...

Oh, and a PS, based on that snippet of code you posted, it looks like they were disallowing the search function. There's a perfectly reasonable and non-sinister reason for this: Search adds a lot of load to a server. In many websites this is the first thing that gets taken off (throttled) when a site gets too much traffic.

Much better to use an external Google search function...saves on bandwidth.

And if they truly had anything they wanted to hide, they wouldn't put it up on a website in the first place.

Common sense, really.

So please...enough with the conspiracies or kudos. If the government wants to hide stuff from you, they'll be more savvy than just using a robots.txt file.

10:19:00 PM  
Blogger Libby Spencer said...

Anon, I'm a complete technodope. None of this means anything to me. I went by the post because the guy seemed to know what he was talking about.

In any event, I'm hardly swooning every time Obama sneezes. I've criticized him on the few things that I thought were really bad. Otherwise, this is the hardly the biggest step towards transparency he's made so far. Let's not discount the progress because we're too stuck in criticism mode from the last 8 years to believe that change is even possible.

For the moment, it's been three weeks now and the I imagine the economy and the occupations are rather daunting problems that require a whole lot of attention, so if Obama is not perfect and OMG can't change everything in a month, I'm willing to cut him a little slack on the small stuff.

As I've been saying repeatedly, if and when the admin shows a serious pattern of obfuscation and or miscreant behavior, I'll be the on the front lines of the torch and pitchfork crowd.

10:25:00 AM  

Post a Comment

<< Home