Planet GIntern

Manas Tungare: Personal Information Backup - Twitter, Gmail Contacts, Google Calendar, Reader

Manas — Fri, 02 Jan 2009 05:17:54 +0000

Make a New Year’s Resolution to start backing up your data regularly. Not just local files, but even data from the cloud. Here’s how to backup your data from a few of the most common online services. More importantly, I’ve also included instructions on how to restore from that backup.

Twitter

How to backup?

Copy this URL to a new browser window.

http://twitter.com/statuses/user_timeline/USERNAME.xml?count=10000

Then replace the string USERNAME with your actual Twitter username. Press enter to start downloading. If your browser does not prompt you with a file download box, but instead opens the file showing a bunch of text, choose File > Save As to save your backup to a secure location.

How to restore?

You cannot restore this data into Twitter (neither to your own account, nor to a different account.) But you will have access to your witticisms and interesting web links that you posted to amuse your friends. Do with it as you please.

Gmail Messages

How to backup?

Use an IMAP client such as Mail.app on the Mac, or Thunderbird on any platform. Make sure it’s configured to download and cache every email and every attachment.

How to restore?

You can access your messages from these programs even if Gmail is down. If you need to transfer messages to another account, add that new account in the same program as a new IMAP account, then drag-and-drop messages from your old account to your new account to transfer them there.

Gmail Contacts

How to backup?

Login to your Gmail / Google Apps email account, then open the Contact Manager. Click on the Export button in the top-right corner. For maximum compatibility with other applications, choose the third option for data format, vCard format. (It’s a standard format for contact information exchange.)

How to restore?

The vCard format is fairly standard. Gmail itself can read back the same file without trouble. To import into Mac OS X Address Book, simple double-click the .vcf file and let the import proceed. Microsoft Outlook also supports importing addresses from vCard files.

Google Calendar

How to backup?

Google Calendar publishes feeds of your calendar in the iCal format. If you save this feed to a file, you can use it as a backup. On the left side of your main calendar, there is a list labeled “My Calendars”. For each calendar that you want to backup, click on the little downward-pointing arrow next to the calendar name, and select “Calendar Settings”.

Calendar Settings" /> Calendar Settings" /> Calendar Settings" title="My Calendars > Calendar Settings" width="327" height="265" class="alignnone size-full wp-image-248" />

On the Settings page, under Calendar Details, locate the section labeled “Private Address”. Click on the button labeled ICAL and copy the URL there. Open a new browser window and paste the URL there. This will start downloading a file; save it to a safe location — this is your calendar backup. Lather, rinse, repeat for each calendar you want to backup.

Private URL > ICAL" /> Private URL > ICAL" /> Private URL > ICAL" title="Calendar Details > Private URL > ICAL" width="788" height="106" class="alignnone size-full wp-image-249" />

How to restore?

The iCalendar format (also abbreviated as iCal or .ics) is a standard calendar format. You can import the backed up calendar file into Google Calendar, Apple iCal or Microsoft Outlook simply by opening it.

Google Reader

How to backup?

Login to Google Reader, then come back here and click on this link: Export Google Reader subscriptions as OPML. Save the file that your browser will prompt you to download. This is your backup.

How to restore?

Google Reader and lots of other feed readers know how to import OPML files. In case of Google Reader, go to Settings > Import/Export to import it back. For desktop software, try looking for an “Import from OPML” menu item somewhere.

Need instructions for more services? Write a comment and I’ll try to provide them.

David Anderson: Levels of abstraction

David Anderson — Mon, 29 Dec 2008 05:43:36 +0000

I promise, I can explain. It started out rather simply, then got a little out of hand. A week or so later, I'm still having an immense amount of fun.

It all started when Google gave me an awesome Christmas present: An HTC Dream. It's a very shiny mobile phone, and what's more, it's an unlocked developer edition. It's hacking time!

This is where things get a bit complicated. Lemme take you through the reasoning.

My first idea when I saw this beast was to try to get emulators running on it. A phone is nice, but a phone that can play vintage games is even better. I decided on playing with the Sega Genesis first, as I have rather fond memories of Sonic the Hedgehog.

First obstacle: Android (the open source OS that Google develops) currently can run only Java code. There is currently no open source Genesis emulator written in Java. Most of them are written in C, or in extreme cases, even in x86 assembler. There is currently no official way to execute native code on the android phone. I'd like to make this software available to everyone.

I therefore need to write a Genesis emulator in Java.

Okay, that should be simple. The Genesis is a relatively old console, so it can't be too elaborate. I mean, it's no PS3. All I need is an emulator for the CPU, a decoder for the ROM format, and some audio and graphics hookups within Android, and I should be good to go.

Well, first, the Genesis has three processors. A Motorola 68000 CPU, a Z80 sound processor, and a custom made graphics processor. Let's start with the 68k CPU. Apparently, there is no well known open source Java emulator for the 68k.

I therefore need to write a Motorola 68k emulator in Java.

But Java sucks. I mean, it's obviously a successful language, but I find no pleasure at all programming in Java. It is pure pain without an IDE on the level of Eclipse or Netbeans, and those IDEs aggravate me in various ways. The the the language language language is is is way way way too too too verbose verbose verbose (verbose verbose).

Plus, after consulting the 68k specs and sampling a few C implementations, it looks like an extremely repetitive task: most opcodes have around 20 variants, depending on addressing modes and various flag bits. It would be very tedious to implement this by hand, not only because it'd be in Java, but because it'd be even more mind numbing and uninteresting. However, the kinds of variants that are needed are quite amenable to be described at a high level, leaving the repetitive task of actual implementation to a program. And I can use a language that I enjoy for that, say, Python.

I therefore need to write a Motorola 68k emulator generator in Python.

After a bit of prototyping, I came to the conclusion that implementing this in Python would also be rather tedious, for a variety of reasons. First, I started off badly by writing a generator that goes straight from high level description language to a Java source code string, mushing several levels of abstraction together. Second, the description needed to generate the variants quickly lead me to combinatorial explosions, or to independent components that began interacting with each other in hilarious ways. Not good.

Plus, one day, when Android does have a supported way of running native code, I'd probably want an emulator in C or C++, running on the CPU directly, instead of under the Dalvik virtual machine. At which point all my work will have been for nothing.

I therefore need something of an emulator compiler, that parses the high level description into an execution tree for the opcode implementations, which a code generator then translates into a variety of output languages, such as Java, C++ or Brainfuck.

Python is nice, but I don't think that writing compilers is one of its fortes, despite what the PyPy folks appear to think. Manipulating the code representations is cumbersome at best, and the divide between the living Python code and the dead data it manipulates is rather wide. Manipulating code as data and vice versa is one of the often described merits of the Lisp family of programming languages. I've been wanting to get back to Common Lisp as a language and poke around with it more, and it feels like the ideal language in which to build a compiler. What could be more code-as-data-as-code than a program that takes apart a description of a program and puts it back together again in another form?

I therefore need to write an m68k emulator compiler in Common Lisp, at first targeting the Java programming language, and later possibly other languages.

And that is how, a week after Google gives me a mobile phone, I find myself writing Common Lisp code, for a compiler that compiles a lisp-like language into Java code, that will be compiled into Dalvik VM bytecode, running on an ARM-based embedded system, which when executed will emulate a Motorola 68000 CPU.

I feel like I've just had a Wikipedia attack. You know, that thing where you go to Wikipedia to look up something very specific, say how Tesla coils can be used to play music, and end up three hours later reading through an analysis of 14th century persian battle tactics, with no idea how you got there. That's kinda how I felt when I came up for air yesterday and looked back. "So, I got a phone... And now I'm writing a compiler... I'm pretty sure I have a good reason..."

Oh, and the emulator compiler is starting to work. I was very rusty in Common Lisp, but in a couple of days of hacking and prototyping, I'm starting to get somewhere. I can already generate the implementation of the simplest variant of the 68k ADD opcode. The "source" looks like this, with comments added

(instruction
   ;; Instruction name, with variant information.
   "add_dreg_to_dreg"

   ;; The description of the meaning of the 16 bits of the opcode.
   ((:literal 4 #b1101)      ; 4 constant bits, with the given binary value
    (output-register 3 dest) ; The output register, whose number is coded
                             ; over 3 bits. Its value is available in the
                             ; 'dest' variable.
    (:literal 6 #010000)     ; More constant bits, describing the addressing mode.
    (input-register 3 src))  ; Input register, similar declaration to dest.

   ;; How to perform this operation, in a subset of Common Lisp.
   (setf dest (+ src dest)))

The above instruction definition produces an opcode object that contains two things: information for the instruction decoder, so that it can identify this instruction, and the intermediate representation of the implementation of that instruction:

;; The constant bit values in the opcode, and the mask to test them,
;; for the instruction decoder.
(debug-print-opcode-mask the-above-opcode-object)

--> Output: 1101---010000---

;; The intermediate representation of the implementation.
(opcode-ast the-above-opcode-object)

--> (LET ((SRC (REGISTER-VALUE :DATA
                               (VM-OPCODE-BITS 3 0)))
          (DEST (WRITABLE-REGISTER-VALUE :DATA
                                         (VM-OPCODE-BITS 3 9))))
      (SETF DEST (+ DEST SRC)))

This opcode can then be fed to the Java code generator, to produce the output implementation:

(java-gen-opcode the-above-opcode-object)

--> public static void op_add_dreg_to_dreg(unsigned short opcode) {
      unsigned long src = mDataRegisters[(opcode >> 0) & 0x7];
      unsigned long dest = mDataRegisters[(opcode >> 9) & 0x7];
      dest = dest + src;
      mDataRegisters[(opcode >> 9) & 0x7] = dest;
    }

There is still a lot to be done. For one, the ADD opcode is supposed to update the CPU's state flags with information about the result of the addition. After that, the addressing modes other than to/from a numbered register must be supported. Implementing more opcodes will surely bring more things that need to be implemented.

Once a solid base is laid, a higher-still level of description must be layered on, so that all the variants of an instruction are produced from a single implementation definition. Once all that is done, a C++ backend would be nice. And why not attempt to generalize the compiler infrastructure, so as to support the compilation of emulators other than m68k CPUs?

By the time I get anywhere near that, I suspect that it will have become possible to easily write and release native code for Android, making all of my efforts unnecessary. But I don't care, this is damn fun!

Some people are of the opinion that I should get my head examined.

Vincent Cheung: Hong Kong, China, and Australia

noreply@blogger.com (Vince) — Sat, 20 Dec 2008 04:24:00 +0000

Bye! See you in 2009!

Talk amongst yourselves.

Vincent Cheung: Random observations 2

noreply@blogger.com (Vince) — Fri, 19 Dec 2008 18:32:00 +0000

There are *way* more asians in Winnipeg then there used to be. It used to be that you'd go to a bar and you could literally count the number of non-white people on one hand. I don't know where all these people came from. It seems that the generation just a few years younger than mine is a lot more multicultural, even in Winnipeg.

Girls definitely dress a lot skankier here than in Toronto. Even with a windchill of -40, girls wear super short skirts and dresses. Girls in Toronto seem to dress rather conservative in comparison.

Tonight it'll warm up to -18 with a windchill of -28. I think we've finally seen the end of this cold snap! (This is *not* a sarcastic statement, this is the actual view of people over here)

Oh, and here are some amusing photos showing how cold it is:

Oh wait, these aren't actually pictures from Winnipeg where we've seen windchills as low as -49 in the last few days and there's permanently 2 feet of snow on the ground. My friend in Seattle took these and it was -3 outside and it had snowed a little overnight.

Vincent Cheung: No more orthodontist!

noreply@blogger.com (Vince) — Thu, 18 Dec 2008 21:38:00 +0000

I got my braces back in Grade 6... 1992! My orthodontist is in the Medical Arts Building on the 15th floor and I clearly remember the elevator being a novelty because it was the only time that I would ride the elevator. Nowadays, I ride the elevator every day in my apartment, but my elevator manners have Winnipeg roots.

I got my braces off after maybe a year and a half, but ever since then, I've still been going to my orthodontist every 6 or 12 months just for him to check on my teeth and how my retainer fits, which takes a whole 2 minutes. He tells me to just pop it in overnight once every few months or so just to make sure things stay where they are. The only reason I keep going to see him is that I haven't had to pay for the visits for the past 14 years! I don't know anyone that still wears their retainer, let alone still visits their childhood orthodontist!

I just went today and he told me that he's letting me go! After 16 years, I no longer have to go to my orthodontist!

Manas Tungare: Book-as-Blog: Encouraging Reading by Posting a Chapter at a Time

Manas — Wed, 17 Dec 2008 20:57:34 +0000

I realized I haven’t picked up a book in weeks, (non-academic book, that is), but I’ve read more than my fair share of blogs in that same time. I wonder if part of the reason is the longer time commitment required by a book. This prevents it from being read quickly and keeps it forever on my wish list. If so, then how about a service that breaks down books into blog-post-sized chunks and publishes them every few days?

The idea is inspired by, — nay, stolen from — Kevin Kelly, who is reissuing his 10-yr old book as a blog (hat-tip to Seth Godin’s post on the topic). His reasons are different, though. The book is out-of-print, and is already available as a downloadable PDF from his web site. Making it available as a blog is just another way of spreading his ideas wider, which is a great idea.

But apart from that, I like the idea of chopping up a book into chapter-sized chunks and making them available to readers one at a time. Not for any economic reasons, but because attentional resources are so scarce these days. A few times during the day, I have some free time which I use to read a few blog posts. If I ever thought about picking up a book during these breaks, I wouldn’t do it, simply because of the (arguably artificial) time commitment issues it raises in my mind. But talk about a chapter-sized, or even smaller blog post, and I’d read it.

Of course, not all book content has an affordance for this kind of splicing and dicing. If it takes several minutes for a reader to re-establish context from the last blog post, the purpose is lost. Some authors would consider their books a work of art too precious(ssss) to split it up into anything smaller. That’s also the reason why bands are often reluctant to sell singles instead of entire albums (apart from the record labels preferring to sell you 9 lame tracks bundled with 1 great track for $10 instead of $1, thank you very much.) But several non-fiction books could verily adapt to such a format.

The book-as-blog need not be free (as in no charge.) Sure, charge me for it. Implementation would be easy, charge me a micropayment and give me a secret watermarked feed URL. With so much new content licensed under a Creative Commons attribution license, it’s also possible to develop a web service that does this for liberally-licensed and public domain works. This is compatible with Creative Commons Attribution (BY), Attribution-ShareAlike (BY-SA), Attribution-Noncommercial (BY-NC), and Attribution Non-commercial Share-Alike (BY-NC-SA) licenses (but I’m not a lawyer, this is not legal advice, blah blah.)

Maybe something like this will finally get me back to the several-books-a-month club I used to be a member of, until I discovered this newfangled shiny thing called the Internet.

Kaitlin Duck Sherwood: Throwing Blagojevich out

ducky — Tue, 16 Dec 2008 15:59:18 +0000

Am I the only person who is concerned that justice is moving too hastily on Blagojevich? He stands accused of some pretty appalling stuff, but the key word is accused. I realize that many civil liberties have been badly compromised in the past eight years, but I thought that the US still (mostly) believed in “innocent until proven guilty” for its citizens. To throw him out of office before a trial would be unfair.

There is also some speculation that he didn’t do anything illegal. He sure looks like a stupid, arrogant slimebucket, but that isn’t illegal. He wanted to use the appointment to his advantage, sure, but there is lots of influence-trading that doesn’t get prosecuted, e.g. people donating to a candidate being rewarded with ambassadorships. (It is always less shocking to discover how much illegal activity goes on, than how much is perfectly legal.)

It is important for civil liberties to ensure that the government not be allowed to deny anyone — even people we don’t like — fair, equitable process under the law, including the presumption of legal innocence.

Vincent Cheung: Random observations

noreply@blogger.com (Vince) — Fri, 12 Dec 2008 21:43:00 +0000

Actual flight path and door-to-door transportation as recorded by my GPS device
(This is just a screenshot b/c Google Maps couldn't handle that big of a file)

The taxi driver was giving me career advice. He thought that I should learn Mandarin.

Cabbie was talking to me in Cantonese. I hadn't heard that much Chinese for a long time.

It's still harrowing walking through the airport metal detector carrying a cat without a leash or anything.

Tiki didn't move or make a sound the entire flight. I don't think the woman on the aisle seat knew she was even there.

During the initial announcement, the flight attendant tried to make a reference to the commercial where the guy thinks he's on a flight to Hawaii, but is actually on a flight to Winnipeg and goes "...going to Winnipeg..." No one laughed.

When asked who was from Winnipeg, the entire flight went in an uproar. No one goes to Winnipeg at this time of the year unless they are from there.

The games on the Zune are actually pretty fun.

Upon arrival, the flight attendant neglected to notify us of the local time (one hour behind)

When it was announced that the current temperature in Winnipeg was -27, it wasn't a big deal or Earth shattering news.

While waiting for my luggage, I saw a woman drinking a slurpee. This is why Winnipeg is the slurpee capital of the world.

I miss the feeling of the cold fresh air in your lungs.

When driving here, if it's not a big road, you actually drive on top of snow since the roads aren't cleared right to the asphalt and salt doesn't work b/c it's too cold.

There are 3 cats within arm's reach of me right now as I am sitting on my bed.

Mi-Mi has reclaimed her spot on my bed and refuses to budge. Tiki is confused. Tiggy is indifferent.

The fish is big now.

The dog is big now.

Kaitlin Duck Sherwood: Canadians worrying about US water appropriation, wtf?

ducky — Wed, 10 Dec 2008 07:35:58 +0000

I am baffled by a concern that seems endemic in Canada: that the US is going to steal Canadian water. The way they talk about it, it’s almost like they think there are already secret contingency plans drawn up that one more dry season in California will trigger.

This seems totally preposterous to me:

I have never heard anyone in the US talk about routing Canadian water to the US. I remember about ten or twenty years ago, hearing people talk about a canal to Oregon, to the Columbia river, but it wasn’t something that people were taking seriously. It was sort of like how in the late seventies there were people talking about building space colonies. There were a few people thinking about the theoretical possibility, but there wasn’t any real thought that they were practical.
Which states might run out of water? Let’s suggest California, Arizona, New Mexico, and Texas. States in the South. Where is Canada? Way way north. Where is the closest water to California? Oregon. Don’t need to go any farther. Where is the closest water to Texas? The Mississippi. Don’t need to go any farther. What is the easiest water to get to from Arizona and New Mexico? Probably the Mississippi again. Maybe you’d object that the Mississippi water isn’t very nice by the time it gets to Louisiana. Maybe it is, but if they are out of water, they can’t be that choosy. The next place they could look would be Lake Michigan; you don’t have to go over any mountains and you don’t have to cross any international borders.
Why should the Canadians worry about water when they could worry about oil instead? The US has a history of belligerency related to oil; I don’t know of any US belligerency related to water.
- One Canadian, in response to that question, said “Yeah, but we already sell the oil.” Yeah, but Canada could sell the water, too. And Canada has a lot more water than it has oil.

So I find it a very odd concern. I am not suggesting that Canadians should think of the US as an entirely and always consistently benevolent country. I’m sure there are things Canadians should be nervous about. But water? That is so far down on the list of things that I would worry about that I find it very odd.

To be fair, I also have heard a man from Michigan be concerned about California stealing Michigan’s water. (He went on an extended rant about how people shouldn’t live in places that required importing large quantities of natural resources. I wonder how he would enjoy winter in Michigan without large imports of fossil fuels.) Maybe they got this idea from the Canadians.

Kaitlin Duck Sherwood: Wright ad that didn’t play

ducky — Wed, 10 Dec 2008 00:26:38 +0000

An ad that the McCain team put together featuring Rev. Wright has surfaced, with some questioning why McCain didn’t run the ad. Deep in the dusty corners of my mind, a bored little clot of neurons is sure she knows why McCain didn’t run the Wright ad.

In my scenario, at some point early in the campaign, Obama talks to McCain about running a honorable campaign. “John”, says Obama, “let’s rise above the pettiness. Let’s agree right now, before we get really started, that our private lives should stay private. Let’s agree that our families are off-limits, and our religions are off-limits.”

In my imagination, McCain can’t believe his ears! Obama is pretty much offering him a get-out-of-jail free card! Obama is volunteering to not talk about McCain’s serial infidelity, Cindy’s drug addiction, or about how McCain’s religious intensity is um low. What a cool deal! In my head, McCain is having a hard time containing his glee as he agrees to the deal.

Later, when the Wright tapes surface, and Michelle makes a few missteps, McCain realizes that he’s been had. He got suckered, big time. In my story, he fell for it: hook, line, and sinker. McCain’s a little peeved at himself for being such a fool, but it’s emotionally easier to put the blame on Obama than to place the blame on himself. Thus McCain decides that this proves that Obama is nothing but a smooth-talking snake. (This explains why McCain sure acted like he couldn’t stand Obama.)

He can’t actually say that Obama is a smooth-talking snake, without telling the story of how McCain had gotten suckered, and that doesn’t look good on McCain. So instead he alludes to it with the “celebrity ad”: don’t trust this guy, he’s just a pretty face. But because the populace didn’t know the back story, they don’t recognize that the ad is a story of Obama’s treachery. So the ad fizzles.

And Wright is radioactive, but in my story, McCain can’t talk about Wright without going back on his word. (And if he goes back on his word, then he has to admit to himself that he’s no better than that treacherous snake Obama, so he can’t do that.) So instead, McCain uses Ayers as a proxy for Wright. They talk about Ayers over and over again hoping that somebody will make the link between Ayers and Wright. Unfortunately, when the McCain camp says, “Ayers, Ayers, Ayers, Ayers”, the populace hears, “Ayers, Ayers, Ayers, Ayers”.

Meanwhile, because I am an Obama supporter, I can paint Obama as completely innocent in this matter. In my story, I can have Obama approaching McCain with complete sincerity and noble intent, and the story still hangs together.

Now, I have no hotline to Obama’s or McCain’s brain. This is just a story that I made up to help me make sense of the world. But I like the story.

Kaitlin Duck Sherwood: times are a-changin’

ducky — Mon, 08 Dec 2008 10:32:23 +0000

From James Fallow’s blog posting about Shinseki:

the first Asian-American in a military-related cabinet position

I love how we are starting to have to pile on the adjectives. It’s no longer enough to say, “the first Asian-American in a cabinet position”. (That would be Norm Mineta.)

I look forward to the day when I see something like:

the first left-handed Asian-American woman from a university in the Minnesota named to be the National Security Advisor under a Republican president

And what will be really cool is the day when I don’t even notice the adjectives.

Manas Tungare: Evolving Similes

Manas — Sun, 07 Dec 2008 21:24:15 +0000

A buggy is like a car, but drawn by a horse.

Radio is like TV, but with no picture, only sound.

A letter is like email, but written on paper and delivered in days or weeks.

The Encyclopedia Britannica is like Wikipedia, but printed in 26 volumes, and occupies half a room.

A telegram is like SMS, but you need to send it from the telegraph office.

A newspaper is like Google News, but is only updated once a day.

TV is like YouTube, but you can only watch what’s playing right now.

A Victrola is like an iPod, but with fewer songs and lower fidelity.

Kaitlin Duck Sherwood: Digital public works

ducky — Sat, 06 Dec 2008 18:33:42 +0000

Mitch Kapor suggests that some of Obama’s pledged massive public infrastructure project be digital. While I would hesitate a little bit — I would want to make sure that it wouldn’t cannibalize projects to repair the decaying US physical infrastructure – I think it’s a really intriguing idea.

For some projects, a lot of good could be done by setting up a structure to make it easy for volunteers to contribute to. For an example near and dear to my heart, the government already provides a bunch of mapping data. However, that mapping data is incomplete and erroneous. Fixing an erroneous data point requires very local knowledge, but not much effort.

Suppose there were an iPhone app that would alert when you came close to a questionable point, and ask you to check it out. For example, if there was a street that had no name entered in the database, if you were close to/on that street, it could ping you and ask you what the name of the street was. If earlier, it had had trouble finding the location of “234 West Wilpole St, Hoopston, IL” and you live in Wilpole, the app could ping you and ask you where it is.

This will only work if you have a lot of people signed up, and the app works well. You would need some sort of public awareness campaign (which takes resources), some money to develop the application, and some money to do “customer support”.

I care a lot about map data, so naturally I think of that. However, there is probably lots of other data that it would be useful to collect.

Vincent Cheung: Shape Collage

noreply@blogger.com (Vince) — Fri, 05 Dec 2008 03:08:00 +0000

Some collages of Tiki made using Shape Collage

Shape Collage is the latest side project that I've been working on. It is an automatic photo collage making program that can make collages in different shapes. Just look at the collages above and at the end of the post to see what it can do. There are a lot more examples on the Shape Collage web page.

It's really easy to make a collage. You can make one with just 2 mouse clicks and in as little as a couple seconds! It's quite customizable as well, as you can change the shape to really, anything you want. You can even draw your own shape! You can also customize the border, background, etc.

Arranging photos into a collage

The photos are automatically placed in the desired shape of the collage using a machine learning algorithm that I came up with that is simple, but fast and effective. Basically what it does is put the photos on a page and jiggle them around so they are arranged nicely. This is a much harder problem than you may think. My readers with some knowledge of computer science will be familiar with NP problems. My algorithm isn't perfect, but it does a pretty good job most of the time. If you're really anal about it, you can save the collage as a Photoshop PSD and edit the collage in Photoshop if you don't like the automatic placement of the photos.

I started working on this project b/c I read a paper that automatically created a photo collage and thought that it was really cool, especially since it took me so long to make my Californian photo collage. Since they didn't actually release a program or their source code, I decided to just implement their approach, but it was overly complicated and rather slow - it took several minutes to create a collage with just 30 photos. So, I tried something simpler that was much faster - my program can easily create a collage with thousands of photos. Then I realized that my algorithm didn't have to only create rectangular collages and could also create collages in different shapes, something that I had never seen before, and so Shape Collage was created!

Anyways, check it out and let me know what you think! It's available for Windows, Mac OS X, Linux, and anything that supports Java.

Snowboarding snowman collage

Volleyball collage

Kaitlin Duck Sherwood: Times they are a-changing

ducky — Mon, 01 Dec 2008 22:59:27 +0000

Wow. Obama announced his six national security advisers today, and white men were in the minority. Eric Holder and Susan Rice are not white; Hillary Clinton, Janet Napolitano, and Susan Rice are not men.

A few days ago, I read that Condoleeza Rice phoned Barak Obama twice during the Mumbai terrorist attacks and was profoundly moved by the mental image. It’s not that there haven’t been black people in positions of power before. I’m sure that Rice phoned General Powell more than once. (While I dislike almost everything about G. W. Bush, I do have to give him props for not being afraid to appoint people of colour to high positions.)

What struck me was that it was a very powerful black person in one administration phoning a very important black person in the next administration. This demonstrates that it is not tokenism, nor a fluke of one administration. It says that having people of colour in positions of high responsibility is not odd or unusual. And that’s the way it should be.

Kaitlin Duck Sherwood: Programming persistence

ducky — Mon, 01 Dec 2008 07:21:03 +0000

Warning: this is a long and geeky post.

From time to time in the past few years, I have mentioned that I was a little puzzled as to why more people didn’t render tiles on-the-fly for Google Maps, as I do in my U.S. Census Bureau/Google Maps mashup.

I have reappraised my attitude. I have been redoing my mapping framework to make it easier to use. I have reminded myself of all the hurdles I had to overcome, and discovered a number of annoying new ones.

First pass

I originally wrote my mapping framework in an extreme hurry. It was a term project, and a month before the end of the term, I realized that it would be good for personal reasons to hand it in a week early. The code functioned well enough to get me an A+, but I cut a huge number of corners.

Language/libraries/database choice

It was very important to minimize risk, so I wrote the framework in C++. I would have liked to use a scripting language, but I knew that I would need to use a graphics library and a library to interpret shapefiles. The only ones I found that looked reasonable were C-based libraries (Frank Warmerdam’s Shapelib library andThomas Boutell’s gd library). I knew it was possible using a tool called SWIG, but I hadn’t ever used SWIG and had heard that it was touchy. Doing it in C++ was guaranteed to be painful, but I knew what the limits of that pain were. I didn’t know what the limits of pain of using SWIG would be.

Projection

I also had problems figuring out how to convert from latitude/longitude to pixel coordinates in the Google tile space. At the time (December 2005), I had a hard time simply finding out what the mathematics of the Mercator transformation were. (It is easier to find Mercator projection information now.) I was able to figure out something that worked most of the time, but if you zoomed out past a certain level, there would be a consistent error in the y-coordinates. The more you zoomed out, the bigger the error. I’m pretty sure it’s some sort of rounding error. I looked at it several times, trying to figure out where I could possibly have a roundoff error, but never did figure it out. I just restricted how far people could zoom out. (It also took a very long time to render tiles if you were way zoomed out, so it seemed reasonable to restrict it.)

Polygon intersection

I remember that I spent quite a lot of time on my polygon intersection code. I believe that I looked around the Web and didn’t find any helpful code, so developed it from scratch on little sleep. (Remember, I was doing this in a frantic, frantic hurry.) I ended up with eight comparisons that needed to be done for each polygon in the database for every tile. More on this later.

Rendering bug

The version I handed in had a bug where horizontal lines would show up at the bottom of tiles frequently, as you can see in the bottom left tile:

It was pretty obvious that the bug was my fault, as gd is a very mature and well-used graphics library. My old office partner Carlos Pero had used it way back in 1994 to develop Carlos’ Coloring Book, so it was clear to me that the problem was my fault.

After I handed in my project, I spent quite a lot of time going through my code trying to figure out where the problem was with no luck. Frustrated, I downloaded and built gd so that I could put breakpoints into the gd code. Much to my surprise, I discovered that the bug was in the gd library! I thus had to study and understand the gd code, fix it, report the bug (and patch), and of course blog about it so that other people wouldn’t have the same problem.

Pointing my code to the fixed gd

Then, in order to actually get the fix, I had to figure out how to statically link gd into my binaries. I like my ISP (Dreamhost) and wasn’t particularly interested in changing, but that meant I couldn’t use the system-installed gd libraries. Statically linking wasn’t a huge deal, but it took me at least several hours to figure out which flag to insert where in my makefile to get it to build statically. It was just one more thing.

Second pass

I have graduated, but haven’t found a job yet, so I decided to revamp my mapping framework. In addition to the aesthetic joy of making nice clean code:

It would be an opportunity to learn and demonstrate competence in another technology.
I had ideas for how I could improve the performance by pre-computing some things.
With a more flexible framework, I would be able to do some cool new mashups that I figured would get me more exposure, and hence lead to some consulting jobs.

Language/libraries/database choice

Vancouver is a PHP town, so I thought I’d give PHP a shot. I expected that I might have to rewrite my code in C++ eventually, but that I could get the basics of my improved algorithms shaken out first. (I’m not done yet, but so far, I have been very very pleased with that strategy.)

I also decided to use MySQL. While the feeling in the GIS community is that the Postgres‘ GIS extensions (PostGIS) are better than the GIS extensions to MySQL, I can’t run Postgres on my ISP, and MySQL is used more than Postgres.

I had installed PHP4 and MySQL 4 on my home computer some time ago, when I was working on Mapeteria. However, I recently upgraded my home Ubuntu installation to Hardy Heron, and PHP4 was no longer supported. That meant I need to install a variety of packages, and I went through a process of downloading, trying, discovering I was missing a package, downloading/installing, discovering I was missing a package, lather, rinse, repeat. I needed to install mysql-server-5.0, mysql-client-5.0, php5, php5-mcrypt, php5-cli, php5-gd, libgd2-xpm-dev, php5-mysql, and php5-curl. I also spent some time trying to figure out why php5 wouldn’t run scripts that were in my cgi-bin directory before realizing/discovering that with mod_php, it was supposed to run from everywhere but the cgi-bin directory.

Note that I could have done all my development on my ISP’s machines, but that seemed clunky. I knew I’d want to be able to develop offline at some point, so wanted to get it done sooner rather than later. It’s also a little faster to develop on my local system.

I did a little bit of looking around for a graphics library, but stopped when I found that PHP had hooks to the gd library. I knew that if gd had not yet incorporated my horizontal lines bug fix, then I might have to drop back to C++ in order to link in “my” gd, but I figured I could worry about that later.

Projection

I made a conscious decision to write my Mercator conversion code from scratch, without looking at my C++ code. I did this because I didn’t want to be influenced in a way that might lead me to get the same error at way-zoomed-out that I did before. I was able to equations on the Wikipedia Mercator page for transforming Mercator coordinates to X-Y coordinates, but those equations didn’t give a scale for the X-Y coordinates! It took some trial and error to that out.

Data

For the initial development, I decided to use country boundaries instead of census tract boundaries. The code wouldn’t care which data it was using, and it would be nice to have tiles that would render faster when way-zoomed-out. I whipped up a script read a KML file with country boundaries (that I got from Valery Hronusov and used in my Mapeteria project) and loaded it into MySQL. Unfortunately, I had real problems with precision. I don’t remember whether it was PHP or MySQL, but I kept losing some precision in the latitude and longitude when I read and uploaded it. I eventually converted to uploading integers that were 1,000,000 times the latitude and longitude, and so had no rounding difficulties.

One thing that helped me enormously when working on the projection algorithm was to gather actual data from Google. I found a number of places on the Google maps where three territories (e.g. British Columbia, Alberta, and Montana) came together. I would determine the latitude/longitude of those points, then figure out what the tile coordinates, pixel X, and pixel Y of that point were for various zoom levels. That let me assemble high-quality test cases, which were absolutely essential in figuring out what the transformation algorithm should be, but it was very slow, boring, and tedious to collect that data.

Polygon intersection

When it came time to implement my polygon bounding box intersection code again, I looked at my old polygon intersection code again, saw that it took eight comparisons, and thought to myself, “That can’t be right!” Indeed, it took me very little time to come up with a version with only four comparisons, (and was now able to find sources on the Web that describe that algorithm).

Stored procedures

One thing that I saw regularly in employment ads was a request for use of stored procedures, which became available with MySQL 5. It seemed reasonable that using a stored procedure to calculate the bounding box intersection would be even faster, so I ran some timing tests. In one, I used PHP to generate a complex WHERE clause string from eight values; in the other, I passed eight values to a stored procedure and used that in the WHERE clause. Much to my suprise, it took almost 20 times more time to use the stored procedure! I think I understand why, but it was interesting to discover that it was not always faster.

GIS extensions

My beloved husband had been harping on me to use the built-in GIS extensions. I had been ignoring him because a large part of the point of this exercise was to learn more about MySQL, including stored procedures, but now that I found that the stored procedure was slow, it was time to time the built-in bounding box intersection routine. If I stored the bounding box as a POLYGON type instead of as two coordinate pairs, then it took half the time. Woot!

Rendering

I discovered that despite my having reported the horizontal lines bug fifteen months ago, the gd team hasn’t done anything with it yet. Needless to say, this means that the version of libgd.a on Dreamhost has the bug in it. I thought about porting back to C++. I figured that porting back would probably take at minimum a week, and would raise the possibility of nasty pointer bugs, so it was worth spending a few days trying to get PHP to use my version of gd.

It is possible to compile your own version of PHP and use it, though it means using the CGI version of PHP instead of mod_php. I looked around for information on how to do that, and found a Dreamhost page on how to do so.. but failed utterly when I followed the directions. I almost gave up at that point, but sent a detailed message to Dreamhost customer support explaining what I was trying to do, why, and what was blocking me. On US Thanksgiving Day, I got a very thoughtful response back from Robert at Dreamhost customer support which pointed me at a different how-to-compile-PHP-on-Dreamhost page that ultimately proved successful. (This is part of why I like Dreamhost and don’t really want to change ISPs.)

Compiling unfamiliar packages can be a real pain, and this was no different. The Dreamhost page (on their user-generated wiki) had a few scripts that would do the install/build for me, but they weren’t the whole story. Each of the scripts downloaded a number of projects (like openSSL, IMAP, CURL, etc) in compressed form, extracted the files, and built them. The scripts were somewhat fragile — they would just break if something didn’t work right. They were sometimes opaque — they didn’t always print an error message if something broke. If there was a problem, they started over from the beginning, removing everything that had been downloaded and extracted. Something small — like if the mirror site for mcrypt was so busy that the download timed out — would mean starting from scratch. (I ended up iteratively commenting out large swaths of the scripts so that I wouldn’t have to redo work.)

There was some problem with the IMAP build having to do with SSL. I finally changed one of the flags so that IMAP built without SSL — figuring that I was unlikely to be using this instance of PHP to do IMAP, let alone IMAP with SSL — but it took several false starts, each taking quite a long time to go through.

Finally, once I got it to build without my custom gd, I tried folding in my gd. I uploaded my gd/.libs directory, but that wasn’t enough — it wanted the gd.h file. I suppose I could have tried to figure out what it wanted, where it wanted it, but I figured it would be faster to just re-build gd on my Dreamhost account, then do a make install to some local directory. Uploading my source was fast and the build was slow but straightforward. However, I couldn’t figure out how to specify where the install should go. The makefiles were all autogenerated and very difficult to follow. I tried to figure out where in configure the install directory got set, but that too was hard to decipher. Finally, I just hand-edited the default installation directory. So there. That worked. PHP built!

Unfortunately, it wouldn’t run. It turned out that the installation script had a bug in it:

cp ${INSTALLDIR}/bin/php ${HOME}/${DOMAIN}/cgi-bin/php.cgi

instead of

cp ${INSTALLDIR}/bin/php.cgi ${HOME}/${DOMAIN}/cgi-bin/php.cgi

But finally, after all that, success!

Bottom line

So let me review what it took to get to tile rendering:

Choose a database and figure out how to extract data from it, requiring reading and learning.
Find and load boundary information into the database, requiring trial and error.
Choose a graphics library and figure out how to draw coloured polygons with it, requiring reading and learning.
Gather test cases for converting from latitude/longitude into Google coordinate system, requiring patience.
Figure out how to translate from latitude/longitude pairs into the Google coordinate system, requiring algorithmic skills.
Diagnose and fix a bug in a large-ish C graphics library, requiring skills debugging in C.
Download and install PHP and MySQL, requiring system administration skills.
Figure out how to build a custom PHP, requiring understanding of bash scripts and makefiles.

So now, I guess it isn’t that easy to generate tiles!

Note: there is an entirely different ecosystem for generating tiles, one that comes from the mainline GIS world, one that descends from the ESRI ecosystem. I expect that I could have used PostGIS and GeoTools with uDig look like fine tools, but they are complex tools with many many features. Had I gone that route, I would have had to wade through a lot of documentation of features I didn’t care about. (I also would have had to figure out which ISP to move to in order to get Postgres.) I think that it would have taken me long enough to learn / install that ecosystem’s tools that it wouldn’t have been worth it for the relatively simple things that I needed to do. Your milage may vary.

Vincent Cheung: Encrypted Blog Posts Ver. 2

noreply@blogger.com (Vince) — Fri, 28 Nov 2008 23:03:00 +0000

Over 2 years ago, I developed an encryption system that you could use to encrypt blog posts.

Since then, I have written several encrypted blog posts that were about particularly personal things and I didn't want the general public or certain people reading them. You can ask me for one of the keys if you want, but I can't guarantee that you'll get it :p

I just released a new version of my JavaScript Encryption and Decryption system.

Try it:
Show encrypted text (the decryption key is: password)

The new encryption code is a lot faster than the old version, the webpage decryption code now uses a fancy dialog box to ask you for the key, and I fixed some bugs. The entire process has been greatly simplified and the encryption page now automatically generates code that you can copy and paste into your website.

I'm starting to actually make use of my personal website, VincentCheung.ca, which is where I'm hosting my random side projects. I'm just finishing up the new design on that site in preparation for an upcoming announcement.

Kaitlin Duck Sherwood: Cost of the bailout

ducky — Thu, 27 Nov 2008 18:48:10 +0000

There has been discussion about how the current financial system bailout is the most expensive government program ever, according to numbers from Jim Bianco (as I saw it reported by Barry Ritholtz).

Bianco’s numbers are adjusted for inflation, which is good, but that isn’t a complete picture. There are an awful lot more Americans now than there used to be. If you look at the bailout in terms of per capita cost or as a percentage of GNP, you’ll see that there were a few other programs that were comparably expensive. So yeah, it’s bad. Yeah, it’s a big deal. But we have seen worse.

Program	Inflation-adjusted cost (billions)	Cost per capita (thousands)	% of GNP
Marshall Plan	115.3	0.78	4.7%
Savings and Loan crisis	256	0.95	1.7%
Moon shot	237	1.2	4.3%
Iraq war	587	2.0	4.4%
Korean war	454	2.9	15.0%
Vietnam war	698	3.4	11.2%
New Deal	500	4.0	55.1%
Current bailout	4616	15.3	33.2%
World War II	3600	26.3	150%
Louisiana Purchase	217	40.9	NA

Notes: It was surprisingly hard to find historical GNP figures. It only started being recorded in 1947, and the sources aren’t always clear if the figures are inflation-adjusted or not. Also, most of these things spanned several years; I picked a year near the middle for the calculations. Bottom line: take the % of GNP numbers with a grain of salt. They are close, but not exact.

I used the Flow of Funds Accounts of the United States for 1947-2007, and the a very poorly annotated list from Duke for the New Deal and WW2 numbers. Sorry, there are no GNP numbers from 1803.

Kaitlin Duck Sherwood: Obama’s middle-class values

ducky — Thu, 27 Nov 2008 17:27:34 +0000

One of the things I really like about Obama is what, for lack of a better term, I will call middle-class values. He does things like clean up after himself at an ice cream shop, carry his own luggage (pictures here and here and here) and says he turns off lights and will make his kids do chores in the White House. I don’t recall ever seeing any of the presidents in my adult lifetime — Reagan, Bush 41, Clinton, or Bush 43 — ever carrying anything, even when they were campaigning. I suspect that Bush 41 never washed a dish or picked up dog poop — ever. I can’t imagine that either of the Clintons would do so now.

People in power frequently have other people do mundane things for them. There is a potential that, by doing things himself, Obama could make himself seem less powerful. Jimmy Carter once spent the night in a private home, and it was reported in all the newspapers that he made the bed himself. My recollection of that is that people were kind of incredulous at him diminishing himself that way. However, Jimmy Carter ran with a persona of folksiness. (He was Jimmy Carter, not James Earl Carter, Jr.) He had to struggle a little against being perceived as a rube, a southern bumpkin.

I don’t think Obama really risks debasing himself in the public eye by doing mundane things for himself. In contrast to Jimmy Carter, Obama has a public persona that is a bit cold and standoffish. He even got attacked for being elitist for a little while. Doing mundane things for himself counters that perception.

Maybe he carefully does these mundane things for show. Maybe he’s conscious of it and wants to “keep it real”. But maybe it’s part of his value system that he is not inherently better than other people, and should play by the same rules as the rest of the world. (Unlike, say, Arnold Schwarzenegger, who has demonstrated a pattern throughout his life of acting like rules were for other people.) I hope so.

Kaitlin Duck Sherwood: Marriage equality a threat to men’s self-image?

ducky — Wed, 26 Nov 2008 22:40:05 +0000

Salon has an interesting interview with Richard Rodriguez, who says — as I do — that the fight over “protecting traditional marriage” is really about protecting traditional gender roles. However, he spotted something that I missed: the role of male insecurity.

And the majority of American women are now living alone. We are raising children in America without fathers. I think of Michael Phelps at the Olympics with his mother in the stands. His father was completely absent. He was negligible; no one refers to him, no one noticed his absence.

The possibility that a whole new generation of American males is being raised by women without men is very challenging for the churches. I think they want to reassert some sort of male authority over the order of things. I think the pro-Proposition 8 movement was really galvanized by an insecurity that churches are feeling now with the rise of women.

I have been struck in the past at how when The Loyal Opposition talks about gay and lesbian people adopting, they usually emphasize, “a child needs a mother and a father”. It’s usually men I see saying this; Rodriguez’ interview makes me think that what they are really saying is, “Men are important! We are! We are! We are!”, trying to convince both us and themselves that it is true.

(By the way, children do just fine with same-gender parents.)

Kaitlin Duck Sherwood: Anti-marriage-equality piece reflects values

ducky — Fri, 21 Nov 2008 18:01:23 +0000

I found an anti-marriage-equality piece (via Andrew Sullivan) that was very interesting to me because of how it reflected its values.

I saw a striking example of what Jonathan Haight has found about differences in morality between liberals and conservatives. Haight found that conservatives are more likely to value “moral purity”, which basically says “if it feels icky to me, then it must be morally wrong”.

In his essay, Rod Dreher quotes University of Virginia sociologist James Davison Hunter:

“The momentum is toward experience and emotions and feelings. People are saying, ‘I feel, therefore I am.’ This is how more and more people are deciding what is real and right and true.”

Dreher complains that liberals don’t value that:

You can see this in the remarkable unwillingness of many gay-marriage defenders to grant their opponents any moral standing. To disagree with them is to reveal yourself to be a “bigot” (I heard a married, straight young Republican in Texas use that word to describe those who voted for Prop 8; he was far from the only one). Bigots are by definition people whose prejudices are irrational. Bigots are moral cretins who can’t be talked to, only coerced. One is under no obligation to compromise with a bigot, only to smash him.

I think he’s absolutely right. Liberals cannot understand the value that “if it feels icky, it must be wrong” (especially if “it” doesn’t feel icky to the liberals). Furthermore, there is no arguing with such a “moral purity” value. Joe Liberal cannot reason their way to making Joe Conservative feel less icky; Joe Liberal sees it as non-rational irrational because it is not rational by definition. It is emotional.

Dreher also laments the loss of the “meaning of marriage”:

Though no consensus on gay marriage now exists, the trend lines are not in traditionalists’ favor, in large part because our culture has lost its understanding of what marriage is for. That is, marriage no longer has a settled meaning beyond a nominalist one: it is a contract formalizing the positive emotions two people (for now) have for one another, and binding them in a legal and social framework.

I, a liberal, read that, and go, “yes, that is exactly what civil marriage is”. (I even have an old blog posting titled “Civil marriage is a contract“!)

Dreher doesn’t explain what the “meaning of marraige” is, but Andrew Sullivan (who perhaps is more familiar with Dreher’s corpus) says:

Rod longs, as many do, for a return to the days when civil marriage brought with it a whole bundle of collectively-shared, unchallenged, teleological, and largely Judeo-Christian, attributes. Civil marriage once reflected a great deal of cultural and religious assumptions: that women’s role was in the household, deferring to men; that marriage was about procreation, which could not be contracepted; that marriage was always and everywhere for life; that marriage was a central way of celebrating the primacy of male heterosexuality, in which women were deferent, non-heterosexuals rendered invisible and unmentionable, and thus the vexing questions of sexual identity and orientation banished to the catch-all category of sin and otherness, rather than universal human nature.

This is exactly what I was getting at in this post and in the first paragraph of this post. Marriage equality is not a threat to traditional marriage. It is a threat to traditional gender roles.

Kaitlin Duck Sherwood: LOLcats representing the human spirit

ducky — Thu, 20 Nov 2008 20:39:53 +0000

A while back, I wrote about LOLcats being a stand-in for ethnic groups, allowing us the humour of shared stereotypes but without having to saddle an ethnic group with those stereotypes.

Jay Dixit has a more expansive, romantic take on it: LOLcats are stand-ins for humans in all their glory and pathos. By being stand-ins, they are less emotionally dangerous:

By articulating profound feelings through cats and marine mammals speaking garbled English, we’re able to shroud genuine emotions in pseudo-irony — which means those animals can evoke deeper emotions without fear of mockery or cheapness.

I’ll put it more simply: humour is pain at a distance. Using cats (or dogs or walruses) lets us put even more distance between us and the pain. We can thus tolerate situations in LOLcats that would be too painful if it were about humans.

Hmm, I wonder if this is why animated cartoons so frequently starred animals (e.g. Mickey Mouse, Roadrunner, Foghorn Leghorn)…

Kaitlin Duck Sherwood: email tool

ducky — Thu, 20 Nov 2008 04:01:00 +0000

I spoke a while back with Jason Gallic, the Product Marketing Manager for Email Center Pro. They have a product designed for improving email-based customer service, including automatic reply templates.

Fourteen years ago, I got to use a webmail system @ATS, developed for the National Center for Supercomputing Applications by the talented Ben Johnson. (Ben? Email me!) @ATS let you set up filters that would suggest a response if the condition you specified was met. When you read a message, after the message at the bottom, there would be a few checkboxes next to titles of suggested responses. I had the option of selecting any or none of the checkboxes, then pressing either a “Send as is” button or “Edit response” button. @ATS would include the responses that I checked, and send/let me edit it.

For example, I had one filter set up to suggest the “Undergraduate admissions” answer if the word “admissions” was in the body of the message. I had another filter which suggested the “Graduate admissions answer” if the word “admissions” was in the body of the message. By reading the message, I could sometimes tell if they were interested in graduate or undergraduate admissions, in which case I would click the appropriate box and send it on. Sometimes I couldn’t tell, so I would click both boxes and send it on. Sometimes I wanted to add a little extra information that I happened to know — if, for example, they asked about who would be a good advisor for research on hydrogen embrittlement in high-carbon steels — I would check the “graduate admissions” box and add the additional information before sending it on.

I ranted to Jason about how useful auto-suggest is; we’ll see if he manages to get it into his product.

Ben Maurer: Amazon's CloudFront CDN: disappointing

noreply@blogger.com (Ben Maurer) — Tue, 18 Nov 2008 17:46:00 +0000

I took a look at CloudFront today. They have really good intentions. The CDN space is quite a mess -- it could easily be a pay-as-you-go, self-service industry. However, players such as Akamai try to make a large profit. The CDN space is especially hard for small sites -- you can't get any reasonable pricing unless you are doing high levels of traffic.

Amazon wants to change all of that. However, I think they made a number of missteps in their initial offering.

They aren't using it on amazon.com. They use Level(3)'s CDN! Why should anybody consider using a service Amazon isn't using themselves. This is a chance to prove your CDN in real life.
Tiered pricing. In a self-service model, it doesn't make sense to offer different prices for different bandwidth usages. One customer with 100 TB of traffic is the same as 10 customers with 10 TB of traffic.
Pay per request. For S3, this made sense. Every request was one disk seek on the servers, and people need to pay for that. However, in a CDN, you are expected to serve from memory. The 1 cent per 10,000 requests effectively adds 6 KB of data to every file. So if you serve a 1 KB file, this increases your cost by 6x. At the very least, the fixed cost per request should be less than that with s3 to account for the lack of disk seeks
Lack of peering. Doing a traceroute to cloudfront from a few locations (Carnegie Mellon, colos in New York and LA), it appeared that all of my traffic was going over transit links. In contrast, traffic to amazon.com went over fast and cheap peering links.

I do hope that Amazon fixes up CloudFront. It's a fantastic concept. They have the power to force reason into the market.

David Anderson: Black box says yes

David Anderson — Tue, 18 Nov 2008 03:10:09 +0000

This just in: following the discovery of the "diagnostic LED" of my black box, it took mere minutes to home in on the bug and eradicate it.

It's alive! ALIVE I SAY!

Oh, what was the bug? Let's just say that when you check, in the code of a driver, whether you properly told the power management driver to power up the chip you're driving, it would be wise to also check the code of the power management driver to make sure the power-up code is right. Because a chip with no power ain't gonna be driven nowhere.

In other news, powering up random peripherals unrelated to what you want to drive doesn't work either. No, really.

David Anderson: Debugging the NXT startup: a binary printf()

David Anderson — Tue, 18 Nov 2008 02:36:50 +0000

(Warning: very nerdy rant about very geeky topic ahead)

Debugging a NXT that crashes during the bootup sequence is hard. Before the main AVR link comes up, there is no way to even get any sound. I've already done debugging by sound: during the early stages of NxOS a couple of years back, I would debug by playing bytes I wanted to check as morse-code-like dits and daas, one bit at a time, over the brick's speaker. It's extremely basic, but it's how I got the display driver to work.

But debugging a crash before the sound driver is in a working state is hard. You have a large binary black box. Either it boots and the sound driver works, in which case you don't have a problem, or it doesn't and you only get The Beep Of Death, the sound of the coprocessor periodically blipping the speaker to say "Your OS is screwed, I'm not playing any more".

Just now, attempting to debug one such crash, I discovered something interesting. If I initialize the sound controller and start an infinite loop of playing a tone, for some reason the pitch of the Beep Of Death changes by a few kHz for 2 beeps, then returns to its regular pitch.

This gives me a more basic equivalent of the morse code byte "printer": if the tone changes, I know that the brick booted at least up to the point of my infinite loop. If it doesn't, I know it crashed before that point. It's an audio diagnostic LED that tells me either "I managed to initialize the kernel up until this point", or "Nope, the crash occurs before execution gets to the bruteforce sound loop".

Therefore, by moving the sound loop around in the init code, I should be able to zero in on the exact crash site. The initialization black box is no longer completely black. A little information leaks out. Instead of "Everything works/doesn't work", I now have "Everything works/doesn't work up to the following intermediate point of my choosing".

And, sometimes, when debugging embedded systems without proper hardware debugging hardware, that tiny insignificant diagnostic LED is the difference between hope and despair.

Manas Tungare: Software and the Democratization of Production

Manas — Mon, 17 Nov 2008 00:05:08 +0000

The availability of consumer software in this century has democratized the production of … well, everything. Parts of the current creative landscape seem no different than Marxist philosophies of workers owning the means of production, with one exception: the workers aren’t doing it for money, they’re doing it for fun.

I recently watched Be Kind, Rewind, that’s what has inspired this post — at least the spark behind it. In the movie, two video store employees recreate popular movies using a video camera when the original tapes get erased by a mysterious magnetic force. Their videos were, of course, of very low production quality, but the general idea was still valid: that amateur-grade equipment is approaching professional grade equipment.

That made me realize how easy (or at least, possible) it is to create movies with affordable software on consumer hardware. When things were still in the analog domain, you would need specialized hardware to be able to shoot on film, capture audio on expensive multi-track recording equipment, and edit it all by splicing film together. Now, all you need is a digital video camera and a general-purpose high-end computer (which, incidentally, can also be used for other tasks, so is cheap.) The barrier to entry for amateur film-makers has almost been removed.

Ditto with music production — it is possible for a musician to set up a studio in his/her basement with cheap equipment that doesn’t cost an arm and a leg. The quality of recordings made with these tools is comparable to what the studios churn out.

Publishing is no longer the domain of the publishing house — Gutenberg’s printing press now inhabits every single computer that has a printer attached to it. High quality design tools and cheap reproduction has made publishers out of everyone: flyers, posters, announcements, articles, books — all of them required professional assistance in the past. Newer genres such as wiki articles, blog posts and Usenet postings have been made possible by the Internet.

An entire “prosumer” grade of still cameras has made its way into the hands of millions of photographers. Shooting digitally has minimized the variable costs associated with photography, thus unshackling the amateur from budgetary constraints that professionals never had to bother about. That brings them one step closer to competing with professionals, e.g. by selling their shots as stock photos online.

As the economy turns digital, distribution is also taken care of democratically. Earlier, it would take a promoter, someone who could invest the initial millions, to take a creation to market. Today, it’s as easy as uploading it to YouTube or selling it on iTunes or printing a book on Lulu or making a shirt at CafePress. If it’s good, it will go viral. Simple as that.

Nowhere is this change more apparent than in the recently-concluded Presidential Election in the United States. Barack Obama is spoken of by many as the first YouTube president. Indeed, the numerous amateur videos posted by his fans to YouTube and Twitter and on their blogs played a major part in spreading the word about his ideas — in a way that pre-Internet generations could never have.

If I were an anthropologist circa 3000 AD, the last three decades would show up as a significant inflection point in a graph of human achievement and creativity.

Here’s to software!

Kaitlin Duck Sherwood: Blogs fomenting partisanship? No, conservatives.

ducky — Thu, 06 Nov 2008 01:10:25 +0000

In his post today, Scott Rosenberg suggests that there are people who blame the blogosphere for how intensely nasty and partisan our political world is right now.

Excuse me????

Partisan nastiness has been going on far longer than people have been blogging. The Web was pretty well unknown during Clinton’s first term, and in its infancy during the second. I seem to recall a whole lot of partisan bickering back then.

I don’t think that divisiveness is due to the Web, I believe that it’s due to conservatives.

That’s a little hard for me to write because I want to be fair. But I really think it is true.

I recently read an article on research in morality that points up values differences between liberals and conservatives. One thing that researchers found was that liberals put a much higher value on fairness than on group loyalty, while conservatives value them about equally. This research suggests that a liberal is more likely to sacrifice group loyalty in the name of fairness than a conservative, e.g. to help a conservative do the work to send in an absentee ballot. This research suggests conservative is more likely to toe the party line, even if he/she doesn’t believe in it.

When Palin was insinuating that Barack Obama wasn’t a “real” American, she was exploiting her white audiences’ high value on group loyalty. By making it look like Obama had a different in-group, Palin made her audience worry that they might end up as the out-group.

(Being a member of an out-group might be particularly scary if you have yourself treated out-groups unfairly. I’m just sayin’.)

I was totally unconcerned about being in Obama’s out-group. You would think that I, a 45-year old, hot, white woman with an upper Midwest accent, who lives above the 48th parallel, might identify more strongly with Sarah Palin. However, I am a liberal, and I believe that Obama is a liberal. As such, I absolutely believe he will be fair. I absolutely do not believe that Palin will be fair. And I think that is part of her appeal to her base.

(P.S. I was kidding about being hot.)

Kaitlin Duck Sherwood: Are we moving back to the US?

ducky — Wed, 05 Nov 2008 21:45:20 +0000

Several people have asked me, “So are you and Jim moving back to California now?”

The answer is “No, not yet. Maybe never.”

I had six reasons to move to Canada:

I was devastated that my fellow Americans could elect G.W. Bush for a second term. That said to me that my fellow Americans and I were not at all on the same page, and that maybe I didn’t belong in the US.
I was upset at how my government shredded civil liberties for both citizens (e.g., illegal wiretapping) and non-citizens (e.g., torture and abuse).
I was unnerved by an almost willful neglect/disinterest in some major, fundamental structural problems in the US and Californian economies. In particular, the US has been, as Lloyd Bentsen famously put it in a 1988 VP debate, been “writing hot checks” for a very long time: spending a lot but not paying enough in taxes to support those costs.
UBC was more nurturing than Stanford, my other choice for grad school.
We have lots of relatives close to Vancouver, just across the border in Bellingham.
Canada’s health system is not tied to employment. It is highly likely that we will, at some point, earning money but not be employed. Living in Canada, that’s not a problem. (Like right now. I’m looking for work and Jim is consulting.) Living in the US, that might be a problem.

The fact that my compatriots turned out in such droves for Obama lessens the feeling that I am out of step with the rest of America. I was shocked and appalled by the divisive tactics used by the McCain/Palin campaign, but enormously heartened at the number of Republicans who have publicly voiced being likewise shocked and appalled. So Obama’s election knocks off #1 pretty well.

I have finished my graduate degree, so #4 is off the list.

Our families are still in Bellingham. We could move to Seattle and be slightly closer to our families, but California would be quite a bit farther away. So #5 favours Vancouver or Seattle, but still disfavours California.

I think Obama will probably make #2 better. Issuing an executive order banning torture at one minute past noon on Jan 20, 2009 would be a good start, but to see how he does on #2, I’ll have to see him govern.

Likewise, on #3, I won’t know if he will make things better until I see him govern. However, it’s not likely that he will be able to avoid “hot checks” in his term because of the horrible horrible financial problems. He also can’t do much about California’s problems due to Prop 13.

There are more factors to consider now.

Ducky Watching Election Returns

I like many things about Canada and Vancouver.
- I have friends here. (It was really nice to watch the election last night surrounded by a bunch of friends!)
- It is really cool to live in the heart of downtown. We are able to walk to everything (so much so that we only use our car about twice per month).
- I like, in theory, that there is skiing so close. We have season passes this year to a mountain that we can see from our apartment. It takes about 30 minutes to drive there.
- By and large, Canadian government services have far better customer service than in California. It takes me about twenty minutes to renew my Social Insurance Number (like a Social Security Number in the US). It took me about fifteen minutes to move my driver’s license to BC.
It is not a perfect fit.
- In particular, I still have ambitions to change the world, while I think Vancouver puts more value on having fun. I’m trying to get the “fun” attitude, but it’s swimming upstream for me. (Hopefully the ski passes this winter will help!) Silicon Valley is all about changing the world, and so that is a huge magnet attracting me south.
- I don’t like maple syrup, I have never played hockey, and I thought Anne of Green Gables was a boring book. I did not spend many years steeped in the Canadian cultural stew, absorbing the Canadian value system, shared experiences, and etiquette. I will never be fully Canadian. (At the same time, the longer I stay in Canada, the less time I spend in the American cultural stew; the less American I become.)
Somewhat to my surprise, I discovered that I still love my country.
I am growing to love Canada.
I haven’t found a job yet.

So. Will I return to the ever return to the US? To California? I’m not sure.

Kaitlin Duck Sherwood: Prop 8 looks like it will pass

ducky — Wed, 05 Nov 2008 19:40:50 +0000

It’s looking like California’s Prop 8 is going to pass, and that’s a very sad thing.

However, I think it was far, FAR more important that Obama get elected than that Prop 8 fail. If McCain/Palin had won, we would have seen a significant shift in the Supreme Court to the right. We could have kissed goodbye to any hopes of getting marriage equality through the Supreme Court for twenty-five or thirty years.

With Obama in office, it will probably stay roughly the same in liberal/conservative makeup, but get younger. I expect that we will now see a federal Supreme Court case in five to ten years about marriage equality. And we will win that one — not just for California, but for everybody.

There is no good legal argument against marriage equality. Let me repeat that: there is no good legal argument against marriage equality. The arguments are emotional or religious, not rational. The rational arguments — the one on which our legal system is founded — say that citizens get equal protection under the law. It’s in the Constitution. It’s fundamental to the constitution. So unless the SCOTUS has people whose judgement is influenced by religion or emotion, we will win that fight. (This will be especially true after five or ten more years of seeing same-sex marriages function in Massachusetts, New York, and Connecticut, Canada, Netherlands, Belgium, Spain, Israel, and South Africa without destroying the fabric of society.)

So yes, it is disappointing. It would have been nice to put this issue to rest in California forever. However, it is not dead. We will overcome.

(Update: Andrew Sullivan has a similar post, written with eloquence.)

Kaitlin Duck Sherwood: I nominate Fivethirtyeight.com

ducky — Wed, 05 Nov 2008 18:52:54 +0000

I nominate fivethirtyeight.com for a Pulitzer Prize for their absolutely outstanding electoral poll coverage.

Vincent Cheung: Congratulations President (Elect) Obama!

noreply@blogger.com (Vince) — Wed, 05 Nov 2008 07:15:00 +0000

After months and months of campaigning, it's finally over! And the US didn't screw things up this time around! I know the polls pretty much had called the election for Obama, but I was still a bit worried that something was gonna go wrong. I was impressed by the polling; they were pretty accurate. I was following FiveThirtyEight and RealClearPolitics quite closely and while each individual poll is not that reliable, they are pretty reliable if you average them all together and weight them appropriately (as 538 did).

Anyways, nothing I can say would be anything more than what has already been said by all the pundits and newscasters. Well, except for the fact that even here in Toronto, people are very interested in the election. After volleyball, I went to the Wheat Sheaf for wings and beer, a pub that seems to have a bit of a sports inclination. Even there, the tv's (maybe 6 - 8 of them) were on CNN for the election coverage and when John McCain was giving his concession speech, the place went quiet.

Congratulations President David Palmer.... err... I mean President Barack Obama ;)

Kaitlin Duck Sherwood: Yes we did!

ducky — Wed, 05 Nov 2008 06:59:36 +0000

President Barack Obama.

Yay! Thank you, America!

Manas Tungare: A Heads-Up Display for Social Networks

Manas — Tue, 04 Nov 2008 23:12:17 +0000

I often find myself talking to people who I should know (in theory), but for some reason, in practice, my neurons refuse to make the right connections to remember these connections. Wouldn’t it be great if someone designed a heads-up display based on your social network?

This is how it would work: when I activate it, and it notices I’m talking to someone, it would do a quick scan and tell me his/her name. That would be a life-saver, and would avoid the first five minutes of the 20-Questions game I have to play every time this happens (while making sure that the other guy (or girl!) doesn’t notice I’m playing the game in my mind.)

It could also tell me how I know that person, because sometimes I remember the name, but nothing else. Wouldn’t it be helpful to know that I’m talking to John Doe, who went to the same high school as I did, and who is now President and CEO of a Fortune 100 company (note to self: graduate soon.)

Not just names, it could even tell me more about the person I didn’t already know (or, in the more likely case, I’ve forgotten.) I’d love to know that my friend John Doe is no longer with his (now ex-) girlfriend Jane, so that would cut out a lot of awkward conversation. Knowing that he just went on a cruise to Alaska would instantly give us a topic to chat about. Knowing that the lady on his arm is not his wife would probably also help. I could ask him about our common friends and if he were in touch with any of them. And then he could use his heads-up display to pull their details up and tell me what I’d already looked up, but that’s another story.

So why isn’t something like this on the market yet? I’m sure there would be throngs of people lined up outside the offices of the company that makes the first such thing. And if they try to patent it, you can cite my blog post as prior art. You’re welcome.

Update: A picture is worth a thousand words. A movie, perhaps a million?

Kaitlin Duck Sherwood: For Barack Obama

ducky — Mon, 03 Nov 2008 18:55:50 +0000

Andrew Sullivan wrote an endorsement of Barack Obama that made me cry. It wasn’t that his prose was so poetic that I got a form of Stendhal Syndrome. It wasn’t that he inspired me. It was that he reminded me, in clear and vivid detail, just how badly Bush messed up the country. He brought up all of my grief and dismay about — and all of my shame for — my government’s actions. He reminds me why I left my beautiful country.

One of my best friends is Lebanese. In about 1996, I asked him why he never talked about Lebanese politics. Had he just written it all off? No, he said that it was too painful to talk about. At the time, I didn’t understand. Now I do.

My productivity in the past few weeks has gone way down as I continually hunt for more stories about the election. It’s a destructive, addictive, action. It’s not like me reading the stories are going to change the outcome of the election. ( I voted several weeks ago, so it’s not like the stories are going to change my vote.) I know that it is pointless to read about the election, but I can’t help it, I must read — because every story that I read about Obama leading gives me a tiny flicker of hope.

I left. I turned my back and walked away. So why does it still matter? The best analogy I can think of is of being in love with an absolutely wonderful man who two or three times per year beats the crap out of me. I’ve metaphorically walked away and found another — one who is incredibly sweet and nice, but who isn’t as good a fit as my ex. There is nothing wrong with my new beau, and I admit to a little bit of excitement at something novel. But the fit isn’t quite right: he puts the toilet paper on backwards, he really likes foods I can’t stand, and he just doesn’t have the same shared context that I do with my ex. I have to keep explaining things to him that my ex understood right away. My new beau is certainly a fine and wonderful person, and I could be very content with him for the rest of my life, but there isn’t that same level of passion. Really I want a reformed version of my ex, one who fits but who doesn’t beat me.

I want Obama to win. Very much. I then want him to get my beautiful country out of this mess. (Er, these messes.) I want that very, very much. I’d like to think that someday I might have the option of coming home.