February 2008 Archives

Macbook Air vs. iPhone: A Multi-touch Mystery

Continuing my theme of Macbook Air trackpad related posts, today we're going to look at some intriguing differences in behavior between the trackpad and the device that made multi-touch all the rage -- the iPhone. The particular behavior I'm talking about is the swipe behavior in Preview.app. The behavior in Preview is obviously an attempt to mimic how the iPhone allows one to browse through all of the open documents in Preview. In this example we'll use a group of photos to exhibit both behaviors.

Let's start with the iPhone.

On the iPhone, the idea is that the user is reaching out and touching the photo, and "flicking" it to the left or right. So, let's say you reach out to the right side of a photo, and slide your finger to the left. The image then follows your finger, sliding off to the left, and the next image slides in from the right. This is pretty expected -- you have flicked the photo off the screen to the left, and it follows your finger off the edge of the screen. Even if you reach out and touch the middle of the photo and flick just a little bit to the left, the same expected behavior happens. You've grabbed the photo, pushed/flicked it off to the left, and a new image comes in from the right. Of course, the images aren't unlimited and if you attempt to do this when there is not a next image the image attempts to move off to the left, and then bounces back to show you that no image to the right exists, and that you are at the end. This all makes perfect sense, is pretty intuitive, and seems like the logical behavior to model the swipe behavior in Preview after, right?

Apparently not.

As an iPhone user, I approached the Macbook Air with a great bit of excitement. While the applications that currently support gestures don't exactly make me jump for joy, the potential for these gestures in third party applications is simply put, awesome. However, I'd at least expect some consistency between the two platforms.

Unfortunately, Preview falls short.

In Preview, swipes aren't as sophisticated, and in my opinion, not as well thought out as on the iPhone. Instead of the reaching out, touching, and flicking metaphors that work so well on the iPhone, swipes on the Macbook Air's trackpad simply move in the direction of the swipe rather uninterestingly. Sure, they animate nicely, taking full advantage of Core Animation, and when they reach the end of the set the image bounces back. However, Preview decides that reality, and thus platform parity, should fly out the window.

A swipe to the right in Preview, moves to the photo to the right. A swipe to the left in Preview, moves to the photo to the left. This is the opposite behavior you'd expect if you've ever used an iPhone, and sure enough, the opposite behavior I think you'd actually want. Using a multi-touch trackpad is not analogous to using the arrow keys. I'm not trying to arrow through my pictures, my mind is expecting a higher level experience -- I am reaching out and literally swiping through my photos, just like I might turn the pages of a book. Similarly if I rotate, or magnify my pictures, I'd be expecting the computer to act like I'm picking up a photo, and pulling the photo closer to me respectively, and indeed this is what happens.

Now then, it's possible that this functionality is actually intended. If we look at another application that implements multi-touch capability, the current behavior is actually correct. In Safari, moving to the right should in fact send us forward, and moving to the left, should send us backwards. That makes sense after using it a couple of times. There are thus two arguments that can be leveraged for the case of Preview's behavior. The first is system parity. Other applications act this way, thus, Preview should as well. I'm guessing that's the one they chose, and that's why it's implemented this way. I argue that is should be changed, because it just feels wrong. Even more so after you've used an iPhone for about two minutes. The second reason, is that Preview is not necessarily running a full screen context. Thus, the idea of flicking pictures isn't especially clear. However, I get the feeling that's exactly what they were attempting to mimic. When you reach the end of a set of photos, the image has a nice bounce effect, just like on the iPhone. However it bounces the wrong way! That's right, when you get to the end of a group of photos, and continue to scroll as if another photo should be there, the image slides to the right and then bounces back. This is completely wrong. Since the photo should be sliding off to the left, it should do so, and then bounce back, very similarly to how the iPhone works.

I think this ultimately shows one of the interesting aspects of these trackpads being integrated into our computers. It's very easy to get confused about how they should work. Multi-touch shouldn't be just an alternative way to arrow through things. It's an advanced and sophisticated input method, and requires programmers to really think about how users are going to interact with their computer when using gestures. Assuming Apple continues to integrate this technology into their laptops, we will soon be at a point where multi-touch is a big deal. Both platform parity and system parity need to be considered, but more importantly a great deal of thought needs to go into this process, and whether through guidance from the top as to how users should be expected to interact with the multi-touch trackpad, or through a standards system of our own, right now it's up to the developers using the technology to shape the way users use this technology.

All in all, there are far more iPhones and iPod touches around than Macbook Airs. Far more people are going to be used to using these devices, rather than a Macbook Air, and thus I think it's clear the paradigms we should strive to mimic.

Peter Hosey has pointed out, and I agree with him really, that Safari should mimic the iPhone behavior as well. After all, who doesn't want to flick through webpages?

Bugs have been filed on Preview and Safari: here and here.

Ciaó

⌘

NSResponder Modifications: Swipe, Rotate, and Magnify

So, I'm sure you've heard of the Macbook Air and it's revolutionary multi-touch trackpad, borrowed from the iPhone technology. The basic gist of it is that it provides application specific gestures that can be triggered by performing some gestures on the new trackpad. With that in mind, every Cocoa developer should be asking themselves this question: What has Apple done to NSEvent [and friends] to facilitate gestures in their own applications and how can I do it in mine?

With the help of my local Apple Store's Macbook Air, and some NSEvent knowledge, I'm going to answer exactly that.

Alright, since we're dealing with what are presumably new events, the first place we should check is the system version of NSEvent, hoping that maybe, just maybe Apple has defined them for us. Since this a blog post of decent length, you can infer what I did -- they aren't documented at all. Hooray.

This is furthermore supported by a small test application that I wrote which logs all of the events coming out of NSApplication by replacing my NSPrincipalClass key in Info.plist with the name of my subclass of NSApplication - GestureTest. By overriding -(void)sendEvent:(NSEvent *)anEvent, and logging every event that comes out, I can get a quick overview of every single event that gets passed to my application. This is very handy in a lot of situations, particular this one. Upon closer inspection, we have five new events - three events for each gesture, and two events corresponding to the beginning and end of a gesture respectively.

The gestures are as follows, grouped with their corresponding NSEventType.

BeginGesture: 19 (I think, I forgot to double check this, woops!)
Swipe: 31
Magnify: 30
Rotate: 18
EndGesture: 20

Since these events are standard NSEvents, they are affected by flag modifiers (i.e. we can differentiate between a swipe with Command held down and a swipe with Command-Option held down). 

Thus, we have a basic way to check for these events -- we can check the flag and handle gestures appropriately. However, this is pretty inefficient - we have to check every event for this type and it isn't very flexible - we have to intercept at the NSApplication level. No good.

If you recall from Chapter 16 of Aaron Hillegass' excellent book Cocoa Programming for Mac OS X (Preorder the third edition here), every object that is going to respond to events needs to descend from NSResponder. NSResponder implements the public interface for responding to various mouse and keyboard events - our good friends -(void)mouseDown:(NSEvent *)anEvent, -(void)keyDown:(NSEvent *)anEvent, and friends. However, this documentation hasn't been updated for the Macbook Air, and as expected, the header does not reveal any additional information about these new events.

However, we have a solution for this. Steve Nygard's excellent class-dump which allows us to inspect a particular framework for it's symbols. Naturally, being the curious person I am, I ran this against the Macbook Air's version of AppKit (949.18.0 if you're curious). After a short wait (AppKit is very large after all), the results poured out into a freshly pressed text file. I immediately searched for the word "swipe", chosen completely at random. What I found, was worth it's weight in gold... assuming you want to implement gestures.

- (void)magnifyWithEvent:(id)fp8;
- (void)rotateWithEvent:(id)fp8;
- (void)swipeWithEvent:(id)fp8;
- (void)beginGestureWithEvent:(id)fp8;
- (void)endGestureWithEvent:(id)fp8;

These new methods belong to NSResponder, which means that every single responder now supports these gesture events. That's pretty awesome. My immediate reaction was to set symbolic breakpoints for these symbols and to perform some gestures using my little test application. As expected, they were hit, and I checked out the backtrace. The only thing of interest was a new private method for NSApplication -- _handleGestureEvent: 
called from NSApplication's -(void)sendEvent:(NSEvent *)anEvent method.

Just to confirm this assumption, I overrode this method and just logged events from it. Turns out the only events that come through this method are gestures. Awesome. Interestingly enough, there is a small observation to be made here. Since every gesture passes through here, one could actually replace these gesture events with any other event, so you could get some free custom gestures by replacing the gestures with say, a keyboard shortcut event before passing it down the responder chain. Nothing really fantastic, but nice to note if you feel like playing with private methods. 

So finally, we have everything we need to implement gestures for any NSResponder based object. Simply override the appropriate method for your gesture (swipe, pinch [magnify], or rotate), you can grab the value out of the NSEvent using the corresponding methods:

Swipe : -(float)deltaY; -(float)deltaX;
Magnification : -(float)deltaZ; -(float)magnification; 
Rotation : -(float)rotation;

These methods return various values for each type of event depending on the type of event, velocity of movement and/or direction of movement. Some interesting notes here - swipes can be detected to the left or right as expected, but also up and down. Pretty cool. I also suspect that the two methods under magnification actually return the same value, but I haven't checked. The value logged with the event is deltaZ, however. Rotation is pretty self explanatory. The actual values these methods return is a little bit more of a mystery. Swipe is straight forward, 1.0 for right, -1.0 for left, 1.0 for up, and -1.0 for down, depending on whether you check deltaX or deltaY respectively. Magnification and rotation both return various positive or negative values depending on the rotation or scale of magnification. You'll just kind of have to play with these, I couldn't discern a good pattern. However, there are two methods on NSEvent: -(float)standardRotationThreshold and -(float)standardMagnificationThreshold which I'm sure return some magic values that allow you to determine the angle and scale for the gestures, which I imagine can help you figure out what you need to do to your responder as a result of the gesture. Additionally, NSEvent adds -(BOOL)isGesture, which as expected, lets you determine if an event is a gesture. NSResponder also adds gesture masks via -(unsigned long long)gestureEventMask and -(void)setGestureEventMask:(unsigned long long)aMask which presumably deal with the specific masks regarding events. 

So, that should pretty much do it. You should be armed and ready to go to write some amazing code to take advantage of the trackpad inside the Macbook Air, and hopefully coming soon to a Macbook and Macbook Pro near you. Before anyone asks, this code will NOT work on current Macbook models, at least not on the current build of Leopard. It's possible when this AppKit version begins shipping standard (10.5.2, I suspect) that we may be able to fake these events with the old trackpads, depending on how much information we can squeeze out of AppKit. 

Finally, I've built a small project based on the all of this code that will allow those of you lucky enough to have your Macbook Air already to see it all in action. It's fairly simple, just filling a simple custom view with a color based on the gesture you perform, but it is set up as a great test bed for playing around with third party gestures. In addition it has a commented out private method from NSApplication to log all the gesture events if you want to see them. Obviously if you uncomment this method, the events won't be processed and the color won't change since it's not sending them down the chain. I'll leave it as an exercise to the reader to add that. ;)

Thanks to everyone at my local Apple Store for their help, and the Macbook Air. Hopefully this will help everyone get their applications ready for multi-touch trackpads before they are everywhere.

Be back soon.


« January 2008 | Main Index | Archives | March 2008 »