I attended Apple’s WWDC 2013 event in San Francisco last week, and I spent a lot of time talking to fellow developers and designers. As we shared tips and techniques, there were a few topics that repeatedly came up regarding my Halftone 2 app. This post covers the user interaction model for graphical elements like this BOOM stamp:

Hopefully, I can inspire others to experiment with new techniques and to reevaluate mouse input methods that many of us take for granted.

Background

Halftone 2 uses an enhanced version of the interaction model that was used in the original Halftone. That model was the result of reconsidering how to manipulate objects in a touch-first environment. From an earlier post, titled I Have No Idea:

In almost every graphics application, you know that you can click on an element to enter an editing mode and use “handles” to scale and rotate the element. These are such common interactions that you might not even think about it. Step back for a moment, though, and you’ll realize that these handles were created for your mouse, and over the years, you’ve been trained to adapt to the computer more than the computer has adapted to you.

In Halftone and Halftone 2, you move a graphic element by dragging near its center. To resize or rotate an element, you drag near its outer edge. While this may not be obvious without experimentation, once you’ve seen it demonstrated (via an in-app tip or video), it’s an easy interaction to recall. These are similar to the actions you’d perform if you manipulated a physical piece of paper on a desk, and in fact, that’s how it was developed. I frequently receive e-mail about the simplicity of this technique.

For users who skip the in-app video (a sizable percentage), the standard pinch-to-zoom/rotate gesture is also supported, and they likely don’t know that an alternate interaction method even exists.

In the original version of Halftone, only stamps (like WHAM, BAM, etc.) use this technique. All stamps are drawn as vectors using Quartz 2D, and in addition to the drawing commands, there is also a “touch path” that defines the touchable region of the element. This is necessary, because some of the stamps have fine lines, and simply detecting the alpha channel of the generated artwork doesn’t work well with comparatively large finger tips. Unfortunately, this also means that new stamp shapes need a custom touch path, and bitmap stamps have the same limitation.

A “swoosh” stamp from Halftone 2 with relatively thin lines:

For Halftone 2, I knew that I needed to remove the touch path limitation (so that users can eventually add their own artwork), and I wanted the technique to work equally well with bitmap elements and captions/balloons. While the current version of Halftone 2 doesn’t contain any bitmap elements, support is already built in, and a related feature will be enabled in a future update.

Implementation

Without going into too much detail, all graphic elements in Halftone 2 inherit from a JBGraphicElement base class, and their associated views inherit from JBGraphicElementView. The interaction model is implemented in JBGraphicElementView, so all elements inherit the same behavior.

Like the original version of Halftone, all elements in Halftone 2 are drawn as vectors using Quartz 2D. I’ve created a custom vector file format that is encapsulated in a JBVectorImage class. After creating a vector image ([JBVectorImage imageNamed:@”file.jbv”]), the image can be rendered to any graphics context with a call to its renderInContext: method. This rendering path is used by JBVectorImageView to display the element on-screen (as a cached bitmap for faster manipulation) and during final output to ensure that the vectors render at the highest possible quality.

To detect touches, JBGraphicElementView uses another class called JBAlphaMap that creates a scaled, bitmasked version of the rendered element’s alpha channel. This is done for a few reasons.

First, obtaining the alpha channel of a bitmap image on iOS requires drawing that image to a CGContext. This isn’t something that you want (or need) to do frequently, especially while you’re detecting touches and trying to maintain a responsive user interface. JBAlphaMap performs this step once during initialization.

Second, the resolution of the alpha map doesn’t need to match the high resolution of the original image. Fingers aren’t precise instruments—certainly not to the pixel level. Even when the element is scaled up, this loss of “detection resolution” isn’t noticeable. So, JBAlphaMap scales down the original image for mapping.

The scaled-down alpha channel of the BOOM bitmap image:

Third, if the alpha channel is used in its unmodified state, you run into the problem that I mentioned earlier: fine lines and detail become difficult to touch. To mitigate this issue, JBAlphaMap runs the scaled down version of the original image through a GPU-accelerated blur filter before converting the alpha channel to its internal bitmasked format. The logic is simply: if the alpha channel of the blurred image is non-zero, set its internal alpha bitmask to 1. This technique makes fine lines and image detail easy to touch, as most of the detail simply blends together into a larger region.

The blurred alpha channel:

A visual representation of the internal JBAlphaMap bitmask:

Finally, JBAlphaMap exposes a simple API that makes it easy to test for touches. A call to pointIsOpaque:forBounds: scales the incoming point and tests against its internal bitmask to detect a touch.

During interaction, JBGraphicElementView asks JBAlphaMap if the user is touching the element, and if so, it computes the distance from its center to the users touch point. If the touch point is within a certain distance, the touch is detected as a move gesture. Otherwise, it is considered a rotate/resize gesture.

The blue area represents the touch region that initiates a move gesture:

For elements like balloons and captions that dynamically change their shape during resize, the JBAlphaMap is invalidated and regenerated with a new bitmask representation. This ensures that the touch information is always up-to-date.

Feel free to contact me if you have other topics that you’d like me to cover.