Archive for August, 2008

Ubiquity In Depth

Tuesday, August 26th, 2008

An experiment into connecting the Web with language.

Ubiquity is an experiment two parts. It’s both an interface and a development platform. Ubiquity 0.1 focuses on the platform aspects, while beginning to explore language-driven methods of controlling the browser.

Read about the release here, or download it.

In this post, we’ll talk first about the interface, and then the platform. For those who are really impatient, and just want to see how the prototype version works, check out all of the pretty screenshots and use-cases in the Ubiquity Tutorial.

The Problem: The Web is Disconnected

You’re writing an email to invite a friend to meet at a local San Francisco restaurant that neither of you has been to.  You’d like to include a map. Today, this involves the disjointed tasks of message composition on a web-mail service, mapping the address on a map site, searching for reviews on the restaurant on a search engine, and finally copying all links into the message being composed.  This familiar sequence is an awful lot of clicking, typing, searching, copying, and pasting in order to do a very simple task.  And you haven’t even really sent a map or useful reviews—only links to them.

This kind of clunky, time-consuming interaction is common on the Web. Mashups help in some cases but they are static, require Web development skills, and are largely site-centric rather than user-centric.

It’s even worse on mobile devices, where limited capability and fidelity makes this onerous or nearly impossible.

Most people do not have an easy way to manage the vast resources of the Web to simplify their task at hand. For the most part they are left trundling between web sites, performing common tasks resulting in frustration and wasted time.

A Solution: Universal Access

Ubiquity’s interface goal is to enable the user to instruct the browser (by typing, speaking, using language) what they want to do. The end goal is something like this:

A concept sketch for the future of Ubiquity.

We aren’t there yet. Instead, we have the rudimentary systems of structured natural language commands. You can select something and Ubiq “translate this to French”, or “email it to Jono”. In both cases, Ubiquity is smart enough to realize what “this” and “it” refers to, as well as knowing who Jono is (by talking with my web-mail’s contact list). It’s also smart enough to be able to understand commands like “map Chicago Comics” and “yelp Tapas near SF” and give you rich previews and search results to get you where you want to be quickly. Even better, both of those commands let you insert results directly into, say, an email you’re writing so that you never have to interrupt your chain of thought.

There’s a long way to go with this interface, though. We aren’t even prioritizing the command suggestions we give. The interface looks messy and is visually cluttered. We have made the ultimate faux pas of putting hyphens into what should be natural language commands. It’s hard to know what you can and can’t type. It’s certainly something I don’t think everyday users would be comfortable with. Yet kernel of the idea is right. It needs thought and a lot of refinement. We’ll need your help to shape the future in the web.

A big direction that we know we are going to move is suggestions based on data-type recognition. We should be able to select an address and Ubiquity should then suggest commands that make sense to apply to an address (like map it, get directions there, find restaurants near there, etc.). Similarly, we should be able to select a phone number and prompt actions like “call”, a time and date should prompt actions like “add to calendar”.

Although this starts to move into the into the direction of talking about the platform, the language-based method isn’t the only way of connecting the web. For example, Ubiquity also provides a context menu to access functionality. You can easily select some text, right click, and translate. Or put it on a map. Or look it up on Ebay.

The point is not that the context-menu is a great way of exposing functionality. It isn’t. The point is that with the Ubiquity platform, it is easy to expose functionality in a variety of ways. Given modular functionality, we are given a great expressiveness in how let users harness its power. Again, thinking of new types of interaction — universally available functionality — is where we need help.

The Problem: Extending the Browser is Too Hard

Being relatively new to the Mozilla world, we found it difficult and time-consuming to write extensions to Firefox. There is something largely last-decade about requiring restarts to add a new feature to your browsing experience. It’s ironic that the entire Web is on a push model, yet the browser—the most fundamental tool of interacting with the Web—is on a pull model.

The fundamental problem is that extending the browser, and hence the web, is too difficult. The closer new browser functionality can be packaged to look like standard HTML and JS, the larger and more diverse a community will create it. The desktop paradigm for extension development, while powerful, has a high cost of adoption. Right now we have a short tail of browser functionality with thousands of add-ons. There should be millions. We can get to that long tail using a more web-like model for functionality development — tools that are accessible to hobbyists and tinkerers, but that scales to professionals.

A Solution: The Ubiquity Platform

Ubiquity treats extending the browser like writing websites. It’s an experiment in lowering the barrier to fundendemental enhancing the browsing experiment.

We’ve tried to factor out all the boilerplate code into the Ubiquity core, so that writing a useful Ubiquity command is as simple as this:

name: "tinyurl",
takes: {"url to shorten": noun_arb_text},
preview: "Replaces the selected URL with a TinyUrl.",
execute: function( urlToShorten ) {
var baseUrl = "";
var params = {url: urlToShorten.text};
jQuery.get( baseUrl, params, function( tinyUrl ) {
CmdUtils.setTextSelection( tinyUrl );

or this

name: "insert-email",
takes: {"person": noun_type_contact},
preview: "Inserts an email address, by name, form your contact list.",
execute: function( email ) {
CmdUtils.setSelection( email.text );

You can learn how to write commands in the Ubiquity Author Tutorial. The command development API is not carved in stone; we’re hoping to get your feedback on how to make it even more convenient. Once you’ve written a command, sharing it is as easy as putting up a web page. For the end user, getting that command is as easy as bookmarking. No downloads and no clunky updates mechanism. The hope is to empower content providers with ways of bettering the web as a whole. To empower innovation at the edges.

If you have a web-application or website that provides a service, you can easily write a small amount of JavaScript glue to expose your functionality thorugh a Ubiquity command.  Ubiquity can help you extend the reach of your service beyond the bounds of your domain to find more users in the wider Web world. Ubiquity increases the surface area of innovation for the browser many-fold, by making anyone who can write simple Javascript into an agent for bettering the browser and the open Web. You are one of those agents.

If you do write any commands, please add them to the wiki-based command repository. You can also look for them on the opt-in statistics dashboard.

Get Involved

We are a virtual lab, so there are many ways to join the team to get involved:

Firefox Proposal: A Better New Tab Screen

Monday, August 25th, 2008

I’ve been thinking a lot recently about what it means to design interfaces at scale. The amount of time saved when the tasks we do often are streamlined is staggering when multiplied by a quarter of a billion users. At the moment, I’m interested in what we can streamline in the new-tab workflow.

Right now, when you open a new tab, you get a blank screen. While clean, it has a 100% probability of not getting you where what you want to be. While it’s good to not intimidate with an explosion of information, we can get a much more streamlined workflow—thereby saving huge amounts of aggregate time—by showing something. The question is, “What?”.

Let’s look at using the power of context and contextual actions to enhance the browsing experience through a smarter new tab. The main goals are to:

(1) Simplify common actions. Right now to use some text you’ve selected on a page, you have to copy the text, open a new tab, go to a new web service, paste it in, execute, wait for the page load, and the go back to the other tab. If the browser knows you’ve just selected an address and then opened a tab, it knows you’ll probably want to map it. Let’s give the user one-click access to map it.

(2) Streamline your habits. If you always visit TechCrunch after reading Slashdot, the browser can offer you one-click navigation from the new tab.

(3) Super-charge search. You often go to a new tab to start a search action: Make that front and center. Additionally, we can save the user’s time by integrate all of the services I care about (like my mail, delicious, and the web) into one search box.

Here’s a video explaining the concept:

Thoughts: How I Arrived Here

90% of the time, when I hope a new tab it’s do do a search. Why not put a search box there? The search could be Google or Yahoo or whatever your favorite provider is. As the browser, we can automatically detect this by just watching where you do your searches&mash;as the browser we can be smart. Let’s mix the results of the search engine with the results of the Awesome Bar. There are lots of ways of doing this: an easy way is to have the suggestions be from the awesome bar, and if you don’t select something from the awesome bar, you just get a search from the search provider.

Firefox Mobile Design Session: Bookmarks

Sunday, August 24th, 2008

With over two thousand watches of the find-in-page session, and a lively discussion, the experiment into broadcasting our internal face-to-face meetings is a continuing success. It exceeded our expectations for participation and yield of quality dialogue; we’ll be trying to do more of these at Mozilla—not just for Mobile—in the future.

For the third installment of the Firefox Mobile design discussions, it’s bookmark time. Bookmarks have been a fundamental feature in the browser since Mosaic days. Do they have a place in Mobile? Are there better paradigms? How are they displayed? What’s the difference between a tab and a bookmark? Is it only a matter of one being in the device’s memory?

These are some of the questions are asked, and some of questions answered in this thrilling video installment of Firefox Mobile Design Sessions. We are looking forward to hearing your thoughts. PDF of the slides are again unavailable because the whiteboard software ate them. We’ll work to resolve that for future sessions.

Firefox 3.1 Proposal: Tabs in the Awesome Bar

Friday, August 22nd, 2008

At the end of my last post on tabs, I mentioned Madhava’s concept for putting tabs into the Awesome Bar.

It’s a scalable and elegant solution. It has the hallmark of a good interface: it’s almost invisible, with no training required to gain the benefits of the feature. In one fell swoop it solves the twin problem of (1) it being easier to open a new page than it is to find an existing tab (2) the cognitive overload (and Hick’s Law penalty) of being required to think about whether something is already open in a tab, or whether I need to open it in a new tab.

I simply type where I want to go, and Firefox does the right thing.

After getting some insightful comments from Jenny Boris, who has been leading the charge in good thinking about tabs, we think it’s a feature we should try to get into Firefox 3.1. In particular, she asked me to try my hand at some mockups.

Here are three ideas on a theme; my preference is for the last. Although not pictured well, matching tabs should be ranked above plain URLs.

In terms of interaction there is only one complication: What happens if you want to open a new tab to a location that’s already open in another tab?

There are two solutions. The first is to make explicit a behavior that already exists in Firefox; holding down option while selecting a URL from the location bar opens it in a new tab. That’s the type of solution I’ve included in the mockups.

The second solution, proposed by Alex Faaborg, is much more subtle. If the user has opened a new tab, and then entered a URL, they probably mean to start a new navigation task instead of switching. Note that this solution can be used in conjunction with the first. While we both like how unobtrusive this solution, we worry that it might make Firefox feel too “magic”, or unreliable — not knowing why your browser has done something is unsettling.

Luckily, this is an edge-case that won’t happen often and any of the solutions will work well enough to be quickly polished to perfection with user testing.

My hunch is that adding tabs to the awesome bar will drastically simplify the day-to-day hunt-and-peck for tabs with a minimal change to Firefox. It’s a huge win. Another benefit is that it can work seamlessly with Weave in the future—which will be especially useful for calling up information that you were just looking at on your desktop, on your mobile device. What does everyone else think? This feature is still in fledgling mockup stage—although we hope to move it forward quickly. Madhava proposed a couple other sketch ideas for how a tab-awesome bar integration could look. We’d love other sketches if you think there are other directions to explore.

Ambient Information in the Browser

Friday, August 22nd, 2008

My browser knows a lot about me. It knows what I’m typing, what I buy, what websites I go to and how often, my calendar, and with whom I communicate. In fact, my computer probably knows more about me than I do myself. Yet, despite the wealth of information about my interests and habits, my browser rarely uses that knowledge to make my life easier.

In Mozilla Labs, we’ve been starting to think about how to harness the data the browser has to make your browsing experience better—while also maintaining privacy and security. We are particularly interested in “ambient” use of this information. That is, we don’t want to add extra features to the browser that require interaction. The data should be able to gracefully enhance our browsing experience without getting in our way.

Take, for example, RSS feeds. Despite being widely evangelized, RSS is still a technology mainly for the technorati. Why is that? Because it is a technology that is meant for consumption by computers instead of humans, and (more importantly) there is a high time-cost of setting up an RSS reader and populating it with feeds.

My browser knows which sites I visit frequently and which sites I check compulsively. Instead of manually having to subscribe to an RSS feed, the browser should augment my experience with the information it knows I want. By doing so, it takes the power of RSS and bite-sized updates and pushes it mainstream. It gracefully enhances my online experience — helping to reduce information overload, without bothering me.

Where should we display this information? Well, there’s that tempting giant white area displayed every time a new tab is opened…

There are other ways that the browser can use ambient information to silently enhance my browsing experience. The Awesome Bar is currently the best example of using information in an ambient way to enhance the browsing experience by tailoring its behavior, in a more or less tacit way, to my own. There are others. My browser knows where I read my web-mail; why not bubble the number of unread messages up into the interface (a sort of mail-pressure indicator)? My browser knows which tabs I’m switching between frequently; why not use that information to make my workflow more efficient? Why not pre-fetch the pages I visit often, and pre-render them so that my standard browsing actions become instant-fast? The browser knows I visit certain sites in the morning, and different sites in the evening. There’s something that the browser should do with that information.

One of my favorite uses of tacit use of such information is in Bret Victor’s Bart Widget, which learns where you are going from and to every day. It does this by silently watching your behavior; if you are traveling to East Bay in the morning and back to San Francisco in the evening, when you launch the widget it will default to showing the train information you most likely want.

We should be doing this kind of no-cost action more often in the browser. In Labs, we’ve got some specific thoughts of how to apply this in Firefox, but before we taint the thinking with our thoughts, I’d love to open the discussion.

How should ambient information, and all of the meta-data available to the browser, be used to enhance our everyday browsing experience?

UPDATE: Atul Varma has released Ambient News, which is a add-on that begins to explore the RSS idea from this post. In fact, this post was entirely inspired by Atul (which he says was inspired by some Humanized work we did for a company called OpenEnd—inspiration is hard to track…).

Firefox 3.1: Control-Tab Woes

Wednesday, August 20th, 2008

Atul recently had a great post about the problems of Control-Tab, which is currently in the nightly for Firefox 3.1. While I agree with everything Atul said, and I’d go a couple steps further. The problem that Control-Tab addresses exists—we waste a lot of time switching between tabs, an in particular switching back and forth between a set of tabs. It is also great that we are taking the steps to ensure a better user experience without being scared of adding animations and escaping standard widgets. Control-Tab, as it stands now, is a good starting place for discussion.

Besides the points Atul points out (unexpected results due to tab-state modality as well as breaking natural mappings), the new interface has been completely seduced by interaction seduction. It has a low information density, which is particularly apparent in that only only 3 tabs are displayed at a time, and even then only the current tab’s title is displayed. Low information density results in a high-interaction (fidgety) interface that feels constraining. It’s like trying to solve a maze in which you can never see further than the next turn: you can’t see where you are going or where you come from. I give a talk called Don’t Make Me Click, where I show a marginally humorous redesign of Google—the apropos bit starts at minute 10—you’ll see a lot of not-as-humorous parallels to the current Control-Tab design.

There are two levels of problems here. The first is of the interface for feature as-is. The second is questioning the assumption for why we need the feature in the first place, and whether we can get a better product by re-asking the question differently.


Most of the interaction problems have to do with with the visual design of Control-Tab. Let’s take a cue from Tufte to find a high information-density solution, that uses macro/mico scaling to not overwhelm.

Let me propose one solution. It draws inspiration from the semi-circle visualizations that highlight similarities.

The basic idea is to extend the tab bar down, to include thumbnails combined with the favicon (the later of which is an Alex idea). Using some sort of arrow, Firefox indicates which tab you are jumping to in the context of the current tab strip. It can also show further jumps. This attempts to solve the problem of the current interface by not providing a confusing new tab ordering by giving a strong visualization of which/where the tabs you are switching between. You also end up with a much higher information density, and the ability to easily visually browse without much interaction.

The main problem to be solved with this approach is extreme scale. What happens when you have 100 tabs, and are Control-Tab’ing between the 2nd and the 98th? There are lots of ways of dealing with this, but the easiest is ellipsification to contract the space between the separated tabs.

I’d love someone with a bit more visual design talent to take a stab.

The Bigger Picture

Jenny Boriss recently added new mockups to the design process. The design addresses many of the information-density concerns of first problem, although it doesn’t look at the problem of an ever-changing order that frustrates my spatial memory (which is my personal bane when using Command-Tab on the Mac).

Before we dive too deeply, it may also be worth looking at entirely different solutions. The solution proposed by Madhava of adding tabs to the Awesome Bar is a much more scalable and elegant solution. With barely any new interface, it provides a weak form of visual search coupled with a strong form of textual search. It also streamlines my work flow by not incurring a Hick’s Law penalty for making me think about whether something is already open, or whether I need to open it in a new tab. In addition, it solves the problem that it is easier to open a new page than it is to find an existing tab—leading to multiple duplicate tabs.

I’d add one thing to Madhava’s proposal: a bit of semantics. Let’s add a “last viewed tab” as something you can type into the Awesome bar. That way, to switch back to the last tab I was viewing, I’d hit Command-L and type “last viewed tab”, return. As the awesome bar is, indeed, awesome, soon I’d only have to type the first two characters, “la”, to get there.

Firefox Mobile Design Session: In-page Find

Saturday, August 16th, 2008

This is the second installment in the ongoing experiment to open the face-to-face design meetings we have at Mozilla. This design meeting centered around in-page find for Fennec (Firefox Mobile). This feature is as invaluable tool on the desktop, letting you home in quickly on the information you’re looking for once a search engine has got you to the right page. It’s incremental nature means you never have to type a key more than you need. Conspicuously missing from mobile Safari, incremental in-page find on mobile may prove to be even more of a killer feature than on it’s desktop counterpart.

We’ve tried to learn from the feedback to the last video (which, at the time of writing, has almost 350 views!—I was expected a number that could be counted on two hands). We changed the format of our white-boarding to be more like slides: Each board contains a title with its purpose, and has more focused content.

Video is a low information density medium, so for this video we are trying Viddler which allows viewers to make in-line comments to increase its time-information density. The hope is that you guys can annotate, call out interesting sections, or providing feedback on ideas discussed. (Thanks to Patrick and Ian for suggesting this change).

Also, feel free to comment on particular design decisions, or give your own, in the comments here. My favorite feature to come out of this session was the idea of pre-filling your in-page find with the search term, if you had just come from a search engine. When looking for a particular piece of information, the work flow is often (1) search engine the term to get to a potential page, (2) in-page find for the same term to see if the information exists. By defaulting to the search term, it halves the amount of typing (more if you have to check multiple pages), without burdening the user if the guess is wrong.

Here’s the video:

A Glipse Into The Mobile Firefox Design Process

Wednesday, August 13th, 2008

Designing at Mozilla is an interesting process: We are commited to doing design in the open and involving community in a fundemental capacity, yet we must avoid design by committee. To design in the open, we use tools like IRC, wikis, blog posts, and open bug trackers, yet we also have small face-to-face meetings. The tools we have are a great start, and I think we can do better. I’m just not sure how.

For the latest round of user experience iterations for Firefox Mobile, Madhava Enros and I are recording our discussion sessions. It’s an experiment, so want I have a couple questions for you (1) is this is a useful thing to do—do you find value in seeing into our unfiltered face-to-face meetings, and (2) In what ways could we make it more valuable to you?

Here’s the video and the whiteboard drawings for our session on Preferences, Add-ons, and Downloads in Fennec. I apologize for the poor audio quality (it gets better a couple minutes into the recording). If these turns out to be a useful thing to do, our future videos will be higher quality.