Content uploaded by Marc A Smith
Author content
All content in this area was uploaded by Marc A Smith
Content may be subject to copyright.
AURA: A mobile platform for object and location
annotation
Marc Smith, Duncan Davenport, Howard Hwa
Microsoft Research
One Microsoft Way
Redmond, WA 98052 USA
+1 425 706 6896
{masmith, duncand, a-hhwa}@microsoft.com
ABSTRACT
In this paper, we describe a system used to link online
content to physical objects implemented with commercially
available pocket computers using integrated bar code
scanners, wireless networks, and web services. We discuss
our design goals and technical architecture and describe
applications that have been constructed on this architecture.
We describe the role of the related web site to create
communities around scans collected by the handhelds.
Keywords
Laminated reality, mobile object annotation, communities,
mobile devices, bar codes, machine readable object tags,
wireless networks
INTRODUCTION
Every object has a story to tell. However, labels and signs
can only tell part of this story; there is always an enormous
amount more to learn than will fit on a label. Mobile
devices are changing this, allowing physical objects to to be
linked to associated online content. This dramatically
expands the space for commentary and services related to
the places, products, and objects that physically surround
us.
The technical process of linking physical objects to online
content has become increasingly straight-forward. Adding
a tag reading device to a network connected portable
computer shortens the gap between physical objects and
places, and the digital information related to them. This
enables wirelessly networked devices to cheaply and
accurately recognize a wide range of objects and places,
and offer access to information and services pertaining to
those objects. It seems reasonable that some form or forms
of tag detectors will eventually be common features of most
networked information devices. Currently cameras and bar
code readers are widely available for cell phones and
pocket computers.
We created just such a system that combined widely
available wirelessly networked Pocket PC handheld
computers with a laser scanner for reading bar codes.
Client software was created to integrate these components
and connect them with servers available over the public
Internet.
The resulting system has applications in many settings.
Meta-data about objects with UPC codes, found on almost
all consumer products in the United States, can be drawn
from publicly accessible online data services. These
services often provide the name of the object or product, its
size (if it has one) and the name of its manufacturer in
exchange for the object’s bar coded identifier. Our system
uses such a data service to retrieve meta-data that is then
used to construct queries for search engines that yield useful
and highly relevant results. Scanned objects quickly link
back to the web sites for their manufacturers or online
commerce sites that offer those objects for sale. Similarly,
books often bear an ISBN number in the form of a bar
code. These numbers can be used in queries to online book
sellers, making the services offered there like book reviews,
lists of related books, and, of course, purchasing available
with just one scan and a tap.
Figure 1. Mobile device hardware platforms
composed of a Toshiba e740 and a Socket
Compact Flash Bar Code Scanner.
RELATED WORK
Several projects have explored the ways objects and places
can be linked to online content and services. Ljungstrand,
et al. (2000) have built the WebSticker system to link
barcodes to web pages. This was predominantly a desktop
bound system. There is a large body of work on “context-
aware” computing (Schilit, et al., 1994). Context-
awareness refers to the identification of a user’s proximate
environment for the delivery of computing content or
services. Xerox’s PARCTAB system uses custom built
infrared transceivers to help palm-sized computers to
identify their physical environments (Want, et al., 1995).
The Cyberguide uses Palm PDA’s to provide map guides to
tourists (Abowd, et al., 1997). Positioning in Cyberguide is
provided by a combination of custom applications based on
infrared sensing (for indoor) and GPS (for outdoor). MIT
LCS’ Cricket System deploys custom built RF and
ultrasound beacons for indoor navigation (Priyantha, et al.
2000).
The CoolTown Project at HP is building context-awareness
technologies to provide web presences for people, places
and things (Kindberg, et al., 2000). Similar to the MIT
Project Oxygen (MIT, 2002), CoolTown’s main goal is to
enable future “nomadic computing” such that computing
resources follow the human user and customize the human-
computing interaction based on the local human
environment.
Our approach is a more modest and potentially more
broadly deployable in the short term. Our goal is to enable
a light weight way to both access information about
physical objects and places and to add annotations to them.
This focus is different from, but complementary to, efforts
to link physical devices, like printers or projectors to device
based user interfaces.
HARDWARE PLATFORM
The mobile component of our system integrates three core
hardware features: a laser bar code scanner, a wireless
network connection, and a PDA. There are a number of
alternative sensors that could be usefully integrated into this
system, including GPS and wireless network signal strength
detection for location information and readers for the
emerging technology of RFID tags. To date we have only
made use of bar code readers but the system architecture is
extensible, allowing these or other emerging sensor
technologies to generate information that can be used to
identify objects or places.
SERVER
The server is comprised of three components: a web
service, runtime, and local and remote data stores. The
Web Service is the channel the client uses to communicate
with the backend server. This is accomplished entirely
using remote method invocation over HTTP (“web
services”). The web service is the interface to the backend
runtime for the clients. The Runtime provides the business
logic handling event tracing, retrieval, storage, rating
Symbology-
Payload
Mapping
Input
Data
Cache
Figure 2. AURA Architecture diagram
calculations, and other tasks. The local database stores
contain user profiles, barcodes, ratings, written and speech
annotations, which are stored in a SQL2000 database.
Information on books and UPC’s are provided by multiple
remote data stores including the Amazon Web Service for
books and music, and the ServiceObjects Web Service for
UPC lookup.
MOBILE CLIENT SOFTWARE
The client is a standalone application on the Pocket PC (as
opposed to a web front-end) to support improved user
interactivity. Network connectivity is not assumed to be
continuous for the mobile client. The client application
provides queuing and retry services for the storage and
retrieval of data to and from the backend servers. These
services are not possible for a thin web based client.
Caches or local stores on the client can dramatically reduce
the demand on network access for content. In addition, a
client side application allows for a richer user interface.
This is especially true when considering delays and
intermittent network connectivity.
CLIENT INTERFACE COMPONENTS
Users can login to the system by creating a unique
username and password combination either from the mobile
device or through the web portal interface. Without an
account the device can be used to scan objects but the
device creates an Anonymous User account and all the
comments created in that context are by default public.
When a user sees an object that interests them and finds a
bar code printed or affixed to it they point the head of the
device at the bar code from a distance of about 6-12 inches
and press the scan trigger button which we mapped to the
thumb button normally used to invoke the voice recorder
feature of the Pocket PC. If the device acquires the tag’s
data, the application gives the user feedback and based on
some properties of the bar code data and sends a series of
network queries out to appropriate web services.
We have initially created or linked to services to support
three types of bar codes: tags created for a local art gallery,
UPC (Universal Product Code) codes commonly used to
tag consumer products and foods, and ISBN (International
Standard Book Number) codes for books. Any number of
additional or alternate payloads are possible within this
framework to provide services for these or other forms of
object identifiers.
These payloads are linked to the resolution service registry
which contains pairs of pattern matches and pointers to
related web resources. When a tag is scanned it is matched
to an appropriate payload on the basis of the structure of the
identifier string. For example, ISBN codes start with “978”
and have a total of 14 digits. All bar codes starting with
that series of numbers with that number of digits are
assumed to be an ISBN and are submitted to web services
that are listed in the client’s directory of resolution services
that are registered as resolving such codes. We made use of
a web service offered by Amazon.com that returns metadata
about books and music when passed an ISBN number.
Figure 4. UPC Item Display Screen.
When objects with UPC codes are scanned the system
recognizes that the code is not in other classes of codes and
submits the identifier to a UPC mapping service. We made
use of a UPC metadata service provided to the public by
Figure 3. User scenario for grocery and related
retail environments. Query highlighted the recall of
the breakfast cereal by the FDA.
ServiceObjects.Net, a commercial web service provider.
This service returns a set of meta-data about the object and
the client presents this data and creates hyperlinks to search
engines based on the results. For example, when a box of
breakfast cereal is scanned the resulting display provides
two tap access to search results, the first of which notes that
the product has been recalled due to food safety issues
related to undocumented ingredients that might cause fatal
allergic reactions for some people (figure 4 and 5).
Figure 5. Search results linked from UPC meta data
WEB PORTAL
Users can access the system through a web portal as well as
the mobile device. Users can log into the web site and view
their scan history sorted by various properties of the items.
Scans can be sorted by time, by product category (books,
food stuffs, etc.), or by the ratings or comments of other
users or data found in other systems. This creates a simple
way to assemble inventories of tagged objects, for example
a collection of books, videos or music CDs. Alternatively,
it creates a diary-like history of the series of objects
scanned while, for example, browsing through a shopping
mall or museum gallery.
CONCLUSION
A wave of annotation systems for physical objects is likely
to be about to break. Cell phones are already integrating
digital cameras and have the processing power needed to
natively decode bar codes. As pocket computers merge
with cell phones the resulting hybrids will no doubt
combine a vision system with network connectivity and
computation. The widespread distribution of such devices
is likely to have dislocating effects in many sectors of life.
Retail environments seem the most likely to change as
consumers bring the power of the Internet to bear at the
point of sale.
REFERENCES
1.Rheingold, Howard. Smart Mobs: The Next Social
Revolution, Cambridge, MA: Perseus Publishing, 2002.
2.Fiore, Lee Teirnanan and Smith, 2001
3.Service Objects Universal Product Code Web Service
http://www.serviceobjects.com/products/dots_upc.asp?bh
cp=1
4.Abowd, G. D., et al. “Cyberguide: a Mobile Context-
aware Tour guide.” Wireless Networks, vol. 3, (1997) pp.
421-433.
5.Kindberg, T., et al. “People, Places, Things: Web
Presence for the Real World.” Proceedings
WMCSA2000, (2000).
6.Ljungstrand, P. , J. Redström, and L. E. Holmquist.
“Webstickers: Using Physical Tokens to Access, Manage
and Share Bookmarks to the Web.” Proceedings of
Designing Augmented Reality Environments (DARE)
2000 (2000).
7.MIT (2002) Project Oxygen. http://oxygen.lcs.mit.edu/
8.Priyantha, N. B., A. Chakraborty, and H. Balakrishnan.
“The Cricket Location-Support System.” 6th ACM
International Conference on Mobile Computing and
Networking (2000).
9.Schilit, B., and R. Want. “Context-Aware Computing
Applications.” IEEE Workshop on Mobile Computing
Systems and Applications (1994).
10.Want, R., et al. “The PARCTAB Ubiquitous Computing
Experiment.” Technical Report CSL-95-1, Xerox Palo
Alto Research Center (1995).
11.Want, R., et al. “Bridging Physical and Virtual Worlds
with Electronic Tags.” In Proceedings CHI 1999 (1999).
