Bristol Wearable Computing

[ Home | Plans | Meetings | Members | Search | Papers | Links | CyberJacket | LocoMedia]


LocoMedia

What is LocoMedia?

LocoMedia is a wearable computer system that allows users to 'place' media objects such as voice notes at specific locations. These objects are subsequently 'found' by LocoMedia wearables that enter the vacinity of the object. For example a user may record a voice note and associate it with his current location just outside a baker's shop: "This shop sells great doughnuts". The voice note then becomes accessible to other LocoMedia users. When these users pass the same location, the note is played.

Project Objectives

In contrast to its desktop counterpart, the ideal general-purpose wearable computer has four distinctive characteristics: it is unconsciously wearable, environmentally aware, dynamically connected and personalized. Through LocoMedia we plan to demonstrate the value of such a wearable, thus our objectives are:

  • To demonstrate the principles of an unconsciously wearable computer.
  • To show how an environmentally aware wearable computer can interact in radical new ways with its users.
  • To show how a dynamically connected wearable computer can allow users to work effortlessly with whatever peripherals are to hand.
  • To show how a personalized wearable computer can be both an open component in a public system and at the same time a personal device.

Lets be more specific about these objectives:

Unconsciously wearable

We plan to show that a useful wearable computer does not have to offer a visually-biased, desktop style interaction. LocoMedia will initially emphasise speech and audio yet still allowing opportunistic use of visual displays and handheld input as appropriate. An audio-centric wearable is more appropriate in situations where the user must be able to see where they are going and to use their hands for other tasks. Nonetheless in other circumstances a visual channel may be useful (for example, for rapid browsing) and thus we want to experiment with what will essentially be an audio-biased but otherwise flexible device (see dynamic connectivity).

In general, there are many factors that raise people's consciousness of using a wearable, some relate to the interaction media, but others are broader. Click here to see a list of the factors.

Environmentally aware

Situated computing concerns the ability of computers to detect, interpret and respond to the local physical environment. In cases where the user's physical environment reflects their mental context it now becomes possible for the computer to automatically calculate the mental context of the user. With shared context automatically established human computer interaction becomes radically better. This is 'Situated HCI'. The effectiveness of Situated HCI depends on how accurately mental context can be deduced from the user's physical environment. This will vary across different scenarios and thus in order to evaluate Situated HCI we would like LocoMedia to support as many different scenarios as possible. For a description of different sorts of Situated Computing and a discussion of how to support them see: LocoMedia and Situated Computing. Following this discussion, here are some examples we would like to demonstrate with LocoMedia:

  1. A user standing in a square records the sound of a brass band that plays there regularly. Subsequently as other users pass the spot at different times they are still able to hear the band playing. (ta MH)
  2. A user and a friend leave a note in the foyer of a museum discussing which parts of the museum they liked and disliked. Later another user is able to plan thier visit after hearing this note.
  3. A user assigns a personal note to the logical location: "postboxes". The note says: "Don't forget to post those letters". (ta DM)
  4. Further out work. A user walks onto Park St and hears a note about the history and facilities on the street. The difficult thing here is that Park St is not a single physical point or even a fixed radius from a point, it is an irregular shaped object. How do we define such regions and then assign notes to them?
  5. A user walks up to a bus stop. The wearable says how long before the next bus will arrive and where it is going.
  6. Apative interfaces. The wearable detects conversation and suppresses the automatic playing of notes. Instead it sounds a quiet beep and waits for the user to command: "play". Or the device uses ambient sound and travel speed to decide whether the user is travelling on foot, on a bike or in a car and adjusts the range at which notes are triggered.
  7. The user asks for directions to a location. If the user is holding a display, this is used, else audio directions are relayed through the earpiece.
  8. Attach pingers to people and thence attach notes to people. Now we nearly have ContactLense.

Dynamically Connected

We plan to show how a wearable computer can work effortlessly with whatever peripherals are to hand. In contrast to the way that desktop computers work with their peripherals, there must be no need for wires, for software drivers or for any formal human-computer interaction to reconfigure the computer. Local peripherals are automatically and dynamically connected to the wearable computer in response to natural cues such as proximity or touching. Specifically we would like to build a wireless display, wireless earpiece, wireless microphone, wireless data-exchange bracelet, walk up LAN access point and walk up display and keyboard terminal. Here are the scenarios we would like to support:

  1. The user wants to stop using their wearable earpiece. They pull it out of their ear and put it in their pocket. There are no wires or buttons involved. Indeed it is not even necessary to switch off the earpiece.
  2. A user takes a handheld display out of their pocket. It automatically switches on and displays a map showing the current position (and orientation?) of the user. After use the user puts the display back in his pocket there are no wires or buttons involved.
  3. A user can touch an "information post" outside the local Tourist Information Centre and download from the centre a set of "official guide" audio notes stored on the internet.
  4. Two users shake hands. Their wearables automatically negotiate to exchange LocoMedia notes. Should the wearable look for all new notes on only ones originally created by the other user?
  5. Further out work. To record background noises a wireless microphone can be passed between users. Whoever is holding the mike is able to record the notes using the mike.
  6. Support for media other than just audio. A digital camera stores images on the LocoMedia device worn by the person holding the camera. For example, a user photographs balloons taking off at the Bristol Balloon Festival. (ta MH)
  7. A user walks up to a public display device and is able to use it with their wearable. In this case the display may be used as an electronic whiteboard, with users developing a schedule for the remainder of the day. Ok, so they will later want their small personal viewers to review what the note that they agreed!

Personalized

To show how the wearable can be both an open component of a public system and at the same time a personal device. We are not particularly concerned with the sort of personalization that is internal to individual applications or operating systems. Our concern is more about allowing users to perform what is essentially 'programming in the large' (object-level composition and management) without feeling like programmers or system administrators. Our intention is to design and develop easy ways for users to express personal policies based around the idea of publishing and subscribing to a set of domains that reflect the theme of associated LocoMedia objects.

  1. As you leave a coffee shop you pass an information post from which the company offers to add a cappucinno 'earcon' to be sounded whenever you are within range of any of their other outlets. The user is able to accept or decline this offer.
  2. There is an easy way (TBD) of establishing what LocoMedia notes are exchanged when two LocoMedia users shake hands. We need to be more specific here! The simplest approach would be for all LocoNotes to be exchanged unless marked "private". Thereafter the question is whether to enable receipients to impose further restrictions on what they receive. For example, do they want to receive third party authored notes available on the other system and how about a speech command to control the precise domain of notes to accept ("Accept Bristol Tour Guide"). The same transfer controls could apply to information posts as well as to two users shaking hands. Features like these will be necessary in order to avoid overflowing recipient devices.
  3. There is an easy way for LocoMedia users to manage the notes on their system. Note exist within a hierarchy of named domains that allows them to managed in groups. Users can delete existing notes and can chose to hear only notes within a specific set of domains. It may be that most aspects of note management are only supported using a display and pen and not using speech commands. Of all the management commands only domain labelling at the time of note creation really must be possible using speech, for example: "Record 'bristol tourist guide' note...". Further out it: should be possible for notes to exist in multiple domains. Notes are automatically registered in a domain corresponding to the user.

Making LocoMedia Usable and Useful

There are a few other user-level features that we would like to provide that perhaps don't prove anything about wearable computing, but nevertheless make the system more usable and useful.

  1. Users can visit a location "virtually", by using a map displayed on the wireless display, or by speaking the location name. This causes the trigger of any information as if they actually visited the site.
  2. Notes can be partitioned by the author. Normally, authors supply a short "title note" before the contents of a long note. Users can ask for "more" if they wish to continue listening to the main note having heard the title. (LocoMedia itself makes no distinction between such note types.)
  3. Users can also add notes to follow existing notes left by others.
  4. A 'back' command would allow users to back up through a history list and then 'more' would allow them to listen to notes they had previously skipped.
  5. Location and domain names can have aliases. Eg "Clifton Suspension Bridge" or "The Brunel Bridge".
  6. Further out work. Location names can have position dependent interpretation. Eg "visit newsagent" will make a virtual visit to the nearest newsagent.
  7. Triggering conditions can be customized by the user. Eg trigger only if user pauses at a location that has a note. Also trigger only if note has not already been heard within the last 'n' days. Or allow author or user to set a larger trigger range than average (eg, by saying 'yes' at some point as a list of ranges is offered)
  8. We can extend the system to do comparative ranking of sights rather like the Media Lab's film database. Users rank the sights as they tour the locality. A new user is provided with access to these rankings and can chose where to go. As the user enters their own opinions, these are aligned with the registered opinions and subsequent suggestions are biased towards the suggestions of users with similar preferences.
  9. Automatic keyword generation based on the content of a voice note.
  10. A display will mark all the LocoMedia objects available in the locality.
  11. It is possible to find out how many more minutes of audio recording could be added to the remaining memory.

Here are a few more LocoMedia scenarios that bring out some other value points using system features already described:

  1. User says "Visit the Bristol Museum" and is then able to hear details about the museum before visiting it.
  2. Two users arrange a liason point and time before separating. One user subsequently needs extra time and leaves a message at the appointed place, saying that they will be a little late.
  3. Further out work. With a mobile web viewer, web pages of institutions could be brought up autmotically as you pass their premises: Railway stations, Museums, Shops, Restaraunts even home pages (eg house for sale)!
  4. Wireless Microphone (FootnMike). For voice commmands we need a directional mike placed near to the mouth, yet many audio notes might actually be ambient sounds or conversations, so a separate audionote recording mike might be handy.

How will it work?

Storing, transferring and triggering LocoMedia Objects

LocoMedia objects (aka 'notes') are stored on the wearable and triggered according to location. In the first instance we want to support transfer of notes by user handshake using footbridge technology (near-field radio bracelets). We also want to be able to support downloading of voicenotes from "information posts" connected to the internet. We considered offloading LocoMedia objects to servers and then either using some intelligent caching strategy or even downloading them in real time, but this leads to additional hardware and architectural challenges that are better tackled separately from the main programme if at all.

The shape of locations

Eventually locations will correspond to user-defined rather than sensor-dictated regions, but this is seen as further work. Initially we will accept locations defined by the characteristics of GPS or of Pingers. So, a (differential) GPS location will probably correspond to a longditude and latitude defined down to minutes and seconds. Theoretically this will be a square shaped region with sides of a few metres. Pinger locations will correspond to a fixed radius about a point, though it may also be affected by interference and signal obstruction. In both cases we will plan for simple error correcting code to eliminate obviously implausible sensor data caused by noise or signal reflection.

Controlling LocoMedia with Speech

Speech control will be achieved using a speaker dependent recogniser running on the wearable computer. The recogniser will use three vocabulary banks. These will contain the fixed command set, place names and domain names. We accept that with a speaker dependent system every recognisable phrase will have to be trained by the registered user. This will mean that it will be hard for outsiders to get a feel for the system without training. Nonetheless they will be able to witness trained users demonstrating the system and this will allow us to assess how a powerful speaker independent system might feel. By chosing a speaker dependent system we can expect higher recognition rates, especially in noisy environments. Accuracy might also be improved by using a discrete opposed to continuous speech recogniser and a predefined limited grammar. The cost of this is a less naturalistic interface.

Domains

A series of heirarchical domains will be defined in order to represent an endless set of different themes. each domain represents a different 'special interest group'. Authors may publish their LocoMedia objects in one or more domains reflecting the content of the note. Consumers may subscribe to any number of domains. A 'root' domain includes every other domain, thereby providing consumers with a simple way to gain access to all notes available at a location.

The point about domains is that they allow users to stay in control of what will eventually be a highly open device connected to both the environment and the internet. Of course, domains are not the end of the story, we also need notions of acess policy that are expressed in terms of domains. One area where policies are needed is when the user comes into contact with an 'information post'. An information post is a high bandwidth connection to the internet (usually via a PC). It provides an opportunity to both load and save LocoMedia objects (and Java-style applications) onto your wearable. So the kind of policies that you might envisage include:

  • Only play notes from the 'Tourist Board' domain.
  • Only play notes from the 'my friends' domain and the 'commercial' domain.
  • If at an information post accept all LocoMedia objects from the 'Tourist Board' domain.
  • Always reject LocoMedia objects from the 'UK companies' domain.
  • When shaking hands only accept notes from the other user's domain - no third party notes.

If different users are to be able to refer to the same domains, then we need a common naming scheme. This is not to be confused with the fact that each user will need to train a speech recogniser to recognise their way of pronouncing their local name for the domain. The real issue is how do we map one wearable's domain identifier onto the equivalent identifier on another LocoMedia wearable? To solve this issue we might use CORBA object identifiers (but what happens if the wearable is disconnected from the web for a long time?) or an RMI registry object or in general use the URL of some kind of web-based domain-id name server as the first part of any domain id. This is a big issue to pursue, but initially we just need some kind of place holder.

The Speech interface

Here is the complete speech interface for the initial LocoMedia wearable:

Command grammar

  1. stop = stop | cancel | replace
    One of stop, cancel or replace is issued at the end of a piece of free audio, domain name or location name. These are standard delimiters used with different commands as described below. stop is the normal delimiter. cancel causes the preceeding command to abort. replace is used to scrap the preceeding free audio or name and replace it with what follows. replace can be used repeatedly until the user is satisfied with their entry or until they give up with cancel.
  2. publish <domain name> [stop]
    This causes all subsequently created LocoMedia objects to be associated with the domain identified by <domain name>. Note that objects can be associated with as many domains as the user likes. The association continues to apply to newly created objects until a corresponding unpublish command is issued. Note that speech interface provides no way for objects to change their associated domains after the time of creation. This can only be done from the LocoMedia manager application (see below). Note that the stop sequence is optional; after a timeout period of silence the system will say done and assume that command is completed as if the user had finished with stop. If the system does not recognise <domain name> it will fail with Sorry, I dont know the domain <domain name>. If the system is already publishing to that domain (or a parent domain of that domain) then it will silently ignore the request. Note that the system always publishes to a domain corresponding to the user. In this way users can never deny authorship of their own notes. By default this is the only domain that the system will associate with newly created objects.
  3. unpublish <domain name> [stop]
    This withdraws the association of subsequently created objects from the domain identified by <domain name>. If the domain is not currently being associated with created objects then the request is quietly ignored. If the <domain name> is not recognised then it fails with the same error message as for the publish command. Domains can be related in a hierarchy. all is pre-defined as the "root" domain. Thus unpublish all cancels all preceeding publish commands. Note however that the system will always still publish to a domain corresponding to the user (see above).
  4. [at <location name> [publish <domain name>]] note <free audio> stop
    This is the principle command for creating new LocoMedia objects. In its simplest form the user says note followed by the <free audio> contents and finally stop. The optional <location name> field is used to place the note other than at the user's current location. The optional <domain name> field is used to associate just this particular note with the specified domain as well as with any domains declared using the stand alone publish command described in 2. above. Note that the at <location name> and publish <domain name> subsequences can be repeated to register notes with multiple locations and multiple domains. In addition, cancel can be used at any stage to abort the whole command. Also, replace can be used to re-enter any preceeding name or piece of free audio in the command. Finally note that in contrast with most other commands the <free audio> must be terminated, silence will not be detected. This is so that quiet ambient sounds can be used as valid <free audio> without being mistaken for a "null" entry!
  5. cancel
    Aside from the uses given above, cancel can also be used to undo the most recent publish, unpublish, note, at or add command, or attempt to copy notes with another user or an informatin post. If successful the command is always confirmed with done, otherwise the system replies Sorry, I cant cancel the last command!
  6. stop
    This command causes the system to stop the play back of the current note and any "queued" notes that have been triggered but not yet played. Actually any utterance from the user will have the same effect, but this is the official way to do it!
  7. at <location name> [stop]
    This command allows the user to perform a "virtual visit" to the named location. If the location is not recognised the system responds with Sorry, I dont know the location <location name>! If there are no LocoMedia objects at the location the system responds with There are no notes at this location.
  8. play | more | continue
    Either resumes or starts playing the current note. The three terms are actually synonymous, but users might well use different terms depending on their context.
  9. repeat | back
    Repeats the last thing uttered by the computer. This maybe a note or a computer reponse to a command such as space.
  10. revisit
    Stops any current playback and retriggers all notes associated with the most recently visited location.
  11. pause
    Suspends playback of a note, pending play, more or continue. If a different command is issued the pause is treated as a stop.
  12. next
    This skips playback of the current note but plays any following notes associated with the same location. If there are no unplayed notes associated with the same location this functions like stop.
  13. [publish <domain name>] addition <free audio> [stop]
    Stops any current playback and adds a new note associated with the current or most recently visited location. Works in basically the same way as the note command. If the most recently visited location is named, then at...note.. would achieve the same thing. For simplicity, when a user visits a location notes are replayed in order of creation, oldest first. Issuing an addition command at a particular point in the replay of notes associated with a location does not effect the position of the note that you add in the note sequence. We may revise this in the future.
  14. subscribe <domain name> [stop]
    This command together with unsubscribe is used to control the triggering of notes. For a note to be triggered, the user must visit a location at which it is registered and they must subscribe to at least one of the domains with which it is associated. The subscribe command adds the specified domain to a list of subscribed domains. If the domain or a parent domain is already subscribed the command quietly does nothing. If the domain is not recognised, the system replies Sorry, I dont know domain <domain name>! By default the system starts off subscribed to all domains.
  15. unsubscribe <domain name> [stop]
    The unsubscribe command removes the specified domain from a list of subscribed domains. If there is no reference to the domain or any of its parents on the list of subscribed domains, then the command is quietly ignored. If the domain is not recognised, the system replies Sorry, I dont know domain <domain name>! unsubscribe all can be used clear the entire list of subscribed domains. Pending a subscribe command you would hear no further notes, even ones belonging to the user's own domain (in contrast to unpublish all)
  16. new location <location name> [stop <location name> [stop] ]
    This command allows the user to add a name for the current location. The speech system requires that new vocabulary is entered twice for training purposes, hence the format of this command. [In the interesting case where the location is indicated by more than one sensor (eg both a pinger and a GPS fix) we might prefer to associate the location with the GPS fix since the pinger may actually be pinned to a mobile object such as a person, but this isnt very elegant. There are two deeper issues here; firstly if we label a location L where Sensor1=x and Sensor2=y, then if in future we detect Sensor1=x but Sensor2=? or even worse, Sensor2=z then what do we deduce? Secondly, by using the term location opposed to say person in the command syntax we are adding unfortunate semantic overtones. Specifically had we used the term person then we would prefer to use the pinger data since pinger may be attached to people whereas GPS data is fixed at one location. How do we solve these two issues?]
  17. space
    This command allows users to find out how much storage space remains on their wearable computer. The computer replies: There is space for another <n> minutes of notes or when less than a minute remains: There is only space for another <n> seconds of notes

Responses / queries

  1. done
    In reponse to a successful cancel command or following a Copying notes message if all notes were successfully copied.
  2. Sorry, I didnt understand that!
    In response to an utterance that it could not understand or that didnt comply with the grammar described above.
  3. Would you like to accept notes to do with <domain name>?
    If a user contacts another user or an information post, the system can be configured to check with the user before accepting notes from specific domains. If the user responds with an unrelated command, then the notes are declined (why not!) If the user does not respond within a preset period (5 seconds) the system again declines and reponds with...
  4. I assume not
  5. There are no notes at this location
    In response to an at command
  6. There is space for another <n> minutes of notes or ...seconds... when less than a minute
    In response to a space command
  7. Sorry, there is no more space for new notes!
    When a note (or addition) has to be cut short or rejected entirely through lack of space.
  8. Sorry, I dont know the domain <domain name>!
    In response to a note, addition, (un)publish or (un)subscribe command. Ideally the system will interject before the user has attempted to add an entire piece of free audio. Or even better it will accept the note and then allow the user to re-enter the domain name.
  9. Sorry, I dont know the location <location nme>!
    In reponse to a note, addition, at command. Again we must protect the user against the possible loss of a just recorded note.
  10. Sorry, I cant cancel the last command!
    In response to a cancel command
  11. Copying notes
    To indicate that notes are being transferred between the user and another user or an information post
  12. Copy interrupted
    To indicate when communication was abandonned with notes not completely transferred.

The LocoMedia Manager Application

There are other commands to do with more detailed management of LocoMedia objects that are not supported through the speech interface, but through a graphical application called the LocoMedia Manager. This application can be used to:

  1. run new users through command, domain name and location name training
  2. delete notes
  3. change the location and domains associated with a note.
  4. change the playing order of notes associated with the same location
  5. define new location names
  6. relate "physical" locations to positions on a map
  7. relate "logical" locations to other "logical" locations and/or physical locations.
  8. remove, rename or alias location names
  9. define new domain names
  10. relate domains to other parent or child domains
  11. remove, rename or alias location names
  12. Further out work. associate additional media types and URLs with locations
  13. Allow definition of shaped regions as logical locations.

The exact form of this interface is TBD.

The method of invoking this application is also TBD.

Plans

The basic components of the LocoMedia system working with a handheld display are: (click for clearer view)

For plans click here.

Related work

George Fitzmaurice at the University of Toronto has done work on "situated information spaces" [1]. His work focused on a small hand held display with a 6D local position sensor. The thrust is on the labelling of 'objects' rather than places. Though he also labelled places on a wall map. The main differences are in that we are considering doing this for a larger physical space (Bristol) and so this exposes a different set of application values. And that we are contemplating a system where the information consumers can also be information creators, adding their own annotations to places they ahve visited. In this sense what we have is a tool that allows users to create new application values and to use the device for messaging and reminders.

The CyberGuide at Georgie Tech [2] provides audio information for visitors to Georgia Tech. Again there was no facility for visitors to add information at will.

The Lancaster University has also done a tourist system [3], and goes one step further in that the information is stored on the web rather than locally on the wearable, but even so, there is still no facility for tourists to add their own notes into the system.

The Esprit HIPS programme goes all the way in proposing an extensive system that will allow users to add their own media dynamically. This system is not yet complete - the programme is in the first of three years. When this system is complete it will subsume the functionality of the LocoMedia system we are proposing here!

At HP Labs Bristol we have built a prototype situated reminder system based on both audio and text notes and using the Pinger active badge system [4]. Setting aside the incidental application domain of reminders, this system is very similar to the LocoMedia system. The main extension is that notes in LocoMedia can be shared amongst different users. This brings in notions of domains and keywords to organise the notes. The LocoMedia system is also more evolved in that it will incorporate sensor fusion (active badge and differential GPS data is combined) and dynamic connection to communication and IO peripherals.

Also at HP Labs Bristol the (now deceased) Crescent project supported the notion of location-oriented web servers, where web pages were presented as hyperlinks from a geographical map of a particular region (and some other organisation actually built a generic server system for such sites). This system was not necessarily to be linked to users with wearable computers. It could as readily be used from a desktop PC by a would-be visitor to the particular region.

Brad Rhodes has built a Remembrance Agent (RA) and has suggested its extension to wearable computing [5]. Our system can be viewed as an audio remembrance agent, triggered by location. Compared to the work of Fitzmaurice, the RA perspective emphasises that the creator and consumer of notes may be the same person but at different times. The initial RA implementation was entirely tetxually based with no location sensing. Barry Crabtree at BT is building a wearable version with location sensing.

[1] Fitzmaurice G., Situated Information Spaces and Spatially Aware Palmtop Computers, Communications of the ACM, July 1993. Vol 36, no 7.
[2] Long, S. et al, Cyberguide: Prototyping Context-Aware Mobile Applications, CHI 96 Vancouver, proceedings ACM.
[3] Davies, N., Developing a Context Sensitive Tourist Guide, 1st Workshop on HCI for Mobile Devices, Glasgow, 1998.
[4] Hull, R., Neaves, P., Bedford-Roberts, J., Towards Situated Computing, ISWC 97 Boston, IEEE Comp.Soc. Press, 1997.
[5] Rhodes, B.J., The Wearable Remembrance Agent: A system for augmented Memory, ISWC 97, Boston, IEEE Comp.Soc. Press, 1997.


ęCopyright Hewlett-Packard 1997, 1998.
The material displayed is provided 'as is' and is subject to use restrictions. For problems or queries about this web contact web master. Last updated: 14 Jan 2000.