​ IMAGE Project: Tutorial

How do I use IMAGE?


Installation

The IMAGE extension is fully supported on Chrome, but also works on Edge, Brave, Safari, and Opera for Desktop or Laptop. We've noticed that you're using .

After installing the extension, a button labelled "Interpret this graphic with IMAGE" will appear after graphics, charts, and maps. You can choose a specific graphic for which you would like to have a richer experience. By activating these buttons, you'll send the graphic to the IMAGE server. The IMAGE server then sends back one or more interpretations which are presented to you either through audio, text, or haptics. Because the actual graphic is sent to a server, be sure to never use IMAGE on graphics that may contain sensitive or personal information.

If you prefer, you can open an IMAGE Extension Launch Pad by pressing the keys Alt+I on your keyboard. This will take you to a window where you can access this page if you need help, or you can open a file on your computer, or manage the extension options.

For the best experience listening to the interpretations, please use stereo headphones. Additionally, if you wish, you can use haptic devices with IMAGE. This is a work in progress, and entirely optional, but will enhance the experience!


Quickstart

Once you have the IMAGE extension installed, you may activate IMAGE on a few graphic types:

  • ... for Photos
    1. Navigate to the photo
    2. Tab to the button labelled "Interpret this graphic with IMAGE" which follows after the photo.
    3. Activate the button to open a new window with a list of interpretations.
    4. There may be multiple interpretations that are either text, audio, or haptic. You can play the audio by activating the Play button. For photos, you can also listen to a portion of the renderings.

Try it out: A busy street intersection

A street level view of downtown Toronto, Queen Street West in front of City Hall.
  • ... for Maps and Charts
    1. Navigate to the embedded chart or map
    2. Tab to the button labelled 'Interpret this graphic with IMAGE' which follows after the embedded chart or map.
    3. Activate the button to open a new window with a list of interpretations.
    4. You can play the rendering by activating the Play button.

Try it out: The Royal Tyrell Museum in Drumheller, AB.


  • Alternatively, pressing Alt-I on your keyboard will bring up the IMAGE Extension window. From there, you can get more help, open a file on your computer, or open the Options page. For an in-depth explanation of the interpretations, continue to the next section.

Understanding the interpretations

If you do not have IMAGE already installed, here are audio samples to help you understand the interpretations. Note that the audio clips are meant to be illustrative. The IMAGE project is in active development, and therefore the examples below may no longer reflect the current state of experiences.

Photos

IMAGE interprets photos as being comprised of two broad classes: the first is regions, the second is things and people.

Regions

Regions are broad areas of photos made up of similar objects, like trees in a forest, or large things, such as a wall or floor. This is to give you an idea of where the larger components of the photo are relative to other elements of the graphic, rough shape, and spatial extent.

This is what three regions sound like together:

To understand what you are hearing or feeling, imagine that, using the buzzing sound, you are stretching your arm out in front of you to feel the very edge of the region, and tracing the contours of the edge clockwise left to right, with the pitch changing as it traces down and up.

Things and people

Things and people are found by our object detection programs. Things tends to encompass a wide range of objects, and people are humans. These will generally be grouped together.

Here is an example of things and people:

There were two different thing groups, as well as people being identified in this graphic, and you should have heard their relative locations in the scene as popping noises.

Maps

IMAGE uses embedded maps from Google Maps to create point-of-interest type renderings. If you are familiar with Shared Reality Lab's Autour project, this is similar, but also a little different.

Points-of-Interest

If the embedded Google Map has a latitude and a longitude, you will be able to hear the maps with a Points-of-Interest rendering. Imagine that you're standing at the location given by the latitude/longitude and you are facing north. You'll hear a little jingle in a direction relative to due North followed by the name of a place. The volume of the jingle indicates how close it is: louder if its closer, quieter if it's farther.

Here are 5 points of interest centered around a popular tourist spot in Toronto, Ontario. Two of them are to the right, one is behind, and the other two are to the left.

Charts

IMAGE can interpret graphs and charts made using Highcharts.

Line graphs

Currently, IMAGE can turn line graphs with a single variable into spatialized audio. You will hear title of the chart, followed by the website the chart is on, and the variable being measured. You will then hear a noise going left to right as it goes from the start of the x-axis to the end of the x-axis. The noise goes up in pitch as the value it represents goes up and the down in pitch as the value goes down.

Here is an example of what you will hear.

​