​ IMAGE Project: Technical and Developer Information

Technical and Developer Information

Jump to a section on this page:


About IMAGE

IMAGE is an open source project. If you want to modify it for your own uses, or extend it for creating new experiences, our code is available on GitHub for both the Server components and our Chrome browser extension. Please review the README files for the preprocessors in the IMAGE server repository for up to date information on the status of each preprocessor.


Technical overview

When you use the IMAGE extension to select a graphic on a website, it is sent to a server, where a set of microservices (running in a Docker environment) try to figure out what type of graphic it is and what it contains. These "preprocessors" are flexible, and can include, for example, machine learning (AI) models running on the server to determine what objects are in a photograph, or remote databases containing information about the region in a map embedded in a webpage. This information is conveyed as a structured JSON output to another set of microservices we call "handlers", which create thee actual experiences for users. The handlers make use of additional microservices such as an audio rendering engine called SuperCollider that creates spatialized audio sound effects. When the handlers are done, they send the created experiences back to the client app, typically the IMAGE Browser extension, or potentially another device like an iPhone (under development), or touch-based device like a Monarch Braille tablet (also under development).

Since we know that our small team will never be able to create experiences for all kinds of graphics, we have built IMAGE to be extensible. You can create your own preprocessors and handlers, building on top of the ones already created, to focus on just the content that matters most to you. If you are thinking about using IMAGE as a foundation for you project, we would love to hear from you, since we would be delighted to see IMAGE become a playground for researchers, developers, designers and others to create and deploy new ways to help make web graphics more accessible.

For detailed information on the IMAGE architecture, we recommend reading our paper to get an overview of the system as a whole, then consulting the wiki for the IMAGE server repository for more specific implementation details.


Research goals

IMAGE is an ongoing project at McGill University in the Shared Reality Lab. In addition to the core software development and maintenance, we also have multiple ongoing research directions. These are constantly changing as people come in and out of our lab, but in general, we seek to:

  • run user studies looking at the best ways to convey information, which can vary depending on the user's background and needs. For example, can we make a photograph more comprehensible to someone who is blind from birth by transforming the photo into a soundscape based on the relative "depth" of the object it contains, rather than just their location on a 2D plane?
  • optimize learnability of rendering techniques, such as spatialized audio, for people who are not familiar with it
  • experiment with different haptic hardware devices, such as the Haply 2diy force feedback device and the Monarch pin array tablet. Although these are not yet widely available to end users, incorporating touch in conjunction with audio to convey relationships is a very promising direction.
  • seek to create tools to support designers in creating haptic effects without the need for programming. For example, can we use machine learning techniques to guide a haptic designer in creating effects? Can we create a visual representation of haptic effects to help designers more quickly and easily understand the results of their design changes?
  • work with teachers and students who want support for using graphics in a classroom environment. How would we link multiple haptic devices in a classroom to the teacher's IMAGE extension, and create tools for teaching multiple students with these devices? Can we create tools that let teachers manually modify the experiences that IMAGE creates, to tune them for their specific classroom needs, and fill in details that automated methods still miss?


Publications

We look forward to a future where web graphics can be displayed using a variety of tools to match the different needs, and hope that IMAGE can be central part of that ongoing story. If you are a researcher and would like to collaborate, or even just find out more about how IMAGE could support your own research, please get in touch with us!


Extension

Extension (Chrome web store)

GitHub Repositories

Server

Browser

Monarch

iPhone

Safari for iPhone