Interacting with an omni-directionally projected display

• US 8,928,579 B2
• Filed: 02/22/2010
• Issued: 01/06/2015
• Est. Priority Date: 02/22/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew David, Benko, Hrvoje
• Primary Examiner(s): ABDIN, SHAHEDA A
• Application Number:
  US12/709,799
• Publication Number:
  US 20110205147A1
• Time in Patent Office: 1,779 Days
• Field of Search: 345156-157, 345/175, 345/633, 345/168, 345/173, 704/275, 715/866, 463/9, 463/4, 463/30, 463/31, 463/36
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Concepts and technologies are described herein for interacting with an omni-directionally projected display. The omni-directionally projected display includes, in some embodiments, visual information projected on a display surface by way of an omni-directional projector. A user is able to interact with the projected visual information using gestures in free space, voice commands, and/or other tools, structures, and commands. The visual information can be projected omni-directionally, to provide a user with an immersive interactive experience with the projected display. The concepts and technologies disclosed herein can support more than one interacting user. Thus, the concepts and technologies disclosed herein may be employed to provide a number of users with immersive interactions with projected visual information.

Claims

1. A method for detecting interactions with an omni-directional display, the method comprising:

projecting visual information onto a display surface, the visual information being projected by a projection system;

emitting infrared energy toward the display surface;

capturing an image at an image capture device, the image comprising data corresponding to an amount of reflected infrared energy;

analyzing the captured image to detect a movement of a structure located between the projection system and the display surface;

in response to detecting the movement of the structure, analyzing the detected movement to determine if the movement comprises a recognized control command;

in response to determining the movement comprises the recognized control command, determining if a voice command is relevant to execution of the recognized control command;

in response to determining the voice command is relevant to the execution of the recognized control command, determining if the voice command has been received;

in response to determining the voice command has been received, determining if the received voice command is consistent with the recognized control command and if the recognized control command is consistent with the received voice command; and

executing recognized control command if the received voice command and the recognized control command are consistent with each other.

2. The method of claim 1, further comprising ignoring the recognized control command until the voice command is received and is consistent with the recognized control command.

3. The method of claim 1, further comprising in response to determining the detected movement comprises the recognized control command, determining a location of the control command with respect to the projected visual information.

4. The method of claim 3, further comprising determining how to manipulate the projected visual information in response to the recognized control command and the location of the recognized control command, and manipulating the projected visual information in accordance with the determining.

5. The method of claim 4, further comprising determining if movement of the recognized control command has ceased.

6. The method of claim 5, wherein the control command comprises a zoom command, and wherein the method further comprises accelerating the rate at which the zoom command is applied to the projected visual information in response to determining that the movement of the recognized control command has not ceased.

7. The method of claim 3, further comprising:

assigning a gesture point comprising a portion of the projected visual information corresponding to the location of the control command with respect to the projected visual information; and

manipulating the projected visual information by moving the gesture point with the detected movement.

8. The method of claim 1, further comprising:

in response to detecting a plurality of movements and determining that the plurality of movements correspond to a recognized control command, assigning a plurality of gesture points, each of the gesture points comprising a respective portion of the projected visual information and corresponding to a location of a respective one of the plurality of movements; and

manipulating the projected visual information by moving each of the plurality of gesture points with the respective one of the plurality of movements.

9. The method of claim 1, wherein the projecting comprises projecting the visual information onto a display surface, the display surface comprising walls of a room.

10. The method of claim 1, wherein the projecting comprises projecting the visual information onto a display surface, the display surface comprising walls and a ceiling of a room.

11. An omni-directional display system for detecting interactions with an omni-directional display, the system comprising:

a detection system having an infrared energy emitter configured to emit infrared energy toward a display surface, and an image capture device configured to capture an image corresponding to an amount of reflected infrared energy; and

an image analysis system configured to

analyze the captured image to detect a movement of a structure located between the infrared energy emitter and the display surface,

analyze the detected the movement of the structure to determine if the movement comprises a recognized control command,

in response to determining the movement comprises the recognized control command, determine if a voice command is relevant to execution of the recognized control command,

in response to determining the voice command is relevant to the execution of the recognized control command, determine if the voice command has been received,

in response to determining the voice command has been received, determine if the received voice command is consistent with the recognized control command and if the recognized control command is consistent with the received voice command, and

execute the recognized control command if the recognized control command and the received d voice command consistent with each other.

12. The omni-directional display system of claim 11, the image analysis system being further configured to determine a location of the movement of the structure with respect to the projected visual information.

13. The omni-directional display system of claim 12, further comprising a control system in communication with the image analysis system, the control system being configured to determine how to manipulate the projected visual information in response to the recognized control command and the location of the recognized control command with respect to the projected visual information, and to manipulate data corresponding to projected visual information in accordance with the determining.

14. The omni-directional display system of claim 13, wherein the image analysis system is further configured to ignore the recognized command until the voice command is received and is consistent with the recognized control command.

15. The omni-directional display system of claim 13, wherein the control system determines how to manipulate the projected visual information by

analyzing information indicating a location of the control command with respect to the projected visual information,

determining if the recognized control command has been released, and

manipulating the projected visual information in accordance with the analyzing and the determining.

16. A method for detecting interactions with an omni-directional display, the method comprising:

projecting visual information onto a display surface, the visual information being projected by a projection system;

emitting infrared energy toward the display surface;

capturing an image at an image capture device, the image corresponding to an amount of reflected infrared energy;

analyzing the captured image to detect a movement of a structure located between the projection system and the display surface;

in response to detecting the movement of the structure, determining if the movement corresponds to a control command used to manipulate the projected visual information and if so, determining a location of the control command with respect to the projected visual information;

analyzing the control command and the location of the control command to determine how to manipulate the projected visual information;

analyzing the captured image to detect movement of a structure between the projection system and the display surface;

in response to detecting movement, analyzing the detected movement to determine if the movement comprises a recognized control command;

in response to determining the movement comprises the recognized control command, determining if a voice command is relevant to execution of the recognized control command;

in response to determining the voice command is relevant to the execution of the recognized control command, determining if the voice command has been received;

in response to determining the voice command has been received, determining if the received voice command is consistent with the recognized control command and if the recognized control command is consistent with the received voice command

manipulating data corresponding to the projected visual information; and

projecting the manipulated projected visual information toward the display surface.