Systems and methods for immersive interaction with virtual objects

• US 8,009,022 B2
• Filed: 07/12/2010
• Issued: 08/30/2011
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Holmdahl, Todd Eric, Tsunoda, Kudo, Clavin, John, Kipman, Alex, Perez, Kathryn Stone
• Primary Examiner(s): Bugg; George
• Assistant Examiner(s): OTT, FREDERICK R
• Application Number:
  US12/834,723
• Publication Number:
  US 20100302015A1
• Time in Patent Office: 414 Days
• Field of Search: 463/30, 463/36, 463/47
• US Class Current: 340/407.1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/302 specially adapted for receiving control signals not targeted to a display device or game input means, e.g. vibrating driver's seat, scent dispenser
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A system to present the user a 3-D virtual environment as well as non-visual sensory feedback for interactions that user makes with virtual objects in that environment is disclosed. In an exemplary embodiment, a system comprising a depth camera that captures user position and movement, a three-dimensional (3-D) display device that presents the user a virtual environment in 3-D and a haptic feedback device provides haptic feedback to the user as he interacts with a virtual object in the virtual environment. As the user moves through his physical space, he is captured by the depth camera. Data from that depth camera is parsed to correlate a user position with a position in the virtual environment. Where the user position or movement causes the user to touch the virtual object, that is determined, and corresponding haptic feedback is provided to the user.

Claims

1. A system for providing non-visual sensory feedback to a user interacting with a virtual object, comprising:

a capture device that captures data corresponding to a user location;

a feedback device that provides non-visual sensory feedback to the user; and

a component that predicts that a user is likely to interact with the virtual object at a future time based on at least a partial skeletal model of the user generated from the captured data;

that, in response to initially predicting that the user is likely to interact with the virtual object at the future time, computes a non-visual sensory feedback for the predicted future interaction; and

that indicates to the feedback device, prior to the future time, to provide the computed non-visual sensory feedback to the user at the future time.

2. The system of claim 1, further comprising:

a three-dimensional (3-D) display that displays the virtual object to the user.

3. The system of claim 1 wherein the non-visual sensory feedback comprises haptic feedback or olfactory feedback.

4. The system of claim 3 wherein the virtual object corresponds to a second user interacting with the user across a communications network, and the feedback device that provides haptic feedback to the user further provides haptic feedback to the user corresponding to a haptic feedback received by the second user.

5. The system of claim 3 wherein the haptic feedback comprises one from a set, the set comprising haptic feedback produced by an ultrasonic wave, haptic feedback produced by a stream of air, and haptic feedback produced by a device coupled to the location of the user.

6. The system of claim 3 wherein providing haptic feedback to the user comprises:

providing haptic feedback to the user at the location of the user.

7. The system of claim 1 wherein the component that determines that the location of the user corresponds to interacting with a virtual object, and that indicates to the feedback device to provide feedback to the user further:

determines a characteristic of feedback to provide to the location of the user based on at least one quality of the user interaction with the virtual object.

8. The system of claim 7 wherein the quality of user interaction comprises a strength of the user, a task being completed, a velocity of the virtual object or the user, a mass of the virtual object or the user, or a material of the virtual object.

9. The system of claim 1 wherein the feedback device comprises at least one audio speaker, and non-visual sensory feedback comprises:

directed audio from at least one audio speaker substantially centered on the user.

10. The system of claim 9 wherein a second user is present in a scene of the first user, and the directed audio is directed to the user such that it is understandable exclusively to the user.

11. A method for providing non-visual sensory feedback to a user interacting with a virtual object, comprising:

receiving data captured by a capture device, the data reflecting a user location;

predicting, based on the data, that a user is likely to interact with the virtual object at a future time based on at least a partial skeletal model of the user generated from the captured data;

computing, prior to the future time, a non-visual sensory feedback for the predicted future interaction; and

providing the computed non-visual sensory feedback to the user at the future user location at the future time.

12. The method of claim 11, further comprising:

displaying the virtual object to the user via a three-dimensional (3-D) display, wherein the 3-D display is one from a set, the set comprising;

an active 3-D display and a passive 3-D display.

13. The method of claim 11, wherein the non-visual sensory feedback is one from a set, the set comprising:

haptic feedback, and olfactory feedback.

14. The method of claim 13, wherein an amount of haptic feedback corresponds to a strength of the user.

15. The method of claim 13, wherein a type of haptic feedback corresponds to one from a set, the set comprising:

a velocity of the virtual object or the user, a mass of the virtual object or the user, and a material of the virtual object.

16. The method of claim 13, wherein the haptic feedback corresponds to Braille.

17. The method of claim 11, wherein the feedback corresponds to a task being completed.

18. The method of claim 11, wherein the location of the user corresponds to a representation on a display device and the virtual object is displayed on the display device.

19. A computer-readable storage medium excluding signals bearing computer-readable instructions that, when executed on a processor, cause the processor to perform operations comprising:

displaying a virtual object to the user via a three-dimensional (3-D) display;

receiving data captured by a capture device, the data reflecting a user location;

predicting, based on the data, that a user is likely to interact with the virtual object at a future time based on at least a partial skeletal model of the user generated from the captured data;

computing, prior to the future time, a non-visual sensory feedback for the predicted future interaction; and

providing the computed non-visual sensory feedback to the user at the future user location at the future time.