CONTROLLING A VIRTUAL OBJECT WITH A REAL CONTROLLER DEVICE

• US 20130328762A1
• Filed: 06/12/2012
• Published: 12/12/2013
• Est. Priority Date: 06/12/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: McCulloch, Daniel J., Navratil, Arnulfo Zepeda, Steed, Jonathan T., Hastings, Ryan L., Scott, Jason, Mount, Brian J., Hirzel, Holly A., Bennett, Darren, Scavezze, Michael J.
• Granted Patent: US 9,041,622 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Technology is described for controlling a virtual object displayed by a near-eye, augmented reality display with a real controller device. User input data is received from a real controller device requesting an action to be performed by the virtual object. A user perspective of the virtual object being displayed by the near-eye, augmented reality display is determined. The user input data requesting the action to be performed by the virtual object is applied based on the user perspective, and the action is displayed from the user perspective. The virtual object to be controlled by the real controller device may be identified based on user input data which may be from a natural user interface (NUI). A user selected force feedback object may also be identified, and the identification may also be based on NUI input data.

Claims

1. A method for controlling a virtual object displayed by a near-eye, augmented reality display with a real controller device comprising:

receiving user input data from the real controller device requesting an action to be performed by the virtual object;

determining a user perspective of the virtual object being displayed by the near-eye, augmented reality display;

applying the user input data requesting the action to be performed by the virtual object based on the user perspective; and

displaying the action being performed by the virtual object from the user perspective in a display field of view of the near-eye, augmented reality display.

2. The method of claim 1 further comprising identifying a user selected object in the display field of view as the virtual object for control by the real controller device.

3. The method of claim 2 wherein identifying the user selected object in the display field of view as the virtual object for control by the real controller device further comprises:

detecting natural user interface input data including at least one user physical action of a user wearing the near-eye, augmented reality display system by the near-eye, augmented reality display system; and

identifying the user selected object based on the natural user interface input data.

4. The method of claim 2 further comprising:

identifying that the user selected object for control by the real controller device is a real object;

generating a virtual object version of the real object; and

designating the virtual object version as the virtual object to be controlled by the real controller device.

5. The method of claim 4 further comprising

erasing the real object from the display field of view.

6. The method of claim 1 wherein applying the user input requesting the action to be performed by the virtual object based on the user perspective further comprises:

determining any direction associated with the action to be performed by the virtual object based on the user perspective.

7. The method of claim 1 wherein applying the user input data requesting the action to be performed by the virtual object based on the user perspective further comprising:

identifying a user selected portion of the virtual object for performing the action requested in the user input data received from the real controller device;

generating display data of the requested action being performed by the portion of the virtual object; and

displaying the action being performed by the virtual object from the user perspective further comprises displaying the display data of the action being performed by the portion of the virtual object from the user perspective.

8. The method of claim 7 wherein identifying the user selected portion of the virtual object for performing the action requested in the user input data received from the real controller device further comprises:

identifying the user selected portion of the virtual object based on user physical action identified in natural user interface input data.

9. The method of claim 1 wherein determining the user perspective of the virtual object being displayed by the near-eye, augmented reality display further comprises:

determining the user perspective based on a 3D space position and orientation data for the virtual object in the display field of view.

10. The method of claim 1 further comprising:

determining a force value for the action being performed by the virtual object; and

sending the force value to the real controller device for representing the force value.

11. The method of claim 1 further comprising:

playing audio data for the virtual object projected from a 3D space position of the virtual object relative to a 3D space position and an orientation identified for a display device including the near-eye, augmented reality display.

12. A system for controlling a virtual object displayed by a near-eye, augmented reality display with a real controller device comprising:

the near-eye augmented reality display having a display field of view including a real object and the virtual object and being supported by a near-eye support structure;

one or more processors being communicatively coupled to the near-eye, augmented reality display for controlling the display;

the one or more processors being communicatively coupled to at least one capture device for receiving image data and depth data from which the one or more processors determine a three dimensional (3D) mapping of the display field of view and the one or more processors determining a user perspective of the virtual object based on the 3D mapping of the display field of view;

the one or more processors being communicatively coupled to the real controller device for receiving user input data requesting an action to be performed by the virtual object;

the one or more processors applying the user input data requesting the action to be performed by the virtual object based on the user perspective; and

the one or more processors causing the near-eye, augmented reality display to display the action being performed by the virtual object from the user perspective.

13. The system of claim 12 further comprising:

a natural user interface comprising the at least one capture device being supported by the near-eye support structure and a gesture recognition engine being executed by the one or more processors for identifying a user physical action of a gesture selecting a user selected object as the virtual object to be controlled by the real controller device based on the captured image and depth data.

14. The system of claim 12 further comprising:

a natural user interface comprising an eye tracking system being positioned by the support structure and eye tracking software being executed by the one or more processors for identifying a user physical action of an eye based action based on eye data from the eye tracking system; and

the one or more processors identifying a user selected object as the virtual object to be controlled by the real controller device in the display field of view based on the eye based action.

15. The system of claim 12 further comprising:

the real controller device is a hand held controller comprising at least one of the following;

a joystick;

a steering wheel;

a gamepad;

a touch pad;

a multi-point touch pad;

a handheld inertial motion unit (IMU); and

a controller associated with a game console.

16. The system of claim 12 further comprising:

the real controller device comprises a force feedback mechanism;

the one or more processors determine a force value for the action being performed by the virtual object based on physical properties associated with the virtual object; and

the one or more processors sending the force value to the real controller device for activating the force feedback mechanism for representing the force value.

17. The system of claim 12 wherein the near-eye, augmented reality display is an optical see-through, augmented reality display.

18. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for providing force feedback for a user selected force feedback object by a real controller device providing input data to a near-eye augmented reality display device system, the method comprising:

identifying a virtual object to be controlled by a real controller device;

identifying a user selected force feedback object for which force feedback is to be activated by the real controller device based on received user input;

applying a user input command received from the real controller device for controlling action of the virtual object based on a user perspective; and

identifying a collision between the virtual object under control of the real controller device and the user selected force feedback object;

determining a force value for a force acting upon the user selected force feedback object based on physical properties of the virtual object and the user selected force feedback object in the collision;

and sending the force value to the real controller device for representing the force value.

19. The one or more processor readable storage devices of claim 18 wherein identifying the virtual object to be controlled by the real controller device further comprises:

receiving natural user interface input data identifying at least one user physical action by a body part of a user wearing the near-eye, augmented reality display system, and

identifying from the natural user interface input data a user selected control object to be controlled by the real controller device.

20. The one or more processor readable storage devices of claim 18 wherein identifying the user selected force feedback object for which force feedback is to be activated by the real controller device based on received user input further comprises identifying from natural user interface input data the user selected force feedback object.