Controlling a virtual object with a real controller device

• US 9,041,622 B2
• Filed: 06/12/2012
• Issued: 05/26/2015
• Est. Priority Date: 06/12/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: McCulloch, Daniel J., Navratil, Arnulfo Zepeda, Steed, Jonathan T., Hastings, Ryan L., Scott, Jason, Mount, Brian J., Hirzel, Holly A., Bennett, Darren, Scavezze, Michael J.
• Primary Examiner(s): Nguyen, Kevin M
• Application Number:
  US13/494,989
• Publication Number:
  US 20130328762A1
• Time in Patent Office: 1,078 Days
• Field of Search: 345/7, 345/8, 345/9, 345/156
• US Class Current: 345/7
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Technology is described for controlling a virtual object displayed by a near-eye, augmented reality display with a real controller device. User input data is received from a real controller device requesting an action to be performed by the virtual object. A user perspective of the virtual object being displayed by the near-eye, augmented reality display is determined. The user input data requesting the action to be performed by the virtual object is applied based on the user perspective, and the action is displayed from the user perspective. The virtual object to be controlled by the real controller device may be identified based on user input data which may be from a natural user interface (NUI). A user selected force feedback object may also be identified, and the identification may also be based on NUI input data.

Claims

1. A method for controlling a virtual object displayed by a near-eye, augmented reality display of a near-eye, augmented reality display system communicatively coupled with a real controller device comprising:

identifying a user selected object in a display field of view of the near-eye, augmented reality display system for control by the real controller device by one or more processors of the near-eye, augmented reality display system based on user input detected by one or more of the group consisting of one or more sensors of the near-eye, augmented reality display system or the real controller device;

translating one or more physical touch based actions being performed with respect to one or more touch activated parts of the real controller device to user input data;

receiving by the one or more processors of the near-eye, augmented reality display system the user input data from the communicatively coupled real controller device, the user input data representing a request for an action to be performed by the user selected object;

identifying that the user selected object for control by the real controller device is a real object;

generating a virtual object version of the real object;

designating the virtual object version as a virtual object to be controlled by the real controller device;

erasing the real object from the display field of view of the near-eye, augmented reality display system;

determining by the one or more processors a user perspective of the virtual object being displayed by the near-eye, augmented reality display;

applying the user input data requesting the action to be performed by the virtual object based on the user perspective; and

displaying the action being performed by the virtual object from the user perspective in a display field of view of the near-eye, augmented reality display.

2. The method of claim 1 wherein identifying the user selected object in the display field of view for control by the real controller device further comprises:

detecting natural user interface input data including at least one user physical action of a user wearing the near-eye, augmented reality display system by the near-eye, augmented reality display system; and

identifying the user selected object based on the natural user interface input data.

3. The method of claim 1 wherein applying the user input requesting the action to be performed by the virtual object based on the user perspective further comprises:

determining any direction associated with the action to be performed by the virtual object based on the user perspective.

4. The method of claim 1 wherein applying the user input data requesting the action to be performed by the virtual object based on the user perspective further comprising:

identifying a user selected portion of the virtual object for performing the action requested in the user input data received from the real controller device;

generating display data of the requested action being performed by the portion of the virtual object; and

displaying the action being performed by the virtual object from the user perspective further comprises displaying the display data of the action being performed by the portion of the virtual object from the user perspective.

5. The method of claim 4 wherein identifying the user selected portion of the virtual object for performing the action requested in the user input data received from the real controller device further comprises:

identifying the user selected portion of the virtual object based on user physical action identified in natural user interface input data.

6. The method of claim 1 wherein determining the user perspective of the virtual object being displayed by the near-eye, augmented reality display further comprises:

determining the user perspective based on a 3D space position and orientation data for the virtual object in the display field of view.

7. The method of claim 1 further comprising:

determining a force value for the action being performed by the virtual object; and

sending the force value to the real controller device for representing the force value.

8. The method of claim 1 further comprising:

playing audio data for the virtual object projected from a 3D space position of the virtual object relative to a 3D space position and an orientation identified for a display device including the near-eye, augmented reality display.

9. A system for controlling a virtual object displayed by a near-eye, augmented reality display with a real controller device comprising:

the near-eye augmented reality display having a display field of view including a real object and being supported by a near-eye support structure;

one or more processors being communicatively coupled to the near-eye, augmented reality display for controlling the display;

the one or more processors being communicatively coupled to at least one capture device for receiving image data and depth data from which the one or more processors determine a three dimensional (3D) mapping of the display field of view and the one or more processors determining a user perspective of the virtual object based on the 3D mapping of the display field of view;

the real controller device having one or more touch activated parts and translating one or more physical touch based actions with respect to the one or more touch activated parts to user input data;

one or more sensors being supported by the near-eye support structure, including the at least one capture device, for detecting user input;

the one or more processors identifying a user selected object in the display field of view for control by the real controller device based on user input detected by one or more of the group consisting of the one or more sensors being supported by the near-eye support structure or the real controller device, identifying that the user selected object for control by the real controller device is a real object, generating and causing the near-eye display to display a virtual object version of the real object, designating the virtual object version as a virtual object to be controlled by the real controller device, and erasing the real object from the display field of view of the near-eye display;

the one or more processors being communicatively coupled to the real controller device for receiving the user input data, the user input data representing a request for an action to be performed by the virtual object;

the one or more processors applying the user input data requesting the action to be performed by the virtual object based on the user perspective; and

the one or more processors causing the near-eye, augmented reality display to display the action being performed by the virtual object from the user perspective.

10. The system of claim 9 further comprising:

a natural user interface, comprising the one or more sensors being supported by the near-eye support structure, including the at least one capture device, for detecting user input and a gesture recognition engine being executed by the one or more processors for identifying a user physical action of a gesture selecting the user selected object as the virtual object to be controlled by the real controller device based on the captured image and depth data.

11. The system of claim 9 further comprising:

a natural user interface, comprising the one or more sensors being supported by the near-eye support structure, including the at least one capture device, for detecting user input and an eye tracking system being positioned by the support structure and eye tracking software being executed by the one or more processors for identifying a user physical action of an eye based action based on eye data from the eye tracking system; and

the one or more processors identifying the user selected object as the virtual object to be controlled by the real controller device in the display field of view based on the eye based action.

12. The system of claim 9 further comprising:

the real controller device is a hand held controller comprising at least one of the following;

a joystick;

a steering wheel;

a gamepad;

a touch pad;

a multi-point touch pad;

a handheld inertial motion unit (IMU); and

a controller associated with a game console.

13. The system of claim 9 further comprising:

the real controller device comprises a force feedback mechanism;

the one or more processors determine a force value for the action being performed by the virtual object based on physical properties associated with the virtual object; and

the one or more processors sending the force value to the real controller device for activating the force feedback mechanism for representing the force value.

14. The system of claim 9 wherein the near-eye, augmented reality display is an optical see-through, augmented reality display.

15. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for providing force feedback for a user selected force feedback object by a real controller device providing input data to a near-eye augmented reality display device system, the method comprising:

identifying a virtual object to be controlled by a real controller device;

identifying a user selected force feedback object for which force feedback is to be activated by the real controller device based on received user input;

applying a user input command received from the real controller device for controlling action of the virtual object based on a user perspective; and

identifying a collision between the virtual object under control of the real controller device and the user selected force feedback object;

determining a force value for a force acting upon the user selected force feedback object based on physical properties of the virtual object and the user selected force feedback object in the collision;

and sending the force value to the real controller device for representing the force value.

16. The one or more processor readable storage devices of claim 15 wherein identifying the virtual object to be controlled by the real controller device further comprises:

receiving natural user interface input data identifying at least one user physical action by a body part of a user wearing the near-eye, augmented reality display system, and

identifying from the natural user interface input data a user selected control object to be controlled by the real controller device.

17. The one or more processor readable storage devices of claim 15 wherein identifying the user selected force feedback object for which force feedback is to be activated by the real controller device based on received user input further comprises identifying from natural user interface input data the user selected force feedback object.