Augmented reality controls for user interactions with a virtual world

• US 10,025,375 B2
• Filed: 10/01/2015
• Issued: 07/17/2018
• Est. Priority Date: 10/01/2015
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Lazor, Sara, Fidaleo, Douglas A., Ilardi, Michael
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US14/872,991
• Publication Number:
  US 20170097676A1
• Time in Patent Office: 1,020 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

There is provided an augmented reality control for interacting with a virtual world comprising a projector, a display surface for displaying the virtual world, a physical object, a memory storing, and a processor to: project, using the projector, a user control onto the physical object including a visual representation correlated with a virtual element in the virtual world, determine a user input for controlling the virtual element based on a user manipulation of the physical object, and update, using the projector, the user control projected onto the physical object based on the user manipulation of the physical object to maintain the visual representation correlated with the virtual element in the virtual world.

Claims

1. A system comprising:

a display surface;

a virtual world displaying device located below the display surface and configured to display a virtual world on the display surface;

a physical object located above and on the display surface;

an augmented reality front-projector located above the display surface and the physical object;

a memory storing an augmented reality software program; and

a processor executing the augmented reality software program to;

project, using the augmented reality front-projector, a user control projection onto the physical object, the user control projection correlated with a virtual element in the virtual world being displayed on the display surface;

determine a user input for controlling the virtual element in the virtual world based on a user manipulation of the physical object having the user control projection thereon;

update, using the augmented reality front-projector, the user control projection projected onto the physical object based on the user input to maintain the user control projection correlated with the virtual element in the virtual world.

2. The system of claim 1, wherein the user manipulation of the physical object includes at least one of deforming of the physical object, a manipulation of a real-world position of the physical object, and a manipulation of a real-world orientation of the physical object.

3. The system of claim 1, further comprising a plurality of physical objects on the display surface, wherein, prior to the projecting, a user selects the physical object from the plurality of objects on the display surface.

4. The system of claim 1, further comprising a plurality of physical objects on the display surface, wherein the processor is further configured to:

track a position of a first object of the plurality of physical objects on the display surface; and

insert a virtual object into the virtual world at a location in the virtual world corresponding to the position of the first object on the display surface.

5. The system of claim 1, further comprising a plurality of physical objects on the display surface, wherein the augmented reality front-projector projects one or more virtual elements of the virtual world onto each of the plurality of objects.

6. The system of claim 1, wherein the user control projection includes a virtual user interlace having at least one of a virtual button, a virtual trigger, and a virtual slider.

7. The system of claim 1, wherein the user manipulation of the physical object is tracked using one of a compass, a magnetometer, an inertial sensor, and a camera.

8. The system of claim 1, wherein the user control projection is a steering wheel.

9. A method for use with a system having a display surface, a virtual world displaying device, an augmented reality front-projector, a physical object, a memory, and a processor, the method comprising:

displaying, using the virtual world displaying device located below the display surface, a virtual world on the display surface;

projecting, using the augmented reality front-projector located above the display surface and the physical object, a user control projection onto the physical object, the user control projection correlated with a virtual element in the virtual world being displayed on the display surface;

determining, using the processor, a user input for controlling the virtual element in the virtual world based on a user manipulation of the physical object having the user control projection thereon; and

updating, using the augmented reality front-projector, the user control projection projected onto the physical object based on the user input to maintain the user control projection correlated with the virtual element in the virtual world.

10. The method of claim 9, wherein the user manipulation of the physical object includes at least one of deforming of the physical object, a manipulation of a real-world position of the physical object, and a manipulation of a real-world orientation of the physical object.

11. The method of claim 9, further comprising a plurality of physical objects on the display surface, wherein, prior to the projecting, a user selects the physical object from the plurality of objects on the display surface.

12. The method of claim 11, wherein the processor is further configured to:

track a position of a first object of the plurality of physical objects on the display surface; and

insert a virtual object into the virtual world at a location in the virtual world corresponding to the position of the first object on the display surface.

13. The method of claim 9, further comprising a plurality of physical objects on the display surface, wherein the augmented reality front-projector projects one or more virtual elements of the virtual world onto each of the plurality of objects.

14. The method of claim 9, wherein the user control projection includes a virtual user interface having at least one of a virtual button, a virtual trigger, and a virtual slider.

15. The method of claim 9, wherein the user manipulation of the physical object is tracked using one of a compass, a magnetometer, an inertial sensor, and a camera.

16. The method of claim 9, wherein the user control projection is a steering wheel.