Mapping input to hologram or two-dimensional display

• US 9,740,011 B2
• Filed: 08/19/2015
• Issued: 08/22/2017
• Est. Priority Date: 08/19/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Memmott, Robert Courtney, Christen, Jonathan R., Sugden, Benjamin John, Tanner, Marcus, Gallina, Jonathan Paton
• Primary Examiner(s): Pham, Nam
• Application Number:
  US14/830,117
• Publication Number:
  US 20170052373A1
• Time in Patent Office: 734 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G03H 1/2294 Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G03H 2210/20 2D object
• CPC Class Codes: G03H 2210/30 3D object
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Various embodiments relating to editing holograms by extending real world interfaces are disclosed. One embodiment includes a computing device configured to communicatively couple to a head mounted display device having an at least partially see-through display. The computing device includes a non-see-through display, a user input device, and a processor configured to determine whether a user focus is on an image or a hologram, if a determination is made that the user focus is on the image, map the user input to a first coordinate space of the non-see-through display, and if a determination is made that the user focus is the on the hologram, map the user input to a second coordinate space of the head mounted display device.

Claims

1. A computing system comprising:

a computing device configured to communicatively couple to a head mounted display device having an at least partially see-through display configured to display holograms to a user in a three dimensional environment, the computing device including;

a non-see-through display arranged in the three dimensional environment;

a user input device; and

a processor configured to;

display an image on the non-see-through display;

send to the head mounted display device, for display on the at least partially see-through display, a hologram at a location in the three dimensional environment;

receive, via the user input device, a user input;

determine whether a user focus is on the image or the hologram;

if a determination is made that the user focus is on the image, map the user input to a first coordinate space of the non-see-through display; and

if a determination is made that the user focus is the on the hologram, map the user input to a second coordinate space of the head mounted display device.

2. The computing system of claim 1, wherein the image is a two-dimensional visual representation of a data set and the hologram is a three dimensional holographic representation of the data set.

3. The computing system of claim 2, wherein the processor is further configured to:

display, on the non-see-through display, an interface that includes selectable tools, each tool being associated with a programmatic function; and

send to the head mounted display device, for display on the at least partially see-through display, a hologram interface that includes selectable hologram tools, each hologram tool being associated with a hologram programmatic function.

4. The computing system of claim 3, wherein the interface includes a first tool having a first programmatic function, and the hologram interface includes a corresponding first hologram tool having a first hologram programmatic function that corresponds to the first programmatic function.

5. The computing system of claim 3, wherein the processor is further configured to:

if the determination is made that the user focus is on the image;

execute the programmatic function of a selected tool on the data set based on the mapped user input; and

update both the image and the hologram based on the programmatic function executed on the data set;

if the determination is made that the user focus is the on the hologram;

execute the hologram programmatic function of a selected hologram tool on the data set based on the mapped user input; and

update both the image and the hologram based on the hologram programmatic function executed on the data set.

6. The computing system of claim 1, wherein to determine whether the user focus is on the image or on the hologram, the processor is configured to:

determine a gaze direction of the user;

determine whether the gaze direction of the user is towards the non-see-through display or towards the location of the hologram in the three dimensional environment;

based on the determination that the gaze direction of the user is towards the non-see-through display, determine that the user focus is on the image; and

based on the determination that the gaze direction of the user is towards the location of the hologram, determine that the user focus is on the hologram.

7. The computing system of claim 1, wherein if the determination is made that the user focus is on the image, the processor is further configured to:

display a virtual cursor at a two dimensional position in the first coordinate space corresponding to a viewport presented via the non-see-through display;

update the two dimensional position of the virtual cursor based on the mapped user input;

determine whether the updated two dimensional position of the virtual cursor is outside the viewport; and

if the determination is made that the updated dimensional position of the virtual cursor is outside the viewport, determine that the user focus is on the hologram.

8. The computing system of claim 7, wherein if the determination is made that the user focus is the on the hologram, the processor is further configured to:

send to the head mounted display device, for display on the at least partially see-through display, the virtual cursor as a hologram cursor at a three dimensional position in the second coordinate space corresponding to the three dimensional environment;

update the three dimensional position of the hologram cursor based on the mapped user input;

determine whether the updated three dimensional position of the hologram cursor overlaps the non-see-through display arranged in the three dimensional environment; and

if the determination is made that the updated three dimensional position of the hologram cursor overlaps the non-see-through display, determine that the user focus is on the image.

9. The computing system of claim 1, wherein to map the user input to the second coordinate space, the processor is configured to:

determine a gaze direction of the user;

determine a plane that is orthogonal to the gaze direction of the user and is a set distance away from the user; and

map the user input to the plane.

10. A method comprising:

displaying an image on a non-see-through display, the non-see-through display being arranged in a three dimensional environment;

displaying, on an at least partially see-through display of a head mounted display device, a hologram at a location in the three dimensional environment;

receiving, via a user input device, a user input;

determining whether a user focus is on the image or the hologram;

if a determination is made that the user focus is on the image, mapping the user input to a first coordinate space of the non-see-through display; and

if a determination is made that the user focus is the on the hologram, mapping the user input to a second coordinate space of the head mounted display device.

11. The method of claim 10, wherein the image is a two-dimensional visual representation of a data set and the hologram is a three dimensional holographic representation of the data set.

12. The method of claim 11, further comprising:

displaying, on the non-see-through display, an interface that includes selectable tools, each tool being associated with a programmatic function; and

displaying, on the at least partially see-through display, a hologram interface that includes selectable hologram tools, each hologram tool being associated with a hologram programmatic function.

13. The method of claim 12, wherein the interface includes a first tool having a first programmatic function, and the hologram interface includes a corresponding first hologram tool having a first hologram programmatic function that corresponds to the first programmatic function.

14. The method of claim 12, further comprising:

if the determination is made that the user focus is on the image;

executing the programmatic function of a selected tool on the data set based on the mapped user input; and

updating both the image and the hologram based on the programmatic function executed on the data set;

if the determination is made that the user focus is the on the hologram;

executing the hologram programmatic function of a selected hologram tool on the data set based on the mapped user input; and

updating both the image and the hologram based on the hologram programmatic function executed on the data set.

15. The method of claim 10, wherein determining whether the user focus is on the image or on the hologram, further comprises:

determining a gaze direction of a user of the head mounted display device;

determining whether the gaze direction of the user is towards the non-see-through display or towards the location of the hologram in the three dimensional environment;

based on determining that the gaze direction of the user is towards the non-see-through display, determining that the user focus is on the image; and

based on determining that the gaze direction of the user is towards the location of the hologram, determining that the user focus is on the hologram.

16. The method of claim 10, wherein if the determination is made that the user focus is on the image, the method further comprises:

displaying a virtual cursor at a two dimensional position in the first coordinate space corresponding to a viewport presented via the non-see-through display;

updating the two dimensional position of the virtual cursor based on the mapped user input;

determining whether the updated two dimensional position of the virtual cursor is outside the viewport; and

if the determination is made that the updated dimensional position of the virtual cursor is outside the viewport, determining that the user focus is on the hologram.

17. The method of claim 16, wherein if the determination is made that the user focus is the on the hologram, the method further comprises:

displaying the virtual cursor as a hologram cursor at a three dimensional position in the second coordinate space corresponding to the three dimensional environment;

updating the three dimensional position of the hologram cursor based on the mapped user input;

determining whether the updated three dimensional position of the hologram cursor overlaps the non-see-through display arranged in the three dimensional environment; and

if the determination is made that the updated three dimensional position of the hologram cursor overlaps the non-see-through display, determining that the user focus is on the image.

18. The method of claim 10, wherein mapping the user input to the second coordinate space, further comprises:

determining a gaze direction of a user of the head mounted display device;

determining a plane that is orthogonal to the gaze direction of the user and is a set distance away from the user; and

mapping the user input to the plane.

19. A computing device comprising:

a non-see-through display arranged in a three dimensional environment;

a user input device; and

a processor configured to;

display an image on the non-see-through display, the image being a two-dimensional visual representation of a data set;

send, to a head mounted display device, the data set to be presented via a hologram displayed on the head mounted display device at a location in the three dimensional environment;

receive, via the user input device, a user input;

determine whether a user focus is on the image or the hologram;

if a determination is made that the user focus is on the image, map the user input to a first coordinate space of the non-see-through display; and

if a determination is made that the user focus is on the hologram, map the user input to a second coordinate space of the head mounted display device.

20. The computing device of claim 19, wherein to map the user input to the second coordinate space, the processor is further configured to:

determine a gaze direction of a user of the head mounted display device;

determine a plane that is orthogonal to the gaze direction of the user and is a set distance away from the user; and

map the user input to the plane.