Mapping input to hologram or two-dimensional display

• US 10,025,102 B2
• Filed: 08/21/2017
• Issued: 07/17/2018
• Est. Priority Date: 08/19/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Memmott, Robert Courtney, Christen, Jonathan R., Sugden, Benjamin John, Tanner, Marcus, Gallina, Jonathan Paton
• Primary Examiner(s): PHAM, NAM D
• Application Number:
  US15/682,399
• Publication Number:
  US 20170351097A1
• Time in Patent Office: 330 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G03H 1/2294 Addressing the hologram to an active spatial light modulator
• CPC Class Codes: G03H 2210/20 2D object
• CPC Class Codes: G03H 2210/30 3D object
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour

Abstract

Various embodiments relating to editing holograms by extending real world interfaces are disclosed. One embodiment includes a computing device configured to communicatively couple to a head mounted display device having an at least partially see-through display. The computing device includes a non-see-through display, a user input device, and a processor configured to determine whether a user focus is on an image or a hologram, if a determination is made that the user focus is on the image, map the user input to a first coordinate space of the non-see-through display, and if a determination is made that the user focus is the on the hologram, map the user input to a second coordinate space of the head mounted display device.

Claims

1. A computing system comprising:

a computing device configured to communicatively couple to a head mounted display device having a display configured to display three-dimensional objects to a user in a three-dimensional environment, the computing device including;

a computer display device arranged in the three-dimensional environment;

a user input device; and

a processor configured to;

display an image on the computer display device;

send to the head mounted display device, for display on the display, a three-dimensional object at a location in the three-dimensional environment;

receive, via the user input device, a user input;

determine whether a user focus is on the image or the three-dimensional object;

if a determination is made that the user focus is on the image, map the user input to a first coordinate space of the computer display device; and

if a determination is made that the user focus is the on the three-dimensional object, map the user input to a second coordinate space of the head mounted display device.

2. The computing system of claim 1, wherein the head mounted display device further includes one or more outward facing cameras.

3. The computing system of claim 2, wherein the head mounted display device is configured to receive optical sensor information of the three-dimensional environment via the one or more outward facing cameras.

4. The computing system of claim 3, wherein the processor is further configured to generate a virtual model of the three-dimensional environment based on the optical sensor information.

5. The computing system of claim 1, wherein

the three-dimensional object is a hologram displayed on the display so that it appears to overlap the three-dimensional environment.

6. The computing system of claim 5, wherein the image is a two-dimensional visual representation of a data set and the hologram is a three-dimensional holographic representation of the data set.

7. The computing system of claim 6, wherein the processor is further configured to:

display, on the computer display device, an interface that includes selectable tools, each tool being associated with a programmatic function; and

send to the head mounted display device, for display on the display, a hologram interface that includes selectable hologram tools, each hologram tool being associated with a hologram programmatic function.

8. The computing system of claim 7, wherein the interface includes a first tool having a first programmatic function, and the hologram interface includes a corresponding first hologram tool having a first hologram programmatic function that corresponds to the first programmatic function.

9. The computing system of claim 7, wherein the processor is further configured to:

if the determination is made that the user focus is on the image;

execute the programmatic function of a selected tool on the data set based on the mapped user input; and

update both the image and the three-dimensional object based on the programmatic function executed on the data set;

if the determination is made that the user focus is the on the three-dimensional object;

execute the hologram programmatic function of a selected hologram tool on the data set based on the mapped user input; and

update both the image and the three-dimensional object based on the hologram programmatic function executed on the data set.

10. The computing system of claim 1, wherein to determine whether the user focus is on the image or on the three-dimensional object, the processor is configured to:

determine a gaze direction of the user;

determine whether the gaze direction of the user is towards the computer display device or towards the location of the three-dimensional object in the three-dimensional environment;

based on the determination that the gaze direction of the user is towards the computer display device, determine that the user focus is on the image; and

based on the determination that the gaze direction of the user is towards the location of the three-dimensional object, determine that the user focus is on the three-dimensional object.

11. The computing system of claim 1, wherein if the determination is made that the user focus is on the image, the processor is further configured to:

display a virtual cursor at a two dimensional position in the first coordinate space corresponding to a viewport presented via the computer display device;

update the two dimensional position of the virtual cursor based on the mapped user input;

determine whether the updated two dimensional position of the virtual cursor is outside the viewport; and

if the determination is made that the updated dimensional position of the virtual cursor is outside the viewport, determine that the user focus is on the three-dimensional object.

12. The computing system of claim 11, wherein if the determination is made that the user focus is the on the three-dimensional object, the processor is further configured to:

send to the head mounted display device, for display on the display, the virtual cursor as a three-dimensional cursor at a three-dimensional position in the second coordinate space corresponding to the three-dimensional environment;

update the three-dimensional position of the three-dimensional cursor based on the mapped user input;

determine whether the updated three-dimensional position of the three-dimensional cursor overlaps the computer display device arranged in the three-dimensional environment; and

if the determination is made that the updated three-dimensional position of the three-dimensional cursor overlaps the computer display device, determine that the user focus is on the image.

13. The computing system of claim 1, wherein to map the user input to the second coordinate space, the processor is configured to:

determine a gaze direction of the user;

determine a plane that is orthogonal to the gaze direction of the user and is a set distance away from the user; and

map the user input to the plane.

14. A method comprising:

displaying an image on a computer display device, the computer display device being arranged in a three-dimensional environment;

displaying, on a display of a head mounted display device, a three-dimensional object at a location in the three-dimensional environment;

receiving, via a user input device, a user input;

determining whether a user focus is on the image or the three-dimensional object;

if a determination is made that the user focus is on the image, mapping the user input to a first coordinate space of the computer display device; and

if a determination is made that the user focus is on the three-dimensional object, mapping the user input to a second coordinate space of the head mounted display device.

15. The method of claim 14, wherein the head mounted display device further includes one or more outward facing cameras.

16. The method of claim 15, further comprising receiving optical sensor information of the three-dimensional environment via the one or more outward facing cameras.

17. The method of claim 16, further comprising generating a virtual model of the three-dimensional environment based on the optical sensor information.

18. The method of claim 14, wherein

the three-dimensional object is a hologram displayed on the display so that it appears to overlap the three-dimensional environment.

19. The method of claim 18, wherein the image is a two-dimensional visual representation of a data set and the hologram is a three-dimensional holographic representation of the data set.

20. A computing device comprising:

a computer display device arranged in a three-dimensional environment;

a user input device; and

a processor configured to;

display an image on the computer display device, the image being a two-dimensional visual representation of a data set;

send, to a head mounted display device, the data set to be presented via a three-dimensional object displayed on the head mounted display device at a location in the three-dimensional environment;

receive, via the user input device, a user input;

determine whether a user focus is on the image or the three-dimensional object;

if a determination is made that the user focus is on the image, map the user input to a first coordinate space of the computer display device; and

if a determination is made that the user focus is on the three-dimensional object, map the user input to a second coordinate space of the head mounted display device.