Producing and consuming metadata within multi-dimensional data

• US 10,055,888 B2
• Filed: 04/28/2015
• Issued: 08/21/2018
• Est. Priority Date: 04/28/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Christen, Jonathan, Howard, John Charles, Tanner, Marcus, Sugden, Ben, Memmott, Robert C., Ouellette, Kenneth Charles, Kipman, Alex, Omotani, Todd Alan, Reichert, Jr., James T.
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Wu, Chong
• Application Number:
  US14/698,514
• Publication Number:
  US 20160321841A1
• Time in Patent Office: 1,211 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/004 Annotating, labelling

Abstract

A computing system and method for producing and consuming metadata within multi-dimensional data is provided. The computing system comprising a see-through display, a sensor system, and a processor configured to: in a recording phase, generate an annotation at a location in a three dimensional environment, receive, via the sensor system, a stream of telemetry data recording movement of a first user in the three dimensional environment, receive a message to be recorded from the first user, and store, in memory as annotation data for the annotation, the stream of telemetry data and the message, and in a playback phase, display a visual indicator of the annotation at the location, receive a selection of the visual indicator by a second user, display a simulacrum superimposed onto the three dimensional environment and animated according to the telemetry data, and present the message via the animated simulacrum.

Claims

1. A computing system for producing and consuming metadata within multi-dimensional data, comprising:

a see-through display;

a sensor system; and

a processor configured to;

in a recording phase;

receive, from a first user, a user command to generate an annotation at a location in a three dimensional environment;

receive, via the sensor system, a stream of telemetry data recording movement of the first user in the three dimensional environment;

receive a message to be recorded from the first user;

receive, from the first user, a second user command to add a stock animation at a designated point in time during the message, wherein the stock animation includes at least a direct communication animation; and

store, in memory as annotation data for the annotation, the stream of telemetry data and the message;

in a playback phase;

display, to a second user via the see-through display, a visual indicator of the annotation at the location;

receive a selection of the visual indicator by the second user;

display, to the second user via the see-through display, a simulacrum of the first user superimposed onto the three dimensional environment and animated according to the telemetry data and animated to include the stock animation at the designated point in time during the message, wherein to animate the simulacrum with the direct communication animation the simulacrum is animated to adopt a posture of directly communicating to the second user based on a position and orientation of the simulacrum relative to a position and orientation of the second user in the three dimensional environment; and

present, to the second user, the message via the animated simulacrum.

2. The computing system of claim 1, wherein the computing system is a head-mounted display device.

3. The computing system of claim 1, wherein displaying the simulacrum further comprises displaying a gaze ray that is generated based on gaze direction data from the telemetry data of the first user.

4. The computing system of claim 1, wherein the location of the annotation in the three dimensional environment is determined based at least in part on an intersection between a recognized object and a detected gaze direction of the first user.

5. The computing system of claim 4, wherein the recognized object is a virtual object that is displayed as a visual hologram having a position in the three dimensional environment.

6. The computing system of claim 5, wherein the annotation data further includes a received user modification input to modify the virtual object.

7. The computing system of claim 6, wherein the processor is further configured to:

in the playback phase, display a rendering of the modification to the virtual object corresponding to the received user modification input.

8. The computing system of claim 6, wherein the received user modification input is an input to a control point associated with the virtual object.

9. The computing system of claim 1, wherein the processor is further configured to:

display an interaction interface that includes a function selected from the group consisting of adding the direct communication animation, sharing the annotation with another user on a social network, leaving a comment with a nested annotation, capturing a visualization of the three dimensional environment, pausing the playback of the annotation, rewinding the playback of the annotation, and fast forwarding the playback of the annotation.

10. A method for producing and consuming metadata within multi-dimensional data, comprising:

in a recording phase;

receiving, from a first user, a user command to generate an annotation at a location in a three dimensional environment;

receiving a stream of telemetry data recording movement of the first user in the three dimensional environment;

receiving a message to be recorded from the first user;

receiving, from the first user, a second user command to add a stock animation at a designated point in time during the message, wherein the stock animation includes at least a direct communication animation; and

storing, in memory as annotation data for the annotation, the stream of telemetry data and the message;

in a playback phase;

displaying, via a see-through display of a head mounted display device of a second user, a visual indicator of the annotation at the location;

receiving a selection of the visual indicator by the second user via the head mounted display device;

displaying, via the see-through display of the head mounted display device, a simulacrum of the first user superimposed onto the three dimensional environment and animated according to the telemetry data and animated to include the stock animation at the designated point in time during the message, wherein to animate the simulacrum with the direct communication animation the simulacrum is animated to adopt a posture of directly communicating to the second user based on a position and orientation of the simulacrum relative to a position and orientation of the second user in the three dimensional environment; and

presenting the message via the animated simulacrum, via the head mounted display device.

11. The method of claim 10, wherein displaying the simulacrum further comprises displaying a gaze ray that is generated based on gaze direction data from the telemetry data of the first user.

12. The method of claim 10, wherein the location of the annotation in the three dimensional environment is determined based at least in part on an intersection between a recognized object and a detected gaze direction of the first user.

13. The method of claim 12, wherein the recognized object is a virtual object that is displayed as a visual hologram having a position in the three dimensional environment.

14. The method of claim 13, wherein the annotation data further includes a received user modification input to modify the virtual object.

15. The method of claim 14, further comprising, in the playback phase, displaying a rendering of the modification to the virtual object corresponding to the received user modification input.

16. The method of claim 15, wherein the received user modification input is an input to a control point associated with the virtual object.

17. The method of claim 10, further including displaying an interaction interface that includes a function selected from the group consisting of adding the direct communication animation, sharing the annotation with another user on a social network, leaving a comment with a nested annotation, capturing a visualization of the three dimensional environment, pausing the playback of the annotation, rewinding the playback of the annotation, and fast forwarding the playback of the annotation.

18. A computing system for producing and consuming metadata within multi-dimensional data, comprising:

a see-through display;

a sensor system; and

a processor configured to;

in a recording phase;

receive, from a first user, a user command to generate an annotation at a location and a point in time in a virtual scene superimposed on a three dimensional environment, the virtual scene changing over a period of time and including a visualization hologram of a multi-dimensional data set;

receive, via the sensor system, a stream of telemetry data recording movement of the first user in the three dimensional environment;

receive a message to be recorded from the first user;

receive, from the first user, a second user command to add a stock animation at a designated point in time during the message, wherein the stock animation includes at least a direct communication animation;

receive a user modification input to modify the visualization hologram or the multi-dimensional data set; and

store, in memory as annotation data for the annotation, the stream of telemetry data, the message, and the user modification input;

in a playback phase;

display, to a second user via the see-through display, a visual indicator of the annotation at the location and the point in time in the virtual scene;

receive a selection of the visual indicator by the second user;

display, to the second user via the see-through display, a simulacrum of the first user superimposed onto the three dimensional environment and animated according to the telemetry data and animated to include the stock animation at the designated point in time during the message, wherein to animate the simulacrum with the direct communication animation the simulacrum is animated to adopt a posture of directly communicating to the second user based on a position and orientation of the simulacrum relative to a position and orientation of the second user in the three dimensional environment; and

present, to the second user, the message and a rendering of the modification to the visualization hologram or the multi-dimensional data set in synchronization with the animated simulacrum.