Representing a location at a previous time period using an augmented reality display

• US 9,286,711 B2
• Filed: 06/27/2012
• Issued: 03/15/2016
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Perez, Kathryn Stone, Latta, Stephen G., Sugden, Ben J., Vaught, Benjamin I., Kipman, Alex Aben-Athar, Clavin, John, Steed, Jonathan T., Scott, Jason
• Primary Examiner(s): Johnson, M Good
• Application Number:
  US13/535,213
• Publication Number:
  US 20130083011A1
• Time in Patent Office: 1,357 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G08G 1/00 Traffic control systems for road vehicles arrangement of road signs or traffic signals E01F9/00 ; automatic vehicle control B62D
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

Technology is described for representing a physical location at a previous time period with three dimensional (3D) virtual data displayed by a near-eye, augmented reality display of a personal audiovisual (A/V) apparatus. The personal A/V apparatus is identified as being within the physical location, and one or more objects in a display field of view of the near-eye, augmented reality display are automatically identified based on a three dimensional mapping of objects in the physical location. User input, which may be natural user interface (NUI) input, indicates a previous time period, and one or more 3D virtual objects associated with the previous time period are displayed from a user perspective associated with the display field of view. An object may be erased from the display field of view, and a camera effect may be applied when changing between display fields of view.

Claims

1. A method for representing a physical location at a previous time period with three dimensional (3D) virtual data displayed by a near-eye, augmented reality (AR) display of a personal audiovisual (A/V) apparatus comprising:

automatically identifying the personal A/V apparatus is within the physical location based on location data detected by the personal A/V apparatus;

automatically identifying one or more objects in a display field of view of the near-eye, augmented reality display based on a three dimensional mapping of objects in the physical location, and indentifying at least two of the objects as having a relationship;

identifying user input indicating selection of a previous time period from a plurality of different previous time periods; and

displaying three-dimensional (3D) virtual data representing the one or more objects in the physical location for the selected previous time period and the relationship between the at least two objects in the display field of view and based on a user perspective associated wit the display field of view.

2. The method of claim 1 further comprising:

identifying a change in the display field of view; and

updating the displaying of the 3D virtual data associated with the previous time period based on the change in the display field of view.

3. The method of claim 1 further comprising:

displaying virtual data indicating a plurality of previous time periods by the near-eye, augmented reality display; and

identifying user input indicating selection of the previous time period further comprises identifying natural user interface (NUI) input of at least one user physical action of at least one body part indicating selection of the previous time period from the virtual data from data captured by at least one capture device of the personal A/V apparatus.

4. The method of claim 3 wherein the virtual data indicating the plurality of previous time periods is a timeline indicating a plurality of past events.

5. The method of claim 3 wherein the at least one user physical action of at least one body part is a gesture performed by at least one body part.

6. The method of claim 1 wherein automatically identifying one or more objects in the display field of view of the near-eye, augmented reality display based on the three dimensional mapping of objects in the physical location further comprises:

determining a gaze pattern of a user wearing the near-eye, augmented reality display indicating short gaze periods back and forth between the at least two objects at the physical location,

determining whether virtual data showing the relationship between the at least two objects is available for download, and

responsive to virtual data showing the relationship between the at least two objects is available, retrieving the virtual data showing the relationship between the at least two objects.

7. The method of claim 1 further comprising:

identifying a collision between at least one real object and at least one virtual object based on their 3D space positions in the display field of view of the near-eye, augmented reality display;

determining at least one effect on at least one physical property of the at least one virtual object due to the collision based on its one or more physical properties and physical interaction characteristics for the collision;

modifying image data of the at least one virtual object for representing the at least one effect on the at least one physical property of the at least one virtual object; and

displaying the modified image data of the at least one virtual object based on the user perspective associated with the display field of view.

8. The method of claim 1 further comprising:

displaying 3D virtual data of a tour guide;

identifying a question of a wearer of the personal A/V apparatus based on natural user interface input data including at least voice data captured by a microphone of the personal A/V apparatus; and

outputting 3D audiovisual virtual data of the tour guide responding to the question by one or more audio output devices of the personal A/V apparatus and the near-eye, augmented reality display.

9. A personal audiovisual (A/V) apparatus for representing a physical location at a previous time period with three dimensional (3D) virtual data comprising:

a near-eye, augmented reality display having a display field of view and being supported by a near-eye support structure;

one or more processors being communicatively coupled to a network accessible computer system for accessing one or more memories storing a virtual tag associated with one or more objects, 3D virtual data associated with the one or more objects in the physical location and the 3D virtual data representing the one or more objects in the physical location for at least one previous time period from a plurality of different time periods;

the one or more processors receiving a 3D mapping of the physical location identifying one or more 3D space positions of real and virtual objects in the physical location;

a natural user interface (NUI) for interpreting one or more user physical actions of at least one body part as user input indicating a previous time period for representation by 3D virtual data and as user input selecting at least one object in the physical location for augmentation with the 3D virtual data; and

the one or more processors controlling the near-eye, augmented reality display of the personal A/V apparatus for displaying of the 3D virtual data for the at least one object in the at least one previous time period from a user perspective associated with the display field of view and displaying the linked object associated with the virtual tag in response to another user at the physical location previously uploading wit the virtual tag to the memory.

10. The apparatus of claim 9 wherein the near-eye, augmented reality display is an optical see-through, augmented reality display.

11. The apparatus of claim 9 wherein the natural user interface comprises the at least one capture device being supported by the near-eye support structure and a gesture recognition engine being executed by the one or more processors for identifying at least one of the one or more user physical actions as a gesture selecting the at least one object in the physical location to be augmented with the 3D virtual data.

12. The apparatus of claim 9 wherein the natural user interface comprises an eye tracking system being positioned by the support structure and eye tracking software being executed by the one or more processors for identifying at least one of the one or more user physical actions as an eye based action based on eye data from the eye tracking system.

13. The apparatus of claim 9 further comprising:

the one or more processors identifying user input indicating a request for erasing a user selected real object from the display field of view;

the one or more processors retrieving image data of each real object and virtual object behind the user selected real object in the display field of view in an unoccluded state;

identifying a depth position of each real object and each virtual object behind the user selected real object in the display field of view;

ordering in a Z-buffer the image data in the unoccluded state of each real object and each virtual object behind the user selected real object in the display field of view based on its respective depth position from farthest depth position in the display field of view to closest depth position in the display field of view; and

displaying the image data in the Z-buffer in accordance with the ordering from farthest depth position in the display field of view to closest depth position in the display field of view.

14. The apparatus of claim 9 further comprising:

the one or more processors identifying user input indicating one or more allowed sources for audio data in the physical location;

a microphone of the personal A/V apparatus for capturing real sounds made in the physical location, and the one or more processors being communicatively coupled to receive real sound audio data representing the real sounds from the microphone and executing a sound recognition engine for identifying real sound audio data from the one or more allowed sources;

the one or more processors being communicatively coupled to at least one other personal A/V apparatus and registering with the at least one other personal A/V apparatus for receiving real sound audio data from the one or more allowed sources;

the one or more processors receiving real sound audio data identified as being from the one or more allowed sources; and

the one or more processors executing an audio engine for playing the audio data from the one or more allowed sources for the physical location through at least one audio output device of the personal A/V apparatus.

15. The apparatus of claim 9 further comprising:

the one or more processors sending user identification data to the network accessible computer system;

the one or more processors receiving notification of the virtual tag associated with the user identification data and an object in the physical location associated with the tag; and

responsive to identifying the object is in the display field of view of the near-eye, augmented reality display based on the object identifier, displaying virtual data associated with the virtual tag.

16. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for representing a physical location at a previous time period with three dimensional (3D) virtual data displayed by a near-eye, augmented reality display of a personal audiovisual (A/V) apparatus comprising:

automatically identifying the personal A/V apparatus is within the physical location based on location data detected by the personal A/V apparatus;

automatically identifying one or more objects in a first display field of view of the near-eye, augmented reality display based on a three dimensional (3D) mapping of objects in the physical location;

identifying user input indicating selection of a previous time period from a plurality of different time periods displayed as a timeline in the first of view of the near eye, augmented reality display;

displaying (3D) virtual data associated with the previous time period from a user perspective associated with the first display field of view.

17. The one or more processor readable storage devices of claim 16 further comprising:

identifying user input indicating a request for a second display field of view having a different level of detail than the first display field of view and including at least one object of interest of the first display field of view; and

changing the first display field of view to the second display field of view having the different level of detail and including the at least one object of interest.

18. The one or more processor readable storage devices of claim 17 wherein changing the first display field of view to the second display field of view having the different level of detail and including the at least one object of interest further comprises:

implementing a camera effect in changing the first display field of view to the second display field of view having the level of detail and including the at least one object of interest and wherein the camera effect comprises at least one of the following;

a close-up effect;

a zoom effect; and

a flying camera effect.

19. The one or more processor readable storage devices of claim 17 wherein changing the first display field of view to the second display field of view having the different level of detail and including the at least one object of interest further comprises:

based on the different level of detail of the second display field of view and the 3D mapping of objects in the physical location, identifying any real and virtual objects, including the at least one object of interest, to be displayed in the second display field of view;

retrieving virtual data representing any real and virtual objects including the at least one object to be displayed in the second display field of view; and

displaying the retrieved virtual data by the near-eye, AR display from a user perspective of the second display field of view determined based on the user perspective associated with the first display field of view as a reference point.

20. The one or more processor readable storage devices of claim 19 further comprising:

identifying at least one of a gaze change or an orientation change of the near-eye augmented reality display;

updating the user perspective associated with the second display field of view based on the identified at least one of the gaze change or the orientation change and the reference point of the user perspective associated with the first display field of view;

responsive to at least one different object to be displayed in the second display field of view based on the updating of the user perspective associated with the second display field of view, retrieving virtual data representing the at least one different object; and

updating the displaying of the retrieved virtual data by the near-eye, AR display from the updated user perspective associated with the second display field of view.