Composite image associated with a head-mountable device

• US 9,607,440 B1
• Filed: 09/07/2016
• Issued: 03/28/2017
• Est. Priority Date: 11/06/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: The, Richard, Braun, Max Benjamin, McKenzie, Chris, Chen, Alexander Hanbing
• Primary Examiner(s): McCulley, Ryan D
• Application Number:
  US15/258,960
• Time in Patent Office: 202 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

In one aspect, an HMD is disclosed that provides a technique for generating a composite image representing the view of a wearer of the HMD. The HMD may include a display and a front-facing camera, and may be configured to perform certain functions. For instance, the HMD may be configured to make a determination that a trigger event occurred and responsively both generate a first image that is indicative of content displayed on the display, and cause the camera to capture a second image that is indicative of a real-world field-of-view associated with the HMD. Further, the HMD may be configured to generate a composite image that combines the generated first image and the captured second image.

Claims

1. A method comprising:

displaying content on a transparent or semi-transparent display of a head-mountable device (HMD), wherein the displayed content is perceived by a wearer of the HMD as being superimposed over a real-world field-of-view seen through the transparent or semi-transparent display;

making, by the HMD, a determination that a trigger event occurred;

responsive to making the determination that the trigger event occurred, both;

(i) generating, by the HMD, a first image of the displayed content, and (ii) causing, by the HMD, a camera to capture a second image of the real-world field-of-view associated with the HMD; and

generating, by the HMD, a composite image that combines the generated first image, the captured second image, and an effect image by overlaying the generated first image on the effect image and overlaying the effect image on the captured second image, wherein the effect image improves the accuracy of the generated composite image'"'"'s representation of the wearer'"'"'s view, and wherein the effect image is neither displayed on the display of the HMD nor part of the real-world field-of-view associated with the HMD.

2. The method of claim 1, wherein the trigger event comprises at least one event from a group of events comprising:

a gesture being made, a voice command being spoken, and a button being pressed.

3. The method of claim 1, wherein responsive to making the determination that the trigger event occurred, (i) generating, by the HMD, the first image, and (ii) causing, by the HMD, the camera to capture the second image comprises responsive to making the determination, substantially simultaneously (i) generating, by the HMD, the first image, and (ii) causing, by the HMD, the camera to capture the second image.

4. The method of claim 3, wherein causing, by the HMD, the camera to capture the second image causes the HMD to, proximate a time that the camera captures the second image, display a visual effect on the display for a duration.

5. The method of claim 1, wherein the generated first image comprises a screenshot of content displayed on the display.

6. The method of claim 1, wherein the generated first image comprises a modified version of a screenshot of content displayed on the display.

7. The method of claim 1, wherein in the generated composite image, the generated first image is aligned with the captured second image such that the generated composite image is indicative of a view of a user wearing the HMD proximate a time that the trigger event occurred.

8. The method of claim 1, wherein the effect image comprises a glass-texture effect.

9. The method of claim 1, wherein the generating the composite image occurs in response to making the determination.

10. The method of claim 1, further comprising:

responsive to making the determination that the trigger event occurred, determining, by the HMD, data relating to the HMD'"'"'s operation proximate a time that the trigger event occurred, wherein the generated composite image further includes an indication of the determined data.

11. The method of claim 10, wherein the data comprises audio data indicative of audio being played by the HMD proximate a time that the trigger event occurred.

12. The method of claim 10, wherein the data comprises environmental data indicative of the environment in which the HMD is operating proximate a time that the trigger event occurred.

13. The method of claim 10, wherein the data comprises diagnostic or state data related to the HMD proximate a time that the trigger event occurred.

14. The method of claim 1, further comprising displaying on the display the generated composite image.

15. The method of claim 1, wherein the display comprises a transparent display.

16. A head-mountable device (HMD) comprising:

a transparent or semi-transparent display;

a camera positioned for capturing a real-world field-of-view associated with the HMD;

a processor; and

a non-transitory computer-readable medium including program instructions that, when executed by the processor, cause performance of a set of functions comprising;

displaying content on the transparent or semi-transparent display, wherein the displayed content is perceived by a wearer of the HMD as being superimposed over the real-world field-of-view seen through the transparent or semi-transparent display;

making a determination that a trigger event occurred;

responsive to making the determination that the trigger event occurred, (i) generating a first image of the displayed content, and (ii) causing the camera to capture a second image of the real-world field-of-view; and

generating, by the HMD, a composite image that combines the generated first image, the captured second image, and an effect image by overlaying the generated first image on the effect image and overlaying the effect image on the captured second image, wherein the effect image improves the accuracy of the generated composite image'"'"'s representation of the wearer'"'"'s view, and wherein the effect image is neither displayed on the display of the HMD nor part of the real-world field-of-view associated with the HMD.

17. The HMD of claim 16, wherein responsive to making the determination that the trigger event occurred, (i) generating the first image, and (ii) causing the camera to capture the second image comprises responsive to making the determination that the trigger event occurred, substantially simultaneously (i) generating the first image, and (ii) causing the camera to capture the second image.

18. The HMD of claim 16, wherein the generated first image comprises a screenshot of content displayed on the display.

19. The HMD of claim 16, the set of functions further comprising:

responsive to making the determination that the trigger event occurred, determining, by the HMD, data relating to the HMD'"'"'s operation proximate a time that the trigger event occurred, wherein the generated composite image further includes an indication of the determined data.

20. A non-transitory computer-readable medium having stored thereon, program instructions that when executed by a processor, cause performance of a set of acts comprising:

displaying content on a transparent or semi-transparent display of a head-mountable device (HMD), wherein the displayed content is perceived by a wearer of the HMD as being superimposed over a real-world field-of-view seen through the transparent or semi-transparent display;

making, by the HMD, a determination that a trigger event occurred;

responsive to making the determination that the trigger event occurred, both;

(i) generating, by the HMD, a first image of the displayed content, and (ii) causing, by the HMD, a camera to capture a second image of the real-world field-of-view associated with the HMD; and

generating, by the HMD, a composite image that combines the generated first image, the captured second image, and a glass-texture effect image by overlaying the generated first image on the glass-texture effect image and overlaying the glass-texture effect image on the captured second image, wherein the glass-texture effect image is neither displayed on the display of the HMD nor part of the real-world field-of-view associated with the HMD.