Embedding metadata into images and videos for augmented reality experience

• US 10,607,415 B2
• Filed: 08/10/2018
• Issued: 03/31/2020
• Est. Priority Date: 08/10/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Leung, Mira, Landskov, Geoffrey, Veryovka, Oleg
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US16/100,766
• Publication Number:
  US 20200051334A1
• Time in Patent Office: 599 Days
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A method for embedding metadata into images and/or videos for AR experience is described. In one example implementation, the method may include generating a first image/video including an environment captured by a device and a virtually-rendered augmented reality (AR) object composited with the environment. The first image/video may be embedded with a first metadata. The method may further include generating a second image/video by modifying the first image/video. The second image/video may be embedded with a second metadata. The second metadata is generated based on the first metadata.

Claims

1. A method, comprising:

generating a first image including a composite of an environment captured by a device and a virtually-rendered augmented reality (AR) object, the first image embedded with a first metadata including;

a first contextual environment data, and the environment captured by the device without the virtually-rendered AR object; and

generating a second image by modifying the first image, the second image embedded with a second metadata that is different from the first data and generated based on the first metadata, and the second metadata including;

a second contextual environment data, and the environment captured by the device without the virtually-rendered AR object.

2. The method of claim 1, wherein the modifying includes at least one of altering, moving, or replacing the AR object, and the second image including a composite of the environment captured by the device and the modifying includes altered, moved, or replaced AR object.

3. The method of claim 1, wherein the first image is generated by an AR platform on the device.

4. The method of claim 1, wherein the modifying is performed by an AR photo-editing application on the device.

5. The method of claim 1, wherein the first or second contextual environment data includes corresponding pose information.

6. The method of claim 1, wherein the first or second contextual environment data includes corresponding estimated lighting information.

7. The method of claim 1, wherein the first or second contextual environment data includes corresponding planar surface information.

8. The method of claim 1, wherein the first metadata or the second metadata includes the AR object or an identifier associated with the AR object.

9. The method of claim 1, wherein the first metadata is embedded in the first image using a serialization procedure.

10. The method of claim 1, wherein the first image is generated on a first device and the second image is generated on a second device.

11. The method of claim 10, wherein the first image is generated by an AR platform on the first device and the modifying is performed by an AR photo-editing application on the second device.

12. A method, comprising:

generating a first video including a composite of an environment captured by a device and a virtually-rendered augmented reality (AR) object, the first video embedded with a first metadata including;

a first contextual environment data, and the environment captured by the device without the virtually-rendered AR object; and

generating a second video by modifying the first video, the second video embedded with a second metadata, that is different from the first metadata and generated based on the first metadata, and the second metadata including;

a second contextual environment data, and the environment captured by the device without the virtually-rendered AR object.

13. The method of claim 12, wherein the first metadata or the second metadata includes:

frame metadata per each frame of the first video or the second video; and

global metadata for the first video or the second video.

14. The method of claim 13, wherein the global metadata includes one or more of:

camera intrinsics, planar surfaces, point clouds, and global application metadata.

15. The method of claim 13, wherein the frame metadata includes one or more of:

camera pose, lighting estimate information, and per-frame application metadata.

16. The method of claim 12, wherein the first video is generated by an AR platform on the device and the modifying is performed by an AR video-editing application on the device.

17. An apparatus, comprising:

a processor; and

a memory, the memory including instructions configured to cause the processor to;

generate a first image that includes a composite of an environment captured by a device and a virtually-rendered augmented reality (AR) object composited with the environment, the first image embedded with a first metadata including;

a first contextual environment data, and the environment captured by the device without the virtually-rendered AR object; and

generate a second image by modifying the first image, the second image embedded with a second metadata that is different from the first metadata and generated based on the first metadata, and the second metadata including;

a second contextual environment data, and the environment captured by the device without the virtually-rendered AR object.

18. The apparatus of claim 17, wherein the modifying includes at least one of altering, moving, or replacing the AR object of the first image for generating the second image.

19. The apparatus of claim 17, wherein the first or second metadata includes the first or second contextual environment data, respectively, and wherein the first or second contextual environment data includes one or more of pose information, estimated lighting information, and planar surface information.

20. A non-transitory computer-readable storage medium having stored thereon computer executable program code which, when executed on a computer system, causes the computer system to perform a method, comprising:

generating a first image including a composite of an environment captured by a device and a virtually-rendered augmented reality (AR) object composited with the environment, the first image embedded with a first metadata including;

a first contextual environment data, and the environment captured by the device without the virtually-rendered AR object; and

generating a second image by modifying the first image, the second image embedded with a second metadata that is different from the first metadata and generated based on the first metadata, and the second metadata including;

a second contextual environment data, and the environment captured by the device without the virtually-rendered AR object.