Diminished and mediated reality effects from reconstruction

• US 9,269,003 B2
• Filed: 03/25/2014
• Issued: 02/23/2016
• Est. Priority Date: 04/30/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Schmalstieg, Dieter
• Primary Examiner(s): Azarian, Seyed
• Application Number:
  US14/225,157
• Publication Number:
  US 20140321702A1
• Time in Patent Office: 700 Days
• Field of Search: 382/100, 382/103, 382106-107, 382/140, 382/154, 382/162, 382/168, 382/173, 382/181, 382/199, 382/203, 382/209, 382/216, 382/219, 382/254, 382/266, 382/274, 382/276, 382285-291, 382/305, 382/312, 382/321, 345/420, 345/473, 345/419, 455/556.1
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Disclosed embodiments pertain to apparatus, systems, and methods for mixed reality. In some embodiments, a camera pose relative to a tracked object in a live image may be determined and used to render synthetic images from keyframes in a 3D model without the tracked object. Optical flow magnitudes for pixels in a first mask region relative to a subset of the synthetic images may be determined and the optical flow magnitudes may be used to determine pixels in each of the subset of synthetic images that correspond to pixels in the first mask. For each pixel in the first mask, a corresponding replacement pixel may be determined as a function of pixels in the subset of synthetic images that correspond to the corresponding pixel in the first mask.

Claims

1. A processor-implemented method comprising:

obtaining a plurality of live images, the plurality of live images comprising a tracked object;

determining, for a live image in the plurality of live images, a camera pose relative to the tracked object, wherein the camera pose is determined using Simultaneous Localization and Mapping (SLAM) based techniques;

rendering, based on the camera pose, a plurality of synthetic images without the tracked object, each synthetic image obtained from a distinct keyframe in a 3D model;

comparing the live image and at least one of the plurality of synthetic images to determine a first mask region associated with the tracked object;

computing optical flow magnitudes for a plurality of pixels in the first mask region associated with the tracked object relative to a subset of synthetic images in the plurality of synthetic images;

determining, for each of the plurality of pixels in a first mask corresponding to the first mask region, a corresponding pixel in each synthetic image of the subset of synthetic images based, in part, on the optical flow magnitude; and

determining replacement pixels for the plurality of pixels in the first mask, each replacement pixel corresponding to a distinct pixel in the first mask, wherein each replacement pixel is determined as a function of the corresponding pixels in the subset of synthetic images that correspond to the pixel in the first mask.

2. The method of claim 1, further comprising rendering a diminished reality image, the diminished reality image obtained based, at least in part, on the replacement pixels.

3. The method of claim 2, further comprising:

rendering a virtual object in the first mask region, the virtual object being rendered based on the current camera pose.

4. The method of claim 3, wherein rendering the virtual object in the first mask region comprises:

rendering a virtual representation of at least one known real object in the live image, into a virtual z-buffer, the real object at least partially in the first mask;

resolving occlusions between the at least one known real object and the virtual object in the virtual z-buffer; and

warping the virtual z-buffer to match the image, the warping based, in part, on an optical flow between the virtual representation of the at least one known real object and the at least one real object in the image.

5. The method of claim 1, wherein determining the corresponding pixel in each synthetic image of the subset of synthetic images comprises:

obtaining, for each of the plurality of pixels in the first mask, a displacement correction and an intensity correction based, in part, on the optical flow magnitude.

6. The method of claim 1, wherein:

each replacement pixel is determined as a weighted average of corresponding pixels in the subset of synthetic images, wherein a weight associated with a corresponding pixel in a synthetic image in the subset of synthetic images, is based, in part, on the optical flow magnitude for the corresponding pixel, and an angular displacement between the live image and the synthetic image associated with the corresponding pixel.

7. The method of claim 6, wherein the first mask (M′

) region comprises a second mask region (M) around the tracked object and wherein the optical flow magnitude for each replacement pixel in M is determined by scattered data interpolation from a boundary of M′

-M.

8. The method of claim 1, wherein the plurality of synthetic images and the subset of the plurality of synthetic images each comprise three synthetic images.

9. A Mobile Station (MS) comprising:

a camera configured to capture a plurality of live images comprising a tracked object,

a memory, the memory to store a 3D model comprising a plurality of keyframes, and

a processor coupled to the camera and the memory, wherein the processor is configured to;

determine, for a live image in the plurality of live images, a camera pose relative to the tracked object, wherein the camera pose is determined using Simultaneous Localization and Mapping (SLAM) based techniques;

render, based on the camera pose, a plurality of synthetic images without the tracked object, each synthetic image obtained from a distinct keyframe;

compare the live image and at least one of the plurality of synthetic images to determine a first mask region associated with the tracked object;

compute optical flow magnitudes for a plurality of pixels in the first mask region associated with the tracked object relative to a subset of synthetic images in the plurality of synthetic images;

determine, for each of the plurality of pixels in a first mask corresponding to the first mask region, a corresponding pixel in each synthetic image of the subset of synthetic images based, in part, on the optical flow magnitude; and

determine replacement pixels for the plurality of pixels in the first mask, each replacement pixel corresponding to a distinct pixel in the first mask, wherein each replacement pixel is determined as a function of the corresponding pixels in the subset of synthetic images that correspond to the pixel in the first mask.

10. The MS of claim 9, wherein the processor is further configured to:

render a diminished reality image, the diminished reality image obtained based, at least in part, on the replacement pixels.

11. The MS of claim 10, wherein the processor is further configured to:

render a virtual object in the first mask region, the virtual object being rendered based on the current camera pose.

12. The MS of claim 11, wherein to render the virtual object in the first mask region, the processor is configured to:

render a virtual representation of at least one known real object in the live image, into a virtual z-buffer, the real object at least partially in the first mask;

resolve occlusions between the at least one known real object and the virtual object in the virtual z-buffer; and

warp the virtual z-buffer to match the image, the warping based, in part, on an optical flow between the virtual representation of the at least one known real object and the at least one real object in the image.

13. The MS of claim 9, wherein to determine the corresponding pixel in each synthetic image of the subset of synthetic images, the processor is configured to:

obtain, for each of the plurality of pixels in the first mask, a displacement correction and an intensity correction based, in part, on the optical flow magnitude.

14. The MS of claim 9, wherein:

each replacement pixel is determined as a weighted average of corresponding pixels in the subset of synthetic images, wherein a weight associated with a corresponding pixel in a synthetic image in the subset of synthetic images, is based, in part, on the optical flow magnitude for the corresponding pixel, and an angular displacement between the live image and the synthetic image associated with the corresponding pixel.

15. The MS of claim 14, wherein:

the first mask (M′

) region comprises a second mask region (M) around the tracked object and wherein the optical flow magnitude for each replacement pixel in M is determined by scattered data interpolation from a boundary of M′

-M.

16. The MS of claim 9, wherein the plurality of synthetic images and the subset of the plurality of synthetic images each comprise three synthetic images.

17. An apparatus comprising:

imaging means, the imaging means to capture a plurality of live images comprising a tracked object,

storage means, the storage means to store a 3D model comprising a plurality of keyframes, and

processing means, the processing means coupled to the imaging means and the storage means, the processing means further comprising;

means for determining, for a live image in the plurality of live images, an imaging means pose relative to the tracked object, wherein the imaging means pose is determined using Simultaneous Localization and Mapping (SLAM) based techniques;

means for rendering, based on the imaging means pose, a plurality of synthetic images without the tracked object, each synthetic image obtained from a distinct keyframe;

means for comparing the live image and at least one of the plurality of synthetic images to determine a first mask region associated with the tracked object;

means for computing optical flow magnitudes for a plurality of pixels in the first mask region associated with the tracked object relative to a subset of synthetic images in the plurality of synthetic images;

means for determining, for each of the plurality of pixels in a first mask corresponding to the first mask region, a corresponding pixel in each synthetic image of the subset of synthetic images based, in part, on the optical flow magnitude; and

means for determining replacement pixels of the plurality of pixels in the first mask, each replacement pixel corresponding to a distinct pixel in the first mask, wherein each replacement pixel is determined as a function of the corresponding pixels in the subset of synthetic images that correspond to the pixel in the first mask.

18. The apparatus of claim 17, wherein the processing means further comprises:

means for rendering a diminished reality image, the diminished reality image obtained based, at least in part, on the replacement pixels.

19. The apparatus of claim 18, wherein the processing means further comprises:

means for rendering a virtual object in the first mask region, the virtual object being rendered based on the current imaging means pose.

20. The apparatus of claim 19, wherein means for rendering the virtual object in the first mask region comprises:

means for rendering a virtual representation of at least one known real object in the live image, into a virtual z-buffer, the real object at least partially in the first mask;

means for resolving occlusions between the at least one known real object and the virtual object in the virtual z-buffer; and

means for warping the virtual z-buffer to match the image, the warping based, in part, on an optical flow between the virtual representation of the at least one known real object and the at least one real object in the image.

21. The apparatus of claim 17, wherein:

each replacement pixel is determined as a weighted average of corresponding pixels in the subset of synthetic images, wherein a weight associated with a corresponding pixel in a synthetic image in the subset of synthetic images, is based, in part, on the optical flow magnitude for the corresponding pixel, and an angular displacement between the live image and the synthetic image associated with the corresponding pixel.

22. A non-transitory computer-readable medium comprising instructions, which, when executed by a processor, perform steps in a method comprising:

obtaining a plurality of live images, the live images comprising a tracked object;

determining, for a live image in the plurality of live images, a camera pose relative to the tracked object, wherein the camera pose is determined using Simultaneous Localization and Mapping (SLAM) based techniques;

rendering, based on the camera pose, a plurality of synthetic images without the tracked object, each synthetic image obtained from a distinct keyframe in a 3D model based on the camera pose;

comparing the live image and at least one of the plurality of synthetic images to determine a first mask region associated with the tracked object;

computing optical flow magnitudes for a plurality of pixels in the first mask region associated with the tracked object relative to a subset of synthetic images in the plurality of synthetic images;

determining, for each of the plurality of pixels in a first mask corresponding to the first mask region, a corresponding pixel in each synthetic image of the subset of synthetic images based, in part, on the optical flow magnitude; and

determining replacement pixels of the plurality of pixels in the first mask, each replacement pixel corresponding to a distinct pixel in the first mask, wherein each replacement pixel is determined as a function of the corresponding pixels in the subset of synthetic images that correspond to the pixel in the first mask.

23. The computer-readable medium of claim 22, further comprising rendering a diminished reality image, the diminished reality image obtained based, at least in part, on the replacement pixels.

24. The computer-readable medium of claim 23, further comprising, rendering a virtual object in the first mask region, the virtual object being rendered based on the current camera pose.

25. The computer-readable medium of claim 24, wherein rendering the virtual object in the first mask region comprises:

rendering a virtual representation of at least one known real object in the image, into a virtual z-buffer, the real object at least partially in the first mask;

resolving occlusions between the at least one known real object and the virtual object in the virtual z-buffer; and

warping the virtual z-buffer to match the image, the warping based, in part, on an optical flow between the virtual representation of the at least one known real object and the at least one real object in the image.

26. The computer-readable medium of claim 22, wherein determining the corresponding pixel in each synthetic image of the subset of synthetic images comprises:

obtaining, for each of the plurality of pixels in the first mask, a displacement correction and an intensity correction based, in part, on the optical flow magnitude.

27. The computer-readable medium of claim 22, wherein:

each replacement pixel is determined as a weighted average of corresponding pixels in the subset of synthetic images, wherein a weight associated with a corresponding pixel in a synthetic image in the subset of synthetic images, is based, in part, on the optical flow magnitude for the corresponding pixel, and an angular displacement between the live image and the synthetic image associated with the corresponding pixel.

28. The computer-readable medium of claim 27, wherein the first mask (M′

) region comprises a second mask region (M) around the tracked object and wherein the optical flow magnitude for each replacement pixel in M is determined by scattered data interpolation from a boundary of M′

-M.