Method and system for presenting at least part of an image of a real object in a view of a real environment, and method and system for selecting a subset of a plurality of images

• US 10,659,750 B2
• Filed: 07/23/2014
• Issued: 05/19/2020
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Meier, Peter, Wang, Lejing, Misslinger, Stefan
• Primary Examiner(s): Le, Peter D
• Application Number:
  US15/328,349
• Publication Number:
  US 20170214899A1
• Time in Patent Office: 2,127 Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The disclosure relates to a method and system for presenting at least part of an image of a real object in a view of a real environment, comprising providing a first image of at least part of a real object captured by a first camera, determining at least part of the real object in the first image as an object image area, determining a first 3D plane relative to the first camera, the first camera being at a position where the first image is captured, providing at least one image feature related to the real object in the first image, providing at least one first ray passing an optical center of the first camera being at a position where the first image is captured and the at least one image feature, determining, according to a first plane normal direction of the first 3D plane, at least one first angle between the first 3D plane and the at least one first ray, providing a second image of a real environment captured by a second camera, determining a second 3D plane relative to the second camera, the second camera being at a position where the second image is captured, mapping at least part of the object image area to a target space in a view of the real environment captured at a viewpoint, wherein the at least one image feature is mapped to at least one target position in a coordinate system of the view, determining at least one second angle between the second 3D plane and at least one second ray passing the viewpoint of capturing the view and the at least one target position according to a second plane normal direction of the second 3D plane, wherein the at least one image feature is mapped to the at least one target position such that a difference between the at least one first angle and the at least one second angle is minimized.

Claims

1. A method of presenting at least part of an image of a real object in a view of a real environment, comprising:

obtaining a first image of at least part of a real object captured by a camera from a first camera pose, wherein a portion of the first image comprising the real object comprises an object image area;

determining a first 3D plane on which the real object is placed;

determining at least one first ray passing through an optical center of the camera at the first camera pose and the object image area;

determining a first spatial relationship comprising at least one first angle between the first 3D plane and the at least one first ray, and a depth of the first 3D plane;

obtaining a second image of a real environment captured from a second camera pose;

determining a target space in a view of the real environment captured from the second camera pose, wherein the target space comprises a second 3D plane;

determining a second spatial relationship comprising at least one second angle between the second 3D plane and at least one second ray passing from the second camera pose to the target space, and a depth of the second 3D plane;

applying one or more warping functions to the object image area based on the first spatial relationship and the second spatial relationship to obtain a warped object image area; and

blending in the warped object image area into the second image, wherein the at least part of the real object appears from the perspective of the second camera pose.

2. The method according to claim 1, wherein determining a first spatial relationship comprises:

determining a first plane distance between the first 3D plane and the first camera pose according to at least one selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, and vision based pose estimation; and

determining a first plane normal direction of the first 3D plane relative to the first camera pose according to at least one of selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, vision based pose estimation, vanishing points, and at least one orientation sensor,

wherein the at least one first angle is determined according to the first plane normal direction.

3. The method according to claim 2, wherein

the at least one warping function is determined according to at least one of the first plane distance between the first surface and the first camera pose, the first plane normal direction, a second plane distance between the second surface and the second camera pose, and a second plane normal direction,

wherein the object image area is warped according to the at least one warping function.

4. The method according to claim 1, wherein the object image area is determined according to the depth information associated with the first image.

5. The method according to claim 4, wherein applying the one or more warping functions further comprises:

obtaining a plurality of image blocks contained in at least part of the object image area;

determining, for each respective image block of the plurality of image blocks, a respective warping function according to depth information associated with the respective image block; and

warping the respective image block to the target space according to the respective warping function.

6. The method according to claim 5, wherein the respective warping function comprises a homography.

7. The method according to claim 1, further comprising:

displaying the at least part of the warped object image area on a display device in the view of the real environment of the second image.

8. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:

obtain a first image of at least part of a real object captured by a camera from a first camera pose, wherein a portion of the first image comprising the real object comprises an object image area;

determine a first 3D plane on which the real object is placed;

determine at least one first ray passing through an optical center of the camera at the first camera pose and the object image area;

determine a first spatial relationship comprising at least one first angle between the first 3D plane and the at least one first ray, and a depth of the first 3D plane;

obtain a second image of a real environment captured from a second camera pose;

determine a target space in a view of the real environment captured from the second camera pose, wherein the target space comprises a second 3D plane;

determine a second spatial relationship comprising at least one second angle between the second 3D plane and at least one second ray passing from the second camera pose to the target space, and a depth of the second 3D plane;

apply one or more warping functions to the object image area based on the first spatial relationship and the second spatial relationship to obtain a warped object image area; and

blending in the warped object image area into the second image, wherein the at least part of the real object appears from the perspective of the second camera pose.

9. The non-transitory computer readable medium of claim 8, wherein the computer readable code to determine a first spatial relationship comprises computer readable code to:

determine a first plane distance between the first 3D plane and the first camera pose according to at least one selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, and vision based pose estimation; and

determine a first plane normal direction of the first 3D plane relative to the first camera pose according to at least one of selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, vision based pose estimation, vanishing points, and at least one orientation sensor,

wherein the at least one first angle is determined according to the first plane normal direction.

10. The non-transitory computer readable medium of claim 9, wherein the

at least one warping function is determined according to at least one of the first plane distance between the first surface and the first camera pose, the first plane normal direction, a second plane distance between the second surface and the second camera pose, and a second plane normal direction,

wherein the object image area is warped according to the warping function.

11. The non-transitory computer readable medium of claim 8, wherein the object image area is determined according to depth information associated with the first image.

12. The non-transitory computer readable medium of claim 11, wherein the computer readable code to apply the one or more warping functions further comprises computer readable code to:

obtain a plurality of image blocks contained in at least part of the object image area;

determine, for each respective image block of the plurality of image blocks, a respective warping function according to depth information associated with the respective image block; and

warp the respective image block to the target space according to the respective warping function.

13. The non-transitory computer readable medium of claim 12, wherein the respective warping function comprises a homography.

14. The non-transitory computer readable medium of claim 8, further comprising computer readable code to:

display the at least part of the warped object image area on a display device in the view of the real environment of the second image.

15. A system for presenting at least part of an image of a real object in a view of a real environment, comprising:

one or more processors; and

a memory coupled to the one or more processors and comprising computer readable code executable by the one or more processors to cause the system to;

obtain a first image of at least part of a real object captured by a camera from a first camera pose, wherein a portion of the first image comprising the real object comprises an object image area;

determine a first 3D plane on which the real object is placed;

determine at least one first ray passing through an optical center of the camera at the first camera pose and the object image area;

determine a first spatial relationship comprising at least one first angle between the first 3D plane and the at least one first ray, and a depth of the first 3D plane;

obtain a second image of a real environment captured from a second camera pose;

determine a target space in a view of the real environment captured from the second camera pose, wherein the target space comprises a second 3D plane;

determine a second spatial relationship comprising at least one second angle between the second 3D plane and at least one second ray passing from the second camera pose to the target space, and a depth of the second 3D plane;

apply one or more warping functions to the object image area based on the first spatial relationship and the second spatial relationship to obtain a warped object image area; and

blending in the warped object image area into the second image, wherein the at least part of the real object appears from the perspective of the second camera pose.

16. The system of claim 15, wherein the computer readable code to determine a first spatial relationship comprises computer readable code to:

determine a first plane distance between the first 3D plane and the first camera pose according to at least one selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, and vision based pose estimation; and

determine a first plane normal direction of the first 3D plane relative to the first camera pose according to at least one of selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, vision based pose estimation, vanishing points, and at least one orientation sensor,

wherein the at least one first angle is determined according to the first plane normal direction.

17. The system of claim 16, wherein the

at least one warping function is determined according to at least one of the first plane distance between the first surface and the first camera pose, the first plane normal direction, a second plane distance between the second surface and the second camera pose, and a second plane normal direction,

wherein the object image area is mapped to the target space warped according to the warping function.

18. The system of claim 15, wherein the object image area is determined according to depth information associated with the first image.

19. The system of claim 18, wherein the computer readable code to apply the one or more warping functions further comprises computer readable code to:

obtain a plurality of image blocks contained in at least part of the object image area;

determine, for each respective image block of the plurality of image blocks, a respective warping function according to depth information associated with the respective image block; and

warp the respective image block to the target space according to the respective warping function.

20. The system of claim 15, further comprising computer readable code to:

display the at least part of the warped object image area on a display device in the view of the real environment of the second image.