METHOD AND SYSTEM FOR PRESENTING AT LEAST PART OF AN IMAGE OF A REAL OBJECT IN A VIEW OF A REAL ENVIRONMENT, AND METHOD AND SYSTEM FOR SELECTING A SUBSET OF A PLURALITY OF IMAGES

• US 20170214899A1
• Filed: 07/23/2014
• Published: 07/27/2017
• Est. Priority Date: 07/23/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Wang, Lejing, Misslinger, Stefan, Meier, Peter
• Granted Patent: US 10,659,750 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The disclosure relates to a method and system for presenting at least part of an image of a real object in a view of a real environment, comprising providing a first image of at least part of a real object captured by a first camera, determining at least part of the real object in the first image as an object image area, determining a first 3D plane relative to the first camera, the first camera being at a position where the first image is captured, providing at least one image feature related to the real object in the first image, providing at least one first ray passing an optical center of the first camera being at a position where the first image is captured and the at least one image feature, determining, according to a first plane normal direction of the first 3D plane, at least one first angle between the first 3D plane and the at least one first ray, providing a second image of a real environment captured by a second camera, determining a second 3D plane relative to the second camera, the second camera being at a position where the second image is captured, mapping at least part of the object image area to a target space in a view of the real environment captured at a viewpoint, wherein the at least one image feature is mapped to at least one target position in a coordinate system of the view, determining at least one second angle between the second 3D plane and at least one second ray passing the viewpoint of capturing the view and the at least one target position according to a second plane normal direction of the second 3D plane, wherein the at least one image feature is mapped to the at least one target position such that a difference between the at least one first angle and the at least one second angle is minimized.

Claims

1-30. -30. (canceled)

31. A method of presenting at least part of an image of a real object in a view of a real environment, comprising

obtaining a first image of at least part of a real object captured from a first camera pose, wherein the portion of the first image comprising the real object comprises an object image area;

determining a first surface associated with the real object in the first image;

detecting at least one image feature corresponding to the real object in the first image;

determining at least one first ray passing through an optical center of a camera at the first camera pose and the at least one image feature;

determining a first spatial relationship comprising at least one first angle between the first surface and the at least one first ray;

obtaining a second image of a real environment captured from a second camera pose;

determining a second surface in the real environment;

determining a target space in a view of the real environment captured from the second camera pose, wherein the target space comprises the second surface;

determining a second spatial relationship comprising at least one second angle between the second surface and at least one second ray passing from the second camera pose to the at least one target position; and

mapping the at least part of the real object in the target space of the second image so as to optimize the first spatial relationship and the second spatial relationship and such that the at least part of the object image area appears from the second camera pose.

32. The method according to claim 31, wherein determining a first spatial relationship comprises:

determining a first plane distance between the first surface and the first camera pose according to at least one selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, and vision based pose estimation; and

determining a first plane normal direction of the first surface relative to the first camera pose according to at least one of selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, vision based pose estimation, vanishing points, and at least one orientation sensor,

wherein the at least one first angle is determined according to the first plane normal direction.

33. The method according to claim 32, wherein mapping the at least part of the object image area to the target space further comprises:

determining a warping function according to at least one of a first plane distance between the first surface and the first camera pose, the first plane normal direction, a second plane distance between the second surface and the second camera pose, and a second plane normal direction,

wherein the object image area is mapped to the target space according to the warping function.

34. The method according to claim 31, further comprising obtaining depth information associated with the first image, wherein the object image area is determined according to the depth information associated with the first image.

35. The method according to claim 34, wherein mapping the at least part of the object image area to the target space further comprises:

obtaining a plurality of image blocks contained in at least part of the object image area;

determining, for each respective image block of the plurality of image blocks, a respective warping function according to depth information associated with the respective image block; and

mapping the respective image block to the target space according to the respective warping function.

36. The method according to claim 35, wherein the respective warping function comprises a homography.

37. The method according to claim 31, further comprising:

displaying the at least part of the object image area on a display device in the view of the real environment of the second image according to the mapping.

38. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to:

obtain a first image of at least part of a real object captured from a first camera pose, wherein the portion of the first image comprising the real object comprises an object image area;

determine a first surface associated with the real object in the first image;

detect at least one image feature corresponding to the real object in the first image;

determine at least one first ray passing through an optical center of a camera at the first camera pose and the at least one image feature;

determine a first spatial relationship comprising at least one first angle between the first surface and the at least one first ray;

obtain a second image of a real environment captured from a second camera pose;

determine a second surface in the real environment;

determine a target space in a view of the real environment captured from the second camera pose, wherein the target space comprises the second surface;

determine a second spatial relationship comprising at least one second angle between the second surface and at least one second ray passing from the second camera pose to the at least one target position; and

map the at least part of the real object in the target space of the second image so as to optimize the first spatial relationship and the second spatial relationship and such that the at least part of the object image area appears from the second camera pose.

39. The non-transitory computer readable medium of claim 38, wherein the computer readable code to determine a first spatial relationship comprises computer readable code to:

determine a first plane distance between the first surface and the first camera pose according to at least one selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, and vision based pose estimation; and

determine a first plane normal direction of the first surface relative to the first camera pose according to at least one of selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, vision based pose estimation, vanishing points, and at least one orientation sensor,

wherein the at least one first angle is determined according to the first plane normal direction.

40. The non-transitory computer readable medium of claim 39, wherein the computer readable code to map the at least part of the object image area to the target space further comprises computer readable code to:

determine a warping function according to at least one of a first plane distance between the first surface and the first camera pose, the first plane normal direction, a second plane distance between the second surface and the second camera pose, and a second plane normal direction,

wherein the object image area is mapped to the target space according to the warping function.

41. The non-transitory computer readable medium of claim 38, further comprising computer readable code to obtain depth information associated with the first image, wherein the object image area is determined according to the depth information associated with the first image.

42. The non-transitory computer readable medium of claim 41, wherein the computer readable code to map the at least part of the object image area to the target space further comprises computer readable code to:

obtain a plurality of image blocks contained in at least part of the object image area;

determine, for each respective image block of the plurality of image blocks, a respective warping function according to depth information associated with the respective image block; and

map the respective image block to the target space according to the respective warping function.

43. The non-transitory computer readable medium of claim 42, wherein the respective warping function comprises a homography.

44. The non-transitory computer readable medium of claim 38, further comprising computer readable code to:

display the at least part of the object image area on a display device in the view of the real environment of the second image according to the mapping.

45. A system for presenting at least part of an image of a real object in a view of a real environment, comprising:

one or more processors; and

a memory coupled to the one or more processors and comprising computer readable code executable by the one or more processors to cause the system to;

obtain a first image of at least part of a real object captured from a first camera pose, wherein the portion of the first image comprising the real object comprises an object image area;

determine a first surface associated with the real object in the first image;

detect at least one image feature corresponding to the real object in the first image;

determine at least one first ray passing through an optical center of a camera at the first camera pose and the at least one image feature;

determine a first spatial relationship comprising at least one first angle between the first surface and the at least one first ray;

obtain a second image of a real environment captured from a second camera pose;

determine a second surface in the real environment;

determine a target space in a view of the real environment captured from the second camera pose, wherein the target space comprises the second surface;

determine a second spatial relationship comprising at least one second angle between the second surface and at least one second ray passing from the second camera pose to the at least one target position; and

map the at least part of the real object in the target space of the second image so as to optimize the first spatial relationship and the second spatial relationship and such that the at least part of the object image area appears from the second camera pose.

46. The system of claim 45, wherein the computer readable code to determine a first spatial relationship comprises computer readable code to:

determine a first plane distance between the first surface and the first camera pose according to at least one selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, and vision based pose estimation; and

determine a first plane normal direction of the first surface relative to the first camera pose according to at least one of selected from a group consisting of a manual input, user-interaction, depth information produced by the first camera, vision based pose estimation, vanishing points, and at least one orientation sensor,

wherein the at least one first angle is determined according to the first plane normal direction.

47. The system of claim 46, wherein the computer readable code to map the at least part of the object image area to the target space further comprises computer readable code to:

determine a warping function according to at least one of a first plane distance between the first surface and the first camera pose, the first plane normal direction, a second plane distance between the second surface and the second camera pose, and a second plane normal direction,

wherein the object image area is mapped to the target space according to the warping function.

48. The system of claim 45, further comprising computer readable code to obtain depth information associated with the first image, wherein the object image area is determined according to the depth information associated with the first image.

49. The system of claim 48, wherein the computer readable code to map the at least part of the object image area to the target space further comprises computer readable code to:

obtain a plurality of image blocks contained in at least part of the object image area;

determine, for each respective image block of the plurality of image blocks, a respective warping function according to depth information associated with the respective image block; and

map the respective image block to the target space according to the respective warping function.

50. The system of claim 45, further comprising computer readable code to:

display the at least part of the object image area on a display device in the view of the real environment of the second image according to the mapping.