Method and apparatus for image processing in augmented reality systems

• US 9,626,803 B2
• Filed: 12/12/2014
• Issued: 04/18/2017
• Est. Priority Date: 12/12/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Park, Youngmin, Mendez Mendez, Erick, Reitmayr, Gerhard, Wagner, Daniel, Diaz Spindola, Serafin
• Primary Examiner(s): Li, Tracy Y
• Application Number:
  US14/569,549
• Publication Number:
  US 20160171768A1
• Time in Patent Office: 858 Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

Disclosed are a system, apparatus, and method for depth and color camera image synchronization. Depth and color camera input images are received or otherwise obtained unsynchronized and without associated creation timestamps. An image of one type is compared with an image of a different type to determine a match for synchronization. Matches may be determined according to edge detection or depth coordinate detection. When a match is determined a synchronized pair is formed for processing within an augmented reality output. Optionally the synchronized pair may be transformed to improve the match between the image pair.

Claims

1. A method for depth and color camera image synchronization, the method comprising:

obtaining a plurality of input images comprising a first image type and a second image type different from the first image type, wherein an image type is either a depth camera image type or color camera image type, wherein each image type is unsynchronized with respect to images of other types, and wherein each of the plurality of input images of the first image type and of the second image type are without identification of an input image'"'"'s actual creation time;

selecting, from input images of the first image type, a target image;

selecting, from input images of the second image type, a comparison image;

determining whether the comparison image is compatible for synchronization with the target image;

in response to a determination that the comparison image is compatible for synchronization with the target image, identifying the target image and the comparison image as a synchronized image pair; and

processing the synchronized image pair comprising the target image of the first image type and the comparison image of the second image type together for use in an augmented reality output.

2. The method of claim 1, wherein the target image is a color camera image type, and the comparison image is a depth camera image type, and wherein determining whether the comparison image is compatible for synchronization with the target image further comprises:

tracking three-dimensional (3D) coordinates from the plurality of input images of the color camera image type, wherein the 3D coordinates include depth values;

determining, within the comparison image, depth values for each point having a geometrically equivalent point to the coordinates tracked in the target image;

comparing the depth values within the comparison image to depth values within the target image; and

determining a compatibility score according to a number of depth value matches between the target image and the comparison image.

3. The method of claim 1, wherein determining whether the comparison image is compatible for synchronization with the target image further comprises:

performing edge detection on the target image and the comparison image; and

determining a compatibility score according to a number of same edge matches between the target image and the comparison image.

4. The method of claim 1, further comprising:

determining a time of receipt for each of the plurality of input images;

creating a comparison queue comprising a subset of images from the plurality of input images, wherein the subset of images in the comparison queue comprises images of the second image type received within a threshold time from receipt of the target image, and wherein the comparison image is selected from the comparison queue.

5. The method of claim 4, wherein the comparison queue comprises a threshold number of images of the second image type received before and after receipt of the target image.

6. The method of claim 1, further comprising:

determining a timing offset between receipt time of the comparison image and the receipt time of the target image; and

referencing the timing offset to determine a next synchronized pair.

7. The method of claim 6, further comprising:

tracking one or both of an amount of time passed or a number of images processed;

determining one or both of threshold amount of time passed or number of images processed is met;

determining compatibility between a next target image and a next comparison image; and

assigning a next synchronized image pair of together for use in the augmented reality output.

8. The method of claim 1, further comprising:

transforming one or both of the target image or the comparison image according to parameters extracted from one or both of the target image or the comparison image.

9. A device for depth and color camera image synchronization comprising:

memory; and

a processor coupled to the memory and configured to;

obtain, from the memory, a plurality of input images comprising a first image type and a second image type different from the first image type, wherein an image type is either a depth camera image type or color camera image type, wherein each image type is unsynchronized with respect to images of other types, and wherein each of the plurality of input images of the first image type and of the second image type are without identification of an input image'"'"'s actual creation time;

select, from input images of the first image type, a target image;

select, from input images of the second image type, a comparison image;

determine whether the comparison image is compatible for synchronization with the target image;

in response to a determination that the comparison image is compatible for synchronization with the target image, identify the target image and the comparison image as a synchronized image pair; and

process the synchronized image pair comprising the target image of the first image type and the comparison image of the second image type together for use in an augmented reality output.

10. The device of claim 9, wherein the target image is a color camera image type, and the comparison image is a depth camera image type, and wherein to determine whether the comparison image is compatible for synchronization with the target image the processor is further configured to:

track three-dimensional (3D) coordinates from the plurality of input images of the target image, wherein the 3D coordinates include depth values;

determine, within the comparison image, depth values for each point having a geometrically equivalent point to the coordinates tracked in the target image;

compare the depth values within the comparison image to depth values within the target image; and

determine a compatibility score according to a number of depth value matches between the target image and the comparison image.

11. The device of claim 9, wherein to determine whether the comparison image is compatible for synchronization with the target image the processor the processor is further configured to:

perform edge detection on the target image and the comparison image; and

determine a compatibility score according to a number of same edge matches between the target image and the comparison image.

12. The device of claim 9, wherein the processor is further configured to:

determine a time of receipt for each of the plurality of input images;

create a comparison queue comprising a subset of images from the plurality of input images, wherein the subset of images in the comparison queue comprises images of the second image type received within a threshold time from receipt of the target image, and wherein the comparison image is selected from the comparison queue.

13. The device of claim 12, wherein the comparison queue comprises a threshold number of images of the second image type received before and after receipt of the target image.

14. The device of claim 9, wherein the processor is further configured to:

determine a timing offset between receipt time of the comparison image and the receipt time of the target image; and

reference the timing offset to determine a next synchronized pair.

15. The device of claim 14, wherein the processor is further configured to:

track one or both of an amount of time passed or a number of images processed;

determine one or both of threshold amount of time passed or number of images processed is met;

determine compatibility between a next target image and a next comparison image; and

assign a next synchronized image pair of together for use in the augmented reality output.

16. The device of claim 9, wherein the processor is further configured to:

transforming one or both of the target image or the comparison image according to parameters extracted from one or both of the target image or the comparison image.

17. A machine readable non-transitory storage medium having stored therein program instructions that are executable by a processor to:

obtain a plurality of input images comprising a first image type and a second image type different from the first image type, wherein an image type is either a depth camera image type or color camera image type, wherein each image type is unsynchronized with respect to images of other types, and wherein each of the plurality of input images of the first image type and of the second image type are without identification of an input image'"'"'s actual creation time;

select, from input images of the first image type, a target image;

select, from input images of the second image type, a comparison image;

determine whether the comparison image is compatible for synchronization with the target image;

in response to a determination that the comparison image is compatible for synchronization with the target image, identify the target image and the comparison image as a synchronized image pair; and

process the synchronized image pair comprising the target image of the first image type and the comparison image of the second image type together for use in an augmented reality output.

18. The medium of claim 17, wherein the target image is a color camera image type, and the comparison image is a depth camera image type, and wherein to determine whether the comparison image is compatible for synchronization with the target image the program instructions are further executable by the processor to:

track three-dimensional (3D) coordinates from the plurality of input images of the target image, wherein the 3D coordinates include depth values;

determine, within the comparison image, depth values for each point having a geometrically equivalent point to the coordinates tracked in the target image;

compare the depth values within the comparison image to depth values within the target image; and

determine a compatibility score according to a number of depth value matches between the target image and the comparison image.

19. The medium of claim 17, wherein to determine whether the comparison image is compatible for synchronization with the target image the program instructions are further executable by the processor to:

perform edge detection on the target image and the comparison image; and

determine a compatibility score according to a number of same edge matches between the target image and the comparison image.

20. The medium of claim 17, wherein the program instructions are further executable by the processor to:

determine a time of receipt for each of the plurality of input images;

create a comparison queue comprising a subset of images from the plurality of input images, wherein the subset of images in the comparison queue comprises images of the second image type received within a threshold time from receipt of the target image, and wherein the comparison image is selected from the comparison queue.

21. The medium of claim 20, wherein the comparison queue comprises a threshold number of images of the second image type received before and after receipt of the target image.

22. The medium of claim 17, wherein the program instructions are further executable by the processor to:

determine a timing offset between receipt time of the comparison image and the receipt time of the target image;

reference the timing offset to determine a next synchronized pair;

track one or both of an amount of time passed or a number of images processed;

determine one or both of threshold amount of time passed or number of images processed is met;

determine compatibility between a next target image and a next comparison image; and

assign a next synchronized image pair of together for use in the augmented reality output.

23. The medium of claim 17, wherein the program instructions are further executable by the processor to:

transform one or both of the target image or the comparison image according to parameters extracted from one or both of the target image or comparison image.

24. An apparatus to synchronize depth and color camera images, the apparatus comprising:

means for obtaining a plurality of input images comprising a first image type and a second image type different from the first image type, wherein an image type is either a depth camera image type or color camera image type, wherein each image type is received unsynchronized with respect to images of other types, and wherein each of the plurality of input images of the first image type and of the second image type are received without identification of an input image'"'"'s actual creation time;

means for selecting, from input images of the first image type, a target image;

means for selecting, from input images of the second image type, a comparison image;

means for determining whether the comparison image is compatible for synchronization with the target image;

means for identifying that the target image and the comparison image as a synchronized image pair, in response to a determination that the comparison image is compatible for synchronization with the target image; and

means for processing the synchronized image pair comprising the target image of the first image type and the comparison image of the second image type together for use in an augmented reality output.

25. The apparatus of claim 24, wherein the target image is a color camera image type, and the comparison image is a depth camera image type, and further comprising:

means for tracking three-dimensional (3D) coordinates from the plurality of input images of the target image, wherein the 3D coordinates include depth values;

means for determining, within the comparison image, depth values for each point having a geometrically equivalent point to the coordinates tracked in the target image;

means for comparing the depth values within the comparison image to depth values within the target image; and

means for determining a compatibility score according to a number of depth value matches between the target image and the comparison image.

26. The apparatus of claim 25, further comprising:

means for performing edge detection on the target image and the comparison image; and

means for determining a compatibility score according to a number of same edge matches between the target image and the comparison image.

27. The apparatus of claim 25, further comprising:

means for determining a time of receipt for each of the plurality of input images;

means for creating a comparison queue comprising a subset of images from the plurality of input images, wherein the subset of images in the comparison queue comprises images of the second image type received within a threshold time from receipt of the target image, and wherein the comparison image is selected from the comparison queue.

28. The apparatus of claim 27, wherein the comparison queue comprises a threshold number of images of the second image type received before and after receipt of the target image.

29. The apparatus of claim 25, further comprising:

means for determining a timing offset between receipt time of the comparison image and the receipt time of the target image;

means for referencing the timing offset to determine a next synchronized pair;

means for tracking one or both of an amount of time passed or a number of images processed;

means for determining one or both of threshold amount of time passed or number of images processed is met;

means for determining compatibility between a next target image and a next comparison image; and

means for assigning a next synchronized image pair of together for use in the augmented reality output.

30. The apparatus of claim 25, further comprising:

means for transforming one or both of the target image or the comparison image according to parameters extracted from one or both of the target image or the comparison image.