System and method for infinite synthetic image generation from multi-directional structured image array

• US 10,430,995 B2
• Filed: 04/29/2019
• Issued: 10/01/2019
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Rusu, Radu Bogdan, Ren, Yuheng
• Primary Examiner(s): Bader, Robert
• Application Number:
  US16/398,132
• Publication Number:
  US 20190251738A1
• Time in Patent Office: 155 Days
• Field of Search: None
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/265 Mixing

Abstract

This present disclosure relates to systems and processes for interpolating images of an object from a multi-directional structured image array. In particular embodiments, a plurality of images corresponding to a light field is obtained using a camera. Each image contains at least a portion of overlapping subject matter with another image. First, second, and third images are determined, which are the closest three images in the plurality of images to a desired image location in the light field. A first set of candidate transformations is identified between the first and second images, and a second set of candidate transformations is identified between the first and third images. For each pixel location in the desired image location, first and second best pixel values are calculated using the first and second set of candidate transformations, respectively, and the first and second best pixel values are blended to form an interpolated pixel.

Claims

1. A method for interpolating images of an object from a multi-directional structured image array, the method to be performed by a computing device, the method comprising:

obtaining a plurality of images corresponding to a light field using a camera;

determining first, second, and third images in the plurality of images, the first, second, and third images being the closest three images in the plurality of images to a desired image location in the light field;

identify a first set of candidate transformations between the first image and the second image, wherein identifying the first set of candidate transformations includes applying a first interpolation weight value to transformation candidates;

identify a second set of candidate transformations between the first image and the third image, wherein identifying the second set of candidate transformations includes applying a second interpolation weight value to transformation candidates, wherein the first and second interpolation weights are determined according to the desired image location; and

for each pixel location in the desired image location in the light field;

calculate a first best pixel value using the first set of candidate transformations;

calculate a second best pixel value using the second set of candidate transformations;

blend the first and second best pixel values to form an interpolated pixel.

2. The method of claim 1, identifying a first set of candidate transformations between the first image and the second image includes computing at least one candidate transformation from a plurality of tracked keypoints using one or more of the following:

homography, image warping, dense optical flow, similarity, or sparse optical flow.

3. The method of claim 1, wherein calculating the first best pixel value includes computing each candidate transformation in the first set of candidate transformations to generate a first set of transformation pairs, each transformation pair in the first set of transformation pairs including a transformation from the first image and the second image.

4. The method of claim 3, wherein calculating the second best pixel value includes computing each candidate transformation in the second set of candidate transformations to generate a second set of transformation pairs, each transformation pair in the second set of transformation pairs including a transformation from the first image and the third image.

5. The method of claim 4, wherein calculating the first best pixel value and the second best pixel value includes determining a best pair of transformations, the best pair of transformations including a first best transformation from the first set of candidate transformations and a second best transformation from the second set of candidate transformations.

6. The method of claim 5, wherein calculating the first best pixel value includes applying the first best transformation to a pixel in either the first image or the second image, and calculating the second best pixel value includes applying the second best transformation to a pixel in either the first image or third image.

7. The method of claim 1, wherein the interpolated pixel is compared to a pixel in the first, second, or third frame and a determination is made as to which pixel is used for the desired image location.

8. A system for interpolating images of an object from a multi-directional structured image array, the system comprising:

a camera for obtaining a plurality of images corresponding to a light field, wherein each image in the plurality of images contains at least a portion of overlapping subject matter with another image, the subject matter including an object;

a processor, and

memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for;

obtaining a plurality of images corresponding to a light field using a camera;

determining first, second, and third images in the plurality of images, the first, second, and third images being the closest three images in the plurality of images to a desired image location in the light field;

identify a first set of candidate transformations between the first image and the second image, wherein identifying the first set of candidate transformations includes applying a first interpolation weight value to transformation candidates;

identify a second set of candidate transformations between the first image and the third image, wherein identifying the second set of candidate transformations includes applying a second interpolation weight value to transformation candidates, wherein the first and second interpolation weights are determined according to the desired image location; and

for each pixel location in the desired image location in the light field;

calculate a first best pixel value using the first set of candidate transformations;

calculate a second best pixel value using the second set of candidate transformations;

blend the first and second best pixel values to form an interpolated pixel.

9. The system of claim 8, identify a first set of candidate transformations between the first image and the second image includes computing candidate transformations from a plurality of tracked keypoints using similarity 2D parameters.

10. The system of claim 8, wherein calculating the first best pixel value includes computing each candidate transformation in the first set of candidate transformations to generate a first set of transformation pairs, each transformation pair including a transformation from the first image and the second image.

11. The system of claim 10, wherein calculating the second best pixel value includes computing each candidate transformation in the second set of candidate transformations to generate a second set of transformation pairs, each transformation pair including a transformation from the first image and the third image.

12. The system of claim 11, wherein calculating the first best pixel value and the second best pixel value includes determining a best pair of transformations, the best pair of transformations including a first best transformation from the first set of transformations and a second best transformation from the second set of transformations.

13. The system of claim 12, wherein calculating the first best pixel value includes applying the first best transformation to a pixel in either the first image or second image and calculating the second best pixel value includes applying the second best transformation to a pixel in either the first image or third image.

14. The system of claim 8, wherein the interpolated pixel is compared to a pixel in the first, second, or third frame and a determination is made as to which pixel is used for the desired image location.

15. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

obtaining a plurality of images corresponding to a light field using a camera;

determining first, second, and third images in the plurality of images, the first, second, and third images being the closest three images in the plurality of images to a desired image location in the light field;

identify a first set of candidate transformations between the first image and the second image, wherein identifying the first set of candidate transformations includes applying a first interpolation weight value to transformation candidates;

identify a second set of candidate transformations between the first image and the third image, wherein identifying the second set of candidate transformations includes applying a second interpolation weight value to transformation candidates, wherein the first and second interpolation weights are determined according to the desired image location; and

for each pixel location in the desired image location in the light field;

calculate a first best pixel value using the first set of candidate transformations;

calculate a second best pixel value using the second set of candidate transformations;

blend the first and second best pixel values to form an interpolated pixel.

16. The non-transitory computer readable medium of claim 15, wherein calculating the first best pixel value includes computing each candidate transformation in the first set of candidate transformations to generate a first set of transformation pairs, each transformation pair including a transformation from the first image and the second image.

17. The non-transitory computer readable medium of claim 16, wherein calculating the second best pixel value includes computing each candidate transformation in the second set of candidate transformations to generate a second set of transformation pairs, each transformation pair including a transformation from the first image and the third image.

18. The non-transitory computer readable medium of claim 17, wherein calculating the first best pixel value and the second best pixel value includes determining a best pair of transformations, the best pair of transformations including a first best transformation from the first set of transformations and a second best transformation from the second set of transformations.

19. The non-transitory computer readable medium of claim 18, wherein calculating the first best pixel value includes applying the first best transformation to a pixel in either the first image or second image and calculating the second best pixel value includes applying the second best transformation to a pixel in either the first image or third image.

20. The non-transitory computer readable medium of claim 15, wherein the interpolated pixel is compared to a pixel in the first, second, or third frame and a determination is made as to which pixel is used for the desired image location.