Multi-directional structured image array capture on a 2D graph

• US 10,176,592 B2
• Filed: 02/07/2017
• Issued: 01/08/2019
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Holzer, Stefan Johannes Josef, Rusu, Radu Bogdan, Ren, Yuheng
• Primary Examiner(s): Park, Edward
• Application Number:
  US15/427,009
• Publication Number:
  US 20170148186A1
• Time in Patent Office: 700 Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/265 : Mixing

Abstract

This present disclosure relates to systems and processes for capturing an unstructured light field in a plurality of images. In particular embodiments, a plurality of keypoints are identified on a first keyframe in a plurality of captured images. A first convex hull is computed from all keypoints in the first keyframe and merged with previous convex hulls corresponding to previous keyframes to form a convex hull union. Each keypoint is tracked from the first keyframe to a second image. The second image is adjusted to compensate for camera rotation during capture, and a second convex hull is computed from all keypoints in the second image. If the overlapping region between the second convex hull and the convex hull union is equal to, or less than, a predetermined size, the second image is designated as a new keyframe, and the convex hull union is augmented with the second convex hull.

Claims

1. A method for capturing an unstructured light field in a plurality of images, the method including:

identifying a plurality of keypoints on a first keyframe in a plurality of captured images;

computing the convex hull of all keypoints in the plurality of keypoints in the first keyframe to form a first convex hull;

merging the first convex hull with previous convex hulls corresponding to previous keyframes to form a convex hull union;

keeping track of each keypoint from the first keyframe to a second image;

adjusting the second image to compensate for camera rotation during capture of the second image;

computing the convex hull of all keypoints in the second image to form a second convex hull;

if the overlapping region between the second convex hull and the convex hull union is equal to, or less than, half of the size of the second convex hull, designating the second image as a new keyframe; and

if the second image is designated as a new keyframe, augmenting the convex hull union with the second convex hull.

2. The method of claim 1, wherein identifying the plurality of keypoints includes using Harris-style keypoint detection.

3. The method of claim 1, wherein keeping track of each keypoint includes using KLT-style keypoint tracking.

4. The method of claim 1, wherein adjusting the second image to compensate for camera rotation includes using internal measurement unit (IMU) data associated with the second image.

5. The method of claim 1, wherein designating the second image as a new keyframe occurs in real-time during capture of the second image.

6. The method of claim 1, further comprising repeating the steps until a predetermined threshold number of keyframes has been designated for the light field.

7. The method of claim 6, further comprising representing the number of keyframes as nodes on a 2D graph.

8. A system for capturing an unstructured light field in a plurality of images, the system including:

a camera for capturing the plurality of images;

a processor, and

memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for;

identifying a plurality of keypoints on a first keyframe in the plurality of captured images;

computing the convex hull of all keypoints in the plurality of keypoints in the first keyframe to form a first convex hull;

merging the first convex hull with previous convex hulls corresponding to previous keyframes to form a convex hull union;

keeping track of each keypoint from the first keyframe to a second image;

adjusting the second image to compensate for camera rotation during capture of the second image;

computing the convex hull of all keypoints in the second image to form a second convex hull;

if the overlapping region between the second convex hull and the convex hull union is equal to, or less than, half of the size of the second convex hull, designating the second image as a new keyframe; and

if the second image is designated as a new keyframe, augmenting the convex hull union with the second convex hull.

9. The system of claim 8, wherein identifying the plurality of keypoints includes using Harris-style keypoint detection.

10. The system of claim 8, wherein keeping track of each keypoint includes using KLT-style keypoint tracking.

11. The system of claim 8, wherein adjusting the second image to compensate for camera rotation includes using internal measurement unit (IMU) data associated with the second image.

12. The system of claim 8, wherein designating the second image as a new keyframe occurs in real-time during capture of the second image.

13. The system of claim 8, wherein the one or more programs further comprise instructions for repeating the steps until a predetermined threshold number of keyframes has been designated for the light field.

14. The system of claim 13, wherein the one or more programs further comprise instructions for representing the number of keyframes as nodes on a 2D graph.

15. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

identifying a plurality of keypoints on a first keyframe in a plurality of captured images;

computing the convex hull of all keypoints in the plurality of keypoints in the first keyframe to form a first convex hull;

merging the first convex hull with previous convex hulls corresponding to previous keyframes to form a convex hull union;

keeping track of each keypoint from the first keyframe to a second image;

adjusting the second image to compensate for camera rotation during capture of the second image;

computing the convex hull of all keypoints in the second image to form a second convex hull;

if the overlapping region between the second convex hull and the convex hull union is equal to, or less than, half of the size of the second convex hull, designating the second image as a new keyframe; and

if the second image is designated as a new keyframe, augmenting the convex hull union with the second convex hull.

16. The non-transitory computer readable medium of claim 15, wherein identifying the plurality of keypoints includes using Harris-style keypoint detection.

17. The non-transitory computer readable medium of claim 15, wherein keeping track of each keypoint includes using KLT-style keypoint tracking.

18. The non-transitory computer readable medium of claim 15, wherein adjusting the second image to compensate for camera rotation includes using internal measurement unit (IMU) data associated with the second image.

19. The non-transitory computer readable medium of claim 15, wherein designating the second image as a new keyframe occurs in real-time during capture of the second image.

20. The non-transitory computer readable medium of claim 15, wherein the one or more programs further comprise instructions for repeating the steps until a predetermined threshold number of keyframes has been designated for the light field.