Surround image capture and processing

• US 9,998,663 B1
• Filed: 11/20/2015
• Issued: 06/12/2018
• Est. Priority Date: 01/07/2015
• Status: Active Grant

• Current Assignee: Carvana, LLC (Carvana Co.)
• Original Assignee: Car360, Inc. (Carvana Co.)
• Inventors: Franois, Bruno, Schindler, Grant
• Primary Examiner(s): Senfi, Behrooz
• Application Number:
  US14/947,842
• Time in Patent Office: 935 Days
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

Systems and methods are provided to perform surround image capture and processing. A camera device may use physical position and orientation sensors to determine the camera device'"'"'s position along a circular path. The camera device may capture images from positions along the circular path and combine the images into a surround image animation of a central point within the circular path. The camera device may determine feature points within the image frames, and determine a smooth path or track for corresponding feature points to follow from one image to the next. The images can be combined into a smoothly animated 360-degree surround animation, which may be continually rotated without a set beginning or ending point to the animation.

Claims

1. An apparatus comprising:

a positional sensor to determine positional data corresponding to the apparatus relative to an object;

a camera element to capture image data associated with the object; and

a memory configured to store the image data and instructions;

a processor circuit coupled to the memory, the positional sensor, and the camera element, the processor circuit configured to direct the camera element to capture the image data based on the positional data, the image data including a sequence of images, the processor circuit further configured to;

automatically determine one or more feature points in selected images of the sequence of images;

selectively align the one or more feature points across multiple subsets of the selected images of the sequence of images automatically to produce aligned images;

selectively align one or more feature points of a first selected image to corresponding feature points of a last selected image of the sequence of images, automatically, to produce a surround image; and

store the surround image in the memory.

2. The apparatus of claim 1, wherein the processor circuit is further configured to:

determine a position along a closed-loop path based on the positional data; and

direct the camera element to capture the image data at angular intervals along the closed-loop path based on the positional data.

3. The apparatus of claim 1 wherein the processor circuit is further configured to:

determine a position along a closed-loop path based on the positional data; and

direct the camera element to capture the image data at linear distance intervals along the closed-loop path based on the positional data.

4. The apparatus of claim 1, wherein the processor circuit is further configured to:

determine a position along a closed-loop path based on the positional data; and

direct the camera element to capture the image data at periodic intervals along the closed-loop path based on the positional data.

5. The apparatus of claim 1, wherein-the processor circuit is further configured to:

determine selected feature points in selected images of the sequence of images;

calculate a midpoint between the location of the selected feature points in non-consecutive images from the sequence of images; and

perform an image transformation operation on an intervening image from the sequence of images to align the selected feature point of the intervening image with the midpoint.

6. The apparatus of claim 1, wherein the processor circuit is further configured to:

determine selected feature points in selected images of the sequence of images;

calculate a vector connecting the selected feature points in non-consecutive images from the sequence of images; and

perform an image transformation operation on an intervening image from the sequence of images to align the selected feature point of the intervening image with the vector.

7. The apparatus of claim 1, wherein the processor circuit is further configured to generate an animation including a surround image based on processed images from the sequence of images, the animation configured to present the processed images in a sequence to allow continuous rotation without a discontinuity at a starting or ending image of the sequence.

8. The apparatus of claim 1, wherein the processor circuit is further configured to iterate through selected images of the sequence of images to perform image transformation operations to align the selected images to a vector determined from selected feature points within each of the selected images to produce a smooth transition from image to image in the sequence.

9. A processor-readable device embodying instructions that, when executed, cause a processor to:

receive a sequence of images;

automatically determine one or more feature points in selected images of the sequence of images;

automatically track the one or more feature points across selected subsets of the sequence of images to determine one or more tracks extending from a first image to a last image of each subset;

selectively align the one or more feature points across multiple subsets of the selected images of the sequence of images automatically using the one or more tracks to produce aligned images;

selectively align one or more feature points of a first selected image to corresponding feature points of a last selected image of the sequence of images, automatically, to produce a surround image; and

store the surround image in a memory.

10. The processor-readable device of claim 9, wherein the instructions that, when executed, cause the processor to selectively align the one or more feature points across the multiple subsets include instructions that cause the processor to:

selectively align a feature point of an intermediate image, automatically, to a midpoint between a first selected image and a second selected image in one of the multiple subsets of the selected images of the sequence of images.

11. The processor-readable device of claim 9, wherein the instructions that, when executed, cause the processor to selectively align the one or more feature points across the multiple subsets include instructions that cause the processor to:

automatically determine first feature points in a first selected image of one of the multiple subsets;

automatically determine second feature points in a second selected image in the one of the subsets;

automatically determine a vector extending between the first feature points and the second feature points; and

selectively align feature points of an intermediate image of the one of the multiple subsets to the vector, automatically.

12. The processor-readable device of claim 11, wherein the instructions further cause the processor to:

iteratively select a next subset, automatically;

automatically determine first feature points and second feature points in the next subset;

automatically determine a next vector extending between the first feature points and the second feature points;

selectively align feature points of an intermediate image of the next subset of the multiple subsets to the next vector, automatically; and

repeat the selectively align step automatically through selected images of the next subset to produce the surround image.

13. The processor-readable device of claim 9, wherein the instructions to determine one or more feature points in selected images of a sequence of images further cause the processor to:

automatically identify pixels within one or more of the selected images based on a contrast between the identified pixels and adjacent pixels; and

automatically select the identified pixels as feature points.

21. The processor-readable device of claim 9, further embodying instructions that, when executed, cause a processor to:

determine positional data based on signals received from a positional sensor, the positional data corresponding to a position of a camera element relative to an object; and

direct the camera element to capture the image data based on the positional data, the image data including a sequence of images.

14. A method of producing a surround image, the method comprising:

receiving a sequence of images at a processor of a computing device;

automatically identifying, by the processor, one or more feature points in selected images of the sequence of images using the processor;

automatically tracking, by the processor, the one or more feature points across subsets of the selected images to determine a track extending between first and last images of each subset;

automatically aligning, by the processor, at least one intermediate image within each subset to the track;

automatically aligning, by the processor, feature points of first and last images of the sequence of images to produce a surround image; and

storing the surround image in a memory.

15. The method of claim 14, wherein before automatically identifying, the method further comprising:

sensing positional data via one or more sensors coupled to the processor;

automatically determining, by the processor, a position along a closed-loop path based on the positional data; and

automatically directing a camera element by the processor to capture the image data at angular intervals along the closed-loop path based on the positional data.

16. The method of claim 14, wherein automatically identifying the one or more feature points comprises:

automatically determining, by the processor, one or more pixels within a selected image of the sequence of images, the determined one or more pixels having a high contrast relative to adjacent pixels;

automatically selecting, by the processor, a next image of the sequence of images; and

repeating the determining of the one or more pixels with each selected image.

17. The method of claim 14, wherein automatically tracking the one or more feature points comprises:

automatically determining, by the processor, one or more vectors extending between first feature points of a first image and second feature points of a second image of the sequence of images; and

selectively aligning feature points of one or more intermediate images between the first image and the second image to the one or more vectors using the processor.

18. The method of claim 14, further comprising automatically selecting by the processor, a subset of the sequence of images, the subset including a first image, a last image and at least one intermediate image.

19. The method of claim 18, further comprising

automatically determining, by the processor, first feature points in the first image;

automatically determining, by the processor, second feature points in the second image;

automatically determining, by the processor, a vector extending between the first feature points and the second feature points; and

selectively aligning, by the processor, feature points of the at least one intermediate image to the vector.

20. The method of claim 18, further comprising

automatically determining, by the processor, first feature points in the first image;

automatically determining, by the processor, second feature points in the second image;

automatically determining, by the processor, a midpoint between the first feature points and the second feature points; and

selectively aligning, by the processor, feature points of the at least one intermediate image to the midpoint.