Three-dimensional stabilized 360-degree composite image capture

• US 10,284,794 B1
• Filed: 09/21/2017
• Issued: 05/07/2019
• Est. Priority Date: 01/07/2015
• Status: Active Grant

• Current Assignee: Carvana, LLC (Carvana Co.)
• Original Assignee: Car360, Inc. (Carvana Co.)
• Inventors: Francois, Bruno, Schindler, Grant
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US15/711,055
• Time in Patent Office: 593 Days
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors

Abstract

In some embodiments, a device may include a processor and a memory. The memory may include instructions that, when executed, may cause the processor to generate a 360 degree three-dimensional composite image formed from a stabilized sequence of images of an object taken along a closed loop path about the object. In some embodiments, the instructions may cause the processor to rotate the 360 degree three-dimensional composite image about any selected point on the object. Further, in some embodiments, the instructions may cause the processor to provide an interface configured to enable a user to select a feature on the 360 degree three-dimensional composite image and to associate information with the selected feature.

Claims

1. A device comprising:

a display;

a processor coupled to the display; and

a memory coupled to the processor and including instructions that, when executed, may cause the processor to;

determine a direction of gravity in each image of a sequence of images taken along a closed loop path around an object;

estimate a center of mass of the object in each image of the sequence of images based on dimensions of the object;

automatically select feature points within each image of the sequence of images, each feature point corresponding to a sets of pixels of the image that have high contrast relative to nearby pixels, each feature point representing an element of the object;

selectively apply a transformation to each image of the sequence of images to stabilize the feature points associated with the object in the sequence of images such that a three-dimensional line segment passing through the center of mass of the object and parallel to the direction of gravity remains fixed in each image of the sequence of images to produce a stabilized sequence of images;

generate a 360 degree three-dimensional composite image from the stabilized sequence of images; and

provide an interface including the 360 degree three-dimensional composite image to the display.

2. The device of claim 1, wherein the memory further includes instructions that, when executed, cause the processor to:

determine a six-degrees-of-freedom pose including rotation and translation in a global coordinate frame of an optical device for each image of the sequence of images; and

wherein the six-degrees-of-freedom pose is used to selectively adjust each image of the sequence of images to align the three-dimensional line segment.

3. The device of claim 1, wherein the memory includes instructions that, when executed, cause the processor to hide empty image areas in each transformed image of the sequence of images, the empty image area including a plurality of pixels in the transformed image which map onto untransformed pixel locations outside the boundary of the original image.

4. The device of claim 3, wherein the memory includes instructions that, when executed, cause the processor to:

determine an empty image area within a selected image of the sequence of images after application of the transformation, the empty image area including a plurality of pixels in the transformed image which map onto untransformed pixel locations outside the boundary of the original image; and

determine a minor transformation of at least one of scale and translation to apply to each image of the sequence of images so that the empty image area is not visible in the stabilized sequence of images.

5. The device of claim 1, further comprising:

a camera coupled to the processor and configured to capture the sequence of images; and

a gyroscope sensor coupled to the processor to provide signals corresponding to the orientation of the camera relative to the object; and

wherein the memory further includes instructions that, when executed, cause the processor to;

automatically capture a plurality of images of an object during an image capture process as the camera moves along the closed-loop path around the object; and

automatically terminate the image capture process when a signal from the gyroscope sensor indicates that the camera has captured images from a selected number of degrees traveled along the closed-loop path.

6. The device of claim 5, wherein the memory further includes instructions that, when executed, cause the processor to:

provide a current image from the camera to the display together with a first indicator corresponding to an extent of completion of an image capture process about the closed-loop path;

provide dynamic gyro-based alignment aids to the display on opposing sides of the object to assist a user to aim the camera to center the object, to assist the user in adjusting a physical distance from the object in order to frame the object properly in the image, and to assist the user in keeping the camera level;

selectively alter a color of the alignment aids when a signal from the gyroscope sensor indicates that an orientation of the camera has changed by more than a threshold amount relative to a first orientation of the camera determined when a first image of the sequence of images was captured; and

provide a semi-transparent overlay of the first captured image superimposed on top of the live camera feed to assist the user to adjust the distance and angle of the image capture device to the subject in order to achieve close-to-exact alignment between the first and last frames of the image capture process.

7. The device of claim 1, wherein the memory further includes instructions that, when executed, cause the processor to:

receive an input corresponding to a feature tagging option via the interface; and

provide one or more user-selectable elements within the interface that are accessible by a user to identify and label a feature on the 360 degree three-dimensional composite image.

8. The device of claim 7, wherein the memory further includes instructions that, when executed, cause the processor to:

receive a selection corresponding to a feature of the 360 degree three-dimensional composite image; and

selectively rotate the 360 degree three-dimensional composite image about an axis of rotation extending through the selected feature parallel to the direction of gravity.

9. The device of claim 7, wherein the instructions cause the processor to:

determine a polygon mesh of three-dimensional triangles defining a three-dimensional geometrical description of a surface of the object across the stabilized sequence of images;

receive a selection of a point within a selected image of the 360 degree three-dimensional composite image corresponding to a feature point within the interface;

extend a ray from a camera position of the selected image through a point on a focal plane corresponding to the selection until the ray intersects one or more of the three-dimensional triangles in the polygon mesh; and

determine a three-dimensional point corresponding to the selection by computing a closest point of intersection between the ray and the one or more of the three-dimensional triangles, the closest point of intersection representing an intersection point that minimizes a distance between the camera position for the selected image and the one or more of the three-dimensional triangles in the polygon mesh.

10. The device of claim 7, wherein the memory further includes instructions that, when executed, cause the processor to automatically transition between rotating around the center of mass of the object and rotating around each of a plurality of features identified via the feature tagging option to provide an automated tour of the 360 degree three-dimensional composite image.

11. The device of claim 7, wherein the instructions cause the processor to:

receive a selection of a point on the 360 degree three-dimensional composite image corresponding to the feature;

provide a second interface including one or more user-selectable elements to receive information related to the selection, the information including at least one of a text label, a description, a video recording, an audio recording, and an image associated with the three-dimensional point; and

automatically associate the information with the feature within the 360 degree three-dimensional composite image such that the information is accessible from any view of the 360 degree three-dimensional composite image that includes the feature.

12. The device of claim 7, wherein the instructions cause the processor to:

receive a selection of a point within a selected image of the 360 degree three-dimensional composite image corresponding to the feature;

provide, to the interface, a first reference image and a second reference image from the sequence of images that form the 360 degree three-dimensional composite image, the first reference image having a pre-determined amount of rotation to the left of the selected image along the closed loop path, the second reference image have the pre-determined amount of rotation to the right of the selected image along the closed loop path;

receive a first input corresponding to the first reference image and a second input corresponding to the second reference image to designate a location on the object; and

determine a three-dimensional point within the first image, the second image, and at least some of the plurality of images within the sequence of images of the 360 degree three-dimensional composite image that corresponds to the location based on the first input and the second input.

13. The device of claim 12, wherein the memory further includes instructions that, when executed, cause the processor to provide a tag interface to receive data including at least one of a text label, a description, a video recording, an audio recording, and an image associated with the three-dimensional point.

14. The device of claim 1, wherein the instructions, when executed, further cause the processor to:

receive a three-dimensional computer-aided drawing (CAD) model of the object;

correlate the plurality of images to the three-dimensional CAD model to produce the 360 degree three-dimensional composite image;

receive an input to adjust at least one of a color change and a physical modification of the object; and

selectively adjust the 360 degree three-dimensional composite image to provide an augmented-reality visualization of the at least one of the color change and the physical modification of the object in response to receiving the input.

15. The device of claim 1, wherein the instructions, when executed, further cause the processor to apply histogram equalization to each image of the sequence of images to reduce distortion caused by sun glare.

16. The device of claim 1, wherein the instructions, when executed, further cause the processor to determine structure from motion by causing the processor to:

track two-dimensional feature points across the sequence of images, including loop closing features that track the two-dimensional feature points from a last image to a first image in the sequence of images;

determine feature tracks extending through at least three images of the sequence of images, each feature track corresponding to a vector line that couples a two-dimensional feature point in a first image to the two-dimensional feature point in another image of the sequence of images; and

chain the feature tracks together into tracks that cross as many images of the sequence of images as possible.

17. The device of claim 16, wherein the instructions further cause the processor to:

identify an image of the sequence of images that has no tracks passing through it that extend across at least three images; and

identify one or more feature tracks that skip over the identified image to track the two-dimensional feature points across non-consecutive images of the sequence of images to skip the identified image.

18. The device of claim 16, wherein the instructions further cause the processor to:

determine a plurality of three-dimensional feature points where each three-dimensional feature point corresponds to a two-dimensional feature track; and

triangulate an initial estimate for each three-dimensional feature point from its two-dimensional observations based on an initial camera position estimate and a known focal length for the camera; and

reject a three-dimensional feature point when the initial estimate triangulates to a location that is behind the camera location from which the image was captured.

19. The device of claim 1, further including:

at least one optical sensor coupled to the processor and configured to capture images of a view area;

a magnetometer coupled to the processor and configured to provide directional sensor readings; and

a gyroscopic sensor coupled to the processor and configured to provide rotational sensor readings; and

wherein the directional sensor readings and the rotational sensor readings are correlated to each image to provide an estimate of both rotation and location of the camera for each image of the sequence of images.

20. The device of claim 19, further including instructions that, when executed, cause the processor to:

determine an estimated three-dimensional point location of each three dimensional feature point of a plurality of three-dimensional feature points from a plurality of two-dimensional feature tracks extending across multiple images of the sequence of images;

determine estimated camera poses based on the directional sensor readings, the rotational sensor readings, an initial camera position estimate, and a known focal length for the camera; and

perform a bundle adjustment procedure to adjust the three-dimensional camera poses and the three-dimensional point locations to reduce a re-projection error, the re-projection error corresponding to a difference between a two-dimensional feature track in an image and where the two-dimensional feature track projects in the image according to an estimated three-dimensional camera pose and an estimated three-dimensional point location.

21. The device of claim 20, wherein the instructions further cause the processor to:

determine a three-dimensional feature point having a re-projection error greater than a threshold error;

remove the three-dimensional feature point; and

perform the bundle adjustment procedure.

22. The device of claim 20, wherein, during performance of the bundle adjustment, the instructions cause the processor to constrain a first rotation of a first camera of a first image of the sequence of images and a second rotation of a second camera of a last image of the sequence of images to closely match the rotational sensor readings.

23. The device of claim 1, wherein:

the object includes a car; and

the instructions that cause the processor to determine the center of mass include instructions that, when executed, cause the processor to;

identify three-dimensional elements corresponding to four wheels of the car in the sequence of images; and

assume the center of mass of the car passes through a centroid of three-dimensional locations of the four wheels of the car.

24. The device of claim 23, wherein the instructions to identify the three-dimensional elements corresponding to the four wheels includes instructions that, when executed, cause the processor to:

detect two-dimensional circles in each image of the sequence of images using a Hough transform;

perform a RANSAC procedure by randomly sampling one circle detection from each of two separate images of the sequence of images;

triangulate a three-dimensional feature point from centers of the circles detected in the two separate images;

project the three-dimensional feature point into every other image and count how many circles are detected that correspond to the projected three-dimensional feature point;

determine a first three-dimensional vector from the three-dimensional wheel point to a centroid of two camera positions corresponding to the two separate images;

determine a second three-dimensional vector from the projected three-dimensional feature point corresponding to a circle in a selected image to a camera position of the selected image; and

add the circle to a list of candidate wheels when a dot product of the first and second three-dimensional vector is positive.

25. The device of claim 24, wherein the instructions further cause the processor to:

perform a random sampling procedure to sample sets of four three-dimensional wheels from the list of candidate wheels;

evaluate the sampled sets based on an expected symmetry of three-dimensional distances between two front tires and two back tires and between an expected three-dimensional distance between two left tires and two right tires;

pick a group of four wheels that minimizes a sum of the distances between the expected symmetries of the three-dimensional differences as a set of wheels that represents the car in the sequence of images; and

determine a centroid of the set of wheels as the center of mass.

26. The device of claim 23, wherein, when processor is unable to detect a set of four consistent wheels in the sequence of images, the instructions, when executed, cause the processor to:

search for a three-dimensional feature point that, when projected into each image, robustly minimizes a re-projection error between a two-dimensional projection of that three-dimensional feature point and the absolute center of the image assuming that the car is approximately centered in each image; and

select the three-dimensional feature point that minimizes a sum of an absolute value of re-projection errors in a x-dimension and a y-dimension in each of the images.

27. The device of claim 1, wherein the instructions to determine the direction of gravity include instructions that, when executed, cause the processor to:

fit a three-dimensional plane to a set of camera positions such that a line extending normal to the three-dimensional plane points either up or down, the line represents an estimated gravity vector.

28. The device of claim 1, wherein the memory further includes instructions that, when executed, cause the processor to identify edges of the object within each image of the sequence of images and to blur or remove background pixels outside of the edges of the object.

29. The device of claim 1, wherein the instructions cause the processor to:

provide the display with an augmented reality overlay on top of the 360 degree three-dimensional composite image, the augmented reality overlay configured to provide a visualization of selected three-dimensional feature points and any metadata associated with the three-dimensional feature points by projecting the three-dimensional feature points onto corresponding two-dimensional locations in each image;

compute a set of angles, and corresponding images, from which each three-dimensional feature point is visible and not pointing away from a camera or occluded by three-dimensional geometry of the object in the 360 degree three-dimensional composite image;

compute and apply perspective distortion, shadows, and dynamic physical movement to insert each three-dimensional feature point and corresponding metadata into each image of the 360 degree three-dimensional composite image;

provide smooth visual transitions between viewing text, image, and video tags as three-dimensional augmented elements situated around an object and viewing the text, image, and video tags situated on a two-dimensional plane, as a flat list or grid, adjacent to the three-dimensional view of the 360 degree three-dimensional composite image; and

provide user interaction with three-dimensional features and corresponding text, image, and video tags by providing one or more user-selectable elements accessible by a user to select, expand, collapse, reveal, and hide each element of the augmented reality overlay.

30. The device of claim 1, wherein the instructions cause the processor to:

provide the display with two visualizations of the 360 degree three-dimensional composite image, corresponding to left-eye and right-eye views, respectively, of a stereoscopic virtual reality or augmented reality system;

adjust the left-eye and right-eye visualizations based on movement of the device through physical space by automatically selecting and transforming appropriate images from the 360 degree three-dimensional composite image which correspond to angles at which a viewer'"'"'s left and right eyes would view the object if the object were physically positioned at an origin of a global coordinate system of the 360 degree three-dimensional composite image; and

provide the display with an augmented visualization of the two visualizations by integrating tagged three-dimensional features and associated metadata into the two visualizations.