Multi-perspective imaging system and method

• US 10,482,594 B2
• Filed: 09/22/2017
• Issued: 11/19/2019
• Est. Priority Date: 09/23/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Potter, Daniel E., Zhang, Arthur Y.
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US15/712,600
• Publication Number:
  US 20180089816A1
• Time in Patent Office: 788 Days
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A system includes a processor to capture a first image of a scene by a first imaging device of the array of imaging devices, capture a second image of the scene by a second imaging device of the array of imaging devices, perform image analysis of the first image and the second image and determining that an object is present in the first image and the object is present in the second image, the first image representing a first perspective of the object and the second image representing a second perspective of the object different from the first perspective of the object, classify the object with a classification based on a list of known objects and weight an object portion of the first image and an object portion of the second image based on the classification, compress the first image and the second image based on the weighting, encode the first image and the second image as multi-perspective image data, and store the multi-perspective image data in the non-transitory computer-readable medium based on the classification.

Claims

1. A system, comprising:

an array of imaging devices;

a processor; and

a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising;

capturing a first image of a scene by a first imaging device of the array of imaging devices;

capturing a second image of the scene by a second imaging device of the array of imaging devices;

performing image analysis of the first image and the second image and determining that an object is present in the first image and the object is present in the second image, the first image representing a first perspective of the object and the second image representing a second perspective of the object different from the first perspective of the object;

classifying the object with a classification based on a list of known objects and weighting an object portion of the first image and an object portion of the second image based on the classification;

compressing the first image and the second image based on the weighting, the object portion of the first image and the object portion of the second image being compressed at a particular compression level based on at least one of the classification or the weighting;

encoding the first image and the second image as multi-perspective image data; and

storing the multi-perspective image data in the non-transitory computer-readable medium.

2. The system of claim 1, the operations further comprising:

decoding the multi-perspective image data and displaying a representation of the scene using the first image and the second image.

3. The system of claim 1, the operations further comprising:

compressing the first image and the second image at a compression level based on one or more of the classification and the weighting.

4. The system of claim 1, the operations further comprising:

compressing the object portion of the first image and the object portion of the second image at a first particular compression level based on one or more of the classification and the weighting, and compressing a non-object portion of the first image and a non-object portion of the second image at a second particular compression level, the first particular compression level comprising a lossless compression level and the second particular compression level comprising a lossy compression level.

5. The system of claim 1, the operations further comprising:

discarding the first image and the second image when the weighting does not meet a particular threshold.

6. The system of claim 1, the system further comprising a sensor device, the operations further comprising:

determining a distance to the object from the first image device and the second imaging device using the sensor device to determine time-of-flight;

weighting the object portion of the first image and the object portion of the second image based on the distance; and

compressing the first image and the second image based on the weighting, and encoding the first image and the second image as the multi-perspective image data.

7. The system of claim 1, the system further comprising a Global Positioning System (GPS) device, the operations further comprising:

determining a location of the object using the GPS device;

weighting the object portion of the first image and the object portion of the second image based on the location of the object; and

compressing the first image and the second image based on the weighting, and encoding the first image and the second image as the multi-perspective image data.

8. The system of claim 1, the operations further comprising:

determining bidirectional reflectance distribution function (BRDF) information for the object based on the first image and the second image;

weighting the object portion of the first image and the object portion of the second image based on the BRDF information; and

compressing the first image and the second image based on the weighting, and encoding the first image and the second image as the multi-perspective image data.

9. The system of claim 1, the operations further comprising:

activating the array of imaging devices to begin capturing images in response to user input, the user input comprising input to one or more of a touch screen device and a microphone.

10. The system of claim 1, the operations further comprising:

activating the array of imaging devices to begin capturing images in response to user input, the user input comprising one or more of a gesture made on a touch screen and a verbal command.

11. The system of claim 1, the operations further comprising:

activating the array of imaging devices to begin capturing images when the array of imaging devices determine that the object is present in one or more of the first image and the second image.

12. The system of claim 1, the system further comprising a Global Positioning System (GPS) device, the operations further comprising:

determining a location using the GPS device;

activating the array of imaging devices to begin capturing images based on the location.

13. The system of claim 1, the system further comprising a vehicle, the array of imaging devices mounted on an exterior of the vehicle.

14. The system of claim 1, the system further comprising a vehicle, the array of imaging devices mounted on an interior of the vehicle.

15. The system of claim 1, the operations further comprising:

generating a panoramic image of the object using the multi-perspective image data.

16. The system of claim 1, the operations further comprising:

displaying a representation of the scene including displaying the object from a perspective including a front of a vehicle, a first side of the vehicle, a second side of the vehicle, and a rear of the vehicle.

17. The system of claim 1, the operations further comprising:

displaying a representation of the scene including displaying the object from the first perspective of the object and transitioning to displaying the object from the second perspective of the object.

18. The system of claim 1, the image analysis further comprising:

determining the object in the object portion of the first image is a same object as the object in the object portion of the second image by comparing the object portion of the first image and the object portion of the second image to features and patterns in images of objects in the list of known objects.

19. The system of claim 18, the image analysis further comprising determining that the object in the object portion of the first image is the same object as the object in the object portion of the second image based on depth information, BRDF information, map data, and location information.

20. The system of claim 18, the image analysis further comprising determining the object portion of the first image as a subset of the first image representing the object and determining the object portion of the second image as a subset of the second image representing the object.

21. A method comprising:

obtaining a first image of a scene captured by a first imaging device of an array of imaging devices;

obtaining a second image of the scene captured by a second imaging device of the array of imaging devices;

performing, by at least one processor, image analysis of the first image and the second image and determining that an object is present in the first image and the object is present in the second image, the first image representing a first perspective of the object and the second image representing a second perspective of the object different from the first perspective of the object;

classifying, by the at least one processor, the object with a classification based on a list of known objects and weighting an object portion of the first image and an object portion of the second image based on the classification;

compressing the first image and the second image based on the weighting, the object portion of the first image and the object portion of the second image being compressed at a particular compression level based on at least one of the classification or the weighting;

encoding the first image and the second image as a multi-perspective image data; and

storing the multi-perspective image data in a non-transitory computer-readable medium.

22. A non-transitory computer-readable medium comprising instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:

obtaining a first image of a scene captured by a first imaging device of an array of imaging devices;

obtaining a second image of the scene captured by a second imaging device of the array of imaging devices;

performing image analysis of the first image and the second image and determining that an object is present in the first image and the object is present in the second image, the first image representing a first perspective of the object and the second image representing a second perspective of the object different from the first perspective of the object;

classifying the object with a classification based on a list of known objects and weighting an object portion of the first image and an object portion of the second image based on the classification;

compressing the first image and the second image based on the weighting, the object portion of the first image and the object portion of the second image being compressed at a particular compression level based on at least one of the classification or the weighting; and

encoding the first image and the second image as multi-perspective image data, the multi-perspective image data being stored according to at least one of the classification or the weighting.