Image capture device with customizable regions of interest

• US 10,452,926 B2
• Filed: 12/29/2016
• Issued: 10/22/2019
• Est. Priority Date: 12/29/2016
• Status: Active Grant

• Current Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Original Assignee: Uber Technologies, Inc.
• Inventors: Brueckner, Peter, Wellington, Carl, Driscoll, David
• Primary Examiner(s): Hilaire, Clifford
• Application Number:
  US15/393,306
• Publication Number:
  US 20180189574A1
• Time in Patent Office: 1,027 Days
• Field of Search: None
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 19/10 using adaptive coding
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 23/815 for controlling the resolution by using a single image
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An image capture device can include one or more image sensors and one or more image processors. The image sensor(s) can be configured to detect incoming light provided incident to a surface of each image sensor, each image sensor configured to provide full image frames of image capture data at a first resolution. The image processor(s) can be coupled to the image sensor(s) and configured to receive the image capture data at the first resolution, downsample the image capture data outside one or more regions of interest, and provide a digital image output having a second resolution within the one or more regions of interest and a third resolution outside the one or more regions of interest, wherein the third resolution is less than the second resolution.

Claims

1. An image capture device, comprising:

one or more image sensors configured to detect incoming light provided incident to a surface of each image sensor, each image sensor configured to provide full image frames of image capture data at a plurality of pixels at a first resolution; and

one or more image processors coupled to the one or more image sensors and configured to;

receive the image capture data at the first resolution;

identify, based at least in part on one or more image target parameters, one or more regions of interest within the image capture data; and

provide a digital image output having a second resolution within the one or more regions of interest and a third resolution outside the one or more regions of interest by downsampling the image capture data outside the one or more regions of interest, wherein the downsampling comprises retrieving the image capture data at a subset of the plurality of pixels, and wherein the third resolution is less than the second resolution.

2. The image capture device of claim 1, wherein the one or more regions of interest are dynamically determined in real time within a target field of view having a smaller area than the full image frames of image capture data.

3. The image capture device of claim 1, wherein the image capture device is positioned relative to a vehicle, and wherein the image capture device further comprises one or more processors configured to receive one or more portions of environmental factor data comprising real time data indicative of one or more environmental factors defining the operational environment of the image capture device and to dynamically determine the one or more regions of interest based at least in part on the one or more portions of environmental factor data.

4. The image capture device of claim 3, wherein the one or more portions of environmental factor data comprise terrain data that provides elevation values for terrain associated with a current geographic location of the vehicle.

5. The image capture device of claim 3, wherein the one or more portions of environmental factor data comprise sensed object data that provides location data for one or more sensed objects around the vehicle.

6. The image capture device of claim 3, wherein the one or more portions of environmental factor data comprise orientation data that indicates orientation of the image capture device as positioned relative to the vehicle.

7. The image capture device of claim 3, wherein the second resolution within the one or more regions of interest is the same as the first resolution at which the image capture data was obtained by the one or more image sensors.

8. The image capture device of claim 1, wherein the one or more image processors comprise a field-programmable gate array (FPGA) configured to actively control processing of the image capture data in accordance with the one or more regions of interest.

9. The image capture device of claim 1, wherein the one or more regions of interest are defined by a height dimension that is between about 5 percent and about 25 percent of a height dimension of each full image frame of the image capture data detected by the one or more image sensors.

10. A vehicle computing system, comprising:

one or more processors configured to receive one or more portions of environmental factor data comprising real time data indicative of one or more environmental factors defining the operational environment of an image capture device positioned relative to a vehicle, and to dynamically determine one or more regions of interest within a full image frame of image capture data based at least in part on the one or more portions of environmental factor data and on one or more image target parameters; and

an image capture device positioned relative to the vehicle, the image capture device including;

one or more image sensors for detecting incoming light provided incident to a surface of each image sensor, each image sensor configured to provide full image frames of image capture data at a plurality of pixels at a first resolution; and

one or more image processors coupled to the one or more image sensors for receiving the image capture data at the first resolution and providing a digital image output having a second resolution within one or more regions of interest and a third resolution outside the one or more regions of interest by downsampling the image capture data outside the one or more regions of interest, wherein the downsampling comprises retrieving the image capture data at a subset of the plurality of pixels.

11. The vehicle computing system of claim 10, further comprising a vehicle controller configured to detect at least one object of interest within the one or more regions of interest in the digital image output, and to control at least one operational parameter of a vehicle based at least in part on the at least one object of interest detected within the one or more regions of interest in the digital image output.

12. The vehicle computing system of claim 10, wherein the one or more portions of environmental factor data comprise one or more of terrain data that provides elevation values for terrain associated with a current geographic location of the vehicle, sensed object data that provides location data for one or more sensed objects around the vehicle, or orientation data that indicates orientation of the image capture device as positioned relative to the vehicle.

13. The vehicle computing system of claim 10, wherein the one or more regions of interest are dynamically determined in real time within a target field of view having a smaller area than the full image frame of image capture data.

14. The vehicle computing system of claim 10, wherein the second resolution within the one or more regions of interest is the same as the first resolution at which the image capture data was obtained by the one or more image sensors.

15. The vehicle computing system of claim 10, wherein the one or more image processors comprise a field-programmable gate array (FPGA) configured to actively control processing of the image capture data in accordance with the one or more regions of interest.

16. A method, comprising:

detecting, by one or more image sensors of an image capture device, full image frames of raw image capture data at a plurality of pixels and characterized by a first resolution;

determining, by one or more processing devices, one or more regions of interest within the full image frames of raw image capture data, wherein the one or more regions of interest are determined based at least in part on one or more image target parameters and one or more portions of environmental factor data including real time data indicative of one or more environmental factors defining the operational environment of the image capture device as positioned relative to a vehicle;

generating, by the one or more processing devices, a digital image output characterized by a second resolution within the one or more regions of interest and a third resolution outside the one or more regions of interest by downsampling the raw image capture data outside the one or more regions of interest, wherein the downsampling comprises retrieving the raw image capture data at a subset of the plurality of pixels.

17. The method of claim 16, wherein determining, by the one or more processing devices, one or more regions of interest based at least in part on one or more image target parameters comprises:

identifying, by the one or more processing devices, the one or more image target parameters defining a target field of view having a smaller area than a full image frame, wherein the target field of view establishes dimensions and/or boundary limitations within which one or more regions of interest can be determined; and

determining, by the one or more processing devices, the one or more regions of interest within the target field of view, wherein the one or more regions of interest actively shift within the target field of view based on the real time values of the one more portions of environmental factor data.

18. The method of claim 16, wherein determining, by the one or more processing devices, one or more regions of interest comprises:

determining, by the one or more computing devices, a position and orientation of the image capture device relative to a ground surface below the vehicle; and

determining, by the one or more computing devices, the one or more regions of interest based at least in part on the position and orientation of the image capture device relative to the ground surface below the vehicle.

19. The method of claim 16, further comprising:

detecting, by the one or more processing devices, at least one object in the digital image output, wherein the at least one object comprises one or more of a person, a vehicle, a road, or a navigational object; and

controlling, by the one or more processing devices, an operational parameter of a vehicle in response to the detection of the at least one object in the digital image output.

20. The method of claim 16, wherein the one or more regions of interest are defined by a height dimension that is between about 5 percent and about 25 percent of a height dimension of each full image frame of raw image capture data detected by the one or more image sensors, and wherein the second resolution within the one or more regions of interest is the same as the first resolution at which the raw image capture data was obtained by the one or more image sensors, and wherein the third resolution outside the one or more regions of interest is at least about fifty percent less than the first and second resolutions.