Depth perception sensor data processing

• US 10,282,623 B1
• Filed: 09/23/2016
• Issued: 05/07/2019
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Ziyaee, Tarin, Han, Byron B.
• Primary Examiner(s): Bernardi, Brenda C
• Application Number:
  US15/275,175
• Time in Patent Office: 956 Days
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Some embodiments provide a sensor data-processing system which generates a depth data representation of an environment based on sensor data representations which are generated by passive sensor devices. The sensor data-processing system generates the depth data representation via applying an algorithm which includes an model architecture which determines depths of various portions of the represented environment based on detecting features correspond to depth information. The model architecture is established via training an algorithm to generate depth data which corresponds to a sample set of depth data representations of environments, given a corresponding set of image data representations of the environments. As a result, the sensor data-processing system enables depth perception of portions of an environment independently of receiving depth data representations of the environment which are generated by an active sensor device.

Claims

1. An apparatus, comprising:

a sensor data-processing system configured to;

generate a depth data representation of an environment, wherein the depth data representation indicates a depth of one or more portions of the environment relative to a location of a reference point, based on processing an image data representation of the environment in accordance with a sensor data-processing algorithm, wherein the image data representation of the environment is generated by one or more passive sensor devices;

compare the depth data representation and a sensor data representation of the environment, wherein the sensor data representation of the environment is generated by one or more active sensor devices;

iteratively adjust the algorithm based on a comparison of the depth data representation and the sensor data representation until a confidence level for the algorithm satisfies a threshold confidence level, wherein the confidence level is determined based on a magnitude of revisions to the algorithm per iteration of adjustment, and wherein each iteration comprises generation of another depth data representation based on processing another image data representation and comparison of the other depth data representation and another sensor data representation.

2. The apparatus of claim 1, wherein the algorithm comprises at least one of:

at least one traditional machine learning algorithm;

or

at least one deep learning algorithm.

3. The apparatus of claim 1, wherein:

the image data representation of the environment comprises an image of the environment, wherein the image comprises a set of pixels which are each associated with one or more of a color value and a brightness value; and

the depth data representation of the environment comprises an image of the environment, wherein the image comprises a set of pixels which are each associated with at least a depth of the portion of the environment represented by the pixel from a reference point.

4. A method, comprising:

performing, by one or more computer systems;

establishing a sensor data-processing algorithm comprising a deep learning model architecture configured to generate a depth data representation of an environment, which indicates a depth of one or more portions of the environment relative to a location of a reference point, based on processing an image data representation of the environment which is generated by one or more passive sensor devices, wherein the generating the data depth representation comprises;

adjusting the model architecture to cause the algorithm to generate, based on a sample image data representation of an environment, an output depth data representation of the environment, wherein the adjusting the model architecture is performed iteratively until the output depth data representation correlates to a sample depth data representation of the environment within a certain margin of error, wherein the sample depth data representation is generated by one or more active sensor devices, and wherein each iteration comprises generation of another output depth data representation based on processing another sample image data representation and comparison of the other output depth data representation and another sample depth data representation.

5. The method of claim 4, wherein the establishing comprises:

iteratively determining a confidence level for the model architecture based on a magnitude of revisions to the model architecture between iterations, until the confidence level at least meets a threshold confidence level;

wherein the confidence levels, in each given iteration, are inversely proportional to the magnitude of revisions to the model architecture relative to an immediately preceding iteration of the model architecture.

6. The method of claim 5, comprising:

in response to a determination that the confidence level determined for the model architecture at least meets the threshold confidence level, testing the model architecture, wherein the testing comprises;

commanding the algorithm to generate, via the model architecture, an output depth data representation of an environment, based on a sample image data representation of the environment; and

validating the model architecture in response to a determination that a level of error between the output depth data representation and a sample depth data representation of the environment is less than a threshold error level.

7. The method of claim 6, wherein the testing further comprises:

prior to validating the model architecture, and in response to a determination that a level of error between the output depth data representation and a sample depth data representation of the environment at least meets a threshold error level;

reducing the confidence value for the model architecture, and

iteratively performing the adjusting of the model architecture until the confidence level at least meets the threshold confidence level.

8. The method of claim 6, comprising:

in response to validating the model architecture, installing the algorithm in one or more sensor data-processing systems installed in one or more vehicles, wherein the one or more sensor data-processing systems are coupled to one or more passive sensor devices installed in the one or more vehicles, such that the installing configures each vehicle of the one or more vehicles to;

generate depth data representations of environments in which the respective vehicle is located, based on processing image data representations of the environments which are generated by the one or more passive sensor devices installed in the respective vehicle.

9. The method of claim 8, wherein the installing further configures each vehicle of the one or more vehicles to:

identify one or more objects located in the environments in which the respective vehicle is located, based on the generated depth data representations of the environments; and

generate a set of control element command signals which, when executed by a set of control elements installed in the respective vehicle, cause the respective vehicle to be navigated along a driving route which accounts for the identified objects in the environments.

10. The method of claim 9, wherein the installing further configures each vehicle of the one or more vehicles to:

generate a set of control element command signals which, when executed by a set of control elements installed in the respective vehicle, cause the respective vehicle to be navigated along a driving route which avoids intersection with the identified objects in the environments.

11. The method of claim 8, wherein:

each vehicle, of the one or more vehicles, is independent of any active sensor devices configured to generate depth data representations of environments in which the respective vehicle is located, such that the installing configures each vehicle of the one or more vehicles to generate depth data representations of the environments independently of any active sensor devices.

12. The method of claim 4, wherein:

each image data representation of an environment comprises an image of the environment, wherein the image comprises a set of pixels which are each associated with one or more of a color value and a brightness value; and

each depth data representation of an environment comprises an image of the environment, wherein the image comprises a set of pixels which are each associated with at least a depth of the portion of the environment represented by the pixel from a reference point.

13. A non-transitory, computer-readable medium storing a program of instructions which, when executed by at least one computer system, causes the at least one computer system to:

establish a sensor data-processing algorithm comprising a deep learning model architecture configured to generate a depth data representation of an environment, which indicates a depth of one or more portions of the environment relative to a location of a reference point, based on processing an image data representation of the environment which is generated by one or more passive sensor devices, wherein the generating comprises;

adjust the model architecture to cause the algorithm to generate, based on a sample image data representation of an environment, an output depth data representation of the environment, wherein the adjusting the model architecture is performed iteratively until the output depth data representation correlates to a sample depth data representation of the environment within a certain margin of error, wherein the sample depth data representation is generated by one or more active sensor devices, and wherein each iteration comprises generation of another output depth data representation based on processing another sample image data representation and comparison of the other output depth data representation and another sample depth data representation.

14. The non-transitory computer-readable medium of claim 13, wherein the establishing comprises:

iteratively determining a confidence level for the model architecture based on a magnitude of revisions to the model architecture between iterations, until the confidence level at least meets a threshold confidence level;

wherein the confidence levels, in each given iteration, are inversely proportional to the magnitude of revisions to the model architecture relative to an immediately preceding iteration of the model architecture.

15. The non-transitory computer-readable medium of claim 14, wherein the program of instructions, when executed by at least one computer system, causes the at least one computer system to:

in response to a determination that the confidence level determined for the model architecture at least meets the threshold confidence level, test the model architecture, wherein the testing comprises;

commanding the algorithm to generate, via the model architecture, an output depth data representation of an environment, based on a sample image data representation of the environment; and

validating the model architecture in response to a determination that a level of error between the output depth data representation and a sample depth data representation of the environment is less than a threshold error level.

16. The non-transitory computer-readable medium of claim 15, wherein the testing further comprises:

prior to validating the model architecture, and in response to a determination that a level of error between the output depth data representation and a sample depth data representation of the environment at least meets a threshold error level;

reducing the confidence value for the model architecture, and

iteratively performing the adjusting of the model architecture until the confidence level at least meets the threshold confidence level.

17. The non-transitory computer-readable medium of claim 15, wherein the program of instructions, when executed by at least one computer system, causes the at least one computer system to:

in response to validating the model architecture, install the algorithm in one or more sensor data-processing systems installed in one or more vehicles, wherein the one or more sensor data-processing systems are coupled to one or more passive sensor devices installed in the one or more vehicles, such that the installing configures each vehicle of the one or more vehicles to;

generate depth data representations of environments in which the respective vehicle is located, based on processing image data representations of the environments which are generated by the one or more passive sensor devices installed in the respective vehicle.

18. The non-transitory computer-readable medium of claim 17, wherein the installing further configures each vehicle of the one or more vehicles to:

identify one or more objects located in the environments in which the respective vehicle is located, based on the generated depth data representations of the environments; and

generate a set of control element command signals which, when executed by a set of control elements installed in the respective vehicle, cause the respective vehicle to be navigated along a driving route which accounts for the identified objects in the environments.

19. The non-transitory computer-readable medium of claim 18, wherein the installing further configures each vehicle of the one or more vehicles to:

generate a set of control element command signals which, when executed by a set of control elements installed in the respective vehicle, cause the respective vehicle to be navigated along a driving route which avoids intersection with the identified objects in the environments.

20. The non-transitory computer-readable medium of claim 17, wherein:

each vehicle, of the one or more vehicles, is independent of any active sensor devices configured to generate depth data representations of environments in which the respective vehicle is located, such that the installing configures each vehicle of the one or more vehicles to generate depth data representations of the environments independently of any active sensor devices.