Neural network applications in resource constrained environments

• US 10,210,451 B2
• Filed: 04/06/2018
• Issued: 02/19/2019
• Est. Priority Date: 07/22/2016
• Status: Active Grant

• Current Assignee: Alpine Electronics of Silicon Valley, Inc. (Alps Alpine Co., Ltd.)
• Original Assignee: Alpine Electronics of Silicon Valley, Inc. (Alps Alpine Co., Ltd.)
• Inventors: Lin, Rocky Chau-Hsiung, Yamasaki, Thomas, Kanda, Koichiro, Risco, Diego Rodriguez, Ryan, Alexander Joseph, Najeeb, Samah, El Aouar, Samir
• Primary Examiner(s): Chen, Alan
• Application Number:
  US15/947,295
• Publication Number:
  US 20180232639A1
• Time in Patent Office: 319 Days
• Field of Search: None
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Systems and methods are disclosed for applying neural networks in resource-constrained environments. A system may include a sensor located in a resource-constrained environment configured to generate first sensor data and second sensor data of the resource-constrained environment. The system may also include a first computing device not located in the resource-constrained environment configured to produce a neural network structure based on the first sensor data. The system may also include a second computing device configured to determine a state of the resource-constrained environment based on input of the second sensor data to the neural network structure. The system may also include a controller located in the resource-constrained environment configured to control a device in the resource-constrained environment based on the state of the resource-constrained environment determined by the second computing device. The second computing device may be further configured to calculate an activation area for the neural network structure.

Claims

1. A system comprising:

a sensor located in a resource-constrained environment configured to generate first sensor data of the resource-constrained environment and second sensor data of the resource-constrained environment;

a first computing device not located in the resource-constrained environment configured to produce a neural network structure based on the first sensor data;

a second computing device located in the resource-constrained environment configured to provide the second sensor data as input to the neural network structure, wherein the second computing device is further configured to determine a state of the resource-constrained environment based on the input of the second sensor data to the neural network structure; and

a controller located in the resource-constrained environment configured to control a device in the resource-constrained environment based on the state of the resource-constrained environment determined by the second computing device,

wherein the second computing device is further configured to calculate an activation area for the neural network structure,

wherein the second computing device in configured to calculate the activation area for the neural network structure, at least in part, by providing third sensor data as input to the neural network structure,

wherein the third sensor data is formed by placing a first mask at a first location in the second sensor data,

wherein the second computing device in configured to calculate the activation area for the neural network structure, at least in part, by providing fourth sensor data as input to the neural network structure,

wherein the fourth sensor data is formed by placing a second mask at a second location in the second sensor data,

wherein the second computing device in configured to calculate the activation area for the neural network structure, at least in part, by providing fifth sensor data as input to the neural network structure,

wherein the fifth sensor data is formed by placing a third mask at a third location in the second sensor data,

wherein the second computing device in configured to calculate the activation area for the neural network structure, at least in part, by comparing a first result of providing the second sensor data as input to the neural network structure to a second result of providing the third sensor data as input to the neural network structure,

wherein the second computing device in configured to calculate the activation area for the neural network structure, at least in part, by comparing the first result of providing the second sensor data as input to the neural network structure to a third result of providing the fourth sensor data as input to the neural network structure, and

wherein the second computing device in configured to calculate the activation area for the neural network structure, at least in part, by comparing the first result of providing the second sensor data as input to the neural network structure to a fourth result of providing the fifth sensor data as input to the neural network structure.

2. The system of claim 1,

wherein the second computing device is configured to calculate the activation area for a first prediction class of the neural network structure, and

wherein the second computing device is configured to calculate a second activation area for a second prediction class of the neural network structure.

3. The system of claim 2,

wherein the second computing device is configured to compare the activation area to an expected activation area for the first prediction class;

and

wherein the second computing device is configured to compare the second activation area to an expected activation area for the second prediction class.

4. The system of claim 3,

wherein the second computing device is configured, based at least in part on a result of comparing the activation area to the expected activation area for the first prediction class, to instruct the sensor to generate additional sensor data for the first prediction class.

5. The system of claim 1,

wherein the second computing device is configured to compare the first result to the second result, at least in part, by determining if a prediction class provided as part of the second result is different from a prediction class provided as part of the first result.

6. The system of claim 1,

wherein the second computing device is configured to compare the first result to the second result, at least in part, by determining if a second prediction confidence level provided as part of the second result is different from a first prediction confidence level provided as part of the first result.

7. The system of claim 6,

wherein the second computing device is configured to determine if the second prediction confidence level is different from the first prediction confidence level, at least in part, by determining if the difference between the second prediction confidence level and the first prediction confidence level is greater than a predefined threshold value.

8. The system of claim 1,

wherein the second computing device is configured to determine an effectiveness value for the neural network structure based at least in part on the calculated activation area.

9. The system of claim 8,

wherein the second computing device is configured, based at least in part on the determined effectiveness value, to instruct the first computing device to produce a second neural network structure.

10. A method comprising:

generating first sensor data and second sensor data of a resource-constrained environment using a sensor located in the resource-constrained environment;

producing a neural network structure based on the first sensor data using a first computing device not located in the resource-constrained environment;

inputting the sensor data to the neural network structure using a second computing device located in the resource-constrained environment;

determining a state of the resource-constrained environment using the second computing device based on the input of the second sensor data to the neural network structure;

controlling an indicator device in the resource-constrained environment using a controller located in the resource-constrained environment based on the state of the resource-constrained environment determined by the second computing device; and

calculating an activation area for the neural network structure;

wherein calculating the activation area for the neural network structure comprises providing third sensor data as input to the neural network structure,

wherein the third sensor data is formed by placing a first mask at a first location in the second sensor data,

wherein calculating the activation area for the neural network structure comprises providing fourth sensor data as input to the neural network structure,

wherein the fourth sensor data is formed by placing a second mask at a second location in the second sensor data,

wherein calculating the activation area for the neural network structure comprises providing fifth sensor data as input to the neural network structure,

wherein the fifth sensor data is formed by placing a third mask at a third location in the second sensor data,

wherein calculating the activation area for the neural network structure comprises comparing a first result of providing the second sensor data as input to the neural network structure to a second result of providing the third sensor data as input to the neural network structure,

wherein calculating the activation area for the neural network structure comprises comparing the first result of providing the second sensor data as input to the neural network structure to a third result of providing the fourth sensor data as input to the neural network structure, and

wherein calculating the activation area for the neural network structure comprises comparing the first result of providing the second sensor data as input to the neural network structure to a fourth result of providing the fifth sensor data as input to the neural network structure.

11. The method of claim 10, wherein calculating the activation area for the neural network structure comprises calculating the activation area for a first prediction class of the neural network structure, the method further comprising:

calculating a second activation area for a second prediction class of the neural network structure.

12. The method of claim 11, further comprising:

comparing the activation area to an expected activation area for the first prediction class; and

comparing the second activation area to an expected activation area for the second prediction class.

13. The method of claim 12, further comprising:

generating, based at least in part on a result of the comparing the activation area to the expected activation area for the first prediction class, additional sensor data for the first prediction class.

14. The method of claim 10,

wherein comparing the first result to the second result comprises determining if a prediction class provided as part of the second result is different from a prediction class provided as part of the first result.

15. The method of claim 10,

wherein comparing the first result to the second result comprises determining if a second prediction confidence level provided as part of the second result is different from a first prediction confidence level provided as part of the first result.

16. The method of claim 15,

wherein determining if the second prediction confidence level is different from the first prediction confidence level comprises determining if the difference between the second prediction confidence level and the first prediction confidence level is greater than a predefined threshold value.

17. The method of claim 10, further comprising:

determining an effectiveness value for the neural network structure, based at least in part on the calculated activation area.

18. The method of claim 17, further comprising:

instructing the first computing device to produce a second neural network structure, based at least in part on the determined effectiveness value.