Active view planning by deep learning

• US 10,083,369 B2
• Filed: 07/01/2016
• Issued: 09/25/2018
• Est. Priority Date: 07/01/2016
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Tosic, Ivana, Courtney, Logan, Bedard, Noah
• Primary Examiner(s): Potts, Ryan P
• Application Number:
  US15/201,089
• Publication Number:
  US 20180005079A1
• Time in Patent Office: 816 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

A system and method that identifies an object and a viewpoint from an image with a probability that satisfies a predefined criterion is described. An active view planning application receives a first image, performs recognition on the first image to determine an object, a viewpoint and a probability of recognition, determines a first expected gain in the probability of recognition when a first action is taken and a second expected gain in the probability of recognition when a second action is taken, and identifies a next action from the first action and the second action based on an increase in expected gains.

Claims

1. A computer-implemented method comprising:

receiving, by a computing device, a first image;

performing, by the computing device, recognition on the first image using a deep neural network;

determining, by the computing device, a probability of recognition for an object based on performing the recognition on the first image, the probability of recognition for the object identifying an extent of certainty about the image including the object and being captured at a first viewpoint;

determining, by the computing device, whether the probability of recognition for the object satisfies a predetermined threshold;

responsive to determining that the probability of recognition for the object does not satisfy the predetermined threshold, determining, by the computing device, a first expected gain in the probability of recognition when a first action is taken and a second expected gain in the probability of recognition when a second action is taken, the first action and the second action belonging to a set of actions describing receiving a second image for increasing the probability of recognition; and

identifying a next action from the first action and the second action based on an increase in expected gains.

2. The computer-implemented method of claim 1, further comprising performing the next action.

3. The computer-implemented method of claim 1, further comprising:

responsive to determining that the probability of recognition for the object satisfies the predetermined threshold, foregoing the next action.

4. The computer-implemented method of claim 1, wherein the deep neural network is a convolutional neural network.

5. The computer-implemented method of claim 1, wherein the deep neural network determines a class label having an object label and a viewpoint label.

6. The computer-implemented method of claim 1, further comprising:

receiving a set of training data including an original dataset of object images and viewpoints and an augmented dataset of images with ambiguous viewpoints; and

training the deep neural network to recognize the object and the viewpoint from the first image using the set of training data.

7. The computer-implemented method of claim 1, wherein identifying the next action includes:

determining a current belief based on past images evaluated in previous actions and previous time instants;

combining past distributions calculated from the past images and a predicted distribution; and

determining an expected information gain based on the combined distributions.

8. The computer-implemented method of claim 7, further comprising modifying the current belief to compensate a change of a coordinate frame.

9. A system comprising:

one or more processors; and

a memory, the memory storing instructions, which when executed cause the one or more

processors to;

receive a first image;

perform recognition on the first image using a deep neural network;

determine a probability of recognition for an object based on performing the recognition on the first image, the probability of recognition for the object identifying an extent of certainty about the image including the object and being captured at a first viewpoint;

determine whether the probability of recognition for the object satisfies a predetermined threshold;

responsive to determining that the probability of recognition for the object does not satisfy the predetermined threshold, determine a first expected gain in the probability of recognition when a first action is taken and a second expected gain in the probability of recognition when a second action is taken, the first action and the second action belonging to a set of actions describing receiving a second image for increasing the probability of recognition; and

identify a next action from the first action and the second action based on an increase in expected gains.

10. The system of claim 9, wherein the instructions cause the one or more processors to send a command to perform the next action.

11. The system of claim 9, wherein the instructions cause the one or more processors to:

responsive to determining that the probability of recognition for the object satisfies the predetermined threshold, a command to forego the next action is sent.

12. The system of claim 9, wherein the deep neural network is a convolutional neural network.

13. The system of claim 9, wherein the deep neural network determines a class label having an object label and a viewpoint label.

14. The system of claim 9, wherein the instructions cause the one or more processors to:

receive a set of training data including an original dataset of object images and viewpoints and an augmented dataset of images with ambiguous viewpoints; and

train the deep neural network to recognize the object and the viewpoint from the first image using the set of training data.

15. The system of claim 9, wherein to identify the next action, the instructions cause the one or more processors to:

determine a current belief based on past images evaluated in previous actions and previous time instants;

combine past distributions calculated from the past images and a predicted distribution; and

determine an expected information gain based on the combined distributions.

16. The system of claim 15, wherein the instructions cause the one or more processors to modify the current belief to compensate a change of a coordinate frame.

17. A computer program product comprising a non-transitory computer readable medium storing a computer readable program, wherein the computer readable program when executed causes a computer to:

receive a first image;

perform recognition on the first image using a deep neural network;

determine a probability of recognition for an object based on performing the recognition on the first image, the probability of recognition for the object identifying an extent of certainty about the image including the object and being captured at a first viewpoint;

determine whether the probability of recognition for the object satisfies a predetermined threshold;

responsive to determining that the probability of recognition for the object does not satisfy the predetermined threshold, determine a first expected gain in the probability of recognition when a first action is taken and a second expected gain in the probability of recognition when a second action is taken, the first action and the second action belonging to a set of actions describing receiving a second image for increasing the probability of recognition; and

identify a next action from the first action and the second action based on an increase in expected gains.

18. The computer program product of claim 17, wherein the computer readable program causes the computer to perform the next action.

19. The computer program product of claim 17, wherein the computer readable program causes the computer to:

responsive to determining that the probability of recognition the for the object satisfies predetermined threshold, foregoing the next action.

20. The computer program product of claim 17, wherein the deep neural network is a convolutional neural network.