Learning disentangled invariant representations for one-shot instance recognition

• US 10,445,622 B2
• Filed: 09/15/2017
• Issued: 10/15/2019
• Est. Priority Date: 05/18/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kicanaoglu, Berkay, Tao, Ran, Smeulders, Arnold Wilhemus Maria
• Primary Examiner(s): Roz, Mark
• Application Number:
  US15/706,541
• Publication Number:
  US 20180336434A1
• Time in Patent Office: 760 Days
• Field of Search: None
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A method of one-shot joint instance and pose recognition in an artificial neural network includes receiving a first instance of a reference object from a reference image. The reference object has a first identity and a first pose in the first instance. The method also includes generating a first orbit of the reference object comprising additional poses including a second pose for the reference object. The method further includes recognizing a second instance of an example object from an example image. The example object has the first identity and the second pose in the second instance.

Claims

1. A method of one-shot joint instance and pose recognition in an artificial neural network, comprising:

training a generator for generating an orbit, the generator trained using a two-branch encoder-decoder architecture that receives two images of two different objects in a same pose;

receiving a first instance of a reference object from a reference image, the reference object having a first identity and a first pose in the first instance;

generating, via the trained generator, a first orbit of the reference object comprising a plurality of additional poses including a second pose for the reference object; and

recognizing a second instance of an example object from an example image, the example object having the first identity and the second pose in the second instance.

2. The method of claim 1, further comprising:

generating a second orbit of the example object; and

comparing the first orbit with the second orbit.

3. The method of claim 2, further comprising recognizing a pose and identity of the example object based on comparing of the first orbit with the second orbit.

4. The method of claim 1, further comprising:

generating the first orbit based only on the first instance of the object using a generator that is learned from other objects of a same class as the object; and

recognizing the second instance based on the first instance and the first orbit.

5. The method of claim 1, further comprising generating an abstract representation of a first image of the two images, the abstract representation comprising a first pose unit and an identity unit.

6. The method of claim 5, further comprising swapping the first pose unit of the first image with a second pose unit of a second image of the two images having the first pose and a second identity.

7. The method of claim 6, further comprising:

rotating the second pose unit;

concatenating the second pose unit with the identity unit to generate a rotated object; and

repeating the rotating and concatenating for a plurality of poses.

8. An artificial neural network for one-shot joint instance and pose recognition, the artificial neural network comprising:

a memory; and

at least one processor coupled to the memory, the at least one processor configured;

to train a generator for generating an orbit, the generator trained using a two-branch encoder-decoder architecture that receives two images of two different objects in a same pose;

to receive a first instance of a reference object from a reference image, the reference object having a first identity and a first pose in the first instance;

to generate, via the trained generator, a first orbit of the reference object comprising a plurality of additional poses including a second pose for the reference object; and

to recognize a second instance of an example object from an example image, the example object having the first identity and the second pose in the second instance.

9. The artificial neural network of claim 8, in which the at least one processor is further configured:

to generate a second orbit of the example object; and

to compare the first orbit with the second orbit.

10. The artificial neural network of claim 9, in which the at least one processor is further configured to recognize a pose and identity of the example object based on comparing of the first orbit with the second orbit.

11. The artificial neural network of claim 8, in which the at least one processor is further configured:

to generate the first orbit based only on the first instance of the object using a generator that is learned from other objects of a same class as the object; and

to recognize the second instance based on the first instance and the first orbit.

12. The artificial neural network of claim 8, in which the at least one processor is further configured to generate an abstract representation of a first image of the two images, the abstract representation comprising a first pose unit and an identity unit.

13. The artificial neural network of claim 12, in which the at least one processor is further configured to swap the first pose unit of the first image with a second pose unit of a second image of the two images having the first pose and a second identity.

14. The artificial neural network of claim 13, in which the at least one processor is further configured:

to rotate the second pose unit;

to concatenate the second pose unit with the identity unit to generate a rotated object; and

to repeat rotation and concatenation for a plurality of poses.

15. A non-transitory computer-readable medium having program code recorded thereon for one-shot joint instance and pose recognition in an artificial neural network, the program code executed by a processor and comprising:

program code to train a generator for generating an orbit, the generator trained using a two-branch encoder-decoder architecture that receives two images of two different objects in a same pose;

program code to receive a first instance of a reference object from a reference image, the reference object having a first identity and a first pose in the first instance;

program code to generate, via the trained generator, a first orbit of the reference object comprising a plurality of additional poses including a second pose for the reference object; and

program code to recognize a second instance of an example object from an example image, the example object having the first identity and the second pose in the second instance.

16. The non-transitory computer-readable medium of claim 15, in which the program code further comprises:

program code to generate a second orbit of the example object; and

program code to compare the first orbit with the second orbit.

17. The non-transitory computer-readable medium of claim 16, further comprising program code to recognize a pose and identity of the example object based on comparing of the first orbit with the second orbit.

18. The non-transitory computer-readable medium of claim 15, further comprising:

program code to generate the first orbit based only on the first instance of the object using a generator that is learned from other objects of a same class as the object; and

program code to recognize the second instance based on the first instance and the first orbit.

19. The non-transitory computer-readable medium of claim 15, further comprising program code to generate an abstract representation of a first image of the two images, the abstract representation comprising a first pose unit and an identity unit.

20. The non-transitory computer-readable medium of claim 19, further comprising program code to swap the first pose unit of the first image with a second pose unit of a second image of the two images having the first pose and a second identity.

21. The non-transitory computer-readable medium of claim 20, further comprising:

program code to rotate the second pose unit;

program code to concatenate the second pose unit with the identity unit to generate a rotated object; and

program code to repeat rotation and concatenation for a plurality of poses.

22. An apparatus for one-shot joint instance and pose recognition in an artificial neural network, comprising:

means for receiving a first instance of a reference object from a reference image, the reference object having a first identity and a first pose in the first instance;

means for generating a first orbit of the reference object comprising a plurality of additional poses including a second pose for the reference object;

means for recognizing a second instance of an example object from an example image, the example object having the first identity and the second pose in the second instance; and

means for training the means for generating via a two-branch encoder-decoder architecture that receives two images of two different objects in a same pose.

23. The apparatus of claim 22, further comprising:

means for generating a second orbit of the example object; and

means for comparing the first orbit with the second orbit.

24. The apparatus of claim 23, further comprising means for recognizing a pose and identity of the example object based on comparing of the first orbit with the second orbit.

25. The apparatus of claim 22, further comprising:

means for generating the first orbit based only on the first instance of the object using a generator that is learned from other objects of a same class as the object; and

means for recognizing the second instance based on the first instance and the first orbit.

26. The apparatus of claim 22, further comprising means for generating an abstract representation of a first image of the two images, the abstract representation comprising a first pose unit and an identity unit.