VISUAL-SALIENCY DRIVEN SCENE DESCRIPTION

• US 20180165539A1
• Filed: 12/08/2016
• Published: 06/14/2018
• Est. Priority Date: 12/08/2016
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Park, Mi Sun, Lee, Yeongseon, Tickoo, Omesh
• Granted Patent: US 10,438,088 B2
• Time in Patent Office: Days
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

An electronic device for visual-saliency driven scene description is described. The electronic device includes a property identifier to identify properties of a target platform. The properties include target platform device type and target platform device count. The electronic device also includes a salient region detector to detect a salient region of an input in response to the target platform device type being a graphical processing unit (GPU) and the target platform device count being above one. The electronic device further includes a description provider to provide a fine-grained scene description for the salient region.

Claims

1. An electronic device for visual-saliency driven scene description, comprising:

a property identifier to identify properties of a target platform, wherein the properties comprise target platform device type and target platform device count;

a salient region detector to detect a salient region of an input in response to the target platform device type being a graphical processing unit (GPU) and the target platform device count being above one; and

a description provider to provide a fine-grained scene description for the salient region.

2. The electronic device for visual-saliency driven scene description of claim 1, wherein the salient region detector is to detect a salient region of an input in response to the target platform device type being a central processing unit (CPU), and wherein a saliency recognizer is to formulate a single label description of the salient region.

3. The electronic device for visual-saliency driven scene description of claim 1, wherein the scene description is coarse-grained and based on the image as a whole in response to the target platform device type being a GPU and the target platform device count being equal to one.

4. The electronic device for visual-saliency driven scene description of claim 1, wherein the property identifier is a low-level application programming interface (API).

5. The electronic device for visual-saliency driven scene description of claim 1, wherein the properties of the target platform further comprise available memory of the target platform and a resource requirement of a kernel executing on the target platform.

6. The electronic device for visual-saliency driven scene description of claim 1, wherein the input is an image or a sequence of images.

7. The electronic device for visual-saliency driven scene description of claim 1, wherein the salient region detector uses a top-down approach.

8. The electronic device for visual-saliency driven scene description of claim 7, wherein the top-down approach is to:

determine the class the image belongs to; and

compute an image-specific class saliency map by calculating a gradient of a class score.

9. The electronic device for visual-saliency driven scene description of claim 8, wherein determining the class the image belongs to comprises utilizing a classification-trained convolutional neural network (CNN).

10. The electronic device for visual-saliency driven scene description of claim 6, wherein calculating a gradient of a class score comprises determining a plurality of pixels that are changed the least to increase the class score the most.

11. The electronic device for visual-saliency driven scene description of claim 1, wherein the description provider comprises a plurality of deep neural networks.

12. The electronic device for visual-saliency driven scene description of claim 1, wherein the scene description for the salient region comprises natural language text.

13. A method for visual-saliency driven scene description, comprising:

identifying properties of a target platform, wherein the properties comprise target platform device type and target platform device count;

detecting a salient region of an input in response to the target platform device type being a graphical processing unit (GPU) and the target platform device count being above one; and

providing a fine-grained scene description of the salient region.

14. The method of claim 13, comprising formulating a single label description of the salient region in response to the target platform device type being a central processing unit (CPU).

15. The method of claim 13, comprising formulating a coarse-grained scene description based on the image as a whole in response to the target platform device type being a GPU and the target platform device count being equal to one.

16. The method of claim 13, comprising utilizing a low-level application programming interface (API) to identify the properties of the target platform.

17. The method of claim 13, comprising inputting an image or a sequence of images.

18. The method of claim 13, comprising utilizing a top-down approach to detect the salient region of the input.

19. The method of claim 13, comprising:

determining a class the image belongs to; and

computing an image-specific class saliency map by calculating a gradient of a class score.

20. The method of claim 19, comprising utilizing a classification-trained convolutional neural network (CNN) to determine the class the image belongs to.

21. The method of claim 13, comprising utilizing natural language text to describe the salient region.

22. At least one computer-readable medium, comprising instructions to direct a processor to:

identify properties of a target platform, wherein the properties comprise target platform device type and target platform device count;

detect a salient region of an input in response to the target platform device type being a graphical processing unit (GPU) and the target platform device count being above one; and

provide a fine-grained scene description of the salient region.

23. The at least one computer-readable medium of claim 22, comprising instructions to direct the processor to formulate a single label description of the salient region in response to the target platform device type being a central processing unit (CPU).

24. The at least one computer-readable medium of claim 22, comprising instructions to direct the processor to formulate a coarse-grained scene description based on the image as a whole in response to the target platform device type being a GPU and the target platform device count being equal to one.

25. The at least one computer-readable medium of claim 22, comprising instructions to direct the processor to:

determine the class the image belongs to; and

compute an image-specific class saliency map by calculating a gradient of a class score.