Rapid image annotation via brain state decoding and visual pattern mining

• US 9,665,824 B2
• Filed: 10/22/2013
• Issued: 05/30/2017
• Est. Priority Date: 12/22/2008
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Wang, Jun, Jangraw, David, Chang, Shih-Fu, Sajda, Paul, Pohlmeyer, Eric, Hanna, Barbara
• Primary Examiner(s): Starks, Wilbert L
• Application Number:
  US14/060,398
• Publication Number:
  US 20140108302A1
• Time in Patent Office: 1,316 Days
• Field of Search: None
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Human visual perception is able to recognize a wide range of targets but has limited throughput. Machine vision can process images at a high speed but suffers from inadequate recognition accuracy of general target classes. Systems and methods are provided that combine the strengths of both systems and improve upon existing multimedia processing systems and methods to provide enhanced multimedia labeling, categorization, searching, and navigation.

Claims

1. A computer-based method for labeling one or more media objects related to one or more products, comprising:

presenting to a user a first of said one or more media objects corresponding to a first of said one or more products;

receiving a user response from said user corresponding to said first media object;

generating, using a processor, an object label corresponding to said first media object based at least in part on said user response, said object label comprising an interest score for the first media object; and

associating said object label for said first media object with said user.

2. The method of claim 1, further comprising:

performing at least one of refining the object label associated with said first media object or predicting a new object label pertaining to said stored media objects by calculating a classification function; and

presenting to said user a second of said one or more media objects corresponding to a second of said one or more products, said second media object selected based on said interest score of said first product.

3. The method of claim 2, further comprising:

receiving a second user response from said user corresponding to said second media object;

generating, using a processor, a second object label corresponding to said second media object based at least in part on said second user response, said second object label comprising a second interest score for said second media object; and

associating said second object label for said second media object with said user.

4. The method of claim 2, wherein predicting said new object label comprises automatically selecting a most informative media object, predicting its corresponding class and labeling the corresponding media object.

5. The method of claim 2, wherein said refining comprises performing a greedy search among the gradient direction of the classification function.

6. The method of claim 1, further comprising:

storing a media affinity graph in one or more memories, wherein said affinity graph represents media object samples as nodes and comprises edges measuring relatedness among said object samples; and

calculating a classification function based on at least said selected object label using a processor associated with said one or more memories, by iteratively performing at least updating said selected object label relating to said selected media object or predicting a new object label for said stored media objects.

7. The method of claim 1, wherein said user response comprises brain signal response data of said user.

8. The method of claim 7, wherein said generating comprises decoding said brain signal response data of said user.

9. The method of claim. 1, wherein said user response comprises an interaction with a graphical user interface.

10. The method of claim 1, wherein said first media object comprises image data.

11. The method of claim 1, wherein said first media object comprises video data.

12. The method of claim 1, wherein said first media object comprises audio data.

13. The method of claim 1, further comprising presenting said selected media object to said user one or more additional times and receiving a further user response to said selected media object one or more additional times to further refine said object label.

14. The method of claim 1, further comprising presenting navigation data to said user corresponding to said first product if said interest score exceeds a threshold.

15. A computer-based system for labeling one or more media objects related to one or more products, comprising:

one or more memories;

one or more processors coupled to said one or more memories, wherein said one or more processors are configured to;

present to a user a first of said one or more media objects corresponding to a first of said one or more products;

receive a user response from said user corresponding to said first media object;

generate an object label corresponding to said first media object based at least in part on said user response, said object label comprising an interest score for the first media object; and

associate said object label for said first media object with said user.

16. The system of claim 15, wherein said one or more processors are further configured to:

perform at least one of refining the object label associated with said first media object or predicting a new object label pertaining to said stored media objects by calculating a classification function; and

present to said user a second of said one or more media objects corresponding to a second of said one or more products, said second media object selected based on said interest score of said first product.

17. The system of claim 16, wherein said one or more processors are further configured to:

receive a second user response from said user corresponding to said second media object;

generate a second object label corresponding to said second media object based at least in part on said second user response, said second object label comprising a second interest score for said second media object; and

associate said second object label for said second media object with said user.

18. The system of claim 16, wherein said one or more processors are further configured to predict said new object label at least in part by automatically selecting a most informative media object, predicting its corresponding class and labeling the corresponding media object.

19. The system of claim 16, wherein said one or more processors are further configured to refine at least in part by performing a greedy search among the gradient direction of the classification function.

20. The system of claim 15, wherein said one or more processors are further configured to:

store a media affinity graph in one or more memories, wherein said affinity graph represents media object samples as nodes and comprises edges measuring relatedness among said object samples; and

calculate a classification function based on at least said selected object label using a processor associated with said one or more memories, by iteratively performing at least updating said selected object label relating to said selected media object or predicting a new object label for said stored media objects.

21. The system of claim 15, wherein said response comprises brain signal response data of said user.

22. The system of claim 21, wherein said generating comprises decoding said brain signal response data of said user.

23. The system of claim 15, wherein said user response comprises an interaction with a graphical user interface.

24. The system of claim 15, wherein said first media object comprises image data.

25. The system of claim 15, wherein said first media object comprises video data.

26. The system of claim 15, wherein said first media object comprises audio data.

27. The system of claim 15, wherein said one or more processors are further configured to present said selected media object to said user one or more additional times and receive a further user response to said selected media object one or more additional times to further refine said object label.

28. The system of claim 15, wherein the one or more processors are further configured to present navigation data to said user corresponding to said first product if said interest score exceeds a threshold.