RAPID IMAGE ANNOTATION VIA BRAIN STATE DECODING AND VISUAL PATTERN MINING

• US 20120089552A1
• Filed: 08/08/2011
• Published: 04/12/2012
• Est. Priority Date: 12/22/2008
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Chang, Shih-Fu, Sajda, Paul, Wang, Jun, Pohlmeyer, Eric, Hanna, Barbara, Jangraw, David
• Granted Patent: US 8,671,069 B2
• Time in Patent Office: Days
• US Class Current: 706/52
• CPC Class Codes: G06F 16/68 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Human visual perception is able to recognize a wide range of targets but has limited throughput. Machine vision can process images at a high speed but suffers from inadequate recognition accuracy of general target classes. Systems and methods are provided that combine the strengths of both systems and improve upon existing multimedia processing systems and methods to provide enhanced multimedia labeling, categorization, searching, and navigation.

Claims

1. A computer-based method for labeling multimedia objects comprising:

presenting selected multimedia data to a first user;

determining user interest in said selected multimedia data based upon brain signal responses of the first user;

generating selected multimedia label data corresponding to the selected multimedia data based on the determination of user interest;

using a processor and based on at least said generated selected multimedia label data, performing at least one of refining multimedia label data associated with said selected multimedia data or predicting new multimedia label data pertaining to said stored multimedia data by calculating a classification function.

2. The method of claim 1, further comprising

storing a multimedia affinity graph in one or more memories, wherein said affinity graph represents multimedia data samples as nodes and comprises edges measuring relatedness among data samples;

and

calculating a classification function based on at least the selected multimedia label data using a processor associated with said one or more memories, wherein calculating said classification function comprises iteratively performing at least updating selected multimedia label data relating to selected multimedia data or predicting new multimedia label data for stored multimedia data using said processor.

3. The method of claim 1, wherein generating selected multimedia label data comprises decoding the brain signal responses of the user and generating at least one interest score for said selected multimedia data.

4. The method of claim 3, further comprising refining said interest score and generating an updated interest measurement.

5. The method of claim 3, further comprising converting said at least one interest score to at least one interest label.

6. The method of claim 5, wherein converting said at least one interest score to at least one interest label comprises using a binarization function.

7. The method of claim 1, wherein the multimedia data comprises image data.

8. The method of claim 1, wherein the multimedia data comprises video data.

9. The method of claim 1, wherein the multimedia data comprises audio data.

10. The method of claim 1, wherein the multimedia data comprises locations, objects, and events in a virtual world.

11. The method of claim 1, wherein the multimedia data comprises locations, objects, and events in a physical world.

12. The method of claim 1, wherein predicting new multimedia label data comprises automatically selecting a most informative data sample, predicting its corresponding class and labeling the corresponding data sample.

13. The method of claim 1, wherein refining at least a portion of said multimedia label data comprises performing a greedy search among the gradient direction of the classification function.

14. The method of claim 2, wherein at least one multimedia label data sample is further normalized based on a regularization matrix calculated using members of a corresponding class and connectivity degrees of the corresponding nodes in the graph.

15. The method of claim 2, wherein calculating a classification function comprises incremental calculation using graph superposition, wherein a newly added label is incorporated incrementally without calculating a classification function using all labels.

16. The method of claim 1 wherein unreliable label data is replaced.

17. The method of claim 16, wherein replacing unreliable label data comprises adding a multimedia label data sample for every multimedia label data sample that is removed.

18. The method of claim 2 wherein unreliable label data is replaced.

19. The method of claim 18, wherein replacing unreliable label data comprises adding a multimedia label data sample for every multimedia label

20. The method of claim 18, wherein replacing unreliable label data or predicting new multimedia label data comprise updating a node regularization matrix.

21. The method of claim 18, wherein replacing unreliable label data or predicting new multimedia label data comprises minimizing an objective function.

22. The method of claim 1, comprising presenting said selected multimedia data to said first user one or more additional times and determining user interest in said selected multimedia data one or more additional times to further refine said multimedia label data.

23. The method of claim 1, further comprising:

presenting said selected multimedia data to a second user,

determining user interest in said selected multimedia data based on at least brain signal responses of the first user and brain signal responses of the second user.

24. The method of claim 1, further comprising using said refined or predicted multimedia label data to perform a search of said stored multimedia data.

41. The method of claim 16, wherein replacing unreliable label data comprises adding a multimedia label data sample for every multimedia label data sample that is removed.

42. The method of claim 2 wherein unreliable label data is replaced.

43. The system of claim 42, wherein replacing unreliable label data comprises adding a multimedia label data sample for every multimedia label

44. The system of claim 42, wherein replacing unreliable label data or predicting new multimedia label data comprise updating a node regularization matrix.

45. The system of claim 42, wherein replacing unreliable label data or predicting new multimedia label data comprises minimizing an objective function.

65. The method of claim 14, wherein replacing unreliable label data comprises adding a multimedia label data sample for every multimedia label data sample that is removed.

25. A system for labeling multimedia data comprising:

one or more memories;

one or more processors coupled to said one or more memories, wherein said one or more processors are configured to;

present selected multimedia data to a first user;

determine user interest in said selected multimedia data based upon brain signal responses of the first user;

generate selected multimedia label data corresponding to the selected multimedia data based on the determination of user interest;

based on at least said generated selected multimedia label data, perform at least one of refining multimedia label data associated with said selected multimedia data or predicting new multimedia label data pertaining to said stored multimedia data by calculating a classification function.

26. The system of claim 25, wherein said one or more processors are further configured to:

store a multimedia affinity graph in said one or more memories, wherein said affinity graph represents multimedia data samples as nodes and comprises edges measuring relatedness among data samples; and

calculate a classification function based on at least the selected multimedia label data, wherein calculating said classification function comprises iteratively performing at least updating selected multimedia label data relating to selected multimedia data or predicting new multimedia label data for stored multimedia data.

27. The system of claim 25, wherein generating selected multimedia label data comprises decoding the brain signal responses of the user and generating at least one interest score for said selected multimedia data.

28. The system of claim 27, wherein said one or more processors are further configured to refine said interest score and generate an updated interest measurement.

29. The system of claim 27, wherein said one or more processors are further configured to convert said at least one interest score to at least one interest label.

30. The system of claim 29, wherein converting said at least one interest score to at least one interest label comprises using a binarization function.

31. The system of claim 25, wherein the multimedia data comprises image data.

32. The system of claim 25, wherein the multimedia data comprises video data.

33. The system of claim 25, wherein the multimedia data comprises audio data.

34. The method of claim 25, wherein the multimedia data comprises locations, objects, and events in a virtual world.

35. The method of claim 25, wherein the multimedia data comprises locations, objects, and events in a physical world.

36. The system of claim 25, wherein predicting new multimedia label data comprises automatically selecting a most informative data sample, predicting its corresponding class and labeling the corresponding data sample.

37. The system of claim 25, wherein refining at least a portion of said multimedia label data comprises performing a greedy search among the gradient direction of the classification function.

38. The system of claim 26, wherein said one or more processors are further configured to normalize at least one multimedia label data sample on a regularization matrix calculated using members of a corresponding class and connectivity degrees of the corresponding nodes in the graph.

39. The system of claim 26, wherein calculating a classification function comprises incremental calculation using graph superposition, wherein a newly added label is incorporated incrementally without calculating a classification function using all labels.

40. The system of claim 25 wherein unreliable label data is replaced.

46. The system of claim 25, wherein said one or more processors are further configured to present said selected multimedia data to said first user one or more additional times and determine user interest in said selected multimedia data one or more additional times to further refine said multimedia label data.

47. The system of claim 25, wherein said one or more processors are further configured to:

present said selected multimedia data to a second user,

determine user interest in said selected multimedia data based on at least brain signal responses of the first user and brain signal responses of the second user.

48. The system of claim 25, wherein said one or more processors are further configured to use said refined or predicted multimedia label data to perform a search of said stored multimedia data.

49. A computer readable medium containing digital information which when executed cause a processor or processors to:

present selected multimedia data to a first user;

determine user interest in said selected multimedia data based upon brain signal responses of the first user;

generate selected multimedia label data corresponding to the selected multimedia data based on the determination of user interest;

based on at least said generated selected multimedia label data, perform at least one of refining multimedia label data associated with said selected multimedia data or predicting new multimedia label data pertaining to said stored multimedia data by calculating a classification function.

50. The computer readable medium of claim 49 containing digital information which when executed further causes the processor or processors to:

store a multimedia affinity graph in one or more memories, wherein said affinity graph represents multimedia data samples as nodes and comprises edges measuring relatedness among data samples; and

calculate a classification function based on at least the selected multimedia label data, wherein calculating said classification function comprises iteratively performing at least updating selected multimedia label data relating to selected multimedia data or predicting new multimedia label data for stored multimedia data.

51. The computer readable medium of claim 49, wherein generating selected multimedia label data comprises decoding the brain signal responses of the user and generating at least one interest score for said selected multimedia data.

52. The computer readable medium of claim 51 containing digital information which when executed further causes the processor or processors to refine said interest score and generate an updated interest measurement.

53. The computer readable medium of claim 51 containing digital information which when executed further causes the processor or processors to convert said at least one interest score to at least one interest label.

54. The computer readable medium of claim 53, wherein converting said at least one interest score to at least one interest label comprises using a binarization function.

55. The computer readable medium of claim 49, wherein the multimedia data comprises image data.

56. The computer readable medium of claim 49, wherein the multimedia data comprises video data.

57. The computer readable medium of claim 49, wherein the multimedia data comprises audio data.

58. The method of claim 49, wherein the multimedia data comprises locations, objects, and events in a physical or virtual world.

59. The method of claim 49, wherein the multimedia data comprises locations, objects, and events in a physical world.

60. The computer readable medium of claim 49, wherein predicting new multimedia label data comprises automatically selecting a most informative data sample, predicting its corresponding class and labeling the corresponding data sample.

61. The computer readable medium of claim 49, wherein refining at least a portion of said multimedia label data comprises performing a greedy search among the gradient direction of the classification function.

62. The computer readable medium of claim 50, wherein at least one multimedia label data sample is further normalized based on a regularization matrix calculated using members of a corresponding class and connectivity degrees of the corresponding nodes in the graph.

63. The computer readable medium of claim 50, wherein calculating a classification function comprises incremental calculation using graph superposition, wherein a newly added label is incorporated incrementally without calculating a classification function using all labels.

64. The computer readable medium of claim 49 wherein unreliable label data is replaced.

66. The computer readable medium of claim 50 wherein unreliable label data is replaced.

67. The computer readable medium of claim 66, wherein replacing unreliable label data comprises adding a multimedia label data sample for every multimedia label

68. The computer readable medium of claim 66, wherein replacing unreliable label data or predicting new multimedia label data comprise updating a node regularization matrix.

69. The computer readable medium of claim 66, wherein replacing unreliable label data or predicting new multimedia label data comprises minimizing an objective function.

70. The computer readable medium of claim 49 containing digital information which when executed further causes the processor or processors to present said selected multimedia data to said first user one or more additional times and determine user interest in said selected multimedia data one or more additional times to further refine said multimedia label data.

71. The computer readable medium of claim 49 containing digital information which when executed further causes the processor or processors to:

present said selected multimedia data to a second user,

determine user interest in said selected multimedia data based on at least brain signal responses of the first user and brain signal responses of the second user.

72. The computer readable medium of claim 49 containing digital information which when executed further causes the processor or processors to use said refined or predicted multimedia label data to perform a search of said stored multimedia data.