Method and system for semantically labeling data and providing actions based on semantically labeled data

• US 20020198909A1
• Filed: 06/27/2002
• Published: 12/26/2002
• Est. Priority Date: 06/06/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gjerstad, Kevin B., Huynh, Tuan, Wang, Ziyi, Reynar, Jeff, Wolff, Roger, Higashiyama, Nobuya, Ammerlaan, Michael
• Granted Patent: US 7,451,389 B2
• Time in Patent Office: Days
• US Class Current: 715/201
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A method for recognizing data, including strings and media objects, and annotating, or labeling, the data with a type label. After the data is annotated with a type label, application program modules may use the type label to provide users with a choice of actions. If the user'"'"'s computer does not have any actions associated with a type label, the user may be provided with the option to surf to a download Uniform Resource Locator (URL) and download action plug-ins for that type label. One or more recognizer plug-ins perform the recognition of particular strings or media objects in an electronic document. The recognizer plug-ins may be packaged with an application program module or they may be written by third parties to recognize particular data that is of interest. One or more action plug-ins provide possible actions to be presented to the user based upon the type label associated with the data in an electronic document.

Claims

1. A computer-implemented method for semantically labeling data as the data is added to an electronic document created in an application program module, comprising:

accessing data being added to an electronic document;

analyzing said data to recognize semantic information in said data;

if semantic information is recognized, determining a semantic label for said semantic information; and

applying said semantic label to said data.

2. The method of claim 1, wherein accessing data being added to an electronic document includes accessing data related to a media object being added to said electronic document and wherein said semantic label is applied to semantic information recognized in said data related to said media object.

3. The method of claim 2, wherein said data related to a media object is at least one of:

(a) said media object itself;

(b) a pointer to said media object; and

(c) metadata about said media object.

4. The method of claim 3, wherein said metadata about said media object includes a set of properties about said media object.

5. The method of claim 2, wherein said data related to a media object is probabilistic data.

6. The method of claim 5, wherein said probabilistic data is handwriting.

7. The method of claim 6, further comprising applying handwriting recognition to said handwriting and generating a lattice of hypothesis about text associated with said handwriting and wherein accessing data being added to the electronic document includes accessing said lattice of hypotheses.

8. The method of claim 5, wherein said probabilistic data is speech.

9. The method of claim 8, further comprising applying speech recognition to said speech and generating a lattice of hypothesis about text associated with said speech and wherein accessing data being added to the electronic document includes accessing said lattice of hypotheses.

10. The method of claim 2, wherein accessing said data related to a media object includes determining if said media object is a recognized type, and if said media object is not a recognized type, ending the method.

11. The method of claim 2, wherein analyzing said data related to a media object to recognize semantic information includes analyzing data surrounding said media object in said electronic document for semantic information about said media object.

12. The method of claim 2, wherein analyzing said data to recognize semantic information includes comparing said data related to a media object with stored data associated with stored semantic labels to find a match, and if a match is found, determining the stored semantic label associated with said match and applying said stored semantic label to said semantic information recognized in said data related to said media object.

13. The method of claim 2, further comprising modifying the content of said electronic document to reflect said semantic label.

14. The method of claim 2, further comprising displaying an indication that said semantic label has been applied.

15. The method of claim 2, wherein said data related to said media object is accessed by a recognizer dynamic link library and wherein said method also includes said recognizer dynamic link library transmitting said data related to said media object to a plurality of recognizer dynamic link library plug-ins, said plurality of recognizer dynamic link library plug-ins analyzing said data related to said media object to recognize said semantic information, determining said semantic label for said semantic information, and applying said semantic label to said semantic information.

16. The method of claim 2, further comprising:

determining that a user has selected said media object;

in response to determining that a user has selected said media object, determining a plurality of actions based on said semantic label for said selected media object; and

displaying said plurality of actions to the user.

17. The method of claim 16, wherein said semantic label is received by an action dynamic link library and wherein said method also includes said action dynamic link determining said plurality of actions by querying a plurality of installed action dynamic link library plug-ins to determine which of said plurality of action dynamic link library plug-ins apply to said semantic label.

18. The method of claim 16, further comprising:

determining that said user has selected one of said plurality of actions; and

in response to determining that said user has selected said one of said plurality of actions, causing said application program module to execute said selected action.

19. The method of claim 18, wherein causing said application program module to execute said selected action includes:

determining whether an action program module assigned to the selected action is available; and

if an action program module assigned to the selected action is available, receiving instructions from said action program module assigned to the selected action.

20. The method of claim 19, further comprising if an action program module assigned to the selected action is not available, using a Uniform Resource Locator assigned to the selected action to navigate to a Web site and download the action program module.

21. The method of claim 20, wherein said action program module is an action dynamic link library plug-in.

22. The method of claim 2, further comprising determining a plurality of semantic labels and applying said plurality of semantic labels to semantic information recognized in said data related to said media object.

24. The computer-readable medium of claim 23, wherein accessing data being added to an electronic document includes accessing data related to a media object being added to said electronic document and wherein said semantic label is applied to semantic information recognized in said data related to said media object.

25. The computer-readable medium of claim 24, wherein said data related to a media object is at least one of:

(a) said media object itself;

(b) a pointer to said media object; and

(c) metadata about said media object.

26. The computer-readable medium of claim 25, wherein said metadata about said media object includes a set of properties about said media object.

27. The computer-readable medium of claim 24, wherein said data related to a media object is probabilistic data.

28. The computable-readable medium of claim 27, wherein said probabilistic data is handwriting.

29. The computable-readable medium of claim 29, wherein the instructions when executed further cause the computer to apply handwriting recognition to said handwriting and generate a lattice of hypothesis about text associated with said handwriting, and wherein accessing data being added to an electronic document includes said lattice of hypotheses.

30. The computable-readable medium of claim 27, wherein said probabilistic data is speech.

31. The computable-readable medium of claim 30, wherein the instructions when executed further cause the computer to apply speech recognition to said speech and generate a lattice of hypothesis about text associated with said speech and wherein accessing data being added to an electronic document includes said lattice of hypotheses.

32. The computable-readable medium of claim 24, wherein accessing said data related to a media object includes determining if said media object is a recognized type, and if said media object is not a recognized type, ending the method.

33. The computable-readable medium of claim 24, wherein analyzing said data related to a media object to recognize semantic information includes analyzing data surrounding said media object in said electronic document for semantic information about said media object.

34. The computable-readable medium of claim 24, wherein analyzing said data to recognize semantic information includes comparing said data related to a media object with stored data associated with stored semantic labels to find a match, and if a match is found, determining the stored semantic label associated with said match and applying said stored semantic label to said semantic information recognized in said data related to said media object.

35. The computable-readable medium of claim 24, wherein the instructions when executed further cause the computer to modify the content of said electronic document to reflect said semantic label.

36. The method of claim 24, wherein the instructions when executed further cause the computer to display an indication that said semantic label has been applied.

37. The computable-readable medium of claim 24, wherein the instructions when executed further cause the computer to:

determine that a user has selected said media object;

in response to determining that a user has selected said media object, determine a plurality of actions based on said semantic label for said select media object; and

display said plurality of actions to the user.

38. The computable-readable medium of claim 37, wherein the instructions when executed further cause the computer to:

determine that said user has selected one of said plurality of actions; and

in response to determining that said user has selected said one of said plurality of actions, cause said application program module to execute said selected action.

39. The computable-readable medium of claim 38, wherein causing said application program module to execute said selected action includes:

determining whether an action program module assigned to the selected action is available; and

if an action program module assigned to the selected action is available, receiving instructions from said action program module assigned to the selected action.

40. The computable-readable medium of claim 39, further comprising if an action program module assigned to the selected action is not available, using a Uniform Resource Locator assigned to the selected action to navigate to a Web site and download the action program module.

23. A computer-readable medium having computer-executable instructions for semantically labeling data as the data is added to an electronic document created in an application program module, the instructions when executed cause the computer to:

access data being added to an electronic document;

analyze said data to recognize semantic information in said data;

if semantic information is recognized, determine a semantic label for said semantic information; and

apply said semantic label to said data.

41. A computer-readable medium having a computer-executable component for semantically labeling data being adding to an electronic document created in an application program module, the computer-executable component comprising:

a user interface for receiving data to be added to an electronic document and, if said data is recognized as including semantic information, displaying an indication of the location of said recognized semantic information in said data being added.

42. The computer-readable medium of claim 41, wherein said indication includes at least one of:

(a) coloration;

(b) highlighting

(c) brackets; and

(d) an icon.

43. The computer-readable medium of claim 41, wherein if a plurality of semantic information is recognized in said data being added to an electronic document, the user interface displays a nesting of indications of the locations of said plurality of semantic information.

44. The computer-readable medium of claim 41, wherein the data being added includes a media object and the media object includes semantic information.

45. The computer-readable medium of claim 43, wherein the media object includes probabilistic data.

46. The computer-readable medium of claim 45, wherein the probabilistic data is handwriting.

47. The computer-readable medium of claim 45, wherein the probabilistic data is speech.

48. The computer-readable medium of claim 41, wherein in response to the user positioning a cursor over an indication of the location of said recognized semantic information, the user interface displaying a semantic label associated with said recognized semantic information and a plurality of actions associated with said semantic label.

49. The computer-readable medium of claim 48, wherein if a plurality of semantic information is recognized in said data being added to an electronic document the user interface displays a nesting of a plurality of semantic labels and a plurality of actions associated with each of said plurality of semantic labels.

50. The computer-readable medium of claim 48, wherein in response to a user selecting one of said plurality of actions associated with said semantic label, the user interface causing said application program module to execute said selected action.

51. The computer-readable medium of claim 50, wherein said recognized semantic information is a photograph image and wherein said selected one of said plurality of associated actions is at least one of:

showing photographs taken on the same day;

showing photographs with similar descriptions;

electronically mailing a link to the photograph;

inserting photographs taken on the same day; and

inserting photographs having similar descriptions.

52. The computer-readable medium of claim 50, wherein said recognized semantic information is a song and wherein said selected one of said plurality of associated actions is at least one of:

creating a playlist of songs by the same artist; and

showing music liked by people who liked the song.

53. The computer-readable medium of claim 50, wherein causing said application program module to execute said selected action includes:

determining whether an action program module assigned to the selected action is available; and

if an action program module assigned to the selected action is available, receiving instructions from said action program module assigned to the selected action.

54. The computer-readable medium of claim 53, further comprising if an action program module assigned to the selected action is not available, using a Uniform Resource Locator assigned to the selected action to navigate to a Web site and download the action program module.

55. A computer-readable medium having a data structure stored thereon for an electronic document being created in an application program module, comprising:

an amount of data added by a user that includes recognized semantic information; and

a semantic label associated with said recognized semantic information, said semantic label indicating the type of recognized semantic information, said semantic label associated with a plurality of actions.

56. The computer-readable medium of claim 55, wherein said data added by a user to said electronic document including a media object, said semantic label being associated with said media object.

57. The computer-readable medium of claim 55, wherein said data added by a user to said electronic document including probabilistic data, said semantic label being associated with said probabilistic data.

58. The computer-readable medium wherein said probabilistic data is handwriting.

59. The computer-readable medium wherein said probabilistic data is speech.

60. A data processing system operable to semantically label data as the data is entered into the electronic document, comprising:

(a) an application program module for creating an electronic document;

(b) a recognizer component coupled to the application program module for receiving semantic information in data being added to said electronic document and applying a semantic label to any recognized semantic information;

(c) a user interface component for;

(i) displaying an indication that a semantic label was applied to recognized semantic information,

(ii) displaying a plurality of actions associated with said semantic label in response to a user positioning the cursor over said indication, and

(iii) in response to a user selecting one of said plurality of actions, causing said application program to execute said selected action; and

(d) an action component coupled to the application program module for performing said selected action.

61. The data processing system of claim 60, wherein the recognizer component recognizes semantic information in data being added to said electronic document and applies a semantic label to recognized semantic information by comparing said data being added to said electronic document to stored data to determine whether said data being added to said electronic document matches any of the stored data and, if a match is found, labeling the data being added to said electronic document with a label associated with the matching stored data

62. The data processing system of claim 60, wherein the data added to said electronic document includes a media object and wherein if the recognizer component recognizes data related to the media object, the recognizer component compares the data related to the media object to stored data to determine whether the data related to the media object matches any of the stored data and, if the data related to the media object matches any of the stored data, labeling the media object with a label associated with the matching stored data.

63. The data processing system of claim 62, wherein the data added to said electronic document is probabilistic data and wherein if the recognizer component recognizes probabilistic data, the recognizer component compares the probabilistic data to stored data to determine whether the probabilistic data matches any of the stored data and, if the probabilistic data matches any of the stored data, labeling the probabilistic data with a label associated with the matching stored data.

64. The data processing system of claim 63, wherein the probabilistic data is handwriting and the handwriting is semantically labeled as the handwriting is added to said electronic document.

65. The data processing system of claim 64, further comprising a handwriting recognition program module for generating a lattice of hypotheses about text associated with the handwriting and wherein the semantic information received by the recognizer component includes the generated lattices of hypotheses.

66. The data processing system of claim 63, wherein the probabilistic data is speech and the speech is semantically labeled as the speech is added to said electronic document.

67. The data processing system of claim 66, further comprising a speech recognition program module for generating a lattice of hypotheses about text associated with the speech and wherein the semantic information received by the recognizer component includes the generated lattices of hypotheses.

68. The data processing system of claim 60, wherein the recognizer component includes a recognizer dynamic-link library and wherein the data processing system includes at least one recognizer plug-in connected to the recognizer dynamic link library.

69. The data processing system of claim 68, including a plurality of recognizer plug-ins connected to the recognizer dynamic link library, said dynamic link library transmitting said data related to said received semantic information to said plurality of recognizer plug-ins and, if any of said plurality of recognizer plug-ins recognize semantic information in said received semantic information, said recognizer plug-in determining a semantic label, applying said semantic label to said recognized semantic information, and transmitting said semantic label to said dynamic link library.

70. The data processing system of claim 69, wherein said recognizer dynamic-link library transmits said semantic label to said application program module.

71. The data processing system of claim 70, wherein said recognizer dynamic link library synchronizes plural semantic labels received from said plurality of recognizer plug-ins before transmitting said plural semantic labels to said application program module.

72. The computer system of claim 60, further comprising:

an action component connected to the application program module operable to perform said selected action.

73. The data processing system of claim 72, wherein the action component includes an action dynamic-link library and at least one action plug-in connected to the action dynamic link library.

74. The data processing system of claim 73, wherein performing said selected action includes:

said action dynamic link library determining whether an action plug-in dynamic link library assigned to the selected action is available; and

if an action plug-in dynamic link library assigned to the selected action is available, receiving instructions from the action plug-in dynamic link library assigned to the selected action.

75. The data processing system of claim 74, further comprising if an action plug-in dynamic link library assigned to the selected action is not available, using a Uniform Resource Locator associated with the selected action to navigate to a Web site and download the action plug-in dynamic link library.