Method and system for semantically labeling data and providing actions based on semantically labeled data

• US 7,451,389 B2
• Filed: 06/27/2002
• Issued: 11/11/2008
• Est. Priority Date: 06/06/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gjerstad, Kevin B., Huynh, Tuan, Wang, Ziyi, Reynar, Jeff, Wolff, Roger, Ammerlaan, Michael, Higashiyama, Nobuya
• Primary Examiner(s): Bashore; William L
• Application Number:
  US10/184,556
• Publication Number:
  US 20020198909A1
• Time in Patent Office: 2,329 Days
• Field of Search: 715/541, 715/512, 715/516, 715/513, 715/230, 715/210, 715/243
• US Class Current: 715/230
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A method for recognizing data, including strings and media objects, and annotating, or labeling, the data with a type label. After the data is annotated with a type label, application program modules may use the type label to provide users with a choice of actions. If the user'"'"'s computer does not have any actions associated with a type label, the user may be provided with the option to surf to a download Uniform Resource Locator (URL) and download action plug-ins for that type label. One or more recognizer plug-ins perform the recognition of particular strings or media objects in an electronic document. The recognizer plug-ins may be packaged with an application program module or they may be written by third parties to recognize particular data that is of interest. One or more action plug-ins provide possible actions to be presented to the user based upon the type label associated with the data in an electronic document.

Claims

1. A computer-implemented method for semantically labeling data as the data is added to an electronic document created in an application program module, comprising:

accessing data being added to an electronic document, while monitoring to check if said data has been changed, wherein accessing data being added to an electronic document includes accessing data related to a media object being added to said electronic document;

analyzing said data to recognize semantic information in said data;

if semantic information is recognized, reanalyzing said data to determine if the data has been changed;

determining from reanalyzed data a semantic label for said semantic information;

applying said semantic label to said data, wherein said semantic label is applied to semantic information recognized in said data related to said media object; and

modifying the content of said electronic document to reflect said semantic label.

2. The method of claim 1, wherein said data related to a media object is at least one of:

(a) said media object itself;

(b) a pointer to said media object; and

(c) metadata about said media object.

3. The method of claim 2, wherein said metadata about said media object includes a set of properties about said media object.

4. The method of claim 1, wherein said data related to a media object is probabilistic data.

5. The method of claim 4, wherein said probabilistic data is handwriting.

6. The method of claim 5, further comprising applying handwriting recognition to said handwriting and generating a lattice of hypothesis about text associated with said handwriting and wherein accessing data being added to the electronic document includes accessing said lattice of hypotheses.

7. The method of claim 4, wherein said probabilistic data is speech.

8. The method of claim 7, further comprising applying speech recognition to said speech and generating a lattice of hypothesis about text associated with said speech and wherein accessing data being added to the electronic document includes accessing said lattice of hypotheses.

9. The method of claim 1, wherein accessing said data related to a media object includes determining if said media object is a recognized type, and if said media object is not a recognized type, ending the method.

10. The method of claim 1, wherein analyzing said data related to a media object to recognize semantic information includes analyzing data surrounding said media object in said electronic document for semantic information about said media object.

11. The method of claim 1, wherein analyzing said data to recognize semantic information includes comparing said data related to a media object with stored data associated with stored semantic labels to find a match, and if a match is found, determining the stored semantic label associated with said match and applying said stored semantic label to said semantic information recognized in said data related to said media object.

12. The method of claim 1, further comprising displaying an indication that said semantic label has been applied.

13. The method of claim 1, wherein said data related to said media object is accessed by a recognizer dynamic link library and wherein said method also includes said recognizer dynamic link library transmitting said data related to said media object to a plurality of recognizer dynamic link library plug-ins, said plurality of recognizer dynamic link library plug-ins analyzing said data related to said media object to recognize said semantic information, determining said semantic label for said semantic information, and applying said semantic label to said semantic information.

14. The method of claim 1, further comprising:

determining that a user has selected said media object;

in response to determining that a user has selected said media object, determining a plurality of actions based on said semantic label for said selected media object; and

displaying said plurality of actions to the user.

15. The method of claim 14, wherein said semantic label is received by an action dynamic link library and wherein said method also includes said action dynamic link determining said plurality of actions by querying a plurality of installed action dynamic link library plug-ins to determine which of said plurality of action dynamic link library plug-ins apply to said semantic label.

16. The method of claim 14, further comprising:

determining that said user has selected one of said plurality of actions; and

in response to determining that said user has selected one of said plurality of actions, causing said application program module to execute said selected action.

17. The method of claim 16, wherein causing said application program module to execute said selected action includes:

determining whether an action program module assigned to the selected action is available; and

if an action program module assigned to the selected action is available, receiving instructions from said action program module assigned to the selected action.

18. The method of claim 17, further comprising if an action program module assigned to the selected action is not available, using a Uniform Resource Locator assigned to the selected action to navigate to a Web site and download the action program module.

19. The method of claim 18, wherein said action program module is an action dynamic link library plug-in.

20. The method of claim 1, further comprising determining a plurality of semantic labels and applying said plurality of semantic labels to semantic information recognized in said data related to said media object.

21. A computer-readable storage medium having computer-executable instructions for semantically labeling data as the data is added to an electronic document created in an application program module, the instructions when executed cause the computer to:

access data being added to an electronic document, while monitoring to check if said data has been changed, wherein the data being added to an electronic document includes data related to a media object being added to said electronic document;

analyze said data to recognize semantic information in said data;

if semantic information is recognized, reanalyze said data to determine if the data has been changed;

determine a semantic label for said semantic information from said reanalyzed data;

apply said semantic label to said data, wherein said semantic label is applied to semantic information recognized in said data related to said media object; and

modify the content of said electronic document to reflect said semantic label.

22. The computer-readable storage medium of claim 21, wherein said data related to a media object is at least one of:

(a) said media object itself;

(b) a pointer to said media object; and

(c) metadata about said media object.

23. The computer-readable storage medium of claim 22, wherein said metadata about said media object includes a set of properties about said media object.

24. The computer-readable storage medium of claim 21, wherein said data related to a media object is probabilistic data.

25. The computer-readable storage medium of claim 24, wherein said probabilistic data is handwriting.

26. The computer-readable storage medium of claim 25, wherein the instructions when executed further cause the computer to apply handwriting recognition to said handwriting and generate a lattice of hypothesis about text associated with said handwriting, and wherein accessing data being added to an electronic document includes said lattice of hypotheses.

27. The computer-readable storage medium of claim 24, wherein said probabilistic data is speech.

28. The computer-readable storage medium of claim 27, wherein the instructions when executed further cause the computer to apply speech recognition to said speech and generate a lattice of hypothesis about text associated with said speech and wherein the data being added to an electronic document includes said lattice of hypotheses.

29. The computer-readable medium of claim 21, wherein accessing said data related to a media object includes determining if said media object is a recognized type, and if said media object is not a recognized type, ending the method.

30. The computer-readable storage medium of claim 21, wherein analyzing said data related to a media object to recognize semantic information includes analyzing data surrounding said media object in said electronic document for semantic information about said media object.

31. The computer-readable storage medium of claim 21, wherein analyzing said data to recognize semantic information includes comparing said data related to a media object with stored data associated with stored semantic labels to find a match, and if a match is found, determining the stored semantic label associated with said match and applying said stored semantic label to said semantic information recognized in said data related to said media object.

32. The computer-readable storage medium of claim 21, wherein the instructions when executed further cause the computer to display an indication that said semantic label has been applied.

33. The computer-readable storage medium of claim 21, wherein the instructions when executed further cause the computer to:

determine that a user has selected said media object;

in response to determining that a user has selected said media object, determine a plurality of actions based on said semantic label for said select media object; and

display said plurality of actions to the user.

34. The computer-readable storage medium of claim 33, wherein the instructions when executed further cause the computer to:

determine that said user has selected one of said plurality of actions; and

in response to determining that said user has selected said one of said plurality of actions, cause said application program module to execute said selected action.

35. The computer-readable storage medium of claim 34, wherein causing said application program module to execute said selected action includes:

determining whether an action program module assigned to the selected action is available; and

if an action program module assigned to the selected action is available, receiving instructions from said action program module assigned to the selected action.

36. The computer-readable storage medium of claim 35, further comprising if an action program module assigned to the selected action is not available, using a Uniform Resource Locator assigned to the selected action to navigate to a Web site and download the action program module.

37. A computer-readable storage medium having a computer-executable component for semantically labeling data as the data is added to an electronic document created in an application program module, the computer-executable component comprising:

a user interface for receiving data to be added to an electronic document while monitoring to check if said data has been changed, and if said data is recognized as including semantic information, then reanalyzing said data to determine if said data has been changed and determining from said reanalyzed data a semantic label for said semantic information before displaying an indication of a location of said semantic information in said data being added, wherein said indication includes at least one of;

(a) coloration;

(b)highlighting;

(c)brackets; and

(d)an icon, and

wherein if a plurality of semantic information is recognized in said data being added to an electronic document, the user interface displays a nesting of indications of the locations of said plurality of semantic information.

38. The computer-readable storage medium of claim 37, wherein the data being added includes a media object and the media object includes semantic information.

39. The computer-readable storage medium of claim 38, wherein the media object includes probabilistic data.

40. The computer-readable storage medium of claim 39, wherein the probabilistic data is handwriting.

41. The computer-readable storage medium of claim 39, wherein the probabilistic data is speech.

42. The computer-readable storage medium of claim 37, wherein in response to the user positioning a cursor over an indication of the location of said recognized semantic information, the user interface displaying a semantic label associated with said recognized semantic information and a plurality of actions associated with said semantic label.

43. The computer-readable storage medium of claim 37, wherein said recognized semantic information is a song and wherein said selected one of said plurality of associated actions is at least one of:

creating a playlist of songs by the same artist; and

showing music liked by people who liked the song.

44. The computer-readable storage medium of claim 37, wherein causing said application program module to execute said selected action includes:

determining whether an action program module assigned to the selected action is available; and

if an action program module assigned to the selected action is available, receiving instructions from said action program module assigned to the selected action.

45. The computer-readable storage medium of claim 44, further comprising if an action program module assigned to the selected action is not available, using a Uniform Resource Locator assigned to the selected action to navigate to a Web site and download the action program module.

46. A computer-readable storage medium having a computer-executable component for semantically labeling data as the data is added to an electronic document created in an application program module, the computer-executable component comprising:

a user interface for receiving data to be added to an electronic document while monitoring to check if said data has been changed, and if said data is recognized as including semantic information, then reanalyzing said data to determine if said data has been changed and determining from said reanalyzed data a semantic label for said semantic information before displaying an indication of a location of said semantic information in said data being added,

wherein in response to the user positioning the cursor over an indication of the location of said recognized semantic information, the user interface displaying a semantic label associated with said recognized semantic information and a plurality of actions associated with said semantic label; and

wherein if a plurality of semantic information is recognized in said data being added to an electronic document the user interface displays a nesting of a plurality of semantic labels and a plurality of actions associated with each of said plurality of semantic labels.

47. The computer-readable storage medium of claim 46, wherein the data being added includes a media object and the media object includes semantic information.

48. The computer-readable storage medium of claim 47, wherein the media object includes probabilistic data.

49. The computer-readable storage medium of claim 48, wherein the probabilistic data is handwriting.

50. The computer-readable storage medium of claim 49, wherein the probabilistic data is speech.

51. A computer-readable storage medium having a computer-executable component for semantically labeling data as the data is added to an electronic document created in an application program module, the computer-executable component comprising:

a user interface for receiving data to be added to an electronic document while monitoring to check if said data has been changed, and if said data is recognized as including semantic information, then reanalyzing said data to determine if said data has been changed and determining from said reanalyzed data a semantic label for said semantic information before displaying an indication of a location of said semantic information in said data being added,

wherein in response to the user positioning the cursor over an indication of the location of said recognized semantic information, the user interface displaying a semantic label associated with said recognized semantic information and a plurality of actions associated with said semantic label; and

wherein said recognized semantic information is a photograph image and wherein said selected one of said plurality of associated actions is at least one of;

showing photographs taken on the same day;

showing photographs with similar descriptions;

electronically mailing a link to the photograph;

inserting photographs taken on the same day; and

inserting photographs having similar descriptions.

52. The computer-readable storage medium of claim 51, wherein the data being added includes a media object and the media object includes semantic information.

53. The computer-readable storage medium of claim 52, wherein the media object includes probabilistic data.

54. The computer-readable storage medium of claim 53, wherein the probabilistic data is handwriting.

55. The computer-readable storage medium of claim 54, wherein the probabilistic data is speech.

56. A data processing system including a processor and memory operable to semantically label data as the data is entered into the electronic document, comprising:

(a) an application program module for creating an electronic document;

(b) a recognizer component coupled to the application program module for receiving semantic information in data being added to said electronic document, and for monitoring said data to check if it has been changed;

(c) a compiler component coupled to the application program module for compiling said data after reanalyzing said data to determine if the data has been changed and before said recognizer component applies a semantic label to any recognized information;

(d) a user interface component for;

(i) displaying an indication that a semantic label was applied to recognized semantic information,

(ii) displaying a plurality of actions associated with said semantic label in response to a user positioning the cursor over said indication, and

(iii) in response to a user selecting one of said plurality of actions, causing said application program to execute said selected action; and

(e) an action component coupled to the application program module for performing said selected action.

57. The data processing system of claim 56, wherein the recognizer component recognizes semantic information in data being added to said electronic document and applies a semantic label to recognized semantic information by comparing said data being added to said electronic document to stored data to determine whether said data being added to said electronic document matches any of the stored data and, if a match is found, labeling the data being added to said electronic document with a label associated with the matching stored data.

58. The data processing system of claim 56, wherein the data added to said electronic document includes a media object and wherein if the recognizer component recognizes data related to the media object, the recognizer component compares the data related to the media object to stored data to determine whether the data related to the media object matches any of the stored data and, if the data related to the media object matches any of the stored data, labeling the media object with a label associated with the matching stored data.

59. The data processing system of claim 58, wherein the data added to said electronic document is probabilistic data and wherein if the recognizer component recognizes probabilistic data, the recognizer component compares the probabilistic data to stored data to determine whether the probabilistic data matches any of the stored data and, if the probabilistic data matches any of the stored data, labeling the probabilistic data with a label associated with the matching stored data.

60. The data processing system of claim 59, wherein the probabilistic data is handwriting and the handwriting is semantically labeled as the handwriting is added to said electronic document.

61. The data processing system of claim 60, further comprising a handwriting recognition program module for generating a lattice of hypotheses about text associated with the handwriting and wherein the semantic information received by the recognizer component includes the generated lattices of hypotheses.

62. The data processing system of claim 59, wherein the probabilistic data is speech and the speech is semantically labeled as the speech is added to said electronic document.

63. The data processing system of claim 62, further comprising a speech recognition program module for generating a lattice of hypotheses about text associated with the speech and wherein the semantic information received by the recognizer component includes the generated lattices of hypotheses.

64. The data processing system of claim 56, wherein the recognizer component includes a recognizer dynamic-link library and wherein the data processing system includes at least one recognizer plug-in connected to the recognizer dynamic link library.

65. The data processing system of claim 64, including a plurality of recognizer plug-ins connected to the recognizer dynamic link library, said dynamic link library transmitting said data related to said received semantic information to said plurality of recognizer plug-ins and, if any of said plurality of recognizer plug-ins recognize semantic information in said received semantic information, said recognizer plug-in determining a semantic label, applying said semantic label to said recognized semantic information, and transmitting said semantic label to said dynamic link library.

66. The data processing system of claim 65, wherein said recognizer dynamic-link library transmits said semantic label to said application program module.

67. The data processing system of claim 66, wherein said recognizer dynamic link library synchronizes plural semantic labels received from said plurality of recognizer plug-ins before transmitting said plural semantic labels to said application program module.

68. The data processing system of claim 56, further comprising:

an action component connected to the application program module operable to perform said selected action.

69. The data processing system of claim 68, wherein the action component includes an action dynamic-link library and at least one action plug-in connected to the action dynamic link library.

70. The data processing system of claim 69, wherein performing said selected action includes:

said action dynamic link library determining whether an action plug-in dynamic link library assigned to the selected action is available; and

if an action plug-in dynamic link library assigned to the selected action is available, receiving instructions from the action plug-in dynamic link library assigned to the selected action.

71. The data processing system of claim 70, further comprising if an action plug-in dynamic link library assigned to the selected action is not available, using a Uniform Resource Locator associated with the selected action to navigate to a Web site and download the action plug-in dynamic link library.