Augmenting and presenting captured data

• US 10,255,253 B2
• Filed: 08/27/2014
• Issued: 04/09/2019
• Est. Priority Date: 08/07/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Akselrod, Gregory, Thiruvengadachari, Prashant, Nam, Eun Ju, Nie, Zaiqing, Cao, Yong, Chilakamarri, Pradeep, Kohlmeier, Bernhard S. J.
• Primary Examiner(s): Zuberi, Mohammed H
• Application Number:
  US14/470,491
• Publication Number:
  US 20150046779A1
• Time in Patent Office: 1,686 Days
• CPC Class Codes: G06F 40/14 Tree-structured documents parsing G06F40/205; validation G06F40/226
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/186 Templates

Abstract

Captured data can be transformed and augmented for a particular presentation in a document, such as a note of a notebook application, based on an identified entity for the captured data. The particular presentation of captured data can be provided based on entity detection, extraction, and knowledge base resolution and retrieval. Methods, systems, and services are provided that identify a primary entity of an item input to a notebook application and create an entity object for the primary entity of the item at least from one or more structured representations for content associated with the item. A template for presenting the entity object can be determined according to the primary entity, where the template is selected from a set of templates corresponding to different primary entities such that an arrangement and presentation for one primary entity is different than that of another primary entity.

Claims

1. A method of augmenting and presenting captured data, comprising:

receiving one or more structured representations for content associated with an item input to a notebook application;

identifying a primary entity of the item, wherein the primary entity defines the content associated with the item;

creating an entity object for the primary entity of the item from at least the one or more structured representations, wherein the entity object comprises additional information relevant to the item input to the notebook application, the additional information being retrieved from one or more sources other than that of the item;

determining a template for presenting the entity object according to the primary entity, the template being selected from a set of templates, wherein each template from the set of templates corresponds to different primary entities so that an arrangement and presentation for one primary entity is different than that of another primary entity;

providing the entity object in a user interface to the notebook application according to the selected template.

2. The method of claim 1, wherein the item comprises a uniform resource locator (URL), the method further comprising:

requesting the one or more structured representations for the content associated with the URL from a web data service; and

receiving, from the web data service, one or more models associated with content of a document referenced by the URL, the one or more models providing the one or more structured representations of the item.

3. The method of claim 2, wherein creating the entity object comprises selecting a single model of the one or more models associated with the content of the document according to a selection criteria.

4. The method of claim 2, wherein creating the entity object comprises merging attributes of at least two models of the one or more models associated with the content of the document.

5. The method of claim 1, wherein the primary entity of the object is identified from a minimum number of attributes of the one or more structured representations for the content associated with the item.

6. The method of claim 5, wherein the primary entity is a recipe;

and the minimum number of attributes comprise at least an ingredient array and a steps array.

7. The method of claim 1, wherein the item comprises an image, the method further comprising:

extracting and classifying information from the image and metadata associated with the image; and

generating at least one of the one or more structured representations by using the extracted and classified information from the image and metadata associated with the image,

wherein identifying the primary entity of the item comprises using the extracted and classified information from the image and metadata associated with the image.

8. The method of claim 7, further comprising:

requesting detailed information corresponding to at least one attribute associated with the primary entity from a knowledgebase or a web data service.

9. The method of claim 7, wherein the image contains an image of a business card.

10. The method of claim 1, wherein the item comprises a clip, the method further comprising:

requesting the one or more structured representations for the content associated with the clip from a web data service.

11. A system comprising:

one or more computer readable storage media;

and program instructions stored on the one or more computer readable storage media for presenting content captured for a notebook application that, when executed by a processing system, direct the processing system to;

request an entity object for a primary entity of content associated with an item input to a notebook application service, wherein the primary entity defines the content associated with the item;

determining a template for presenting the entity object according to the primary entity, the template being selected from a set of templates, wherein each template from the set of templates corresponds to different primary entities so that an arrangement and presentation for one primary entity is different than that of another primary entity, and

in a user interface to the notebook application service, surface an entity-specific presentation of the entity object for the content associated with the item input to the notebook application service, wherein an arrangement for the entity-specific presentation of the entity object is controlled by the selected template for the primary entity, wherein the entity object comprises attributes from one or more structured representations for the content associated with the item and additional information relevant to the item, the additional information being retrieved from one or more sources other than that of the item.

12. The system of claim 11, wherein the item comprises a URL.

13. The system of claim 11, wherein the item comprises an image.

14. The system of claim 11, wherein the item comprises a clip.

15. One or more computer readable storage media having program instructions stored thereon for augmenting and presenting captured data, when executed by a processing system, direct the processing system to:

identify a primary entity of an item input to a notebook application, wherein the primary entity defines content associated with the item;

create an entity object for the primary entity of the item at least from one or more structured representations for content associated with the item, wherein the entity object comprises additional information relevant to the item, the additional information being obtained from sources accessible over the Internet or one or more knowledge bases; and

determine a template for presenting the entity object according to the primary entity, the template being selected from a set of templates, wherein each template from the set of templates corresponds to different primary entities so that an arrangement and presentation for one primary entity is different than that of another primary entity;

provide the entity object in a user interface to the notebook application according to the selected template.

16. The media of claim 15, wherein the item comprises a URL, wherein the program instructions further direct the processing system to:

request the one or more structured representations for the content associated with the URL from a web data service; and

receive, from the web data service, one or more models associated with content of a document referenced by the URL, the one or more models providing the one or more structured representations of the item.

17. The media of claim 16, wherein the entity object comprises a single model of the one or more models associated with the content of the document selected according to a selection criteria.

18. The media of claim 16, wherein the entity object comprises merged attributes of at least two models of the one or more models associated with the content of the document.

19. The media of claim 15, wherein the item comprises an image, wherein the program instructions further direct the processing system to:

extract and classify information from the image and metadata associated with the image; and

generate at least one of the one or more structured representations by using the extracted and classified information from the image and metadata associated with the image,

wherein the primary entity of the item is identified by using the extracted and classified information from the image and metadata associated with the image.