AUGMENTING AND PRESENTING CAPTURED DATA

• US 20150046779A1
• Filed: 08/27/2014
• Published: 02/12/2015
• Est. Priority Date: 08/07/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Akselrod, Gregory, Nam, Eun Ju, Nie, Zaiqing, Cao, Yong, Chilakamarri, Pradeep, Kohlmeier, Bernhard S.J., Thiruvengadachari, Prashant
• Granted Patent: US 10,255,253 B2
• Time in Patent Office: Days
• US Class Current: 715/202
• CPC Class Codes: G06F 40/14 Tree-structured documents parsing G06F40/205; validation G06F40/226
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/186 Templates

Abstract

Captured data can be transformed and augmented for a particular presentation in a document, such as a note of a notebook application, based on an identified entity for the captured data. The particular presentation of captured data can be provided based on entity detection, extraction, and knowledge base resolution and retrieval. Methods, systems, and services are provided that identify a primary entity of an item input to a notebook application and create an entity object for the primary entity of the item at least from one or more structured representations for content associated with the item. A template for presenting the entity object can be determined according to the primary entity, where the template is selected from a set of templates corresponding to different primary entities such that an arrangement and presentation for one primary entity is different than that of another primary entity.

Claims

1. A method of augmenting and presenting captured data, comprising:

receiving one or more structured representations for content associated with an item input to a notebook application;

identifying a primary entity of the item;

creating an entity object for the primary entity of the item from at least the one or more structured representations.

2. The method of claim 1, further comprising:

determining a template for presenting the entity object according to the primary entity, the template being selected from a set of templates corresponding to different primary entities.

3. The method of claim 1, wherein the item comprises a uniform resource locator (URL), the method further comprising:

requesting the one or more structured representations for the content associated with the URL from a web data service; and

receiving, from the web data service, one or more models associated with content of a document referenced by the URL, the one or more models providing the one or more structured representations of the item.

4. The method of claim 3, wherein creating the entity object comprises selecting a single model of the one or more models associated with the content of the document according to a selection criteria.

5. The method of claim 3, wherein creating the entity object comprises merging attributes of at least two models of the one or more models associated with the content of the document.

6. The method of claim 1, wherein the primary entity of the object is identified from a minimum number of attributes of the one or more structured representations for the content associated with the item.

7. The method of claim 6, wherein the primary entity is a recipe;

and the minimum number of attributes comprise at least an ingredient array and a steps array.

8. The method of claim 1, wherein the item comprises an image, the method further comprising:

extracting and classifying information from the image and metadata associated with the image; and

generating at least one of the one or more structured representations by using the extracted and classified information from the image and metadata associated with the image,

wherein identifying the primary entity of the item comprises using the extracted and classified information from the image and metadata associated with the image.

9. The method of claim 8, further comprising:

requesting detailed information corresponding to at least one attribute associated with the primary entity from a knowledgebase or a web data service.

10. The method of claim 8, wherein the image contains an image of a business card.

11. The method of claim 1, wherein the item comprises a clip, the method further comprising:

requesting the one or more structured representations for the content associated with the clip from a web data service.

12. A system comprising:

one or more computer readable storage media; and

program instructions stored on the one or more computer readable storage media for presenting content captured for a notebook application that, when executed by a processing system, direct the processing system to;

request an entity object for content associated with an item input to a notebook application service; and

in a user interface to the notebook application service, surface an entity-specific presentation of the entity object for the content associated with the item input to the notebook application service, wherein the entity object comprises attributes from one or more structured representations for the content associated with the item.

13. The system of claim 12, wherein the item comprises a URL.

14. The system of claim 12, wherein the item comprises an image.

15. The system of claim 12, wherein the item comprises a clip.

16. One or more computer readable storage media having program instructions stored thereon for augmenting and presenting captured data, when executed by a processing system, direct the processing system to:

identify a primary entity of an item input to a notebook application;

create an entity object for the primary entity of the item at least from one or more structured representations for content associated with the item; and

determine a template for presenting the entity object according to the primary entity, the template being selected from a set of templates corresponding to different primary entities.

17. The media of claim 16, wherein the item comprises a URL, wherein the program instructions further direct the processing system to:

request the one or more structured representations for the content associated with the URL from a web data service; and

receive, from the web data service, one or more models associated with content of a document referenced by the URL, the one or more models providing the one or more structured representations of the item.

18. The media of claim 17, wherein the entity object comprises a single model of the one or more models associated with the content of the document selected according to a selection criteria.

19. The media of claim 17, wherein the entity object comprises merged attributes of at least two models of the one or more models associated with the content of the document.

20. The media of claim 16, wherein the item comprises an image, wherein the program instructions further direct the processing system to:

extract and classify information from the image and metadata associated with the image; and

generate at least one of the one or more structured representations by using the extracted and classified information from the image and metadata associated with the image,

wherein the primary entity of the item is identified by using the extracted and classified information from the image and metadata associated with the image.