Dynamic presentation of targeted information in a mixed media reality recognition system

• US 8,452,780 B2
• Filed: 03/08/2012
• Issued: 05/28/2013
• Est. Priority Date: 01/06/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Hull, Jonathan J.
• Primary Examiner(s): Lee, Wilson
• Application Number:
  US13/415,756
• Publication Number:
  US 20120166435A1
• Time in Patent Office: 446 Days
• Field of Search: 707706-710, 707722-725, 707728-740, 707747-758, 707769-772
• US Class Current: 707/748
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0256 User search
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/1918 Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A context-aware targeted information delivery system comprises a mobile device, an MMR matching unit, a plurality of databases for user profiles, user context and advertising information, a plurality of comparison engines and a plurality of weight adjusters. The mobile device is coupled to deliver an image patch to the MMR matching unit which in turn performs recognition to produce recognize text. The recognized text is provided to a first and second comparison engines to produce relevant topics and relevant ads. The relevant topics and relevant ads are adjusted with information from a user context database including information such as location, date, time, and other information from a user profile. The third comparison engine compares the relevant topics and relevant ads to produce a set of final ads that are most related to the topics of interest for the user and delivered for display on to the mobile device.

Claims

1. A system for retrieving and generating targeted information comprising:

one or more processors;

a matching unit stored on a memory and executable by the one or more processors, the matching unit receiving an input image, performing document recognition on the input image and producing recognized text;

a first comparison engine coupled to the matching unit, the first comparison engine receiving the recognized text from the matching unit, receiving user profile information and generating a list of relevant topics based on the recognized text and the user profile information;

a second comparison engine coupled to the matching unit, the second comparison engine receiving the recognized text from the matching unit, receiving targeted information and generating a list of relevant information;

a first weight adjuster coupled to the first comparison engine, the first weight adjuster adjusting a weight of at least one of the relevant topics in the list of relevant topics using user context information, wherein adjusting the weight of the at least one of the relevant topics includes determining word relevancy by calculating a distance from a center of the input image outward using a distance measure; and

a third comparison engine coupled to the first comparison engine and the second comparison engine, the third comparison engine generating a final list of targeted information by comparing the list of relevant topics to the list of relevant information.

2. The system of claim 1, wherein the targeted information is relevant advertisements and the final list of targeted information is a final list of relevant advertisements.

3. The system of claim 1, wherein the second comparison engine compares keywords in the recognized text to advertisements in the targeted information to produce a list of relevant advertisements and wherein the final list of targeted information is a final list of relevant advertisements.

4. The system of claim 1, further comprising sending the final list of targeted information for display on the mobile device.

5. The system of claim 1, wherein the user context information includes one from the group of a patch image;

a recognized document;

a location and page of a recognized document;

recognized text from a patch image;

surrounding text for a patch image;

image features of a patch image;

hotspot information;

physical location when image captured;

time of day when the image was captured;

time of year when the image was captured;

history of previous matches;

other images that were captured before the latest one; and

a user context.

6. The system of claim 1, further comprising a second weight adjuster coupled to the second comparison engine and the third comparison engine, the second weight adjuster adjusting a weight of the at least one piece of relevant information in the list of relevant information using user context information, the adjusting of the weight including determining a feature match between the user context information and the at least one piece of relevant information based on at least one of a proximity search and a time search.

7. The system of claim 6, wherein the user context information includes at least one of a location of a mobile device that provided the input image and a time the input image was captured, the targeted information is relevant advertisements, the final list of targeted information is a final list of relevant advertisements and wherein the second weight adjuster increases the weight of relevant advertisements that are in the proximity of at least one of the location of the mobile device and the time the input image was captured.

8. The system of claim 1, further comprising a database for storing the user profile, the user context information and advertisements.

9. The system of claim 2, wherein the third comparison engine generates the final list of relevant advertisements that evolves over time based at least in part on trend information.

10. The system of claim 3, wherein the final list of targeted information is dynamically displayed on the mobile device using a visually distinctive marker.

11. The system of claim 10, wherein the visually distinctive marker is expanded to show content of the final list of targeted information in an extended window.

12. A computer-implemented method for retrieving and generating targeted information, the method comprising:

receiving an input image;

performing, with one or more computers, document recognition on the input image to produce recognized text;

receiving user profile information;

generating, with the one or more computers, a list of relevant topics based on the recognized text and the user profile information;

receiving targeted information;

generating a list of relevant information based on the recognized text and the targeted information;

adjusting a weight of at least one of the relevant topics in the list of relevant topics using user context information, wherein adjusting the weight of the at least one of the relevant topics includes determining word relevancy by calculating a distance from a center of the input image outward using a distance measure; and

generating a final list of targeted information by comparing the list of relevant topics to the list of relevant information.

13. The method of claim 12, wherein the targeted information is relevant advertisements and the final list of targeted information is a final list of relevant advertisements.

14. The method of claim 12, further comprising comparing keywords in the recognized text to advertisements in the targeted information to produce a list of relevant advertisements and wherein the final list of targeted information is a final list of relevant advertisements.

15. The method of claim 12, further comprising sending the final list of targeted information for display on the mobile device.

16. The method of claim 12, wherein the user context information includes one from the group of a patch image;

a recognized document;

a location and page of a recognized document;

recognized text from a patch image;

surrounding text for a patch image;

image features of a patch image;

hotspot information;

physical location when image captured;

time of day when the image was captured;

time of year when the image was captured;

history of previous matches;

other images that were captured before the latest one; and

a user context.

17. The method of claim 12, further comprising adjusting a weight of the at least one piece of relevant information in the list of relevant information using user context information, the adjusting of the weight including determining a feature match between the user context information and the at least one piece of relevant information based on at least one of a proximity search and a time search.

18. The method of claim 17, wherein the user context information includes at least one of a location of a mobile device that provided the input image and a time the input image was captured, the targeted information is relevant advertisements, the final list of targeted information is a final list of relevant advertisements and wherein adjusting the weight increases the weight of relevant advertisements that are in the proximity of at least one of the location of the mobile device and the time the input image was captured.

19. The method of claim 12, further comprising storing a the user profile, the user context information and advertisements.

20. A computer program product for retrieving an electronic document in response to receipt of an image patch comprising a non-transitory computer usable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive an input image;

perform document recognition on the input image to produce recognized text;

receive user profile information;

generate a list of relevant topics based on the recognized text and the user profile information;

receive targeted information;

generate a list of relevant information based on the recognized text and the targeted information;

adjust a weight of at least one of the relevant topics in the list of relevant topics using user context information, wherein adjusting the weight of the at least one of the relevant topics includes determining word relevancy by calculating a distance from a center of the input image outward using a distance measure; and

generate a final list of targeted information by comparing the list of relevant topics to the list of relevant information.