DYNAMIC PRESENTATION OF TARGETED INFORMATION IN A MIXED MEDIA REALITY RECOGNITION SYSTEM

• US 20090125510A1
• Filed: 12/23/2008
• Published: 05/14/2009
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Hull, Jonathan J.
• Granted Patent: US 8,156,116 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0241 Advertisements
• CPC Class Codes: G06Q 30/0256 User search
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/1918 Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion
• CPC Class Codes: G06V 30/414 : Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A context-aware targeted information delivery system comprises a mobile device, an MMR matching unit, a plurality of databases for user profiles, user context and advertising information, a plurality of comparison engines and a plurality of weight adjusters. The mobile device is coupled to deliver an image patch to the MMR matching unit which in turn performs recognition to produce recognize text. The recognized text is provided to a first and second comparison engines to produce relevant topics and relevant ads. The relevant topics and relevant ads are adjusted with information from a user context database including information such as location, date, time, and other information from a user profile. The weight adjusted relevant topics and relevant ads to a third comparison engine. The third comparison engine compares the weighted relevant topics and relevant ads to produce a set of final ads that are most related to the topics of interest for the user and delivered for display on to the mobile device.

Claims

1. A method for retrieving and generating targeted information, the method comprising:

receiving an image;

performing document recognition using the captured image to produced recognized text;

producing a list of relevant topics using the recognized text and user profile information;

producing a list of relevant information using the recognized text and target information; and

comparing the list of relevant topics to the list of relevant information to produce a final list of information.

2. The method of claim 1, wherein the relevant targeted information is relevant advertisements.

3. The method of claim 1, comprising:

capturing the image with a mobile device; and

displaying an indicator for the final list of information on the mobile device.

4. The method of claim 3, wherein the indicator is a colored rectangle positioned near the outer edge of the screen of the mobile device.

5. The method of claim 3, comprising receiving user input selecting the indicator and sending enhanced content for display on the mobile device.

6. The method of claim 1, comprising sending final list of information for display on the mobile device.

7. The method of claim 1, further comprising adjusting a weight of one of the relevant topic in the list of relevant topics based on user context information.

8. The method of claim 7, wherein the user context information includes one from the group of a patch image;

a recognized document;

a location and page of a recognized document;

recognized text from a patch image;

surrounding text for a patch image;

image features of a patch image;

hotspot information;

physical location when image captured;

time of day when the image was captured;

time of year when the image was captured;

history of previous matches;

other images that were captured before the latest one; and

a user context.

9. The method of claim 1, comprising adjusting a weight of one of the list of relevant information based on user context information.

10. The method of claim 9, wherein the user context information includes one from the group of a patch image;

a recognized document;

a location and page of a recognized document;

recognized text from a patch image;

surrounding text for a patch image;

image features of a patch image;

hotspot information;

physical location when image captured;

time of day when the image was captured;

time of year when the image was captured;

history of previous matches;

other images that were captured before the latest one; and

a user context.

11. The method of claim 9, wherein adjusting a weight of one of the list of relevant information based on user context information comprises determining word relevancy by calculating a distance from a center of a current image outward using a distance measure.

12. The method of claim 1, comprising storing the final list of information in a user context database.

13. A system for retrieving and generating targeted information for an input image patch, the system comprising:

a matching unit having an input and an output for generating recognized text from the input image, the input of the matching unit adapted to receive the input image;

a first comparison engine having a first input, a second input and an output for generating a list of relevant topics, the first input of the first comparison engine coupled to receive recognized text from the matching unit, the second input of the first comparison engine adapted to receive user profile information;

a second comparison engine having a first input, a second input and an output for generating a list of relevant information, the first input of the second comparison engine coupled to receive recognized text from the matching unit, the second input of the second comparison engine adapted to receive target information; and

a third comparison engine having a first input, a second input and an output for generating produce a final list of targeted information by comparing the list of relevant topics to the list of relevant information, the first input of the third comparison engine coupled to the output of the first comparison engine, and the second input of the third comparison engine coupled to the output of the second comparison engine.

14. The system of claim 13, wherein the targeted information is relevant advertisements.

15. The system of claim 13, wherein the third comparison engine sends the final list of targeted information to a mobile device for display.

16. The system of claim 13, further comprising a weight adjuster having an input and an output for adjusting a weight of one of the relevant topics in the list of relevant topics using user context information, the input of the weight adjuster coupled to the output of the first comparison engine, and the output of the weight adjuster coupled to the first input of the third comparison engine.

17. The system of claim 16, wherein the user context information includes one from the group of a patch image;

a recognized document;

a location and page of a recognized document;

recognized text from a patch image;

surrounding text for a patch image;

image features of a patch image;

hotspot information;

physical location when image captured;

time of day when the image was captured;

time of year when the image was captured;

history of previous matches;

other images that were captured before the latest one; and

a user context.

18. The system of claim 13, further comprising a weight adjuster having an input and an output for adjusting a weight of one of the list of relevant information using user context information, the input of the weight adjuster coupled to the output of the second comparison engine, and the output of the weight adjuster coupled to the first input of the third comparison engine.

19. The system of claim 18, wherein the user context information includes one from the group of a patch image;

a recognized document;

a location and page of a recognized document;

recognized text from a patch image;

surrounding text for a patch image;

image features of a patch image;

hotspot information;

physical location when image captured;

time of day when the image was captured;

time of year when the image was captured;

history of previous matches;

other images that were captured before the latest one; and

a user context.

20. The system of claim 18, further comprising a database for storing a user profile, user context information and advertisements.