Mixed media reality retrieval of differentially-weighted links

• US 9,176,984 B2
• Filed: 10/17/2008
• Issued: 11/03/2015
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Moraleda, Jorge
• Primary Examiner(s): BURKE, JEFF A
• Application Number:
  US12/253,715
• Publication Number:
  US 20090100048A1
• Time in Patent Office: 2,573 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1847 specifically adapted to static storage, e.g. adapted to flash memory or SSD
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/00 Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

An MMR system for publishing comprises a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher integrated into a network with an advertiser, an ad broker, and an MMR service bureau. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR service bureau uses the result to retrieve advertising or other links associated with the location on the page. The list of results and links are sent back to the MMR gateway for presentation on the mobile device. The present invention also includes a number of novel methods including a differentially weighting links associated with an MMR document.

Claims

1. A computer-implemented method of differentially weighting links associated with a mixed media document, comprising:

receiving, with one or more processors, an image query;

identifying, with the one or more processors, a recognition result based on the image query;

receiving, with the one or more processors, the recognition result, the recognition result identifying at least a portion of a document and a location of the image query on at least the portion of the document;

dividing, with the one or more processors, the recognition result into a plurality of zones using a segmented version of at least the portion of the document;

retrieving, with the one or more processors, an overlap preference for each of the plurality of zones from a hotspot database;

calculating, with the one or more processors, an overlap percentage for each of the plurality of zones based on the location of the image query on at least the portion of the document and a percent of the image query that overlaps each zone;

calculating, with the one or more processors, a zone valuation for each of the plurality of zones by multiplying the overlap percentage and the overlap preference for a respective zone;

retrieving, with the one or more processors, one or more links to external media from the hotspot database, the one or more links associated with the plurality of zones;

ranking, with the one or more processors, the one or more links using the zone valuation for each of the plurality of zones, the zone valuation indicating a likelihood of displaying a link associated with a zone of the plurality of zones at a top of a list of links and the ranking being higher for a higher zone valuation; and

transmitting, with the one or more processors, the list comprising the ranked links.

2. The method of claim 1, wherein the zone valuation is calculated using one or more of the plurality of zones, overlaps, the location, and distance information.

3. The method of claim 1, wherein calculating the overlap percentage comprises comparison of the location of the image query on at least the portion of the document and locations of the plurality of zones using an R-tree data structure.

4. The method of claim 1, wherein the location of the image query on at least the portion of the document comprises one or more x,y coordinates.

5. The method of claim 1, further comprising:

retrieving, with the one or more processors, a non-overlap preference for each of the plurality of zones; and

calculating, with the one or more processors, the zone valuation based on the non-overlap preference for each of the plurality of zones having a zero overlap percentage.

6. The method of claim 1, wherein calculating the zone valuation for each of the plurality of zones is also based on a normalized distance between a center of the location of the image query on at least the portion of the document and a center of each zone.

7. The method of claim 1, wherein retrieving the one or more links comprises:

retrieving portion links associated with at least the portion of the document;

retrieving document links associated with the document including at least the portion of the document; and

retrieving collection links associated with a collection including at least the portion of the document.

8. The method of claim 7, further comprising:

retrieving overlap preferences associated with at least the portion of the document;

retrieving overlap preferences associated with documents of the collection; and

retrieving overlap preferences associated with the collection.

9. The method of claim 7, wherein ranking the one or more links further comprises calculating a valuation for each zone, portion, document, and collection as a product of a respective overlap preference and a distance between the location of the image query on at least the portion of the document and relative locations of each zone, portion, document, and collection.

10. The method of claim 7, further comprising ordering the zone links, the portion links, the document links, and the collection links according to a calculated valuation for each of the plurality of zones, portions, documents, and collections.

11. An apparatus for differentially weighting links associated with a mixed media document, comprising:

one or more processors;

a matching unit stored on a memory and executable by the one or more processors, the matching unit for receiving an image query;

a recognition unit coupled to the matching unit, the recognition unit for identifying a recognition result based on the image query;

a result receiver coupled to the recognition unit, the result receiver for receiving the recognition result, the recognition result identifying at least a portion of a document and a location of the image query on at least the portion of the document;

an acquisition unit coupled to the result receiver, the acquisition unit for dividing the recognition result into a plurality of zones using a segmented version of at least the portion of the document;

a link retriever coupled to the acquisition unit, the link retriever for retrieving one or more links to external media from the hotspot database, the one or more links associated with the plurality of zones;

a rank generator coupled to the link retriever, the rank generator for retrieving the overlap preference for each of the plurality of zones from the hotspot database, calculating an overlap percentage for each of the plurality of zones based on the location of the image query on at least the portion of the document and a percent of the image query that overlaps each zone, calculating a zone valuation for each of the plurality of zones by multiplying the overlap preference and the overlap percentage for a respective zone, and ranking the one or more links using the zone valuation for each of the plurality of zones, the zone valuation indicating a likelihood of displaying a link associated with a zone of the plurality of zones at a top of a list of links and the ranking being higher for a higher zone valuation; and

a rankings provider coupled to the rank generator, the rankings provider for transmitting the list comprising the ranked links.

12. The apparatus of claim 11, wherein the zone valuation is calculated using one or more of the plurality of zones, overlaps, the location, and distance information.

13. The apparatus of claim 11, wherein the rank generator is further configured for:

retrieving a non-overlap preference for each of the plurality of zones; and

calculating the zone valuation based on the non-overlap preference for each of the plurality of zones having a zero overlap percentage.

14. The apparatus of claim 11, wherein the rank generator is further configured for:

calculating the zone valuation for each of the plurality of zones based on a normalized distance between a center of the location of the image query on at least the portion of the document and a center of each zone.

15. The apparatus of claim 11, wherein the link retriever is further configured for:

retrieving portion links associated with at least the portion of the document;

retrieving document links associated with the document including at least the portion of the document; and

retrieving collection links associated with a collection including at least the portion of the document.

16. The apparatus of claim 11, wherein the link retriever is further configured for:

retrieving overlap preferences associated with portions of the document;

retrieving overlap preferences associated with documents of the collection;

retrieving overlap preferences associated with the collection.

17. The apparatus of claim 15, wherein the rankings provider is further configured for ordering the zone links, the portion links, the document links, and the collection links according to a calculated valuation for each of the plurality of zones, portions, documents, and collections.

18. A system for integrating publishing and advertising content, the system comprising:

one or more processors;

a gateway stored on a memory and executable by the one or more processors, the gateway for receiving a recognition result, the recognition result identifying at least a portion of a document and a location of the image query on at least the portion of the document, dividing the recognition result into a plurality of zones using a segmented version of at least the portion of the document, retrieving an overlap preference for each of the plurality of zones from a hotspot database, calculating an overlap percentage for each of the plurality of zones based on the location of the image query on at least the portion of the document and a percent of the image query that overlaps each zone, calculating a zone valuation for each of the plurality of zones by multiplying the overlap preference and the overlap percentage for a respective zone, retrieving one or more links to external media from the hotspot database, the one or more links associated with the plurality of zones, and ranking the one or more links using the zone valuation for each of the plurality of zones, the zone valuation indicating a likelihood of displaying a link associated with a zone of the plurality of zones at a top of the list of links and the ranking being higher for a higher zone valuation;

a publisher executable by the one or more processors and coupled to the gateway, the publisher for providing publication content and hierarchical layout information;

an advertising broker executable by the one or more processors and coupled to the publisher, the advertising broker for providing advertising content linked to locations within the publication content and preference information indicating when the linked advertising content should be displayed in conjunction with display of the publication content; and

a service bureau executable by the one or more processors and coupled to the advertising broker, the service bureau for receiving a portion of the publication content as the recognition result after the recognition result is identified from a received image query, and providing a ranked list of the linked advertising content associated with the recognition result.

19. The system of claim 18, wherein the recognition result identifies at least the portion of the document and the location of the image query on at least the portion of the document, and wherein the service bureau is further configured for transmitting the list comprising the ranked links.

20. The system of claim 18, wherein the service bureau is further configured for calculating the zone valuation using one or more of the plurality of zones, overlaps, the location, and distance information.