Embedded media markers and systems and methods for generating and using them

• US 9,245,043 B2
• Filed: 12/23/2009
• Issued: 01/26/2016
• Est. Priority Date: 12/23/2009
• Status: Expired due to Fees

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Fujifilm Holdings Corporation)
• Inventors: Liu, Qiong, Liao, Chunyuan, Wilcox, Lynn, Dunnigan, Anthony, Liew, Bee
• Primary Examiner(s): TSUI, WILSON W
• Application Number:
  US12/646,841
• Publication Number:
  US 20110154174A1
• Time in Patent Office: 2,225 Days
• Field of Search: 715/205, 715/208, 715230-233
• US Class Current: 1/1
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06K 19/06103 the marking being embedded in a human recognizable image, e.g. a company logo with an embedded two-dimensional code

Abstract

Embedded Media Markers (EMM) are optical-filter-like overlaid marks that can be printed on a part of a paper document that signify the existence of media associated with that part of the document. Users take a picture of an EMM-signified document patch using a cell phone camera, and the media associated with the EMM-signified document location is retrieved and displayed on the phone. Unlike bar codes, EMMs are nearly transparent and thus do not interfere with the document contents. Retrieval of media associated with an EMM is based on image local features of the captured EMM-signified document patch. A semi-automatically technique places an EMM at a location in a document, in such a way that the EMM encompasses sufficient identification features with minimal disturbance to the original document.

Claims

1. A computer-implemented method for providing a digital medium by making use of a database and an article, the database storing a set of information, corresponding to a portion of content visualized in a predetermined space, the set of information including (i) a plurality of local image features existing prior to placement of at least one embedded media marker and within the portion extracted from a local spatial arrangement of the content and (ii) an embedded digital media corresponding to the portion, wherein the portion of content has an area which includes the plurality of local image features sufficient to identify the portion of content, the article holding the portion of content and a placement of the at least one embedded media marker on said portion of content, the embedded media marker including a visible feature boundary enclosing the portion of the content, the method comprising:

a. receiving, from a requestor, an image of a patch comprising an image of the placed embedded media marker, including the visible feature boundary, on the article;

b. determining a location of the visible feature boundary in the image of the patch;

c. based on the determined location of the visible feature boundary, extracting only local image features that are within the determined location of the visible feature boundary, from the image of the patch, and identifying the embedded media marker from the extracted local image features by comparing the extracted local image features to the plurality of local image features existing prior to the placement of the at least one embedded media marker of the set of information;

d. identifying the embedded digital media corresponding to the embedded media marker;

e. retrieving the identified embedded digital media from the database; and

f. providing the retrieved digital media to the requestor.

2. The computer-implemented method of claim 1, wherein the embedded media marker further comprises a media-type indicator which comprises a media-type icon disposed within a media-type boundary, the media-type icon representing the type of media of the embedded digital media.

3. The computer-implemented method of claim 1, wherein the embedded media marker is alpha-blended with the visualized content of the article.

4. An article comprising:

a. content visualized on a surface of the article; and

b. an embedded media marker placed on the surface of the article, the embedded media marker corresponding to an embedded digital object, the embedded media marker overlaying the visualized content, the embedded media marker comprising a visible feature boundary enclosing a plurality of local image features of the visualized content extracted from a local spatial arrangement of the visualized content,

wherein the embedded media marker is located such that the embedded media marker encloses ones of the plurality of the local image features within the feature boundary and causes a computer to identify the embedded digital object corresponding to the embedded media marker in response to the computer identifying the embedded media marker by (i) determining a location of the visible feature boundary in the image of the patch, (ii) based on the determined location of the visible feature boundary, extracting only local image features that are within the determined location of the visible feature boundary, and (iii) identifying the embedded media marker by comparing the extracted local image features to a plurality of local image features existing prior to the placement of the embedded media marker that are stored as index information.

5. The article of claim 4, wherein the article is a two dimensional surface medium.

6. The article of claim 4, wherein the embedded media marker is alpha-blended with the visualized content of the article.

7. The article of claim 4, wherein the embedded media marker further comprises a media-type indicator comprising a media-type boundary, wherein the media-type indicator is disposed within the feature boundary.

8. The article of claim 7, wherein the media-type indicator of the embedded media marker further comprises a media-type icon disposed within the media-type boundary, wherein the media-type indicator is a graphical indication of the type of media of the embedded media object.

9. The article of claim 7, wherein the media-type indicator further comprises an arrow pointing to a location within the visualized content and wherein the embedded media marker indicates availability of the digital media object associated with the location within the visualized content.

10. The article of claim 7, wherein the feature, wherein the media-type indicator is disposed within the feature boundary.

11. The article of claim 4, wherein one or more edges of the embedded media marker are defined by a drop shadow graphical effect.

12. The article of claim 4, wherein the embedded media marker comprises graphical effects that appear three-dimensional.

13. A computer-implemented method for producing an article including visualized content and an embedded media marker comprising a visible feature boundary on the content, the method comprising:

a. receiving a user selection of a location on the visualized content of the article;

b. determining a position and a size of the embedded media marker, including a location of the visible feature boundary, based on having at least a threshold quantity of local image features extracted from a local spatial arrangement of the visualized content near the selected location within the embedded media marker; and

c. positioning the embedded media marker on the article based on the determined position and size, wherein the embedded media marker indicates availability of a digital media object associated with the location within the visualized content,

wherein the location of the visible feature boundary is determined such that the selected portion of the content may be identified by extracting only a plurality of local image features that are within the determined location of the visible feature boundary.

14. The computer-implemented method of claim 13, further comprising alpha-blending of the embedded media marker with the visualized content of the article.

15. The computer-implemented method of claim 13, wherein the embedded media marker further comprises a media-type indicator comprising a media-type boundary, wherein the media-type indicator is disposed within the feature boundary.

16. The computer-implemented method of claim 15, wherein the media-type indicator of the embedded media marker further comprises a media-type icon disposed within the media-type boundary, wherein the media-type icon is a graphical representation of a type of media of the digital media object.

17. The computer-implemented method of claim 15, wherein the media-type indicator further comprises an arrow pointing to the location within the visualized content to which the digital medium object relates.

18. The computer-implemented method of claim 15, wherein the determining the position and size of the embedded media marker comprises determining at least one of a feature boundary circle center, a feature boundary circle radius, and media type circle center.

19. The computer implemented system of claim 18, wherein the determining a position and a size of the embedded media marker further comprises determining if sufficient local image features exist within the embedded media marker by estimating the number of local image features within the determined position and size of the embedded media marker.

20. The computer-implemented method of claim 13, wherein one or more edges of the embedded media marker are defined by a drop shadow effect.

21. A computer-implemented method for producing an article including visualized content and an embedded media marker comprising a visible feature boundary on the content, the method comprising:

a. receiving a user selection of a location on the visualized content of the article;

b. determining a position and a size of the embedded media marker, including a location of the visible feature boundary, based on having at least a threshold quantity of local image features extracted from a local spatial arrangement of the visualized content near the selected location within the embedded media marker; and

c. positioning the embedded media marker on the article based on the determined position and size, wherein the embedded media marker indicates availability of a digital media object associated with the location within the visualized content,

wherein the location of the visible feature boundary is determined such that the selected portion of the content may be identified by extracting only a plurality of local image features that are within the determined location of the visible feature boundary,

wherein the positioning of the embedded media marker on the article comprises;

a. projecting a shape of the embedded media marker in two perpendicular directions to obtain a first projection and a second projection;

b. for each of the first projection and the second projection, generating a fixed-aspect-ratio bounding rectangle of the shape of the embedded media marker to form a plurality of bounding rectangles;

c. determining a reference circle that can be covered by each of the plurality of bounding rectangles;

d. generating a two-dimensional vector between a center of the reference circle and the location;

e. determining a radius of the reference circle;

f. using a ratio of a radius of the embedded media marker to the radius of the reference circle to scale the two dimensional vector and the embedded media marker for positioning the embedded media marker.

22. A computer-implemented method for producing an article including visualized content and an embedded media marker comprising a visible feature boundary on the content, the method comprising:

a. receiving a user selection of a location on the visualized content of the article;

b. determining a position and a size of the embedded media marker, including a location of the visible feature boundary, based on having at least a threshold quantity of local image features extracted from a local spatial arrangement of the visualized content near the selected location within the embedded media marker; and

c. positioning the embedded media marker on the article based on the determined position and size, wherein the embedded media marker indicates availability of a digital media object associated with the location within the visualized content,

wherein the location of the visible feature boundary is determined such that the selected portion of the content may be identified by extracting only a plurality of local image features that are within the determined location of the visible feature boundary,

wherein the embedded media marker further comprises a media-type indicator comprising a media-type boundary, wherein the media-type indicator is disposed within the feature boundary,

wherein the determining the position and size of the embedded media marker comprises determining at least one of a feature boundary circle center, a feature boundary circle radius, and media type circle center,

wherein the determining a position and a size of the embedded media marker further comprises determining if sufficient local image features exist within the embedded media marker by estimating the number of local image features within the determined position and size of the embedded media marker, and

wherein the estimating the number of local image features comprises deriving a square approximating the feature boundary circle; and

determining the number of local image features within the derived square based on a cumulative keypoint distribution map of the content.

23. A non-transitory computer readable storage medium storing instructions for providing a digital medium by making use of a database and an article, the database storing a set of information corresponding to a portion of content visualized in a predetermined space, the set of information including (i) a plurality of local image features existing prior to placement of at least one embedded media marker and within the portion extracted from a local spatial arrangement of the content and (ii) an embedded digital media corresponding to the portion, the portion of content has an area which includes the plurality of local image features sufficient to identify the portion of content, the article holding the portion of content and a placement of the at least one embedded media marker on said portion of content, the embedded media marker including a visible feature boundary enclosing the portion of the content, the instructions when executed by a processor performing a method comprising:

a. receiving, from a requestor, an image of a patch comprising an image of the placed embedded media marker, including the visible feature boundary, on the article;

b. determining a location of the visible feature boundary in the image of the patch;

c. based on the determined location of the visible feature boundary, extracting only local image features that are within the determined location of the visible feature boundary, from the image of the patch, and identifying the embedded media marker from the extracted local image features by comparing the extracted local image features to the plurality of local image features existing prior to the placement of the at least one embedded media marker of the set of information;

d. identifying the embedded digital media corresponding to the embedded media marker;

e. retrieving the embedded digital media from the database; and

f. providing the retrieved digital media to the requestor.

24. The non-transitory computer readable storage medium of claim 23, wherein the embedded media marker further comprises a media-type indicator which comprises a media-type icon disposed within a media-type boundary, the media-type icon representing the type of media of the embedded digital media.

25. The non-transitory computer readable storage medium of claim 23, wherein the embedded media marker is alpha-blended with the visualized content of the article.

26. A computer-implemented method for providing a digital medium, the digital medium including a document having content and an embedded media maker signifying a digital media object in association with index information based on a plurality of local image features existing prior to placement of the embedded media marker within a patch of the document extracted from a local spatial arrangement of the content;

receiving an image, from a requestor, of at least a portion of the document including an image of a placement of the embedded media marker, the embedded media marker having a visible feature boundary enclosing a portion of the document and defining the patch of the document, and a media type boundary within the feature boundary, enclosing a media icon;

determining a location of the visible feature boundary in the image of the patch;

based on the determined location of the visible feature boundary, extracting only local image features of the content that are within the determined location of the visible feature boundary;

identifying the digital media object corresponding to the embedded media marker by comparing the local image features extracted from within the visible feature boundary to the plurality of local image features existing prior to the placement of the embedded media marker of the index information stored in association with the digital media object;

retrieving the identified digital media object; and

providing the retrieved digital media object to the requestor.

27. The computer-implemented method of claim 26, further comprising identifying the media icon within the embedded media marker and identifying the digital media object using the identified media icon.