IMAGE ANNOTATION ON PORTABLE DEVICES

• US 20120300089A1
• Filed: 02/25/2011
• Published: 11/29/2012
• Est. Priority Date: 09/02/2008
• Status: Active Grant

• Current Assignee: Jolly Seven, Series 70 of Allied Security Trust I (Allied Security Trust)
• Original Assignee: Ecole Polytechnique Federal de Lausanne
• Inventors: Sbaiz, Luciano, Vetterli, Martin
• Granted Patent: US 9,953,438 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: H04N 1/00323 with a measuring, monitoring or signaling apparatus, e.g. for transmitting measured information to a central location
• CPC Class Codes: H04N 1/32203 Spatial or amplitude domain methods H04N1/32309 takes precedence
• CPC Class Codes: H04N 1/32229 with selective or adaptive application of the additional information, e.g. in selected regions of the image H04N1/32213 takes precedence
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/0089 Image display device
• CPC Class Codes: H04N 2201/3226 of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
• CPC Class Codes: H04N 2201/3253 Position information, e.g. geographical position at time of capture, GPS data
• CPC Class Codes: H04N 2201/3266 of text or character information, e.g. text accompanying an image H04N2201/3226 takes precedence
• CPC Class Codes: H04N 2201/3273 Display

Abstract

A system for automated annotation of images and videos points a mobile device towards an object of interest, such as a building or landscape scenery, for the device to display an image of the scene with an annotation for the object. An annotation can include names, historical information, and links to databases of images, videos, and audio files. Different techniques can be used for determining positional placement of annotations, and, by using multiple techniques, positioning can be made more precise and reliable. The level of detail of annotation information can be adjusted according to the precision of the techniques used. A trade-off can be taken into account between precision of annotation and communication cost, delay and/or power consumption. An annotation database can be updated in a self-organizing way. Public information as available on the web can be converted to annotation data.

Claims

1. A computerized method for placing an annotation on an image of a scene, comprising the steps of:

(a) obtaining an original image of said scene with a camera;

(b) obtaining an annotation for a feature of said scene represented in said image;

(c) for each position of a plurality of positions in said image, determining a probability of having said feature at said position; and

(d) forming the annotated image by including said annotation at an annotation position in said image where said probability is the highest further comprising

(e) determining the position of said camera with a location sensor,

(f) using computer vision means for determining a similarity between said feature and reference images or models of said feature;

(g) computing a probability distribution function indicating the probability of having said feature at each of said positions in said image, based on information from said location sensor and on information from said computer vision means.

2. The method of claim 1, wherein a plurality of methods is used for determining said annotation position.

3. The method of claim 2, wherein each of said methods delivers a probability distribution function depending on the method,

and wherein the annotation is included at an annotation position determined by combining a plurality of said probability distribution functions given by different methods.

4. The method of claim 2, wherein different precisions of positioning of said annotations are associated with each method, and wherein the computation of said annotation position is based on the most precise methods.

5. The method of claim 4, wherein a precision distribution function is determined for indicating the precision of positioning given by at least one method at different points of said image.

6. The method of claim 2, wherein different reliabilities of positioning said annotation are associated with each method, and wherein the computation of said annotation position is based on the most reliable methods.

7. The method of claim 6, wherein a reliability distribution function is determined for indicating the reliability of at least one method at different points of said image.

8. The method of claim 2, wherein the cost associated with each method in term of communication costs between mobile devices (1) and servers (3) and delay in communication between said mobile devices (1) and said servers (3) is computed, and wherein the computation of said position is based on the less expensive methods.

9. The method of claim 1, comprising a step of pre-processing said image based on said position and the orientation of said camera in order to compensate for geometrical deformation of said image.

10. The method of claim 1, wherein a precision of location of said feature is determined;

and the level of resolution of said annotation is adjusted according to said precision.

14. A computer program carrier having a computer program stored therein, said computer program being arranged for causing data processing means to perform the steps of claim 1 when said computer program is executed by said data processing means.

11. An annotating device for placing an annotation on an image of a scene comprising:

(a) a camera for obtaining an original image of said scene;

(b) computation means for obtaining an annotation for a feature of said scene represented in said image;

(c) computation means for determining a probability for said feature to be placed at each position of a plurality of positions in said image, and for forming the annotated image by including said annotation at an annotation position where said probability is highest

further comprising;

(d) a location sensor for determining the position of said annotating device;

(e) computer vision means for determining the similarity between said feature and reference images or models of said features;

(f) computation means for computing a probability distribution function indicating the probability that said feature is at each said position in said image, based on information from said location sensor and on information from said computer vision means.

12. An annotating system comprising:

(a) at least one annotating device (1) for placing an annotation on an image of a scene comprising

a camera for obtaining an original image of said scene;

computation means for obtaining an annotation for a feature of said scene represented in said image;

computation means for determining a probability for said annotation to be placed at each position of a plurality of positions in said image, and for forming the annotated image by including said annotation at an annotation position where said probability is highest

further comprising;

a location sensor for determining the position of said annotating device;

computer vision means for determining the similarity between said feature and reference images or models of said features;

computation means for computing a probability distribution function indicating the probability that said feature is associated with each said position in said image, based on information from said location sensor and on information from said computer vision means;

(b) at least one server (3).

13. The annotating system of claim 12, wherein the computation of said annotation is distributed between said mobile device (1) and said server (3),

and wherein the distribution is automatically and dynamically adjusted by taking into account the precision of said annotation, the communication cost between said mobile device (1) and said server (3), the delay in communication between said mobile device (1) and said server (3) and/or the power consumption of said mobile device (1).