Mobile image search and indexing system and method

• US 8,483,519 B2
• Filed: 12/30/2009
• Issued: 07/09/2013
• Est. Priority Date: 12/22/2008
• Status: Expired due to Fees

• Current Assignee: IPOINTER, INC.
• Original Assignee: IPOINTER, INC.
• Inventors: FRANK, Christopher Edward, Caduff, David
• Primary Examiner(s): Bhatnagar, Anand
• Assistant Examiner(s): PARK, SOO JIN
• Application Number:
  US12/649,845
• Publication Number:
  US 20100306200A1
• Time in Patent Office: 1,287 Days
• Field of Search: 382/305, 382/306
• US Class Current: 382/306
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass
• CPC Class Codes: H04W 4/185 by embedding added-value information into content, e.g. geo-tagging

Abstract

A computer-implemented system and method are described for image searching and image indexing that may be incorporated in a mobile device that is part of an object identification system. A computer-implemented system and method relating to a MISIS client and MISIS server that may be associated with mobile pointing and identification system for the searching and indexing of objects in in situ images in geographic space taken from the perspective of a system user located near the surface of the Earth including horizontal, oblique, and airborne perspectives.

Claims

1. A computer-implemented method for image searching and image indexing for identifying image content in at least one digital image representation of a real world visual scene, comprising the steps of

(A) determining for a mobile pointing device a mobile device geographic location, pointing direction, and a viewing angle;

(B) retrieving from a first storage location the digital image that relates to the real world visual scene that is associated with the mobile pointing device geographic location;

(C) generating at least a first viewing content cone that includes the mobile pointing device pointing direction, with the mobile pointing device geographic location being the index of the first viewing content cone, and overlaying the digital image with the first viewing content cone;

(D) determining potential objects of the digital image that are within limits of the viewing content cone;

(E) retrieving object-based datasets from a second storage location, with the object-based datasets including at least information relating to potential objects within the viewing content cone determined at Step (D);

(F) determining each object within the limits of the viewing content cone at Step (D) by linking object-based dataset information retrieved at Step (E) with each such object by indexing with an indexing mechanism each object potentially contained within the viewing content cone and assessing at least thematic information associated with each such object; and

(G) electronically annotating the digital image with an image position and directional orientation, and with respect to at least each object determined to be within the viewing content cone annotating thematic information.

2. The method as recited in claim 1, wherein the method further includes the step of:

(H) updating the indexing mechanism at Step (F) with information about available images for objects determined to be within the viewing content cone.

3. The method as recited in claim 1, wherein the mobile pointing device geographic location includes being determined by a global positioning system (GPS).

4. The method as recited in claim 1, wherein the mobile pointing device pointing direction includes being determined by a digital compass.

5. The method as recited in claim 1, wherein the mobile pointing device viewing angle includes being determined by an optical system associated with the mobile pointing device.

6. The method as recited in claim 1, wherein the mobile pointing device viewing angle includes being determined as a system default setting.

7. The method as recited in claim 1, wherein indexing with an indexing mechanism includes indexing using a Binary Space Partitioning (BSP) Tree, Octree, R-Tree, or KD-Tree (k-dimensional).

8. The method as recited in claim 1, wherein the first and second storage locations include being a single storage location.

9. The method as recited in claim 1, wherein the method further includes determining for each object within the limits of the viewing content cone a numeric value determination for the completeness of the object'"'"'s representation within the digital image.

10. The method as recited in claim 9, wherein the numeric value determination includes being determined based on nearness of the object to the index of the viewing content cone, a location of the object with respect to the pointing direction determined at Step (A), or a percentage of the object that is within the limits of the viewing content cone.

11. A computer implemented system for image searching and image indexing for identifying image content in at least one digital image representation of a real world visual scene, comprising:

a mobile pointing device having at least sensors for determining a mobile device geographic location, pointing direction, and a viewing angle, and with the mobile pointing device having a capability to generate a picture of at least a portion of the visual scene that can be processed into a digital image of at least that portion of the visual scene;

a system server that is connected to the mobile pointing device that is capable of receiving the digital image generated by the mobile pointing device, with the system server for generating at least a first viewing content cone that includes the mobile pointing device pointing direction, with the mobile pointing device geographic location being the index of the first viewing content cone, and overlaying the digital image with the first viewing content cone, determining potential objects of the digital image that are within limits of the viewing content cone, retrieving object-based datasets from a second storage location, with the object-based datasets including at least information relating to potential objects within the viewing content cone, determining each object within the limits of the viewing content cone by linking retrieved object-based dataset information with each such object by indexing with an indexing mechanism each object potentially contained within the viewing content cone and assessing at least thematic information associated with each such object, and electronically annotating the digital image with an image position and directional orientation, and with respect to at least each object determined to be within the viewing content cone annotating thematic information; and

system storage that connects to the system server that includes at least the first and second storage locations.

12. The system as recited in claim 11, wherein a first mobile pointing device sensor includes a global positioning system (GPS) sensor for determining the mobile pointing device geographic location.

13. The system as recited in claim 11, wherein a second mobile pointing device sensor includes a digital compass for determining the mobile pointing device pointing direction.

14. The system as recited in claim 11, wherein a third mobile pointing device sensor includes an optical system for determining a mobile pointing viewing angle.

15. The system as recited in claim 14, wherein the optical system includes a camera lens system.

16. The system as recited in claim 11, wherein the mobile pointing device includes a camera for generating the picture of at least a portion of the visual scene.