System and Method for Exploring 3D Scenes by Pointing at a Reference Object

• US 20100306707A1
• Filed: 12/22/2009
• Published: 12/02/2010
• Est. Priority Date: 12/22/2008
• Status: Active Grant

• Current Assignee: IPOINTER, INC.
• Original Assignee: Intelligent Spatial Technologies, Inc.
• Inventors: CADUFF, David
• Granted Patent: US 8,745,090 B2
• Time in Patent Office: Days
• US Class Current: 715/849
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass
• CPC Class Codes: H04W 4/185 by embedding added-value information into content, e.g. geo-tagging

Abstract

The system and method are described for enhancing location-based services by enabling spatial database systems to respond to or answer spatial queries that use a reference object to identify objects or features of interest in environmental scene before a system user. The system and method present invention enhances pointing technology by permitting system users to use queries to identify objects or features within the system user'"'"'s field of view by pointing at the reference object or feature, and linking it to the object of interest by using spatial prepositions.

Claims

1. A computer-implemented method for responding to spatial queries about a real-world object of interest using a reference object to identify the object of interest, comprising the steps of:

(A) generating a query statement relating to at least one object of interest with an electronic pointing device, with the query statement including at least a query topic and a spatial preposition, and generating a pointing device geodetic position and pointing direction, and transmitting the query statement and pointing device geodetic position and pointing direction to a system server;

(B) mapping a three-dimensional representation of a real-world scene including at least the object of interest and the reference object, and transmitting the three-dimensional representation to the system server;

(C) a system server performing the Substeps of,

(1) generating a two-dimensional digital representation of the three-dimensional scene mapped at Step (B), with the two-dimensional digital representation including at least a digital representation of the object of interest and the reference object,

(2) identifying the reference object and the reference object position according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation,

(3) annotating the two-dimensional digital representation for identifying the object of interest according to the position of the reference object, the query topic, and spatial preposition,

(4) identifying the object of interest according to be annotated two-dimensional digital representation at Substep (C)(3) by linking the query topic and spatial preposition of the query statement to an object in the two-dimensional digital representation, and

(5) the system server communicating with the pointing device and transmitting at least the identification of the object of interest to the pointing device.

2. The method as recited in claim 1, wherein the pointing device and system server communicate wired or wirelessly.

3. The method as recited in claim 1, wherein the pointing device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the pointing device.

4. The method as recited in claim 1, wherein the query statement includes the object of interest as a query topic.

5. The method as recited in claim 1, wherein the spatial preposition includes a spatial relationship between the object of interest and the reference object.

6. The method as recited in claim 5, wherein the spatial preposition identifies the object of interest according to an egocentric frame of reference according to the pointing device geodetic position.

7. The method as recited in claim 6, wherein the egocentric frame of reference identifies the object of interest according to the reference object position and pointing device geodetic position.

8. A computer-implemented method for responding to spatial queries about a real-world object of interest using a reference object to identify the object of interest, comprising the steps of:

(A) generating a query statement relating to at least one object of interest with an electronic pointing device, with the query statement including at least a query topic and a spatial preposition, and generating a pointing device geodetic position and pointing direction, and transmitting the query statement and pointing device geodetic position and pointing direction to a system server;

(B) mapping a three-dimensional representation of a real-world scene including at least the object of interest and the reference object, and transmitting the three-dimensional representation to the system server;

(C) a system server performing the Substeps of,

(1) generating a two-dimensional digital representation of the three-dimensional scene mapped at Step (B), with the two-dimensional digital representation including at least a digital representation of the object of interest and the reference object,

(2) identifying the reference object and the reference object position according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation,

(3) identifying the object of interest according to the position of the reference object, and the query topic and spatial preposition by linking the query topic and spatial preposition of the query statement to an object in the two-dimensional digital representation, and

(4) the system server communicating with the pointing device and transmitting at least the identification of the object of interest to the pointing device.

9. The method as recited in claim 8, wherein the pointing device and system server communicate wired or wirelessly.

10. The method as recited in claim 8, wherein the pointing device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the pointing device.

11. The method as recited in claim 8, wherein the query statement includes the object of interest as a query topic.

12. The method as recited in claim 8, wherein the spatial preposition includes a spatial relationship between the object of interest and the reference object.

13. The method as recited in claim 12, wherein the spatial preposition identifies the object of interest according to an absolute frame of reference applied to the reference object position.

14. The method as recited in claim 12, wherein the spatial preposition identifies the object of interest according to an intrinsic frame of reference applied to the reference object position.

15. A system for responding to spatial queries about a real-world object of interest using a reference object to identify the object of interest, comprising:

a pointing device for pointing at a reference object in a real-world scene and generating a query statement including a query topic and spatial preposition, relating to the object of interest according to a position of the reference object, with the pointing device determining a pointing device geodetic position and pointing direction, and with the pointing device communicating the query statement, and the pointing device geodetic position and pointing direction to a system server; and

a system server further comprising,

a mapping module for receiving and processing a three-dimensional representation of an environment that contains a real-world scene including the reference object and the object of interest,

a scene generator that connects to the mapping module and receives an output from the mapping module for generating a two-dimensional digital representation of the real-world scene including the reference object and object of interest, and identifies the reference object according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation,

a scene annotator for annotating the two-dimensional digital representation for identifying the object of interest according to the position of the reference object, and the query topic and spatial preposition of the query statement by linking the query topic and spatial preposition to an object in the two-dimensional digital representation, and communicating to the pointing device the identification of the object of interest.

16. The system as recited in claim 15, wherein the pointing device includes a mobile device.

17. The system as recited in claim 15, wherein the scene annotator includes using a egocentric frame of reference for identifying the object of interest.

18. The system as recited in claim 17, wherein the egocentric frame of reference identifies the object of interest according to the reference object position and pointing device geodetic position.

19. The system as recited in claim 15, wherein the pointing device and system server communicate wired and wirelessly.

20. A system for responding to spatial queries about a real-world object of interest using a reference object to identify the object of interest, comprising:

a pointing device for pointing at a reference object in a real-world scene and generating a query statement including a query topic and spatial preposition, the query statement relating to the object of interest according to a geodetic position of the reference object, with the pointing device determining a pointing device geodetic position and pointing direction, and with the pointing device communicating the query statement, and the pointing device geodetic position and pointing direction to a system server; and

a system server further comprising,

a mapping module for receiving and processing a three-dimensional representation of an environment that contains a real-world scene including the reference object and the object of interest,

a scene generator that connects to the mapping module and receives an output from the mapping module for generating a two-dimensional digital representation of the real-world scene including the reference object and object of interest, and identifies the reference object according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation, and

an identifier module for identifying the object of interest in the two-dimensional digital representation according to the position of the reference object and the query topic and spatial preposition of the query statement by linking the query topic and spatial preposition to an object in the two-dimensional digital representation, and communicating to the pointing device the identification of the object of interest.

21. The system as recited in claim 20, wherein the pointing device includes a mobile device.

22. The system as recited in claim 20, wherein the identifier module includes applying an absolute frame of reference for identifying the object of interest according to the reference object position.

23. The system as recited in claim 20, wherein the identifier module includes applying an intrinsic frame of reference for identifying the object of interest according to the reference object position.

24. The system as recited in claim 20, wherein the pointing device and system serve communicate wired and wirelessly.