System and Method for Initiating Actions and Providing Feedback by Pointing at Object of Interest

• US 20100303339A1
• Filed: 12/22/2009
• Published: 12/02/2010
• Est. Priority Date: 12/22/2008
• Status: Active Grant

• Current Assignee: IPOINTER, INC.
• Original Assignee: Intelligent Spatial Technologies, Inc.
• Inventors: CADUFF, David
• Granted Patent: US 8,675,912 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/185 by embedding added-value information into content, e.g. geo-tagging

Abstract

A system and method as described for compiling feedback in command statements that relate to applications or services associated with spatial objects or features, pointing at such spatial object or feature order to identify the object of interest, and executing the command statements on a system server and attaching feedback information to their representation of this object or feature in a database of the system server.

Claims

1. A computer-implemented method for identifying an object in a real-world scene by pointing at the object with an electronic pointing device and electronically attaching feedback to the object in a digital representation of the object stored in system storage, comprising the steps of:

(A) mapping a three-dimensional representation of the real-world scene containing at least one object and transmitting the three-dimensional representation to a system server;

(B) inputting feedback relating to the at least one object to the pointing device;

(C) inputting command statements respect to the least one object to the pointing device;

(D) generating a request at the pointing device that includes at least the feedback and command statements, and transmitting the request to the system server along with a pointing device geodetic position and pointing direction; and

(E) the system server performing the Substeps of,

(1) generating a two-dimensional digital representation of the three-dimensional scene mapped at Step (A), with the two-dimensional digital representation including at least a digital representation of the at least one object,

(2) identifying the at least one object according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation,

(3) processing the request and separating command statements and feedback from the request,

(4) if the request includes only at least the command statement, executing the at least command statement and then performing Substep (E)(7),

(5) if the request includes only feedback, linking the feedback to the digital representation of the at least one object stored in system storage and then performing Substep (E)(7),

(6) if the request includes at least one command statement and feedback, executing the at least one command statement and linking feedback to the digital representation of the at least one object stored in system storage and then performing Substep (E)(7),

(7) the system server communicating to the pointing device execution of at least one command statement and/or the linking of feedback.

2. The method as recited in claim 1, wherein the pointing device and system server communicate wired or wirelessly.

3. The method as recited in claim 1, wherein the pointing device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the pointing device.

4. The method as recited in claim 1, wherein the request includes context information.

5. The method as recited in claim 4, wherein the method further includes adding context information to feedback.

6. The method as recited in claim 1, wherein the system storage stores environmental information for generating two-dimensional representations of real-world scenes and objects contained therein according to the pointing device geodetic position and pointing direction.

7. The method as recited in claim 1, wherein command statements include commands for affecting a real-world state of the at least one object.

8. The method as recited in claim 5, wherein the method further includes retrieving feedback relating to the at least one object when the at least one object is identified as an object of interest.

9. The method as recited in claim 8, wherein retrieving feedback includes retrieving feedback linked to the least one object and stored in system storage.

10. The method as recited in claim 9, wherein retrieving feedback includes retrieving context information.

11. The method as recited in claim 1, wherein linking feedback includes linking feedback to a subpart of the at least one object.

12. The method as recited in claim 1, the pointing device includes a mobile device.

13. A computer-implemented method for identifying an object in a real-world scene by pointing at the object with an electronic pointing device and electronically attaching feedback information to the object in a digital representation of the object stored in system storage, comprising the steps of:

(A) mapping a three-dimensional representation of the real-world scene containing at least one object and transmitting the three-dimensional representation to the system server;

(B) inputting feedback relating to the at least one object to the pointing device;

(C) generating a request at the pointing device that includes at least the feedback input at Step (B) and transmitting the request to the system server along with a geodetic pointing device position and pointing direction; and

(D) the system server performing the Substeps of,

(1) generating a two-dimensional digital representation of the three-dimensional scene mapped at Step (A), with the two-dimensional digital representation including at least a digital representation of the at least one object,

(2) identifying the at least one object according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation,

(3) linking the feedback to the digital representation of the at least one object stored in system server, and

(4) the system server communicating to the pointing device the linking of feedback.

14. The method as recited in claim 13, wherein the pointing device and system server communicate wired or wirelessly.

15. The method as recited in claim 13, wherein the pointing device geodetic position includes a position according to a latitude, longitude, elevation, pitch, and yaw of the pointing device.

16. The method as recited in claim 13, wherein the request includes context information.

17. The method as recited in claim 16, wherein the method further includes adding context information to feedback.

18. The method as recited in claim 13, wherein the system storage stores environmental information for generating two-dimensional representations of real-world scenes and objects contained therein according to the pointing device geodetic position and pointing direction.

19. The method as recited in claim 17, wherein the method further includes retrieving feedback relating to the at least one object when the at least one object is identified as an object of interest.

20. The method as recited in claim 19, wherein retrieving feedback includes retrieving at least feedback linked to the least one object and stored in the system storage.

21. The method as recited in claim 20, wherein retrieving feedback includes retrieving context information.

22. The method as recited in claim 13, wherein linking feedback includes linking feedback to a subpart of the at least one object.

23. The method as recited in claim 13, the pointing device includes a mobile device.

24. A system for identifying and attaching feedback to an object in a real-world scene, comprising,

a pointing device for pointing at an object of interest in a real-world scene and having input therein feedback and/or command statements with respect to the object of interest, with the pointing device determining a pointing device geodetic position and pointing direction, and generating a request including feedback and/or command statements, and communicating the request and pointing device geodetic position and pointing direction to a system server;

the system server further comprising,

a mapping module for receiving and processing a three-dimensional representation that contains the real-world scene including the object of interest,

a scene generator that connects to the mapping module and receives an output from the mapping module for generating a two-dimensional digital representation of the real-world scene including the object of interest and identifies the object of interest according to the pointing device geodetic position and pointing direction applied to the two-dimensional digital representation,

a linker module for linking feedback from the request to the digital representation of the object of interest stored in system storage,

a command parser module for parsing command statements from the request,

a communications module for receiving requests, and the pointing device geodetic position and pointing direction from the pointing device, and transmitting at least retrieved feedback and command statement execution status to the pointing device; and

system storage in communication with the system server for storing feedback relating to objects, including the object of interest, and two-dimensional digital environment information relating to the real-world scene containing objects, including the object interest; and

an execution module in communications with the system server for executing command statements that affect a real-world state of the object of interest or subpart of the object of interest.

25. The system as recited in claim 24, wherein the pointing device includes a mobile device.

26. The system as recited in claim 24, wherein system storage includes system databases.

27. The system as recited in claim 24, wherein the execution module executes commands associated with appliances, applications, or services.

28. The system as recited in claim 24, wherein the system server and pointing device communicates wired or wirelessly.