Determining a position of a pointing device

• US 8,456,419 B2
• Filed: 04/18/2008
• Issued: 06/04/2013
• Est. Priority Date: 02/07/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): ABDIN, SHAHEDA A
• Application Number:
  US12/106,091
• Publication Number:
  US 20080192007A1
• Time in Patent Office: 1,873 Days
• Field of Search: 345/158, 345/157, 345/163, 345174-178, 345/156, 345/179, 340/825.72, 340/825.69, 178/18.09, 178/19.05, 463/37, 715/863, 715/856, 715/848
• US Class Current: 345/158
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1. A system for determining a location of a pointing device in three-dimensional space and a location to where the pointing device is directed, the system comprising:

one or more infrared light emitting diodes (LEDs);

a control system coupled to the one or more infrared LEDs, the control system capable of causing the one or more infrared LEDs to emit infrared light;

one or more cameras, capable of capturing one or more frames at substantially the same time as the one or more infrared LEDs are caused to emit the infrared light, at least one of the one or more frames recording an existence of an emission of the infrared light;

a processing system, capable of;

determining the location of the pointing device in three-dimensional space;

determining the location to where the pointing device is directed;

computing a similarity between an input sequence of sensor values output by the pointing device and at least one stored prototype sequence, each stored prototype sequence representing a sequence of sensor values that is generated if the pointing device is used to perform a unique gesture; and

determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold; and

a communication interface, capable of transmitting a message across a transmission medium to a receiving unit, the message including data that can be used to determine the location of the pointing device in three-dimensional space and the location to where the pointing device is directed, and data relating to gesture determination, which the receiving unit can use to manipulate an object displayed on a display screen.

2. The system of claim 1, wherein a portion of the system is coupled with the pointing device.

3. The system of claim 2, wherein the portion of the system is located at a front end of the pointing device.

4. The system of claim 1, wherein the processing system determines the location of the pointing device in three-dimensional space by identifying a high intensity area in one or more of the frames.

5. The system of claim 4, wherein the high intensity area comprises an image coordinate of a pixel having a highest intensity.

6. The system of claim 1, wherein a portion of the system is coupled with the pointing device and a portion of the system is coupled with the receiving unit.

7. The system of claim 1, wherein the processing system determines the location of the pointing device in three-dimensional space by applying a stereo imaging technique.

8. The system of claim 1, wherein the processing system determines the location to where the pointing device is directed using the location of the pointing device in three-dimensional space and an orientation of the pointing device.

9. The system of claim 1, wherein the transmission medium is a wireless transmission medium.

10. A method for determining a location of a pointing device in three-dimensional space, the method comprising:

emitting light from a light emitting diode (LED) in response to a control signal;

recognizing the emission of the light from the LED with an imaging device having a lens;

capturing a frame with the imaging device, wherein the frame includes a representation of the light emitted from the LED;

determining a location of the representation in the frame;

determining the location of the pointing device in three-dimensional space based upon the location of the representation in the frame;

computing a similarity between an input sequence of sensor values output by the pointing device and at least one stored prototype sequence, each stored prototype sequence representing a sequence of sensor values that is generated if the pointing device is used to perform a unique gesture; and

determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold.

11. The method of claim 10, further comprising utilizing infrared filters to filter the frame.

12. The method of claim 10, wherein the step of determining the location of the representation in the frame further comprises eliminating at least a portion of background infrared light in the frame.

13. The method of claim 10, wherein the step of determining the location of the representation in the frame further comprises:

determining a high intensity area in the frame; and

determining a coordinate position of the high intensity area.

14. The method of claim 10, further comprising determining an area to which the pointing device is directed based at least on the location of the pointing device in three-dimensional space.

15. The method of claim 14, further comprising implementing a training procedure associated with the area to which the pointing device is directed.

16. The method of claim 14, further comprising providing data to a receiving unit, the data being associated with the area to which the pointing device is directed to, the data being for manipulating a screen object at the area.

17. The method of claim 10, further comprising performing a stereo imaging technique.

18. A system for determining a location of a pointing device in three-dimensional space, comprising:

a light emitting diode (LED) in a first position, the LED capable of emitting infrared light;

a controller coupled to the LED capable of causing the LED to emit the infrared light for a pre-defined time period;

a camera capable of detecting the emitted infrared light; and

a processor capable of;

determining a first position of the emitted infrared light, wherein the first position corresponds with a first area of high intensity in an image, wherein the location of the pointing device in three-dimensional space is based at least in part on the first position;

determining an orientation of the pointing device;

computing a similarity between an input sequence of sensor values output by the pointing device and at least one stored prototype sequence, each stored prototype sequence representing a sequence of sensor values that is generated if the pointing device is used to perform a unique gesture; and

determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold.

19. The system of claim 18, wherein the camera captures the emitted infrared light contemporaneously with additional data.

20. The system of claim 18, wherein the processor is further capable of determining a display screen region to which the pointing device is directed using the location of the pointing device in three-dimensional space in conjunction with additional orientation related data.

21. The system of claim 18, wherein the processor utilizes a pre-defined coordinate system in conjunction with the image to establish a first coordinate position for the first area of high intensity.

22. The system of claim 21, wherein the processor utilizes triangulation calculations associated with at least the first coordinate position to determine the location of the pointing device in three-dimensional space.

23. The system of claim 21, wherein the processor utilizes stereo imaging techniques associated with at least the first coordinate position to determine the location of the pointing device in three-dimensional space.

24. The system of claim 18, wherein a portion of the system is connected to the pointing device.