MANIPULATING OBJECTS DISPLAYED ON A DISPLAY SCREEN

• US 20080192070A1
• Filed: 04/18/2008
• Published: 08/14/2008
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: WILSON, ANDREW
• Granted Patent: US 7,823,089 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1. One or more computer-readable storage media in a computing device, having computer-useable instructions embodied thereon for performing a method of determining a location where a pointing device is directed, the method comprising:

receiving a message at the computing device from the pointing device;

extracting sensor data from the message;

identifying a position of the pointing device in three-dimensional space;

identifying an orientation of the pointing device in three-dimensional space using the sensor data;

determining a location to which the pointing device is directed by utilizing the identified position of the pointing device and the identified orientation of the pointing device; and

altering an object on a display screen at the location where the pointing device is directed.

2. The one or more computer-readable storage media of claim 1, wherein the step of identifying a position further comprises receiving data at the computing device associated with one or more infrared lights detected by a camera.

3. The one or more computer-readable storage media of claim 1, wherein the step of identifying a position further comprises:

receiving data at the computing device associated with one or more infrared lights detected by a camera, wherein the data describes one or more images; and

analyzing the one or more images to identify the position.

4. The one or more computer-readable storage media of claim 1, wherein the sensor data includes data provided by an output of an accelerometer.

5. The one or more computer-readable storage media of claim 4, wherein the received message includes acceleration data and the step of extracting further comprises extracting the acceleration data from the received message.

6. The one or more computer-readable storage media of claim 5 further comprising receiving normalization data at the computing device prior to receiving the message.

7. The one or more computer-readable storage media of claim 6, wherein the step of receiving normalization data further comprises receiving calibration data at the computing device via an output of an accelerometer in the pointing device.

8. The one or more computer-readable storage media of claim 6, wherein the step of identifying an orientation further comprises identifying data associated with a pitch in the extracted sensor data of the message.

9. The one or more computer-readable storage media of claim 6, wherein the step of identifying an orientation further comprises identifying data associated with a roll in the extracted sensor data of the message.

10. The one or more computer-readable storage media of claim 6, wherein the step of identifying an orientation further comprises identifying data associated with a yaw in the extracted sensor data of the message.

11. The one or more computer-readable storage media of claim 6, wherein the step of identifying an orientation further comprises identifying data associated with a motion about a x-axis, a y-axis, and a z-axis with respect to a pre-defined coordinate system.

12. The one or more computer-readable storage media of claim 1 further comprising waiting to receive a subsequent message from the pointing device.

13. The one or more computer-readable storage media of claim 12 further comprising:

identifying a new orientation of the pointing device in three-dimensional space using new sensor data in the subsequent message; and

determining a change in orientation of the pointing device using the orientation and the new orientation.

14. One or more computer-readable storage media in a computerized base station, the computer-readable storage media having computer-useable instructions embodied thereon for performing a method of manipulating an object displayed on a display screen, the method comprising:

receiving a message at the computerized base station, the message including orientation data;

identifying acceleration data from the orientation data;

determining a motion of the pointing device in 3-dimensional space, based upon the acceleration data; and

manipulating the object displayed on the display screen in accordance with the motion of the pointing device.

15. The one or more computer-readable storage media of claim 14 further comprising:

receiving button data in the message from the pointing device; and

determining from the button data, if a button on the pointing device is selected.

16. The one or more computer-readable storage media of claim 14, wherein the step of manipulating further comprises manipulating the object displayed on the display screen in accordance with the motion of the pointing device and the selection of the button.

17. The one or more computer-readable storage media of claim 14 further comprising recording the orientation data in the computerized base station.

18. The one or more computer-readable storage media of claim 14 further comprising identifying if the orientation data exceeds one or more thresholds.

19. The one or more computer-readable storage media of claim 14, wherein the motion of the pointing device comprises an upward motion, a downward motion, a lateral motion, or a rotation motion.

20. The one or more computer-readable storage media of claim 19, wherein a rotation motion comprises rotating the pointing device about an axis.

21. The one or more computer-readable storage media of claim 14 further comprising receiving calibration data at the computerized base station via an output of an accelerometer in the pointing device prior to the step of receiving orientation data.

22. The one or more computer-readable storage media of claim 21, wherein the step of determining a motion uses the calibration data and the orientation data.

23. A device for manipulating an object displayed on a display screen comprising:

a transceiver, which receives a signal;

a demodulator, which demodulates the received signal;

a controller, which outputs a data message including acceleration data from the demodulated signal;

a memory, which stores the outputted data message; and

a processor, which accesses the memory to process the outputted data message, the processing including a determination of a motion of a pointing device in three-dimensional space based upon the acceleration data in the outputted data message, and wherein the processing further includes manipulating the object displayed on the display screen in accordance with the motion of the pointing device.

24. The device of claim 23, wherein the data message includes information which is output by the controller, the information indicating whether a button on the pointing device has been selected.

25. The device of claim 24, wherein the processor records the acceleration data in the outputted data message if the information indicates that the button on the pointing device has been selected.

26. The device of claim 25, wherein the memory further stores one or more thresholds and wherein the processor further processes the one or more thresholds and the acceleration data in the outputted data message.

27. The device of claim 25, wherein the memory further stores calibration data and wherein the processor further processes the calibration data and the acceleration data in the outputted data message in order to determine the motion of the pointing device in three-dimensional space.

28. The device of claim 27 further comprising instructions in the memory, which perform a calibration routine, the execution of the calibration routine by the processor configured to produce the calibration data that is stored in the memory.

29. The device of claim 23, wherein when the processor processes the outputted data message to determine a motion of a pointing device in three-dimensional space, the processing includes processing data associated with a motion about an x-axis, a y-axis, and a z-axis with respect to a pre-defined coordinate system.

30. A system for manipulating an object displayed on a display screen comprising:

a transceiver, which receives a signal;

one or more infrared lights that are detected by a camera in order to produce position data;

a demodulator, which demodulates the received signal;

a controller, which outputs a data message including acceleration data and the position data from the demodulated signal;

a memory, which stores the outputted data message; and

a processor, which accesses the memory to process the outputted data message, the processing including the determination of a motion of a pointing device in three-dimensional space based upon the acceleration data in the outputted data message and a position of the pointing device in three-dimensional space based upon the position data, and wherein the processing further includes manipulating the object displayed on the display screen in accordance with the motion and the position of the pointing device.