System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration

• US 20030149803A1
• Filed: 05/31/2002
• Published: 08/07/2003
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew
• Granted Patent: US 6,990,639 B2
• Time in Patent Office: Days
• US Class Current: 710/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

The present invention is directed toward a system and process that controls a group of networked electronic components using a multimodal integration scheme in which inputs from a speech recognition subsystem, gesture recognition subsystem employing a wireless pointing device and pointing analysis subsystem also employing the pointing device, are combined to determine what component a user wants to control and what control action is desired. In this multimodal integration scheme, the desired action concerning an electronic component is decomposed into a command and a referent pair. The referent can be identified using the pointing device to identify the component by pointing at the component or an object associated with it, by using speech recognition, or both. The command may be specified by pressing a button on the pointing device, by a gesture performed with the pointing device, by a speech recognition event, or by any combination of these inputs.

Claims

1. A multimodal electronic component control system comprising an object selection subsystem, a gesture recognition subsystem, and a speech control subsystem, each of which provides inputs to an integration subsystem that integrates said inputs to arrive at a unified interpretation of what component a user wants to control and what control action is desired.

2. The system of claim 1, wherein the integration subsystem comprises a dynamic Bayes network which determines from the individual inputs of the object selection subsystem, gesture recognition subsystem, and speech control subsystem, the identity of a component the user wants to control (i.e., the referent), a command that the user wishes to implement (i.e., the command), and the appropriate control action to be taken to affect the identified referent in view of the command.

3. The system of claim 2, wherein the dynamic Bayes network comprises input, referent, command and action nodes, wherein the input nodes include said individual inputs which provide information as to their state to at least one of a referent, command, or action node, said inputs determining the state of the referent and command nodes, and wherein the states of the referent and command nodes are fed into an action node whose state indicates the action that is to be implemented to affect the referent, and wherein said referent, command and action node states comprise probability distributions indicating the probability that each possible referent, command and action is the respective referent, command and action.

4. The system of claim 3, wherein the object selection subsystem is pointer based in that an electronic component is selected by a user pointing a pointing device at an object that corresponds to, or is associated with, the electronic component, and wherein the input nodes providing information to the referent node comprise a pointing target node which indicates to the referent node what electronic component is associated with the object that the user is currently pointing at.

5. The system of claim 3, wherein the speech control subsystem identifies an electronic component corresponding to a prescribed component name spoken by a user, and wherein the input nodes providing information to the referent node comprise a speech referent node which indicates to the referent node what electronic component the user currently identified by speaking the prescribed name for that component.

6. The system of claim 3, wherein the state of all nodes of the dynamic Bayes network are updated on a prescribed periodic basis to incorporate new information, and wherein the input nodes providing information to the referent node comprise a prior state node whose state reflects the referent node'"'"'s state in the update period immediately preceding a current period, and wherein the input from the prior state node is weighted such that its influence on the state of the referent node decreases in proportion to the amount of time that has past since the prior state node first acquired the state being input.

7. The system of claim 3, wherein the object selection subsystem is pointer based in that an electronic component is selected by a user pointing a pointing device at an object that corresponds to, or is associated with, the electronic component, and wherein the input nodes providing information to the command node comprise a button click node which indicates to the command node that a user-operated switch resident on the pointer of the object selection subsystem has been activated.

8. The system of claim 3, wherein the gesture recognition subsystem identifies a command indicative of a desired action for affecting an electronic component which corresponds to a prescribed gesture performed by the user with a pointing device, and wherein the input nodes providing information to the command node comprise a gesture node which indicates to the command mode what electronic component the user currently identified by performing the gesture associated with that that component.

9. The system of claim 3, wherein the speech control subsystem identifies a command indicative of a desired action for affecting an electronic component corresponding to a prescribed command word or phrase spoken by a user, and wherein the input nodes providing information to the command node comprise a speech command node which indicates to the command node what command the user currently identified by speaking the prescribed word or phrase for that command.

10. The system of claim 3, wherein the state of all nodes of the dynamic Bayes network are updated on a prescribed periodic basis to incorporate new information, and wherein the input nodes providing information to the command node comprise a prior state node whose state reflects the command node'"'"'s state in the update period immediately preceding a current period, and wherein the input from the prior state node is weighted such that its influence on the state of the command node decreases in proportion to the amount of time that has past since the prior state node first acquired the state being input.

11. The system of claim 3, wherein input nodes providing information to the action node comprise one or more device state nodes whose state is set to reflect the current condition of an electronic component associated with the node, and wherein each device state node provides information as to the current state of the associated component to the action node whenever the referent node probability distribution indicates the referent is the component associated with the device state node.

12. A computer-implemented multimodal electronic component control process comprising a pointer-based object selection process module, a gesture recognition process module, and a speech control process module, each of which provides inputs to an integration process module that integrates the inputs to arrive at a unified interpretation of what component a user wants to control and what control action is desired.

13. The process of claim 12, wherein integration process module is a dynamic Bayes network.

14. The process of claim 13, wherein the dynamic Bayes network determines from the individual inputs of the object selection module, gesture recognition module, and speech control module, the identity of a component the user wants to control (i.e., the referent), a command that the user wishes to implement (i.e., the command), and the appropriate control action to be taken to affect the identified referent in view of the command.

15. The process of claim 14, wherein the dynamic Bayes network has an process flow architecture comprising a series of input nodes including said individual inputs which provide information as to their state to at least one of a referent node, a command node and an action node, said inputs determining the state of the referent and command nodes, and wherein the state of the referent and command node is fed into an action node whose state is determined by the input from the referent, command and input nodes, and whose state indicates the action that is to be implemented to affect the referent, and wherein said referent, command and action node states comprise probability distributions indicating the probability that each possible referent, command and action is the respective referent, command and action.

16. The process of claim 15, wherein the state of all nodes of the Bayes network are updated on a prescribed periodic basis to incorporate new information, and wherein said input nodes further comprise nodes whose state reflects the same node'"'"'s state in the update period immediately preceding a current period, wherein the prior node state is input into the same node in the current period thereby allowing the prior states of the nodes to influence the node states in the current period so to preserve prior incomplete or ambiguous inputs until enough information is available to the network to determine the state of an undecided node.

17. The process of claim 16, wherein the input from an input node associated with a prior node state is weighted such that its influence on the state of the corresponding node in the current period decreases in proportion to the amount of time that has past since the node first acquired the state being input.

18. The process of claim 15, wherein said input nodes further comprise one or more device state nodes whose state is set to reflect the current condition of an electronic component associated with the node, and wherein each device state node provides information as to the current state of the associated component to the action node whenever the referent node probability distribution indicates the referent is the component associated with the device state node.

19. The process of claim 18, wherein the device state nodes are set to indicate whether the associated component is activated or deactivated.

20. The process of claim 19, wherein only two action node states are possible for a particular electronic component in that when the component is activated the only action is to deactivate it, and when the component is deactivated the only action is to activate it, and wherein whenever a device state node provides information as to the current state of the associated component as to whether the component is activated or deactivated to the action node, the action node is set to a state indicating the component is to be activated if it is currently deactivated or a state indicating the component is to be deactivated if it is currently activated, regardless of the state of the command node.

21. A computer-readable medium having computer-executable instructions for controlling electronic components using multimodal integration, said computer-executable instructions comprising:

accepting inputs from at least one of a pointer-based object selection process module, a gesture recognition process module, and a speech control process module; and

integrating said inputs to arrive at a unified interpretation of what electronic component a user wants to control and what control action is desired.

22. The computer-readable medium of claim 21, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the object selection process module an indication as to what electronic component a user wants to affect via a network connection to a host computer based on what object a pointer is being pointed at, wherein the object corresponds to, or is associated with, the indicated electronic component.

23. The computer-readable medium of claim 21, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the object selection process module an indication as to whether a user-activated switch on the pointer has been activated.

24. The computer-readable medium of claim 21, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the gesture recognition process module an indication of what command the user wants implemented based on a gesture performed by the user.

25. The computer-readable medium of claim 21, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the speech control process module an indication as to what electronic component a user wants to affect via a network connection to a host computer based on a word or phase spoken by the user.

26. The computer-readable medium of claim 21, wherein the instruction for accepting inputs, comprises a sub-instruction for inputting from the speech control process module an indication as to what command the user wants implemented based on a word or phase spoken by the user.

27. A computer-implemented process for controlling a user-selected electronic component within an environment using a pointing device, comprising using a computer to perform the following process actions:

computing the similarity between an input sequence of sensor values output by the pointing device and recorded over a prescribed period of time and at least one stored prototype sequence, wherein each prototype sequence represents the sequence of said sensor values that are generated if the user performs a unique gesture representing a different control action for the selected electronic component using the pointing device;

determining if the computed similarity between the input sequence and any prototype sequence exceeds a prescribed similarity threshold; and

whenever it is determined that one of the computed similarities exceeds the similarity threshold, implementing a command represented by the gesture.

28. The process of claim 27, wherein the prescribed period of time in which the sensor values output by the pointing device are recorded is long enough to ensure that characteristics evidenced by the sensor values which distinguish each gesture from any other gesture representing a different control action are captured in the input sequence.

29. The process of claim 28, wherein the prescribed period of time in which the sensor values output by the pointing device are recorded is within a range of approximately 1 second to approximately 2 seconds.

30. The process of claim 27, the process action of computing the similarity between the sequence of sensor values output by the pointing device and recorded over the prescribed period of time and a prototype sequence, comprises an action of computing the similarity using a squared Euclidean distance technique.

31. The process of claim 27, wherein the prescribed similarity threshold is large enough to ensure that characteristics in a prototype sequence as the evidenced by the sensor values thereof that distinguish the gesture associated with the prototype sequence from any other gesture represented by a different prototype sequence, exist in the input sequence, thereby causing the similarity computed between the input sequence and the prototype sequence to exceed the threshold, and causing the similarity computed between the input sequence and any other prototype sequence associated with a different gesture to not exceed the threshold.

32. The process of claim 27, wherein the process is performed continuously for each consecutive portion of the inputted sequence of sensor values output by the pointing device having the prescribed period of time for as long as the electronic component remains selected.

33. The process of claim 27, wherein each gesture is a series of movements of the pointing device in arbitrary directions, and wherein the sensor output or outputs recorded are those that are sure to characterize the motion of the pointing device associated with the gesture.

34. The process of claim 33, wherein the pointing device sensors comprise a magnetometer which has outputs that characterize the movement of the pointer in pitch, roll and yaw, and are used exclusively as the sensor outputs that are recorded.

35. The process of claim 33, wherein the pointing device sensors comprise an accelerometer which has outputs that characterize the movement of the pointer in pitch and roll only, and a gyroscope which has outputs that characterize the movement of the pointer in yaw only, and the combination of the outputs from these two sensors are used as the sensor outputs that are recorded.

36. The process of claim 35, wherein the pointing device sensors comprise a magnetometer which has outputs that characterize the movement of the pointer in pitch, roll and yaw, and wherein the outputs of the magnetometer are used in conjunction with the outputs of the accelerometer and gyroscope as the sensor outputs that are recorded.

37. A system for controlling a user-selected electronic component within an environment using a pointing device, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

input orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device,

generate and store at least one prototype sequence in a training phase, each prototype sequence comprising prescribed one or ones of the orientation readings that are generated by the pointing device during the time a user moves the pointing device in a gesture representing a particular command for controlling the selected electronic component, wherein each prototype sequence is associated with a different gesture and a different command,

record said prescribed one or ones of the orientation sensor readings from each orientation message inputted for a prescribed period of time to create an input sequence of sensor readings,

for each stored prototype sequence, compute a similarity indicator between the input sequence and each prototype sequence under consideration, wherein the similarity indicator is a measure the similarity between sequences,

identify the largest of the computed similarity indicators,

determine if identified largest computed similarity indicator exceeds a prescribed similarity threshold,

whenever the identified similarity indicator exceeds the prescribed similarity threshold, designate that the user has performed the gesture corresponding to the prototype sequence associated with that similarity indicator, and

implement the command represented by the gesture that the user was designated to have performed.

38. The system of claim 37, wherein the program module for generating and storing at least one prototype sequence in a training phase, comprises, for each prototype sequence generated, the sub-modules for:

(a) inputting a user-specified identity of the electronic component that the gesture associated with the prototype sequence applies to;

(b) inputting a user-specified control action that is to be taken in regard to the identified electronic component in response to the user performing the gesture;

(c) causing periodic requests for orientation messages to be sent to the pointing device;

(d) waiting for an orientation message to be received;

(e) whenever an orientation message is received, determining whether the switch state indicates that the pointing device'"'"'s switch has been activated by the user to indicate that the gesture is being performed;

(f) whenever it is determined that the switch state does not indicate that the pointing devices switch is activated, repeat actions (d) and (e);

(g) whenever it is determined that the switch state does indicate that the pointing devices switch is activated, record a prescribed one or ones of the pointing device sensor outputs taken from the last received orientation message;

(h) waiting for the next orientation message to be received;

(i) whenever an orientation message is received, determining whether the switch state indicates that the pointing devices switch is still activated, and if so repeating actions (g) and (h);

(j) whenever it is determined that the switch state no longer indicates that the pointing device'"'"'s switch is activated thereby indicating that the user has completed performing the gesture, designating the recorded sensor outputs in the order recorded as the prototype sequence; and

(k) storing the prototype sequence.

39. A computer-readable medium having computer-executable instructions for controlling a user-selected electronic component within an environment using a pointing device, said computer-executable instructions comprising:

inputting orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device;

generating and storing at least one prototype sequence in a training phase, each prototype sequence comprising a prescribed one or ones of the orientation readings that are generated by the pointing device during the time a user moves the pointing device in a gesture representing a particular command for controlling the selected electronic component, and wherein each prototype sequence is associated with a different gesture and represents a different command;

recording said prescribed one or ones of the orientation sensor readings from each orientation message inputted for a prescribed period of time to create an input sequence of sensor readings;

for each stored prototype sequence,

computing a match score between correspondingly ordered time steps of the input sequence and each of a plurality of versions of the prototype sequence, wherein each time step refers to the recording of said prescribed one or ones of the orientation sensor readings from a single orientation message, and wherein said match score is a measure the similarity between time steps, and wherein each of the plurality of versions of the prototype sequence are formed by applying a different scaling factor and warping factor to the prototype sequence from a prescribed set of scaling factors for scaling the prototype sequence in amplitude and a prescribed set of warping factors for warping the prototype sequence in time,

respectively averaging the match scores computed between correspondingly ordered time steps of the input sequence and each of a plurality of versions of the prototype sequence to produce an averaged match score for each,

identifying the largest of the averaged match scores, and

designating the largest of the averaged match scores as a similarity indicator for the prototype sequence under consideration;

identifying the largest of the computed similarity indicators;

determining if the identified largest similarity indicator exceed a prescribed similarity threshold;

whenever the identified similarity indicator exceeds the prescribed similarity threshold, designating that the user has performed the gesture corresponding to the prototype sequence associated with the similarity indicator exceeding the prescribed similarity threshold; and

implementing the command represented by the gesture that the user was designated to have performed.

40. The computer-readable medium of claim 39, wherein the instruction for generating and storing at least one prototype sequence in a training phase, comprises, for each prototype sequence generated, the sub-instructions for:

(a) inputting a user-specified identity of the electronic component that the gesture associated with the prototype sequence applies to;

(b) inputting a user-specified control action that is to be taken in regard to the identified electronic component in response to the user performing the gesture;

(c) causing periodic requests for orientation messages to be sent to the pointing device;

(d) waiting for an orientation message to be received;

(e) whenever an orientation message is received, determining whether the switch state indicates that the pointing device'"'"'s switch has been activated by the user to indicate that the gesture is being performed;

(f) whenever it is determined that the switch state does not indicate that the pointing devices switch is activated, repeat sub-instructions (d) and (e);

(g) whenever it is determined that the switch state does indicate that the pointing devices switch is activated, record a prescribed one or ones of the pointing device sensor outputs taken from the last received orientation message;

(h) waiting for the next orientation message to be received;

(i) whenever an orientation message is received, determining whether the switch state indicates that the pointing devices switch is still activated, and if so repeating sub-instructions (g) and (h);

(j) whenever it is determined that the switch state no longer indicates that the pointing device'"'"'s switch is activated thereby indicating that the user has completed performing the gesture, designating the recorded sensor outputs in the order recorded as the prototype sequence;

(k) storing the prototype sequences.

41. A computer-implemented process for controlling a user-selected electronic component within an environment using a pointing device, comprising using a computer to perform the following process actions:

inputting orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device;

recording a prescribed one or ones of the pointing device sensor outputs taken from an orientation message inputted whenever a user indicates performing a gesture representing a control action for the selected electronic component;

for each gesture threshold definition assigned to the selected electronic component, determining whether the threshold if just one, or all the thresholds if more than one, of the gesture threshold definition under consideration are exceeded by the recorded sensor output associated with the same sensor output as the threshold;

whenever it is determined that the threshold if just one, or all the thresholds if more than one, of one of the gesture threshold definitions are exceeded by the recorded sensor output associated with the same sensor output, designating that the user has performed the gesture associated with that gesture threshold definition; and

implementing the command represented by the gesture that the user was designated to have performed.

42. The process of claim 41, wherein the process actions of recording the prescribed one or ones of the pointing device sensor outputs, comprises the actions of:

(a) waiting for an orientation message to be received;

(b) whenever an orientation message is received, determining whether the switch state indicates that the pointing device'"'"'s switch has been activated by the user to indicate that a control gesture is being performed;

(c) whenever it is determined that the switch state does not indicate that the pointing devices switch is activated, repeat actions (a) and (b);

(d) whenever it is determined that the switch state does indicate that the pointing devices switch is activated, recording the prescribed one or ones of the pointing device sensor outputs taken from the last received orientation message.

43. The process of claim 41, wherein each gesture is a movement of the pointer in a single prescribed direction, and wherein the sensor output or outputs included in each gesture threshold definition and so the recorded sensor outputs are those that characterize the motion of the pointing device in the prescribed direction associated with the gesture.

44. The process of claim 43, wherein the pointing device sensors comprise a magnetometer which has outputs that characterize the movement of the pointer in pitch, roll and yaw.

45. The process of claim 43, wherein the pointing device sensors comprise an accelerometer which has outputs that characterize the movement of the pointer in pitch and roll.

46. The process of claim 43, wherein the pointing device sensors comprise a gyroscope which has outputs that characterize the movement of the pointer in yaw only.

47. The process of claim 41, wherein prior to performing the process action of determining if the threshold or thresholds of each gesture threshold definition assigned to the selected electronic component are exceeded, performing a process action of establishing for each gesture representing a control action for the selected electronic component a gesture threshold definition, each of said gesture threshold definitions comprising either (i) a prescribed threshold applicable to a particular single sensor output or (ii) a set of thresholds applicable to a particular group of sensor outputs, which are indicative of the pointing device being moved in a particular direction from a starting point wherein the pointing device is pointed at the object in the environment corresponding to or associated with the selected electronic component.

48. A computer-readable medium having computer-executable instructions for controlling a user-selected electronic component within an environment using a pointing device, said computer-executable instructions comprising:

(a) inputting orientation messages transmitted by the pointing device, said orientation messages comprising orientation sensor readings generated by orientation sensors of the pointing device;

(b) waiting for an orientation message to be received;

(c) whenever an orientation message is received, determining whether the switch state indicates that the pointing device'"'"'s switch has been activated by the user to indicate that a control gesture is being performed;

(d) whenever it is determined that the switch state does not indicate that the pointing devices switch is activated, repeat actions (b) and (c);

(e) whenever it is determined that the switch state does indicate that the pointing devices switch is activated, recording a prescribed one or ones of the pointing device sensor outputs taken from the last received orientation message;

(k) for each gesture threshold definition assigned to the selected electronic component, determining if the threshold if just one, or all the thresholds if more than one, of the gesture threshold definition under consideration are exceeded by the recorded sensor output associated with the same sensor output as the threshold;

(l) whenever it is determined that the threshold if just one, or all the thresholds if more than one, of one of the gesture threshold definitions are exceeded by the recorded sensor output associated with the same sensor output, designating that the user has performed the gesture associated with that gesture threshold definition; and

(m) implementing the command represented by the gesture that the user was designated to have performed.

49. A system for controlling electronic components within an environment using a pointing device, comprising:

a pointing device comprising a transceiver and orientation sensors, wherein the outputs of the sensors are periodically packaged as orientation messages and transmitted using the transceiver;

a base station comprising a transceiver which receives orientation messages transmitted by the pointing device;

a pair of imaging devices each of which is located so as to capture images of the environment from different viewpoints;

a computing device which is in communication with the base station and the imaging devices so as to receive orientation messages forwarded to it by the base station and images captured by the imaging devices, and which,

computes the orientation and location of the pointer from the received orientation message and captured images,

determines if the pointing device is pointing at an object in the environment which corresponds to, or is associated with, an electronic component that is controllable by the computing device using the orientation and location of the pointing device, and if so selects the electronic component,

affects the selected electronic component in accordance with a command received from the pointing device.

50. The system of claim 49, wherein the pointing device further comprises a manually-operated switch whose state with regard to whether it is activated or deactivated at the time an orientation message is packaged for transmission is included in that orientation message, and wherein said command from the pointing device comprises a switch state which indicates that the switch is activated.

51. The system of claim 50, wherein the selected electronic component can be activated or deactivated by the computing device, and wherein a switch state indicating that the pointing device'"'"'s switch is activated received in an orientation message transmitted by the pointing device after the electronic component is selected is interpreted by the computing device as a command to one of (i) activate the selected component if the component is deactivated, or (ii) deactivate the component if the component is activated.

52. The system of claim 49, wherein the pointing device further comprises a pair of visible spectrum light emitting diodes (LEDs) which are visible when lit from outside of the pointer, said visible spectrum LEDs being employ to provide status or feedback information to the user.

53. The system of claim 52, wherein the computing device instructs the pointing device to light one or both of the visible spectrum LEDs by causing the base station to transmit an command to the pointing device instructing it to light one or both of the visible spectrum LEDs.

54. The system of claim 53, wherein the computing device instructs the pointing device to light one of the visible spectrum LEDs whenever the pointer is pointing at an object in the environment which corresponds to, or is associated with, an electronic component that is controllable by the computing device.

55. The system of claim 53, wherein the computing device instructs the pointing device to light a first one of the visible spectrum LEDs whenever the pointer is pointing at an object in the environment which corresponds to, or is associated with, an electronic component that is activated, and to light the other visible spectrum LEDs whenever the pointer is pointing at an object in the environment which corresponds to, or is associated with, an electronic component that is deactivated.

56. The system of claim 53, wherein the computing device instructs the pointing device to light one of the visible spectrum LEDs whenever the pointer is pointing at an object in the environment which corresponds to, or is associated with, an audio device that is controllable by the computing device, and to vary the intensity of that LED in proportion to the volume setting of the audio device.

57. The system of claim 53, wherein the computing device emits a user-audible sound whenever the pointer is pointing at an object in the environment which corresponds to, or is associated with, an electronic component that is controllable by the computing device.