Pointing device and cursor for use in intelligent computing environments

• US 7,038,661 B2
• Filed: 06/13/2003
• Issued: 05/02/2006
• Est. Priority Date: 06/13/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew, Pham, Hubert
• Primary Examiner(s): Chang, Kent
• Assistant Examiner(s): MARESCA, JOSEPH ANDREW
• Application Number:
  US10/461,646
• Publication Number:
  US 20040252102A1
• Time in Patent Office: 1,054 Days
• Field of Search: 345/157, 345/158
• US Class Current: 345/158
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G09G 5/08 Cursor circuits

Abstract

A system and process for directing a laser beam within a space is presented. The system includes a pointing device which periodically outputs orientation data indicative of the direction it is pointing and a cursor device which projects a laser beam. The orientation data is used to compute the direction the pointing device is pointing in terms of yaw and pitch angles. The laser beam is directed to locations in the space relative to the amount of movement of the pointing device. In an absolute mode, the pointed device and the laser beam of the cursor device are pointed at the same location in the space, whereas in a relative mode, the pointing device does not point at the same location as the laser beam. The position of the laser can be used to select a variety of hardware devices known to be in a room for future control actions.

Claims

1. A computer-implemented process for selecting an object within a space by a user causing a laser beam to be shone on the object, comprising using a computer to perform the following process actions:

directing a laser beam generated by a cursor device to shine on said object, wherein the cursor device comprises a laser which when activated projects said laser beam into the space and a pair of servos connected to laser, wherein the first of the servos is configured so as to move the laser in a manner that controls the yaw direction of the laser beam and the other of the servos is configured so as to move the laser in a manner that controls the pitch direction of the laser beam;

determining whether the laser beam is being shone on the object, said determination comprising,

modeling the location of objects within the space as the particular pitch and yaw angles associated with the laser that cause the laser beam to shine on approximately the center of the object being modeled,

modeling the extent of said objects as being a circle which is centered on the location representing said approximate center of the object being modeled; and

whenever the laser beam is being shone on the object, selecting that object for future control actions.

2. The process of claim 1, wherein the process action of directing the laser beam to shine on said object, comprises the actions of:

inputting orientation data from a pointing device which periodically outputs said data, wherein the orientation data is indicative of the direction the pointing device is pointed;

computing the direction the pointing device is pointing from the received orientation data; and

directing the laser beam to shine on said object based on the direction the pointing device is pointing.

3. The process of claim 2, wherein the pointing device comprises a manually-operated switch which is activated and deactivated by the user for the purpose of instructing the computing device to implement a function, and wherein the state of the switch in regard to whether it is activated or deactivated at the time orientation data is output is included in that data, and wherein the process action of modeling the location of objects within the space, comprises, for each object, the actions of:

initiating a training mode in response to a user-specified command to do so;

inputting the periodically output orientation data from the pointing device;

determining if the pointing device'"'"'s switch has been activated by the user each time the orientation data is input, wherein said user activates said switch whenever the cursor device'"'"'s laser beam is shining on approximately the center of the an object being modeled; and

whenever the pointing device'"'"'s switch has been activated,

recording the direction the cursor device'"'"'s laser is pointing in terms of its pitch and yaw angle,

inputting user-entered information that identifies the object being modeled, and

associating the recorded laser direction with the identified object.

4. The process of claim 3, wherein the process action of modeling the extent of objects, comprises, for each object, the actions of:

inputting user-entered information that specifies the size of an object being modeled;

computing an extent for said object in terms of the spherical coordinates of a circle centered about said recorded direction the cursor device'"'"'s laser; and

associating the computed extent with the identified object.

5. The process of claim 4, wherein the process action of determining whether the laser beam is being shone on an object, further comprises the actions of:

initiating an object selection mode in response to a user-specified command to do so;

determining if the pointing device'"'"'s switch has been activated by the user each time the orientation data is input, wherein said user activates said switch whenever the cursor device'"'"'s laser beam is shining on an object in the space the user wants to select; and

whenever the pointing device'"'"'s switch has been activated,

computing the distance between the current pitch and yaw angles of the laser and the pitch and yaw angle associated with each object modeled for the space, and

for each of the modeled objects,

determining if the computed distance is less than a radius of said circle used to model the extent of the object, and

designating that the laser beam is being shone on an object whenever the computed distance is found to be less than a radius of the circle used to model the extent of the object.

6. A computer-implemented process for selecting an object within a space by a user causing a laser beam to be shone on the object, comprising using a computer to perform the following process actions:

directing a laser beam generated by a cursor device to shine on said object, wherein the cursor device comprises a laser which when activated projects said laser beam into the space and a pair of servos connected to laser, wherein the first of the servos is configured so as to move the laser in a manner that controls the yaw direction of the laser beam and the other of the servos is configured so as to move the laser in a manner that controls the pitch direction of the laser beam;

determining whether the laser beam is being shone on the object, said determination comprising modeling objects within the space as polygons, wherein the user shines the laser beam on points associated with an object being modeled to identify them as vertices of the polygon; and

whenever the laser beam is being shone on the object, selecting that object for future control actions.

7. The process of claim 6, wherein the process action of shining the laser beam on a vertex point, comprises the actions of:

inputting orientation data from a pointing device which periodically outputs said data, wherein the orientation data is indicative of the direction the pointing device is pointed;

computing the direction the pointing device is pointing from the received orientation data; and

directing the laser beam to shine on the vertex point based on the direction the pointing device is pointing.

8. The process of claim 7, wherein the pointing device comprises a manually-operated switch which is activated and deactivated by the user for the purpose of instructing the computing device to implement a function, and wherein the state of the switch in regard to whether it is activated or deactivated at the time orientation data is output is included in that data, and wherein the process action of modeling objects within the space as polygons, comprises, for each object, the actions of:

initiating a training mode in response to a user-specified command to do so;

inputting the periodically output orientation data from the pointing device;

determining if the pointing device'"'"'s switch has been activated by the user each time the orientation data is input, wherein said user activates said switch whenever the cursor device'"'"'s laser beam is shining on a vertex of the polygon that is being identified to represent the object; and

whenever the pointing device'"'"'s switch has been activated,

recording the direction the cursor device'"'"'s laser is pointing in terms of its pitch and yaw angle,

associating the recorded laser direction with one of the vertices of the object being modeled,

repeating the inputting, determining, recording and associating actions until all the vertices of the polygon representing the object are recorded.

9. The process of claim 8, wherein the process action of modeling objects further comprises an action of inputting user-entered information that identifies the object being modeled and associating it with the recorded polygon vertices representing the object.

10. The process of claim 8, wherein the process action of determining whether the laser beam is being shone on an object, further comprises the actions of:

initiating an object selection mode in response to a user-specified command to do so;

determining if the pointing device'"'"'s switch has been activated by the user each time the orientation data is input, wherein said user activates said switch whenever the cursor device'"'"'s laser beam is shining on an object in the space the user wants to select; and

whenever the pointing device'"'"'s switch has been activated, employing a point-in-polygon technique to determine if the laser beam is shining on one of the objects modeled with a polygon and if so which one.

11. The process of claim 8, further comprising an action of determining the point on a polygon representing an object that the laser beam is shining on using a transform that maps the pitch and yaw angle coordinates of the laser to orthogonal coordinates of the polygon.

12. The process of claim 11, wherein the polygon representing an object is a rectangle having four recorded vertices, and wherein the process action of determining the point on a polygon representing an object that the laser beam is shining on using a transform that maps the pitch and yaw angle coordinates of the laser to rectangular coordinates of the polygon, comprises employing a projective transform defined by $\left[\begin{array}{c}\mathrm{wx}\\ \mathrm{wy}\\ w\end{array}\right]=\left[\begin{array}{ccc}{p}_{11}& p_{12}& p_{13}\\ p_{21}& p_{22}& p_{23}\\ p_{31}& p_{32}& 1\end{array}\right]$

[θ

cϕ

c1]

where x and y are the rectangular coordinates of the polygon, θ

_c and φ

_c are the yaw and pitch angles of the laser, and p_ij are projective parameters established by solving a linear system of equations given the pitch and yaw coordinates of the vertices of the rectangular polygon and the rectangular coordinates these vertices.

13. A computer-implemented process for selecting an object within a space by a user causing a laser beam to be shone on the object, comprising using a computer to perform the following process actions:

directing a laser beam generated by a cursor device comprising a laser which is capable of projecting a laser beam into the space to shine on said object or whenever a user directs the laser beam to point at a location within the space that is within a prescribed threshold distance from a modeled object, automatically redirecting the laser beam to shine on the object;

determining whether the laser beam is being shone on the object; and

whenever the laser beam is being shone on the object, selecting that object for future control actions.

14. A system for directing a laser beam within a space to act as a cursor, comprising:

a pointing device which periodically outputs orientation data indicative of the direction it is pointing;

a cursor device which comprises a laser which when activated projects said laser beam into the space and a pair of servos connected to laser, wherein a first of the servos is configured so as to move the laser in a manner that controls the yaw direction of the laser beam and the other of the servos is configured so as to move the laser in a manner that controls the pitch direction of the laser beam;

a general purpose computing device which is in communication with the pointing device and the cursor device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

receive the orientation data output by the pointing device and compute the direction the pointing device is pointing from the received orientation data in terms of yaw and pitch angles, and

direct the laser beam generated by a cursor device to a particular location in the space as determined by the direction the pointing device is pointing, such that the laser beam is directed in an absolute pointing mode wherein the location in the space where the laser beam is pointed is substantially the same location that the pointing device is pointed, said directing in the absolute pointing mode comprising,

computing a set of offset angles for the laser of the cursor device comprising respective yaw and pitch angles that define the angular distance between the origin of the spherical coordinate system associated with the cursor device and a prescribed origin of the spherical coordinate system for the space,

computing a set of offset angles for the pointing device comprising respective yaw and pitch angles that define the angular distance between the origin of the spherical coordinate system associated with the pointing device and a prescribed origin of the spherical coordinate system for the space,

computing aligning pitch and yaw angles that define how far the cursor device must move the laser in order to point the laser beam at approximately the same point in the space that the pointing device is pointing at, wherein the aligning pitch angle is defined as the sum of the offset pitch angle of the laser and the computed pitch angle of the pointing device less its offset pitch angle, and wherein the aligning yaw angle is defined as the sum of the offset yaw angle of the laser and the computed yaw angle of the pointing device less its offset yaw angle, and

directing the laser to move to the computed aligning pitch and yaw angles so as to point the laser beam at approximately the same location in the space that the pointing device is pointing to.

15. The system of claim 14, wherein said sets of offset angles comprise initial angle values which are valid for only a part of the space such that when the pointing device is pointed outside the part of the space where the initial offset values are valid the correspondence between the location the pointing device is pointing and the location the laser beam is shining on is lost, and wherein the sub-module for directing the laser beam in an absolute pointing mode comprises a sub-module for maintaining said correspondence between the pointing device and the laser beam.

16. The system of claim 15, wherein the pointing device comprises a manually-operated switch which is activated and deactivated by a user for the purpose of instructing the computing device to implement a function, and wherein the state of the switch in regard to whether it is activated or deactivated at the time orientation data is output is included in that data, wherein the sub-module for maintaining the correspondence between the pointing device and the laser beam, comprises sub-modules for:

determining if the pointing device'"'"'s switch has been activated by the user each time the orientation data is received, wherein said user activates said switch whenever the correspondence between the pointing device and the laser beam is to be re-established;

whenever the pointing device'"'"'s switch has been activated,

ceasing to provide new movement commands to the cursor device so as to stop the motion of the laser beam,

determining if the pointing device'"'"'s switch has been first deactivated by the user and then reactivated using the received orientation data, wherein said user reactivates said switch whenever the pointing device has been pointed at the location on which the stationary laser beam is shining;

whenever the pointing device'"'"'s switch has been reactivated,

recomputing said sets of offset angles,

reinitiating the sending of new movement commands to the cursor device.

17. The system of claim 15, wherein the sub-module for maintaining the correspondence between the pointing device and the laser beam, comprises sub-modules for:

inputting the orthogonal coordinates of the vertices of polygonal surfaces in the space, the orthogonal coordinates of the cursor device, and the orthogonal coordinates of the pointing device in terms of an orthogonal coordinate system of the space;

for a polygon in the space which the orthogonal coordinates of its vertices have been input and at which the pointing device is pointed, determining the location of point on the polygon that the pointing device is pointed at in terms of its orthogonal coordinates in the space;

computing pitch and yaw angles in terms of the spherical coordinate system of the cursor device associated with a line connecting the orthogonal coordinates of said polygon point and the orthogonal coordinates of the cursor device; and

directing the laser to move to the computed pitch and yaw angles so as to point the laser beam at approximately the same location in the space that the pointing device is pointing to.

18. The system of claim 17, wherein the sub-module for inputting the orthogonal coordinates of the pointing device, comprises sub-modules for inputting approximated coordinates of the pointing device.

19. The system of claim 18, wherein the approximated coordinates of the pointing device correspond to a location in the space where the pointing device usually resides.

20. The system of claim 15, wherein the sub-module for maintaining the correspondence between the pointing device and the laser beam, comprises sub-modules for:

inputting the orthogonal coordinates of the vertices of polygonal surfaces in the space and the orthogonal coordinates of the cursor device in terms of a orthogonal coordinate system of the space;

computing the orthogonal coordinates of the pointing device;

for a polygon in the space which the orthogonal coordinates of its vertices have been input and at which the pointing device is pointed, determining the location of point on the polygon that the pointing device is pointed at in terms of its orthogonal coordinates in the space in view of the orthogonal coordinates of the pointing device and the previously computed direction it is pointing;

computing pitch and yaw angles in terms of the spherical coordinate system of the cursor device associated with a line connecting the orthogonal coordinates of said polygon point and the orthogonal coordinates of the cursor device; and

directing the laser to move to the computed pitch and yaw angles so as to point the laser beam at approximately the same location in the space that the pointing device is pointing to.

21. The system of claim 20, wherein the pointing device comprises a manually-operated switch which is activated and deactivated by a user for the purpose of instructing the computing device to implement a function, and wherein the state of the switch in regard to whether it is activated or deactivated at the time orientation data is output is included in that data, wherein the sub-module for computing the orthogonal coordinates of the pointing device, comprises sub-modules for:

(a) determining if the user has performed a clutching operation;

(b) whenever a clutching operation has been performed,

for a polygon in the space which the orthogonal coordinates of its vertices have been input and at which the laser beam of the cursor device is shining on as a result of the clutching operation, determining the location of the point on the polygon where the laser beam is shining in terms of its orthogonal coordinates in the space and in view of the orthogonal coordinates of the pointing device and the previously computed direction it is pointing,

recording the location of said point on the polygon where the laser beam is shining as well as the previously computed direction that the pointing device is pointing at the end of the clutching operation; and

(c) determining if a defined number of clutching operation have been performed,

whenever the defined number of clutching operations have not been performed, repeating actions (a) and (b),

whenever the defined number of clutching operations have been performed, solving a linear system of equation for the orthogonal coordinates of the pointing device in the space using a least squares approach, wherein each equation in the system of equations is associated with one of the clutching operations and is defined as the sum of the unknown orthogonal coordinates of the pointing device and the recorded direction that the pointing device was pointing at the end of the clutching operation under consideration multiplied by a scaling factor, all of which is set equal to the orthogonal coordinates of the recorded location of said point on the polygon where the laser beam was shining at the end of the clutching operation under consideration;

wherein

(d) a clutching operation comprises the actions of,

determining if the pointing device'"'"'s switch has been activated by the user each time the orientation data is received, wherein said user activates said switch whenever the correspondence between the pointing device and the laser beam is to be re-established;

whenever the pointing device'"'"'s switch has been activated,

ceasing to provide new movement commands to the cursor device so as to stop the motion of the laser beam,

determining if the pointing device'"'"'s switch has been first deactivated by the user and then reactivated using the received orientation data, wherein said user reactivates said switch whenever the pointing device has been pointed at the location on which the stationary laser beam is shining;

whenever the pointing device'"'"'s switch has been reactivated,

recomputing said sets of offset angles,

reinitiating the sending of new movement commands to the cursor device.

22. The system of claim 21, wherein the sub-module for determining if a defined number of clutching operation have been performed, comprises a sub-module for determining if a prescribed number of clutching operations have been performed.

23. The system of claim 21, wherein the sub-module for determining if a defined number of clutching operation have been performed, comprise a sub-module for determining if a prescribed number of valid clutching operations have been performed, wherein a clutching operation is valid if the recorded direction that the pointing device was pointing at the end of the clutching operation under consideration differs from the recorded direction that the pointing device was pointing at the end of each previous clutching operation by a prescribed threshold difference.

24. A system for directing a laser beam within a space to act as a cursor, comprising:

a pointing device which periodically outputs orientation data indicative of the direction it is pointing;

a cursor device which projects a laser beam in the space;

a general purpose computing device which is in communication with the pointing device and the cursor device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

receive the orientation data output by the pointing device and compute the direction the pointing device is pointing from the received orientation data in terms of yaw and pitch angles, and

direct the laser beam generated by a cursor device to a particular location in the space as determined by the direction the pointing device is pointing, such that the laser beam is directed in a relative pointing mode wherein the location in the space where the laser beam is pointed is not the same location that the pointing device is pointed, and wherein the relationship between where the pointing device is pointing and where the laser beam is pointing is allowed to vary over time.

25. A system for directing a laser beam within a space to act as a cursor, comprising:

a pointing device which periodically outputs orientation data indicative of the direction it is pointing;

a cursor device which projects a laser beam in the space;

a general purpose computing device which is in communication with the pointing device and the cursor device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

receive the orientation data output by the pointing device and compute the direction the pointing device is pointing from the received orientation data in terms of yaw and pitch angles, and

direct the laser beam generated by a cursor device to a particular location in the space as determined by the direction the pointing device is pointing, such that the laser beam is directed in a relative pointing mode wherein the location in the space where the laser beam is pointed is not the same location that the pointing device is pointed, and wherein a fixed relative pointing condition is created between the pointing device and the laser beam wherein the difference between the direction that the pointing device is pointing and the direction that the laser beam is pointing remains substantially constant over time.

26. The system of claim 25, wherein the cursor device comprises a laser which when activated projects said laser beam into the space and a pair of servos connected to laser, wherein a first of the servos is configured so as to move the laser in a manner that controls the yaw direction of the laser beam and the other of the servos is configured so as to move the laser in a manner that controls the pitch direction of the laser beam, and wherein the sub-module for creating a fixed relative pointing condition between the pointing device and the laser beam, comprises sub-modules for:

computing a set of offset angles for the laser of the cursor device comprising respective yaw and pitch angles that define the angular distance between the origin of the spherical coordinate system associated with the cursor device and a prescribed origin of the spherical coordinate system for the space;

computing a set of offset angles for the pointing device comprising respective yaw and pitch angles that define the angular distance between the origin of the spherical coordinate system associated with the pointing device and a prescribed origin of the spherical coordinate system for the space;

computing pitch and yaw angles that define how far the cursor device must move the laser in order to point the laser beam at approximately the same point in the space that the pointing device is pointing to, wherein the aligning pitch angle is defined as the sum of the offset pitch angle of the laser and the computed pitch angle of the pointing device less its offset pitch angle, and wherein the aligning yaw angle is defined as the sum of the offset yaw angle of the laser and the computed yaw angle of the pointing device less its offset yaw angle;

ascertaining the angular difference that is desired between the pointing device and the cursor device; and

directing the laser to move to the computed aligning pitch and yaw angles plus the ascertained angular difference.

27. The system of claim 26, wherein the pointing device comprises a manually-operated switch which is activated and deactivated by a user for the purpose of instructing the computing device to implement a function, and wherein the state of the switch in regard to whether it is activated or deactivated at the time orientation data is output is included in that data, wherein the sub-module for ascertaining the angular difference that is desired between the pointing device and the cursor device, comprises performing a clutching procedure which comprises sub-modules for:

determining if the pointing device'"'"'s switch has been activated by the user each time the orientation data is received, wherein said user activates said switch whenever the angular difference that is desired between the pointing device and the cursor device is to be set;

whenever the pointing device'"'"'s switch has been activated,

ceasing to provide new movement commands to the cursor device so as to stop the motion of the laser beam,

determining if the pointing device'"'"'s switch has been first deactivated by the user and then reactivated using the received orientation data, wherein said user reactivates said switch whenever the pointing device has been pointed in a direction that defines the angular difference that is desired between the pointing device and the cursor device;

whenever the pointing device'"'"'s switch has been reactivated, reinitiating the sending of new movement commands to the cursor device.

28. The system of claim 27, wherein said sets of offset angles comprise initial angle values which are valid for only a part of the space such that when the pointing device is pointed outside the part of the space where the initial offset values are valid the previously set angular difference between the location the pointing device is pointing and the location the laser beam is shining on is lost, and wherein the sub-module for creating a fixed relative pointing condition between the pointing device and the laser beam further comprises a sub-module for maintaining said angular difference between the pointing device and the laser beam by repeating said clutching procedure and re-computing the sets of offset angles prior to executing the sub-module for reinitiating the sending of new movement commands to the cursor device.

29. A system for establishing a 3D model of a space wherein objects within the space are modeled as a group of vertices defining polygons, comprising:

a pointing device which periodically outputs orientation data indicative of the direction it is pointing and which comprises a manually-operated switch which is activated and deactivated by a user for the purpose of instructing the computing device to implement a function, and wherein the state of the switch in regard to whether it is activated or deactivated at the time orientation data is output is included in that data;

a cursor device which projects a laser beam in the space;

a general purpose computing device which is in communication with the pointing device and the cursor device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program, for each polygon being established in the model of the space, to,

initiate a training mode in response to a user-specified command to do so;

input the orthogonal coordinates of the cursor device and the pointing device in terms of an orthogonal coordinate system of the space,

receive the orientation data output by the pointing device and compute the direction the pointing device is pointing from the received orientation data in terms of yaw and pitch angles,

direct the laser beam generated by a cursor device to point at the same location in the space that the pointing device is pointed toward,

determine if the pointing device'"'"'s switch has been activated by the user each time the orientation data is received, wherein said user activates said switch whenever the cursor device'"'"'s laser beam is shining on a point in the space that corresponds to a vertex of a polygon that is being used to model an object in the space; and

whenever the pointing device'"'"'s switch has been activated when the training mode is active,

compute the orthogonal coordinates of the point in space that the pointing device and the laser beam are pointed at in view of the orthogonal coordinates of the pointing device and the direction it is pointing and the orthogonal coordinates of the cursor device and the direction it is pointing, and

associate the computed orthogonal coordinates of the point in space with the corresponding vertex of the object being modeled,

repeat the determining, computing and associating actions until all the vertices of the polygon representing the object being modeled are determined.

30. The system of claim 29, wherein the computer program further comprises a program module for inputting user-entered information that identifies an object being modeled and associating it with the polygon vertices representing the object.