System and process for selecting objects in a ubiquitous computing environment

• US 7,307,617 B2
• Filed: 12/20/2004
• Issued: 12/11/2007
• Est. Priority Date: 02/07/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew, Wilson, Daniel, Shafer, Steven
• Primary Examiner(s): Dinh; Duc Q
• Application Number:
  US11/019,876
• Publication Number:
  US 20050156883A1
• Time in Patent Office: 1,086 Days
• Field of Search: 345156-171
• US Class Current: 345/156
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

A system and process for selecting objects in an ubiquitous computing environment where various electronic devices are controlled by a computer via a network connection and the objects are selected by a user pointing to them with a wireless RF pointer. By a combination of electronic sensors onboard the pointer and external calibrated cameras, a host computer equipped with an RF transceiver decodes the orientation sensor values transmitted to it by the pointer and computes the orientation and 3D position of the pointer. This information, along with a model defining the locations of each object in the environment that is associated with a controllable electronic component, is used to determine what object a user is pointing at so as to select that object for further control actions.

Claims

1. A computer-implemented process for selecting an object within an environment by a user pointing to the object with a pointing device, comprising using a computer to perform the following process actions:

ascertaining the orientation of the pointing device in relation to a predefined coordinate system of the environment;

ascertaining the location of the pointing device in terms of the predefined coordinate system;

using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer; and

whenever the pointing device is being pointed at a controllable object, selecting that object for future control actions;

wherein the process action of using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer, comprises the actions of,

modeling the location and extent of objects within the environment that are controllable by the computer using 3D Gaussian blobs defined by a location of the mean of the blob in terms of its environmental coordinates and a covariance,

for each Gaussian blob, determining whether a ray originating at the location of the pointing device and extending in a direction defined by the orientation of the device intersects the blob,

for each Gaussian blob intersected by the ray, ascertaining the value of the Gaussian blob at a point along the ray nearest the location of the mean of the blob, said value representing the probability that the pointing device is pointing at the object associated with the Gaussian blob,

identifying which of the probability values associated with the Gaussian blobs is the largest, and

designating the object associated with the Gaussian blob from which the largest probability value was derived as being the object that the pointing device is pointing at.

4. A computer-readable medium having computer-executable instructions for performing the process actions recited in claim 1.

7. The process of claim 1, wherein the process action of designating the object associated with the Gaussian blob from which the largest probability value was derived as being the object that the pointing device is pointing at, comprises the actions of:

determining if a probability value identified as the largest exceeds a prescribed minimum probability threshold; and

whenever it is determined that the probability value identified as the largest exceeds the prescribed minimum probability threshold, designating the object associated with the Gaussian blob from which the probability value was derived as being the object that the pointing device is pointing at.

2. A computer-implemented process for selecting an object within an environment by a user pointing to the object with a pointing device, comprising using a computer to perform the following process actions:

ascertaining the orientation of the pointing device in relation to a predefined coordinate system of the environment;

ascertaining the location of the pointing device in terms of the predefined coordinate system;

using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer; and

whenever the pointing device is being pointed at a controllable object, selecting that object for future control actions;

wherein the process action of using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer, comprises the actions of,

modeling the location and extent of objects within the environment that are controllable by the computer using 3D Gaussian blobs defined by a location of the mean of the blob in terms of its environmental coordinates and a covariance,

for each Gaussian blob,

projecting the Gaussian blob onto a plane which is normal to either a line extending from the location of the pointing device to the mean of the blob or a ray originating at the location of the pointing device and extending in a direction defined by the orientation of the device,

ascertaining the value of the resulting projected Gaussian blob at a point where the ray intersects the plane, said value representing the probability that the pointing device is pointing at the object associated with the projected Gaussian blobs,

identifying which of the probability values associated with the projected Gaussian blobs is the largest, if any, and

designating the object associated with the Gaussian blob from which the largest probability value was derived as being the object that the pointing device is pointing at.

5. A computer-readable medium having computer-executable instructions for performing the process actions recited in claim 2.

3. A computer-implemented process for selecting an object within an environment by a user pointing to the object with a pointing device, comprising using a computer to perform the following process actions:

ascertaining the orientation of the pointing device in relation to a predefined coordinate system of the environment;

ascertaining the location of the pointing device in terms of the predefined coordinate system;

using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer; and

whenever the pointing device is being pointed at a controllable object, selecting that object for future control actions;

wherein the process action of using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer, comprises the actions of,

modeling the location and extent of objects within the environment that are controllable by the computer using 3D Gaussian blobs defined by a location of the mean of the blob in terms of its environmental coordinates and a covariance,

for each Gaussian blob,

projecting the Gaussian blob onto a plane which is normal to either a line extending from the location of the pointing device to the mean of the blob or a ray originating at the location of the pointing device and extending in a direction defined by the orientation of the device,

ascertaining the value of the resulting projected Gaussian blob at a point where the ray intersects the plane, said value representing the probability that the pointing device is pointing at the object associated with the projected Gaussian blob,

identifying which of the probability values associated with the projected Gaussian blobs is the largest, if any,

determining if a probability value identified as the largest exceeds a prescribed minimum probability threshold and

whenever it is determined that the probability value identified as the largest exceeds the prescribed minimum probability threshold, designating the object associated with the projected Gaussian blob from which the probability value was derived as being the object that the pointing device is pointing at.

6. A computer-readable medium having computer-executable instructions for performing the process actions recited in claim 3.