System and process for selecting objects in a ubiquitous computing environment

• US 7,250,936 B2
• Filed: 12/20/2004
• Issued: 07/31/2007
• Est. Priority Date: 02/07/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew, Shafer, Steven, Wilson, Daniel
• Primary Examiner(s): DINH, DUC Q
• Application Number:
  US11/020,064
• Publication Number:
  US 20050110751A1
• Time in Patent Office: 953 Days
• Field of Search: 345156-171
• US Class Current: 345/156
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G08C 17/00 Arrangements for transmitting signals characterised by the use of a wireless electrical link
• CPC Class Codes: G08C 17/02 using a radio link
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 2201/41 Remote control of gateways
• CPC Class Codes: G08C 2201/50 Receiving or transmitting feedback, e.g. replies, status updates, acknowledgements, from the controlled devices

Abstract

A system and process for selecting objects in an ubiquitous computing environment where various electronic devices are controlled by a computer via a network connection and the objects are selected by a user pointing to them with a wireless RF pointer. By a combination of electronic sensors onboard the pointer and external calibrated cameras, a host computer equipped with an RF transceiver decodes the orientation sensor values transmitted to it by the pointer and computes the orientation and 3D position of the pointer. This information, along with a model defining the locations of each object in the environment that is associated with a controllable electronic component, is used to determine what object a user is pointing at so as to select that object for further control actions.

Claims

1. A computer-implemented process for selecting an object within an environment by a user pointing to the object with a pointing device, comprising:

an inputting step for inputting orientation sensor readings generated by orientation sensors of the pointing device and provided by the device in an orientation message;

a deriving step for deriving the orientation of the pointing device in relation to a predefined coordinate system of the environment from the inputted orientation sensor readings;

a second inputting step for inputting digital video from a pair of video cameras each of which is located so as to capture images of the environment from different viewpoints;

an ascertaining step for ascertaining the location of the pointing device at a time substantially contemporaneous with the generation of the orientation message and in terms of the predefined coordinate system using the inputted digital video from the pair of video cameras;

a determining step for using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer; and

whenever the pointing device is being pointed at a controllable object, a selecting step for selecting that object for future control actions.

2. The process of claim 1, wherein the orientation sensor readings comprise the outputs of a three-axis magnetometer, and wherein the process further comprises an establishing step for establishing correction factors to relate the magnetometer outputs to the predefined coordinate system of the environment prior to using them to derive the orientation of the pointing device.

3. The process of claim 1, wherein the orientation sensor readings comprise the outputs of a two-axis accelerometer and a three-axis magnetometer, and wherein the process further comprises an establishing step for establishing normalization factors to normalize the outputs of the accelerometer and magnetometer prior to using them to derive the orientation of the pointing device.

4. The process of claim 3, wherein the accelerometer and magnetometer of the pointing device are oriented therein such that their respective first axis corresponds to an x-axis which is directed laterally to a pointing axis of the pointing device and their respective second axis corresponds to a y-axis which is directed along the pointing axis of the pointing device and the third axis of the magnetometer correspond to a z-axis which is directed vertically upward whenever the pointing device is positioned right-side up with the x and y axes lying in a horizontal plane, and wherein the deriving step, comprises:

a normalizing step for normalizing the accelerometer and magnetometer output values contained in the orientation message using the previously established normalization factors;

a computing step for computing the angles defining the pitch of the pointing device about the x-axis and the roll of the device about the y-axis from the normalized outputs of the accelerometer;

a refining step for refining the normalized magnetometer output values using the pitch and roll angles computed from the normalized accelerometer output values;

an applying step for applying the correction factors previously established for each axis of the magnetometer to the refined magnetometer output values;

a second computing step for computing the yaw angle of the pointing device about the z axis using the corrected and refined magnetometer output values;

a designating step for tentatively designate the computed pitch, roll and yaw angles as defining the orientation of the pointing device at the time the orientation message was generated;

an assessing step for assessing whether the pointing device was in a right-side up or up-side down position at the time the orientation message was generated;

whenever it is determined that the pointing device was in the right-side up position at the time the orientation message was generated, a second designating step for designating the previously computed pitch, roll and yaw angles as defining the finalized orientation of the pointing device; and

whenever it is determined that the pointing device was in the up-side down position at the time the orientation message was generated, a correcting step for correcting the tentatively designated roll angle and designating the pitch, yaw and modified roll angle as defining the finalized orientation of the pointing device.

5. The process of claim 4, further comprising, prior to performing the step for computing the angles defining the pitch and roll of the pointing device from the normalized outputs of the accelerometer, performing the steps for:

determining whether the pointing device has moved in the time between the generation of the orientation message currently under consideration and the generation of the immediately preceding orientation message; and

whenever it is determined that the pointing device has moved in the time between the generation of said sequential orientation messages, ignoring the orientation message currently under consideration and waiting for the next orientation to be received.

6. The process of claim 1, wherein the pointing device comprises an infrared (IR) light emitting diode (LED) which is disposed so as to emit IR light when lit outside of the case and which flashes for a prescribed duration at a prescribed rate, and wherein the pair of video cameras each capture video image frames at a rate approximately twice the prescribed flashing rate of the pointing device'"'"'s IR LED and each have an IR pass filter which causes the cameras to output video image frames that capture only IR tight emitted or reflected in the environment toward the camera including the emissions from the pointing device'"'"'s IR LED, and wherein the step for ascertaining the location of the pointing device using the inputted digital video from the pair of video cameras, comprises:

for each video camera and each pair of video image frames produced by thereby,

a subtracting step for subtracting the pair of frames to produce a difference image, said difference image depicting substantially only the IR emissions and reflections directed toward the camera which appear in one or the other of the pair of frames but not both including the flash from the IR LED of the pointing device as it flashes at a rate approximately one-half the rate at which the image frames are captured and so will appear in only one of each pair of frames produced by the camera, and wherein said flash is presumed to be the predominate feature in the difference image, and

an identifying step for identifying the image coordinates of the pixel in the difference image that exhibits the highest intensity using a peak detection procedure; and

for each pair of approximately contemporaneous pairs of image frames generated by the pair of cameras, a second determining step for determining the 3D coordinates of the flash within the predefined environmental coordinate system using the image coordinates of the flash identified in the difference images associated with said approximately contemporaneous pair of image frames generated by the pair of cameras and predetermined intrinsic and extrinsic camera parameters employing a stereo image technique.

7. The process of claim 1 wherein the step for using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer, comprises:

a modeling step for modeling the location and extent of objects within the environment that are controllable by the computer using 3D Gaussian blobs defined by a location of the mean of the blob in terms of its environmental coordinates and a covariance;

for each Gaussian blob,

a projecting step for projecting the Gaussian blob onto a plane which is normal to either a line extending from the location of the pointing device to the mean of the blob or a ray originating at the location of the pointing device and extending in a direction defined by the orientation of the device,

a second ascertaining step for ascertaining the value of the resulting projected Gaussian blob at a point where the ray intersects the plane, said value representing the probability that the pointing device is pointing at the object associated with the projected Gaussian blob;

an identifying step for identifying which of the probability values associated with the projected Gaussian blobs is the largest, if any;

a second determining step for determining if a probability value identified as the largest exceeds a prescribed minimum probability threshold; and

whenever it is determined that the probability value identified as the largest exceeds the prescribed minimum probability threshold, a designating step for designating the object associated with the projected Gaussian blob from which the probability value was derived as being the object that the pointing device is pointing at.

8. A computer-readable medium having computer-executable instructions for selecting an object within an environment by a user pointing to the object with a pointing device, said computer-readable instructions comprising:

inputting orientation sensor readings generated by orientation sensors of the pointing device and provided by the device in an orientation message;

deriving the orientation of the pointing device in relation to a predefined coordinate system of the environment from the inputted orientation sensor readings;

inputting digital video from a pair of video cameras each of which is located so as to capture images of the environment from different viewpoints;

ascertaining the location of the pointing device at a time substantially contemporaneous with the generation of the orientation message and in terms of the predefined coordinate system using the inputted digital video from the pair of video cameras;

using the orientation and location of the pointing device to determine whether the pointing device is being pointed at an object in the environment that is controllable by the computer; and

whenever the pointing device is being pointed at a controllable object, selecting that object for future control actions.