Easily deployable interactive direct-pointing system and presentation control system and calibration method therefor

• US 9,063,586 B2
• Filed: 08/19/2014
• Issued: 06/23/2015
• Est. Priority Date: 05/28/2004
• Status: Active Grant

• Current Assignee: UltimatePointer, LLC
• Original Assignee: UltimatePointer, LLC
• Inventors: Banning, Erik Jan
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US14/463,405
• Publication Number:
  US 20140354547A1
• Time in Patent Office: 308 Days
• Field of Search: 345156-169, 345173-178, 345204-215, 178 1801- 1809
• US Class Current: 1/1
• CPC Class Codes: G03B 21/00 Projectors or projection-type viewers; Accessories therefor devices for changing pictures G03B23/00
• CPC Class Codes: G03B 21/006 using LCD's
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03549 Trackballs G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

A method for controlling movement of a computer display cursor based on a point-of-aim of a pointing device within an interaction region includes projecting an image of a computer display to create the interaction region. At least one calibration point having a predetermined relation to said interaction region is established. A pointing line is directed to substantially pass through the calibration point while measuring a position of and an orientation of the pointing device. The pointing line has a predetermined relationship to said pointing device. Movement of the cursor is controlled within the interaction region using measurements of the position of and the orientation of the pointing device.

Claims

1. A base station for use in generating first data that is indicative of at least two positional coordinates of an enclosure relative to the base station, said first data being dependent on a first output signal generated by a position sensing device coupled to a first communication device that is configured to generate said first data, the enclosure being separate from the base station and designed to be handheld by a user and operated by the user in mid-air, the position sensing device comprising components contained in at least one of said base station and said enclosure, the base station comprising:

a level sensing device for generating a second output signal that is sensitive to an angle of the base station relative to gravity;

at least one of a first power source and a connection to an external second power source for providing power to devices in the base station; and

a second communication device coupled to said level sensing device and configured to generate second data that is dependent on said second output signal;

wherein the enclosure includes;

a user input device for providing a third output signal and configured to allow the user to manipulate two positional coordinates of a feature on an image displayed by an image display device coupled to a processing unit that is external to said base station and said enclosure and said image display device, the third output signal being independent of orientation of the enclosure;

a third communication device coupled to the user input device and configured to generate third data that is dependent on the third output signal, and

at least one of a third power source and a connection to an external fourth power source for providing power to devices in the enclosure;

wherein the first, second and third communication devices are configured to be coupled to a fourth communication device that is coupled to said processing unit and is for supplying fourth data to the processing unit, the fourth data being dependent on at least one of the first data and the second data and the third data; and

wherein the processing unit is coupled to a non-transitory computer-readable medium or media storing computer-executable instructions for causing the processing unit to perform a method, the method comprising the steps of;

causing communication between the fourth communication device and the third communication device for transferring third data;

causing communication between the fourth communication device and the second communication device for transferring second data;

causing communication between the fourth communication device and the first communication device for transferring first data; and

using the fourth data to control the contents of the image.

2. The base station according to claim 1, wherein the position sensing device comprises an image capturing device that is contained in said base station and is for generating said first output signal,

wherein the second communication device is the first communication device, and

wherein the fourth data depends on the third data.

3. The base station according to claim 1, wherein the fourth data depends on the third data.

4. The base station according to claim 3, wherein the user input device is a trackball.

5. The base station according to claim 1,

wherein the enclosure further includes:

an orientation sensing device for generating a fourth output signal, said orientation sensing device being coupled to the third communication device and comprising;

an accelerometer and a gyro and being sensitive to at least one orientation aspect of the enclosure relative to the base station,

wherein the fourth data is dependent on the first data,

wherein the third communication device is configured to generate fifth data that is dependent on the fourth output signal,

wherein the fourth communication device is configured to generate sixth data that is dependent on the second data, seventh data that is dependent on the third data, and eighth data that is dependent on the fifth data, and

wherein the method further comprises the step of using the sixth data, the seventh data and the eighth data to control the contents of the image.

6. The base station according to claim 5, wherein

the position sensing device comprises an image capturing device that is contained in the base station and is for generating the first output signal, and

wherein the second communication device is the first communication device.

7. A non-transitory computer-readable medium or media storing computer-executable instructions for causing a computer to perform a method for controlling contents of a computer-generated image in conjunction with an enclosure, the computer comprising a first communication device coupled to a processing unit,

the enclosure comprising;

a second communication device for providing first and second data to the first communication device;

an accelerometer for generating a first output signal and coupled to the second communication device; and

a user input device for providing a second output signal, the user input device configured to allow the user to control two positional coordinates of a feature on said image and coupled to the second communication device,

the enclosure configured to be handheld and wielded by a user in mid-air and to develop the first data that is dependent on the first output signal and the second data that is dependent on the second output signal,

the method comprising the steps of;

causing communication between the first communication device and the second communication device to transfer the first data and the second data; and

controlling the contents based on at least one of first data and the second data.

8. The non-transitory computer-readable medium or media according to claim 7, wherein the user input device is a trackball.

9. The non-transitory computer-readable medium or media according to claim 7, wherein the enclosure further comprises:

a light source for providing a light output and configured for use with an image capturing device that is coupled to the processing unit, the image capturing device generating third data based on the light output, the third data being dependent on at least one positional coordinate of the enclosure, the light source being coupled to the second communication device; and

a gyro for generating a third output signal, the gyro being coupled to the second communication device, the first data being dependent on at least one of the first output signal and the third output signal; and

wherein the method further comprises the steps of;

controlling the two positional coordinates of said feature on the image based on the second data; and

controlling the contents of the image based on the first data.

10. The non-transitory computer-readable medium or media according to claim 9, wherein the method further comprising the steps of:

causing communication between the first communication device and a third communication device to transfer the third data, the third communication device being included in a base station that is separate from the enclosure and from the computer, the base station further including the image capturing device which is coupled to the third communication device; and

controlling the contents of the image based on the third data, the third data being dependent on the position of the enclosure relative to the base station.

11. The non-transitory computer-readable medium or media according to claim 10, wherein the base station further includes visible markings designed to allow aligning said base station in a predetermined way relative to horizontal, and wherein the third data is indicative of the three dimensional position of the enclosure relative to the base station.

12. A non-transitory computer-readable medium or media storing computer-executable instructions for causing a computer to perform a method for controlling contents of a computer-generated image in conjunction with an enclosure that is designed to be handheld and wielded by a user in mid air, the computer comprising a first communication device coupled to a processing unit,

the enclosure comprising:

a light source for providing a light output and configured for use with an image capturing device that is coupled to the processing unit, the image capturing device generating first data based on the light output, the first data being dependent on at least one positional coordinates of the enclosure relative to a base station that is separate from the computer and the enclosure;

a manipulation device for providing a first output signal, the manipulation device configured to allow the user to control two mutually independent positional coordinates of a feature on the image; and

a second communication device coupled to the manipulation device and to the light source and configured to allow coupling to the first communication device;

wherein the enclosure is for developing second data that is dependent on the first output signal, and

wherein the second communication device is for providing the second data,

the method comprising the steps of;

causing communication between the first communication device and the second communication device to transfer the second data; and

controlling the contents of the computer generated image based at least in part on at least one of said first data and second data.

13. The non-transitory computer-readable medium or media according to claim 12, wherein the manipulation device is a trackball.

14. The non-transitory computer-readable medium or media according to claim 12, wherein a sensing device comprising components contained in at least one of the enclosure and the base station generates a second output signal that is sensitive to an orientation coordinate of the enclosure, the sensing device being coupled to at least one of the second communication device and a third communication device contained in the base station, with third data that is dependent on the second output signal being developed by at least one of the enclosure and the base station, and

wherein the method further comprises the steps of

causing communication between the first communication device and at least one of the second communication device and the third communication to transfer the third data;

and

controlling the contents of the computer generated image based at least in part on the third data.

15. The non-transitory computer-readable medium or media according to claim 14, wherein said sensing device is contained in said enclosure and comprises an accelerometer and a gyro.

16. The non-transitory computer-readable medium or media according to claim 14, wherein said orientation coordinate is relative to the base station.

17. The non-transitory computer-readable medium or media according to claim 12, wherein the first data is indicative of at least two positional coordinates of the enclosure relative to the base station, and

wherein the step of controlling the contents of the computer generated image based at least in part on at least one of said first data and second data is based at least in part on both the first data and the second data.

18. The non-transitory computer-readable medium or media according to claim 12, wherein the base station includes an accelerometer.

19. The non-transitory computer-readable medium or media according to claim 12, wherein the base station further includes the image capturing device.

20. An apparatus for controlling contents of a computer generated image, the apparatus comprising:

a controller device configured to be handheld and wielded in mid-air by a user and to allow said user to control two positional coordinates of a feature on the computer generated image, said controller device comprising;

a user input device for generating a first output signal that is independent of orientation of said controller device; and

a first communication device coupled to said user input device for generating first data that is dependent on said first output signal;

a base station that is separate from said controller device; and

a position sensing device comprising components contained in at least one of said controller device and said base station for generating a second output signal that is dependent on at least one positional coordinate of said controller device relative to said base station;

a second communication device coupled to said position sensing device for generating second data that is dependent on said second output signal;

a third communication device for coupling to said first and second communication devices for generating third data that is dependent on at least one of said first data and said second data;

a processor for generating the computer generated image and coupled to said third communication device; and

a non-transitory computer-readable medium or media storing computer-executable instructions coupled to and for causing said processor to execute a method comprising the step of using said third data to control the contents of the computer generated image.

21. The apparatus according to claim 20,

wherein said position sensing device comprises:

a light source contained in said controller device; and

an image capturing device,

wherein said second output signal is dependent on three independent positional coordinates of said controller device relative to said base station,

wherein said controller device further comprises;

an orientation sensing device for generating a third output signal that is dependent on orientation of said controller device, said orientation sensing device being coupled to the first communication device and comprising an accelerometer and a gyro,

wherein said first communication device is further for generating fourth data that is dependent on the third output signal,

wherein said third data is dependent on said second data,

wherein said third communication device is further for generating fifth and sixth data, said fifth data being dependent on said first data and said sixth data being dependent on said fourth data, and

wherein said method further comprises the step of using said third data, said fifth data and said sixth data to control the contents of the computer generated image.

22. The apparatus according to claim 21, wherein said base station comprises visible features for allowing alignment of said base station in a predetermined way relative to horizontal, and

wherein said orientation of said controller device comprises at least one angle of said controller device relative to horizontal.

23. The apparatus according to claim 22,

wherein said image capturing device is contained in said base station,

wherein said base station further comprises at least one of a first power source and a connection to an external second power source for providing power to said base station, and

wherein said controller device further comprises at least one of a third power source and a connection to an external fourth power source for providing power to said controller device.

24. The apparatus according to claim 21, wherein said image capturing device is contained in said base station and is for providing said second output signal.

25. The apparatus according to claim 20,

wherein said base station comprises a fourth communication device for generating fourth data and a level sensing device coupled to said fourth communication device and for providing a third output signal, said fourth data being dependent on said third output signal;

wherein said position sensing device comprises an infrared source and an image capturing device;

wherein said third communication device is configured to allow coupling to said fourth communication device and is further for generating fifth data that is dependent on said fourth data; and

wherein the method further comprises the step of using said fifth data to control the contents of the computer generated image.

26. The apparatus according to claim 25,

wherein said level sensing device comprises an accelerometer,

wherein said second output signal is dependent on at least two independent positional coordinates of said controller device relative to said base station,

wherein said third data is dependent on said second data,

wherein said third communication device is further for generating sixth data that is dependent on said first data, and

wherein the method comprises the step of using said sixth data to control the contents of the computer generated image.

27. The apparatus according to claim 26,

wherein said image capturing device is contained in said base station,

wherein said second communication device is said fourth communication device,

wherein said base station further comprises at least one of a first power source and a connection to an external second power source for providing power to said base station, and

wherein said controller device further comprises at least one of a third power source and a connection to an external fourth power source for providing power to said controller device.

28. The apparatus according to claim 25, wherein said image capturing device is contained in said base station and is for providing said second output signal.

29. The apparatus according to claim 20, wherein said user input device is a trackball.

30. The apparatus according to claim 20, wherein said user input device is an analog stick.

31. The apparatus according to claim 20, further comprising:

an orientation sensing device comprising components contained in at least one of the controller device and the base station for generating a third output signal that is dependent on orientation of said controller device, with at least one aspect of said orientation being relative to said base station; and

a fourth communication device coupled to said orientation sensing device, said fourth communication device for generating fourth data that is dependent on said third output signal,

wherein said third data is dependent on said second data,

wherein said third communication device is configured to allow coupling to said fourth communication device;

wherein said third communication device is further for generating fifth data and sixth data, said fifth data being dependent on said first data and said sixth data being dependent on said fourth data; and

wherein the method further comprises the step of using said fifth data and said sixth data to control the contents of the computer generated image.

32. The apparatus according to claim 20,

wherein said base station comprises:

a fourth communication device for generating fourth data; and

an accelerometer coupled to said fourth communication device for providing a third output signal that is dependent on an angle of said base station relative to horizontal, said fourth data being dependent on said third output signal;

wherein said third communication device is configured to allow coupling to said fourth communication device and is for generating fifth data that is dependent on said fourth data; and

wherein the method comprises the step of using said fifth data to control the contents of the computer generated image.

33. An apparatus for controlling a feature on a rectangular computer screen interaction region, there being a fixed coordinate system defined by three axes that are perpendicular to each other, the coordinate system having a first and a second coordinate axis that are horizontal, the apparatus comprising:

a sensing device for generating a sensor output that is sensitive to an angle of said sensing device relative to the coordinate system when the position of said sensing device relative to the coordinate system is constant, said sensing device including an accelerometer for providing a first output signal and a gyro for providing a second output signal, said sensor output being dependent on said first output signal and said second output signal, wherein said sensing device has a fixed relation to a pointing line, the pointing line defining a point-of-aim as the intersection of said pointing line with an interaction structure that is perpendicular to a plane spanned by the first and second coordinate axes;

at least one of an internal first power source and a connection to an external second power source for providing power to devices in said apparatus;

a first communication device coupled to said sensing device, for providing first data based on said sensor output;

a second communication device coupled to said first communication device for receiving said first data;

a processing unit coupled to said second communication device; and

a non-transitory computer-readable medium or media storing computer executable instructions coupled to and for directing said processing unit to perform a method for controlling the feature on the computer screen interaction region using second data developed by said processing unit that is dependent on said first data, wherein for fixed positions of said sensing device the using said second data to control the feature on the computer screen interaction region is independent of the orientation of the computer screen interaction region relative to the coordinate system, and operates so that the feature is kept at substantially the same distance from a first edge of the computer screen interaction region when said second data is indicative of the point-of-aim remaining at substantially the same distance from the plane.

34. The apparatus according to claim 33, wherein said sensor output is indicative of a first and a second angle when the position of said sensing device relative to the coordinate system remains constant, said first and second angle determining orientation of said pointing line relative to the coordinate system, and wherein both of said first and said second output signals are used to establish both of said first and said second angle.

35. The apparatus according to claim 33, wherein said sensing device further includes an image capturing device for providing a third output signal, and

wherein said sensor output further depends on said third output signal.

36. The apparatus according to claim 35, wherein said sensing device further includes a magnetometer for providing a fourth output signal, and wherein said sensor output further depends on said fourth output signal.

37. The apparatus according to claim 35, wherein said sensing device is contained in an enclosure that is designed to be handheld and wielded in mid-air by a user.

38. The apparatus according to claim 37, wherein said interaction structure is a vertical rectangle, and

wherein for fixed positions of said sensing device the step of using said second data to control the feature on the computer screen interaction region operates so that the feature is kept at substantially the same distance from a second edge of the computer screen interaction region if the second data is indicative of the point-of-aim remaining at substantially the same distance from a vertical edge of the interaction structure, said first and second edges of the computer screen interaction region being perpendicular to each other.

39. The apparatus according to claim 33, wherein said sensing device further includes a magnetometer for providing a third output signal, and

wherein said sensor output further depends on said third output signal.

40. The apparatus according to claim 33, wherein said sensing device is contained in an enclosure that is designed to be handheld and wielded in mid-air by a user, and

wherein said sensor output is indicative of a first and a second angle that determine orientation of said pointing line relative to the coordinate system.

41. The apparatus according to claim 40, wherein said sensing device further includes a magnetometer that is contained in the enclosure, said magnetometer for providing a third output signal,

wherein said sensor output further depends on said third output signal, and

wherein said sensor output is indicative of a third angle, said first and second and third angles completely defining the orientation of said sensing device relative to the coordinate system.

42. A controller for controlling the contents of a computer-generated image, the controller designed to be handheld and operated by a user in mid-air, the controller comprising:

a light source for providing a light output and configured for use with an image capturing device, the image capturing device for generating first data based on the light output, said first data being dependent on at least one positional coordinate of the controller;

a user input device for generating second data and designed to allow the user to provide directional inputs, said second data being independent of orientation of the controller;

a controller communication device coupled to said light source and said user input device and configured to allow coupling to a computer that generates the computer-generated image; and

at least one of a first power source and a connection to an external second power source for providing power to said light source and said controller communication device.

43. The controller according to claim 42, wherein the at least one positional coordinate of the controller is relative to a base station that is separate from the computer and the controller, the base station including a base station communication device and a level-sensing device for generating third data that is dependent on an angle of the base station relative to gravity, the level sensing device being coupled to the base station communication device,

wherein the image capturing device is coupled to the computer, and

wherein the computer includes:

a computer communication device configured to allow coupling to said controller communication device and to the base station communication device;

a processing unit coupled to the computer communication device; and

a non-transitory computer-readable medium or media coupled to the processing unit and storing computer-executable instructions for directing the processing unit to perform a method, the method comprising the step of controlling the contents of the computer-generated image based at least in part on at least one of said first data, said second data and said third data.

44. The controller according to claim 43, wherein the base station further includes the image capturing device, and wherein the method further comprises the steps of:

controlling the contents of the computer-generated image based at least in part on said second data; and

controlling the contents of the computer-generated image based at least in part on said third data.

45. The controller according to claim 42, wherein the controller further comprises:

a sensing device coupled to said controller communication device for generating third data that is dependent on an orientational coordinate of the controller, said sensing device including;

an accelerometer for generating a first output signal; and

a gyro for generating a second output signal,

wherein said third data depends on said first and said second output signals.

46. The controller according to claim 45, wherein the at least one positional coordinate of the controller is relative to a base station that is separate from the computer and the controller, the base station including a base station communication device and visible markings designed to allow aligning said base station in a predetermined way relative to horizontal,

wherein the image capturing device is coupled to the computer, and

wherein the computer includes:

a computer communication device configured to allow coupling to the controller communication device and to the base station communication device;

a processing unit coupled to the computer communication device; and

a non-transitory computer-readable medium or media coupled to the processing unit and storing computer-executable instructions for directing the processing unit to perform a method, the method comprising the step of

controlling the contents of the image based at least in part on at least one of said first data and said second data and said third data.

47. The controller according to claim 46, wherein the base station further includes said image capturing device wherein said first data is dependent on three mutually independent positional coordinates of the controller relative to the base station, and wherein the method comprises the steps of:

controlling the contents of the image based at least in part on said first data;

controlling the contents of the image based at least in part on said second data; and

controlling the contents of the image based at least in part on said third data.

48. The controller according to claim 42, wherein said user input device is a trackball.