2D and 3D pointing device based on a passive lights detection operation method using one camera

• US 9,678,583 B2
• Filed: 07/23/2014
• Issued: 06/13/2017
• Est. Priority Date: 07/23/2013
• Status: Active Grant

• Current Assignee: University Of Kentucky Research Foundation (University of Kentucky)
• Original Assignee: University Of Kentucky Research Foundation (University of Kentucky)
• Inventors: Cheng, Fuhua
• Primary Examiner(s): MANDEVILLE, JASON M
• Application Number:
  US14/339,090
• Publication Number:
  US 20150029100A1
• Time in Patent Office: 1,056 Days
• Field of Search: 345156, 345157, 345158, 345163-166
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence

Abstract

Systems for surface-free pointing and/or command input include a computing device operably linked to an imaging device. The imaging device can be any suitable video recording device including a conventional webcam. At least one pointing/input device is provided including first, second, and third sets of actuable light sources, wherein at least the first and second sets emit differently colored light. The imaging device captures one or more sequential image frames each including a view of a scene including the activated light sources. One or more computer program products calculate a two-dimensional or three-dimensional position and/or a motion and/or an orientation of the pointing/input device in the captured image frames by identifying a two-dimensional or three-dimensional position of the activated light sources of the first, second, and/or third sets of light sources. Certain activation patterns of light sources are mapped to particular pointing and/or input commands.

Claims

1. A pointing/input device for effecting two-dimensional and three-dimensional pointing and/or command input using a single imaging device operatively connected to a computing device having at least one processor and at least one memory, comprising:

a housing;

a first set of light sources disposed on a first surface of the housing, at least one light source of the first set emitting a first light color when activated;

a second set of light sources disposed on the first surface of the housing, the second set emitting a second light color when activated that is different from the first color;

a third set of light sources disposed on a second surface of the housing, the third set emitting a light color when activated that may be the same as or different from the first or second color; and

one or more actuators for differently activating one or more of the light sources of the first, second, and third sets of light sources to provide a plurality of light source activation patterns corresponding to a plurality of pointing and/or input commands.

2. The pointing/input device of claim 1, wherein the first color is blue and the second color is red.

3. The pointing/input device of claim 1, including an actuator for activating the first set of light sources.

4. The pointing/input device of claim 1, including an actuator for activating the second set of light sources.

5. The pointing/input device of claim 1, including an actuator for activating the third set of light sources that may be the same as or different from the actuator for activating the first set of light sources.

6. The pointing/input device of claim 1, wherein light emitted by the first set and second set of light sources is visible to an imaging device when either of the first surface of the housing or the second surface of the housing is facing a field of view of the imaging device.

13. A computing system for tracking a motion and/or an orientation of a pointing/input device, comprising:

a computing device having at least one processor, at least one memory, and at least one graphical user interface;

at least one pointing/input device according to claim 1;

an imaging device operably linked to the computing device and configured for capturing a plurality of sequential image frames; and

at least one computer program product operable on the computing device processor, the at least one computer program product including executable instructions for determining at least a two-dimensional or three-dimensional position and/or a motion and/or an orientation of the at least one pointing/input device from digital data of the plurality of sequential image frames providing views of at least one activated light source of the first set, the second set, and the third set of light sources held and/or moved within a field of view of the imaging device, and for rendering on the graphical user interface a visual marker corresponding to the determined position and/or the motion and/or the orientation of the at least one pointing/input device.

14. The system of claim 13, wherein the imaging device is a web cam.

15. The system of claim 13, wherein the at least one computer program product includes executable instructions for mapping an activation pattern of at least one light source of the first set and/or the second set and/or the third set of light sources to a predetermined pointing and/or input command.

16. The system of claim 15, wherein the at least one computer program product further includes executable instructions for mapping and displaying in the graphical user interface a three-dimensional position and/or motion of the at least one pointing/input device, comprising:

determining a three-dimensional coordinate of each activated light source of the third set of light sources in each of said plurality of sequential image frames;

converting the determined three-dimensional coordinate into a three-dimensional position of the visual marker in the graphical user interface; and

displaying the three-dimensional position of the visual marker in the graphical user interface.

17. The system of claim 16, wherein the at least one computer program product includes executable instructions for determining the three-dimensional coordinate in each of said plurality of sequential image frames by:

calculating a depth of each activated light source of the third set of light sources with respect to a pre-determined origin of a three-dimensional coordinate system;

calculating an angle of a light beam emitted by each activated light source of the third set of light sources with respect to a projection plane defining a plane perpendicular to a z-axis of the three-dimensional coordinate system; and

determining a three-dimensional coordinate and orientation of the at least one pointing/input device from said calculated depth and angle.

7. In a computing system environment, a method for two-dimensional and three-dimensional pointing and/or command input, comprising:

providing at least one pointing/input device including a first set of light sources emitting a first light color when activated, a second set of light sources emitting a second light color when activated that is different from the first color, and a third set of light sources emitting a light color that may be the same as or different from the first or second color, the pointing/input device further including one or more actuators for differently activating one or more of the light sources of the first, second, and third sets of light sources to provide a plurality of light source activation patterns corresponding to a plurality of pointing and/or input commands;

while actuating the one or more actuators to provide the plurality of light source activation patterns, holding and/or moving the at least one pointing/input device within a field of view of an imaging device operably connected to a computing device having at least one processor, at least one memory, and at least one graphical user interface;

by the imaging device, capturing a plurality of sequential image frames each including a view of the at least one pointing/input device having at least one light source of the first set, the second set, and/or the third set of light sources activated to emit light;

from one or more views of at least one of the first set, the second set, and/or the third set of light sources in the captured plurality of sequential image frames, calculating a two-dimensional or a three-dimensional position and/or motion and/or orientation of the at least one pointing/input device; and

rendering a visual marker on the graphical user interface corresponding to the calculated position and/or motion and/or orientation of the at least one pointing/input device.

8. The method of claim 7, further including mapping an activation pattern of the first set and/or the second set and/or the third set of light sources to a predetermined pointing and/or input command.

9. The method of claim 7, including mapping and displaying in the graphical user interface a three-dimensional position and/or motion and/or orientation of the at least one pointing/input device, comprising:

determining a three-dimensional coordinate of each activated light source of the third set of light sources in each of said plurality of sequential image frames;

converting the determined three-dimensional coordinate into a three-dimensional position of the visual marker in the graphical user interface; and

displaying the three-dimensional position of the visual marker in the graphical user interface.

10. The method of claim 9, further including determining the three-dimensional coordinate by:

calculating a depth of each activated light source of the third set of light sources with respect to a pre-determined origin of a three-dimensional coordinate system;

calculating an angle of a light beam emitted by each activated light source of the third set of light sources with respect to a projection plane defining a plane perpendicular to a z-axis of the three-dimensional coordinate system; and

determining a three-dimensional coordinate and orientation of the at least one pointing/input device from said calculated depth and angle.

11. The method of claim 7, including calculating from the captured plurality of sequential image frames at least a spatial and a velocity parameter of the at least one pointing/input device when moved through three-dimensional space and rendering on the graphical user interface a visual marker corresponding to the calculated spatial and velocity parameter of the at least one pointing/input device.

12. The method of claim 11, including providing at least one computer program product comprising executable instructions operable on the computing device, for calculating from the captured image frames at least the spatial and velocity parameters of the at least one pointing/input device.