Dynamically controlling a cursor on a screen when using a video camera as a pointing device

• US 8,144,123 B2
• Filed: 08/14/2007
• Issued: 03/27/2012
• Est. Priority Date: 08/14/2007
• Status: Expired due to Fees

• Current Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Denoue, Laurent, Adcock, John E., Golovchinsky, Gene
• Primary Examiner(s): Eisen, Alexander
• Assistant Examiner(s): MARINELLI, PATRICK
• Application Number:
  US11/838,817
• Publication Number:
  US 20090046061A1
• Time in Patent Office: 1,687 Days
• Field of Search: 345156-184, 345/104, 345204-215, 178 1801- 2004, 715/710, 715/711, 715856-862
• US Class Current: 345/157
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A system provides for controlling a cursor on a screen automatically and dynamically when using a video camera as a pointing device. A computer displays static or dynamic content to a screen. A video camera connected to the computer points at the screen. As the video camera films the screen, frames captured by the video camera are sent to the computer. A target image is displayed by the computer onto the screen and marks the position of the screen cursor of the video camera. Frames captured by the video camera include the target image, and the computer dynamically moves the target image on the screen to ensure that the target image stays in the center of the view of the video camera.

Claims

1. A method for controlling a cursor on a screen when using a video camera as a pointing device, the method comprising:

at a computer system including one or more processors and memory storing one or more programs, the one or more processors executing the one or more programs to perform the operations of;

displaying the cursor on the screen;

receiving a respective frame captured by the video camera, wherein the respective frame includes the cursor; and

in response to receiving the respective frame, automatically, without user intervention;

detecting a size and location of the cursor in the respective frame;

moving the cursor on the screen in accordance with the detected location of the cursor in the respective frame; and

adjusting a displayed size of the cursor on the screen in accordance with the detected size of the cursor in the respective frame, wherein the adjusting includes;

increasing the displayed size of the cursor on the screen in accordance with a determination that the detected size of the cursor in the respective frame is smaller than a predefined value, and

decreasing the displayed size of the cursor on the screen in accordance with a determination that the detected size of the cursor in the respective frame is larger than the predefined value.

2. The method of claim 1, further comprising changing the display around the cursor to ensure the cursor remains detectable on the screen.

3. The method of claim 1, further comprising dynamically adjusting the size and opacity of the cursor so that the cursor remains detectable when the video camera is moved away from the screen.

4. The method of claim 1, further comprising estimating, in accordance with an appearance of the cursor in the respective frame, a distance between the video camera and the screen, the estimation used in a computation for moving the cursor.

5. The method of claim 1, further comprising estimating, in accordance with an appearance of the cursor in the respective frame, one or more of yaw, pitch, and roll of the video camera relative to the screen, the estimation used in a computation for moving the cursor.

6. The method of claim 1, further comprising manually initializing the system, after determining that a user has pointed the video camera at the cursor on the screen.

7. The method of claim 1, further comprising automatically initializing the system by displaying the cursor onto different areas of the screen until the cursor appears in a frame captured by the video camera.

8. The method of claim 1, further comprising automatically initializing the system by predicting where to move the cursor by a recursive function that determines a first quadrant of the screen that appears in a particular frame captured by the video camera, then determines a second quadrant of the first quadrant that appears in the particular frame, and so on, until the quadrant size is equal to the cursor size.

9. The method of claim 1, further comprising automatically initializing the system by displaying the cursor at a predefined area on the screen such that the cursor appears in the respective frame captured by the video camera pointed at the predefined area.

10. The method of claim 1, further comprising identifying by the computer for one or more additional screens where the screens are located in relation to each other.

11. The method of claim 10, wherein identifying where the screens are located in relation to each other further comprises determining if the video camera that points to a first screen is moved to point to a second screen, that the cursor follows from the first screen to the second screen.

12. The method of claim 1, further comprising controlling by one or more additional video cameras of their respective cursors, each cursor distinguished by one or more of color, shape, and texture.

13. A system for controlling a cursor on a screen using a video camera as a pointing device, the system comprising:

one or more processors;

memory;

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

displaying the cursor on the screen;

receiving a respective frame captured by the video camera, wherein the respective frame includes the cursor; and

in response to receiving the respective frame, automatically, without user intervention;

detecting a size and location of the cursor in the respective frame;

moving the cursor on the screen in accordance with the detected location of the cursor in the respective frame; and

adjusting a displayed size of the cursor on the screen in accordance with the detected size of the cursor in the respective frame, wherein the adjusting includes;

increasing the displayed size of the cursor on the screen in accordance with a determination that the detected size of the cursor in the respective frame is smaller than a predefined value, and

decreasing the displayed size of the cursor on the screen in accordance with a determination that the detected size of the cursor in the respective frame is larger than the predefined value.

14. The system of claim 13, further comprising instructions for changing the display around the cursor to ensure the cursor remains detectable.

15. The system of claim 13, further comprising instructions for dynamically adjusting the size and opacity of the cursor so that the cursor remains detectable when the video camera is moved away from the screen.

16. The system of claim 13, further comprising instructions for estimating, in accordance with an appearance of the cursor in the respective frame, a distance between the video camera and the screen that is used in a computation of moving the cursor, the distance used in a calculation to determine the movement of the cursor.

17. The system of claim 13, further comprising instructions for estimating, in accordance with an appearance of the cursor in the respective frame, on one or more of yaw, pitch, and roll of the video camera relative to the screen, the position used in a calculation to determine the movement of the cursor.

18. The system of claim 13, further comprising instructions for manually initializing the system, after determining that a user has pointed the video camera at the cursor.

19. The system of claim 13, further comprising instructions for automatically initializing the system by displaying the cursor onto different areas of the screen until the cursor appears in a frame captured by the video camera.

20. The system of claim 13, further comprising instructions for automatically initializing the system by predicting where to move the cursor by a recursive function that determines a first quadrant of the screen that appears in a particular frame captured by the video camera, then determines a second quadrant of the first quadrant that appears in the particular frame, and so on, until the quadrant size is equal to the cursor size.

21. The system of claim 13, further comprising instructions for automatically initializing the system by displaying the cursor at a predefined area on the screen such that the cursor appears in the respective frame captured by the video camera pointed at the predefined area.

22. The system of claim 13, further comprising one or more additional screens such that the computer identifies where the screens are located in relation to each other.

23. The system of claim 22, wherein the identification of the screen locations in relation to each other is performed if the video camera that points to a first screen is moved to point to a second screen, and the cursor follows from the first screen to the second screen.

24. The system of claim 13, further comprising one or more additional video cameras that control their respective cursors, each cursor distinguished by one or more of color, shape, and texture.

25. A non-transitory computer readable storage medium and one or more computer programs embedded therein, the one or more computer programs comprising instructions, which when executed by a computer system, cause the computer system to control a cursor on a screen using a video camera as a pointing device, comprising the steps of:

displaying the cursor on the screen;

receiving a respective frame captured by the video camera, wherein the respective frame includes the cursor; and

in response to receiving the respective frame, automatically, without user intervention;

detecting a size and location of the cursor in the respective frame;

moving the cursor on the screen in accordance with the detected location of the cursor in the respective frame; and

adjusting a displayed size of the cursor on the screen in accordance with the detected size of the cursor in the respective frame, wherein the adjusting includes;

increasing the displayed size of the cursor on the screen in accordance with a determination that the detected size of the cursor in the respective frame is smaller than a predefined value, and

decreasing the displayed size of the cursor on the screen in accordance with a determination that the detected size of the cursor in the respective frame is larger than the predefined value.