System, device, method and computer program product for using a mobile camera for controlling a computer

• US 20070177806A1
• Filed: 02/01/2006
• Published: 08/02/2007
• Est. Priority Date: 02/01/2006
• Status: Abandoned Application

• Current Assignee: Nokia Corporation
• Original Assignee: Nokia Corporation
• Inventors: Vartiainen, Panu
• Application Number:
  US11/345,437
• Publication Number:
  US 20070177806A1
• Time in Patent Office: Days
• US Class Current: 382/201
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06V 10/19 by sensing codes defining pattern positions

Abstract

A system for controlling a computing apparatus includes a mobile camera and a processing device. One of the mobile camera and the processing device is capable of determining a first relationship between a coordinate space of a media surface and a coordinate space of a camera image field at a first point in time, determining a first position of a pointing device relative to the coordinate space of the camera image field at the first point in time, determining a first position of the pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the pointing device relative to the coordinate space of the camera image field, and controlling a computer application based on the first position of the pointing device relative to the coordinate space of the media surface.

Claims

1. A system for controlling a computing apparatus, the system comprising:

a mobile camera; and

a processing device,

wherein one of the mobile camera and the processing device is capable of;

determining a first relationship between a coordinate space of a media surface and a coordinate space of a camera image field at a first point in time;

determining a first position of a pointing device relative to the coordinate space of the camera image field at the first point in time;

determining a first position of the pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on the first position of the pointing device relative to the coordinate space of the media surface.

2. The system of claim 1, wherein one of the mobile camera and the processing device is further capable of:

determining a second relationship between the coordinate space of the media surface and the coordinate space of the camera image field at a second point in time;

determining a second position of the pointing device relative to the coordinate space of the camera image field at the second point in time; and

determining a second position of the pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on a difference between the first and second positions of the pointing device relative to the coordinate space of the media surface.

3. The system of claim 2, wherein the pointing device is a human finger.

4. The system of claim 2, wherein controlling the computer application includes one of moving a cursor on a display screen of the computing apparatus, causing a mouse click, and causing a mouse click and hold and moving the cursor.

5. The system of claim 2, wherein the pointing device is a first pointing device, and wherein one of the mobile camera and the processing device is further capable of:

determining a first position of a second pointing device relative to the coordinate space of the camera image field at the first point in time;

wherein the camera is further capable of determining a second position of the second pointing device relative to the coordinate space of the camera image field at the second point in time;

determining the first position of the second pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the second pointing device relative to the coordinate space of the camera image field;

determining a second position of the second pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the second pointing device relative to the coordinate space of the camera image field; and

transmitting a control signal based on a difference between the first and second positions of the second pointing device relative to the coordinate space of the media surface, based on a difference between first positions of the first and second pointing devices, and based on a difference between the second positions of the first and second pointing devices.

6. A mobile camera for controlling a computing apparatus, the camera comprising a processing element capable of:

determining a first relationship between a coordinate space of a media surface and a coordinate space of a camera image field at a first point in time;

determining a first position of a pointing device relative to the coordinate space of the camera image field at the first point in time;

determining a first position of the pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on the first position of the pointing device relative to the coordinate space of the media surface.

7. The camera of claim 6, wherein one of the mobile camera and the processing device is further capable of:

determining a second relationship between the coordinate space of the media surface and the coordinate space of the camera image field at a second point in time;

determining a second position of the pointing device relative to the coordinate space of the camera image field at the second point in time; and

determining a second position of the pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on a difference between the first and second positions of the pointing device relative to the coordinate space of the media surface.

8. The camera of claim 7, wherein the pointing device is a human finger.

9. The camera of claim 7, wherein the control signal controls the computing apparatus by one of moving a cursor on a display screen of the computer, causing a mouse click, and causing a mouse click and hold and moving the cursor.

10. The camera of claim 7, wherein the pointing device is a first pointing device, and wherein the processing element is further capable of determining a first position of a second pointing device relative to the coordinate space of the camera image field at the first point in time;

wherein the processing element is further capable of determining a second position of the second pointing device relative to the coordinate space of the camera image field at the second point in time;

wherein the processing element is further capable of determining a first position of the second pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the second pointing device relative to the coordinate space of the camera image field;

wherein the processing element is further capable of determining a second position of the second pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the second pointing device relative to the coordinate space of the camera image field; and

wherein the processing element transmits the computer control signal based on a difference between the first and second positions of the second pointing device relative to the coordinate space of the media surface, based on a difference between first positions of the first and second pointing devices, and based on a difference between the second positions of the first and second pointing devices.

11. A method of controlling a computing apparatus, the method comprising:

determining a first relationship between a coordinate space of a media surface and a coordinate space of a camera image field at a first point in time;

determining a first position of a pointing device relative to the coordinate space of the camera image field at the first point in time;

determining a first position of the pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on the first position of the pointing device relative to the coordinate space of the media surface.

12. The method of claim 11, wherein one of the mobile camera and the processing device is further capable of:

determining a second relationship between the coordinate space of the media surface and the coordinate space of the camera image field at a second point in time;

determining a second position of the pointing device relative to the coordinate space of the camera image field at the second point in time; and

determining a second position of the pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on a difference between the first and second positions of the pointing device relative to the coordinate space of the media surface.

13. The method of claim 12, wherein the pointing device is a human finger.

14. The method of claim 12, wherein controlling the computing apparatus comprises one of moving a cursor on a display screen of the computing apparatus, causing a mouse click, and causing a mouse click and hold and moving the cursor.

15. The method of claim 12, wherein the pointing device is a first pointing device, and wherein the method further comprises:

determining a first position of a second pointing device relative to the coordinate space of the camera image field at the first point in time;

determining a second position of the second pointing device relative to the coordinate space of the camera image field at the second point in time;

determining a first position of the second pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the second pointing device relative to the coordinate space of the camera image field;

determining a second position of the second pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the second pointing device relative to the coordinate space of the camera image field; and

controlling the computing apparatus based on a difference between the first and second positions of the second pointing device relative to the coordinate space of the media surface, based on a difference between first positions of the first and second pointing devices, and based on a difference between the second positions of the first and second pointing devices.

16. A computer program product for using a mobile camera for controlling a computer, the computer program product comprising at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion capable of determining a first relationship between a coordinate space of a media surface and a coordinate space of a camera image field at a first point in time;

a second executable portion capable of determining a first position of the pointing device relative to the coordinate space of the camera image field at the first point in time;

a third executable portion capable of determining a first position of the pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the pointing device relative to the coordinate space of the camera image field; and

a fourth executable portion capable of controlling a computer application based on the first position of the pointing device relative to the coordinate space of the media surface.

17. The computer program product of claim 16, wherein the computer-readable program code portions further comprise:

a fifth executable portion capable of determining a second relationship between the coordinate space of the media surface and the coordinate space of the camera image field at a second point in time;

a sixth executable portion capable of determining a second position of the pointing device relative to the coordinate space of the camera image field at the second point in time;

a seventh executable portion capable of determining a second position of the pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the pointing device relative to the coordinate space of the camera image field; and

a eighth executable portion capable of controlling the computer based on a difference between the first and second positions of the pointing device relative to the coordinate space of the media surface.

18. The computer program product of claim 17, wherein the pointing device is a human finger.

19. The computer program product of claim 17, wherein the eighth executable portion controls the computer by one of moving a cursor on a display screen of the computer, causing a mouse click, and causing a mouse click and hold and moving the cursor.

20. The computer program product of claim 17, wherein the pointing device is a first pointing device, and wherein the computer program product further comprises:

an ninth executable portion capable of determining a first position of a second pointing device relative to the coordinate space of the camera image field at the first point in time;

a tenth executable portion capable of determining a second position of the second pointing device relative to the coordinate space of the camera image field at the second point in time;

a eleventh executable portion capable of determining a first position of the second pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the second pointing device relative to the coordinate space of the camera image field;

an twelfth executable portion capable of determining a second position of the second pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the second pointing device relative to the coordinate space of the camera image field; and

a thirteenth executable portion capable of controlling the computer based on a difference between the first and second positions of the second pointing device relative to the coordinate space of the media surface, based on a difference between first positions of the first and second pointing devices, and based on a difference between the second positions of the first and second pointing devices.

21. A processing device for controlling a computer, the device comprising a processing element capable of:

determining a first relationship between a coordinate space of a media surface and a coordinate space of a camera image field at a first point in time;

determining a first position of a pointing device relative to the coordinate space of the camera image field at the first point in time;

determining a first position of the pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on the first position of the pointing device relative to the coordinate space of the media surface.

22. The device of claim 21, wherein one of the mobile camera and the processing device is further capable of:

determining a second relationship between the coordinate space of the media surface and the coordinate space of the camera image field at a second point in time;

determining a second position of the pointing device relative to the coordinate space of the camera image field at the second point in time; and

determining a second position of the pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the pointing device relative to the coordinate space of the camera image field; and

controlling a computer application based on a difference between the first and second positions of the pointing device relative to the coordinate space of the media surface.

23. The device of claim 22, wherein the pointing device is a human finger.

24. The device of claim 22, wherein the processing element controls the computer application by one of moving a cursor on a display screen of the computer, causing a mouse click, and causing a mouse click and hold and moving the cursor.

25. The device of claim 22, wherein the pointing device is a first pointing device, and wherein the processing element is further capable of receiving from the mobile camera a first position of a second pointing device relative to the coordinate space of the camera image field at the first point in time;

wherein the processing element is further capable of receiving from the digital camera a second position of the second pointing device relative to the coordinate space of the camera image field at the second point in time;

wherein the processing element is further capable of determining a first position of the second pointing device relative to the coordinate space of the media surface at the first point in time based on the first relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the first position of the second pointing device relative to the coordinate space of the camera image field;

wherein the processing element is further capable of determining a second position of the second pointing device relative to the coordinate space of the media surface at the second point in time based on the second relationship between the coordinate space of the media surface and the coordinate space of the camera image field and based on the second position of the second pointing device relative to the coordinate space of the camera image field; and

wherein the processing element controls the computer application based on a difference between the first and second positions of the second pointing device relative to the coordinate space of the media surface, based on a difference between first positions of the first and second pointing devices, and based on a difference between the second positions of the first and second pointing devices.