Camera based sensing in handheld, mobile, gaming, or other devices

• US 7,933,431 B2
• Filed: 07/12/2010
• Issued: 04/26/2011
• Est. Priority Date: 07/08/1999
• Status: Expired due to Fees

• Current Assignee: Gesture Technology Partners, LLC
• Original Assignee: Timothy R. Pryor
• Inventors: Pryor, Timothy R.
• Primary Examiner(s): Lu; Tom Y
• Application Number:
  US12/834,281
• Publication Number:
  US 20100277412A1
• Time in Patent Office: 288 Days
• Field of Search: 382/103, 382/104, 348/207.1
• US Class Current: 382/103
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/27 characterised by a large display in a public venue, e.g. in a movie theatre, stadium or game arena
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2009/2435 using a video camera
• CPC Class Codes: A63F 2300/1006 having additional degrees of freedom
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/8023 the game being played by multiple players at a common site, e.g. in an arena, theatre, shopping mall using a large public display
• CPC Class Codes: A63F 3/00643 Electric board games; Electric features of board games electric word or number games A63F3/0421; computer chess G06F; electric raffle games A63F3/081
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 : using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0425 : using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2207/10021 : Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/143 : Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Method and apparatus are disclosed to enable rapid TV camera and computer based sensing in many practical applications, including, but not limited to, handheld devices, cars, and video games. Several unique forms of social video games are disclosed.

Claims

1. A method for controlling a handheld computing device comprising the steps of:

holding said device in one hand;

moving at least one finger in space in order to signal a command to said device;

electro-optically sensing light reflected from said at least one finger using a sensing means associated with said device;

determining from said sensed light the movement of said finger, and

using said sensed finger movement information, controlling said device in accordance with said command.

2. A method according to claim 1, wherein at least one camera is utilized to effect said electro-optical sensing.

3. A method according to claim 1, including the further step of acquiring an image of at least a portion of the user of the device.

4. A method according to claim 1, wherein said movement is sensed in 3 dimensions.

5. A method according to claim 1, wherein movement of one finger relative to another finger is sensed.

6. A method according to claim 1, wherein movement of two fingers is sensed.

7. Handheld computer apparatus comprising:

a housing;

a camera means associated with said housing for obtaining an image using reflected light of at least one object positioned by a user operating said object;

computer means within said housing for analyzing said image to determine information concerning a position or movement of said object; and

means for controlling a function of said apparatus using said information.

8. Apparatus according to claim 7, wherein said object is a finger.

9. Apparatus according to claim 7, further including a display function which is controlled.

10. Apparatus according to claim 9, wherein said display is 3D display.

11. Apparatus according to claim 7, further including means for transmitting information.

12. Apparatus according to claim 7, further including a light source for illuminating said object.

13. Apparatus according to claim 7, wherein said apparatus is a cellular phone.

14. A method for controlling a handheld computing device comprising the steps of:

providing a computer within said device;

associating a camera with said device, said camera viewing at least a portion of the body of a user operating said device or an object held by said user, in order provide image data concerning said portion or object;

using said computer, analyzing said image data to determine information concerning a user input command; and

from said determined information, controlling a function of said device.

15. A method according to claim 14, wherein reflected light from said body portion or object is imaged by said camera.

16. A method according to claim 14, wherein said information includes the position of the portion or object.

17. A method according to claim 14, wherein said information includes the change in position of the portion or object.

18. A method according to claim 14, wherein said information includes the velocity or path of the portion or object.

19. A method according to claim 14, wherein said information is obtained in 3 dimensions.

20. A method according to claim 14, wherein said information includes the pointing direction of the portion or object.

21. A method according to claim 14, wherein a display is controlled.

22. A method according to claim 21, wherein a virtual image on said display is moved or changed.

23. A method according to claim 21, wherein said display is a 3D display.

24. A method according to claim 23, wherein said display is a stereoscopic display.

25. A method according to claim 14, including the further step of transmitting data to a further device.

26. A method according to claim 14, wherein said camera operates at 30 frames per second or greater.

27. A method according to claim 14, wherein said controlled function relates to a game.

28. A method according to claim 14, including the further step of acquiring a picture of the user of the handheld device.

29. A method according to claim 14, wherein two fingers of the user are sensed and a pinching action determined.

30. A method according to claim 14, wherein said body portion indicates an expression of said user.

31. A method according to claim 14, wherein said information provides an aid to speech recognition.