CAMERA BASED SENSING IN HANDHELD, MOBILE, GAMING OR OTHER DEVICES

• US 20110170746A1
• Filed: 03/18/2011
• Published: 07/14/2011
• Est. Priority Date: 07/08/1999
• Status: Active Grant

• Current Assignee: Gesture Technology Partners, LLC
• Original Assignee: Timothy R. Pryor
• Inventors: Pryor, Timothy R.
• Granted Patent: US 8,194,924 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/27 characterised by a large display in a public venue, e.g. in a movie theatre, stadium or game arena
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2009/2435 using a video camera
• CPC Class Codes: A63F 2300/1006 having additional degrees of freedom
• CPC Class Codes: A63F 2300/1025 details of the interface with the game device, e.g. USB version detection
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/8023 the game being played by multiple players at a common site, e.g. in an arena, theatre, shopping mall using a large public display
• CPC Class Codes: A63F 3/00643 Electric board games; Electric features of board games electric word or number games A63F3/0421; computer chess G06F; electric raffle games A63F3/081
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 : using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0425 : using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2207/10021 : Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/143 : Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Method and apparatus are disclosed to enable rapid TV camera and computer based sensing in many practical applications, including, but not limited to, handheld devices, cars, and video games. Several unique forms of social video games are disclosed.

Claims

1. A handheld device comprising:

a housing;

first and second cameras supported by the housing and providing an output; and

a computer within the housing operable to determine position or orientation information concerning an object based on the output of at least one of the first and second cameras.

2. The device of claim 1 wherein the first and second cameras are fixed with respect to the housing.

3. The device of claim 1 wherein the first and second cameras having overlapping fields of view.

4. The device of claim 1 wherein the position or orientation information is determined based on the output of the first and second cameras.

5. The device of claim 1 wherein the object is a finger of the user.

6. The device of claim 1 wherein the device comprises a mobile phone.

7. The device of claim 1 wherein the computer is operable to determine a control input based on the object position.

8. The device of claim 1 wherein the computer is operable to determine a control input based on the object orientation.

9. The device of claim 1 wherein the housing is adapted to be held in the hand of a user.

10. A method for providing a control input, the method comprising:

providing a handheld device including a computer and first and second cameras having overlapping fields of view;

determining, using the computer, information concerning an object viewed by the first and second cameras within the overlapping fields of view; and

controlling the handheld device using the determined information.

11. The method of claim 10 wherein determining information concerning an object includes identifying a gesture.

12. The method of claim 10 wherein the determining step includes determining the orientation or position of the object in three-dimensions.

13. The method of claim 10 wherein the object includes at least a portion of a user of the handheld device.

14. The method of claim 10 wherein the first and second cameras are fixed with respect to the housing.

15. The method of claim 10 wherein the handheld device comprises a mobile phone.

16. The method of claim 10 further including:

obtaining an image of the object using the first and second cameras; and

displaying a three-dimensional image of the object.

17. A handheld device comprising:

a housing;

first and second cameras supported by the housing; and

a computer operable to determine a control input based on an object imaged by at least one of the first and second cameras.

18. The device of claim 17 wherein the first and second cameras having overlapping fields of view.

19. The device of claim 17 wherein the housing is adapted to be held in a hand of a user.

20. The device of claim 17 wherein the first and second cameras are fixed with respect to the housing.

21. The device of claim 17 wherein the object includes a portion of a user of the handheld device.

22. The device of claim 17 wherein the object is a writing instrument held in the hand of a user.

23. The device of claim 17 wherein the device comprises a mobile phone.

24. A handheld device comprising:

a housing;

a computer within the housing;

a first camera oriented to view a user of the handheld device; and

a second camera oriented to view an object other than the user of the device.

25. The handheld device of claim 24 wherein the handheld device comprises a mobile phone.

26. The handheld device of claim 24 wherein the first camera is adapted to acquire an image of a portion of the user.

27. The handheld device of claim 24 wherein the second camera is adapted to acquire an image of the object.

28. The handheld device of claim 24 wherein the second camera is adapted to acquire a video of the object.

29. The handheld device of claim 24 wherein:

at least one of the first and second cameras includes an output; and

the computer is operable to determine a gesture based on the output.

30. The handheld device of claim 24 wherein:

at least one of the first and second cameras includes an output; and

the computer is operable to determine an expression based on the output.