INTERACTING WITH A MOBILE DEVICE WITHIN A VEHICLE USING GESTURES

• US 20130155237A1
• Filed: 12/16/2011
• Published: 06/20/2013
• Est. Priority Date: 12/16/2011
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bahl, Paramvir, Foehr, Oliver H., Paek, Timothy S.
• Application Number:
  US13/327,787
• Publication Number:
  US 20130155237A1
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 2360/148 Instrument input by voice
• CPC Class Codes: B60K 2360/164 Infotainment
• CPC Class Codes: B60K 2360/166 Navigation
• CPC Class Codes: B60K 2360/167 Vehicle dynamics information
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 2360/334 Projection means
• CPC Class Codes: B60K 2360/566 Mobile devices displaying vehicle information
• CPC Class Codes: B60K 2360/573 Mobile devices controlling vehicle functions
• CPC Class Codes: B60K 2360/583 Data transfer between instruments
• CPC Class Codes: B60K 2360/5899 Internet
• CPC Class Codes: B60K 2360/595 Data transfer involving internal databases
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/26 using acoustic output
• CPC Class Codes: B60K 35/265 Voice
• CPC Class Codes: B60K 35/28 characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
• CPC Class Codes: B60K 35/80 Arrangements for controlling instruments
• CPC Class Codes: B60K 35/85 Arrangements for transferring vehicle- or driver-related data
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/1686 : the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/048 : Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A mobile device is described herein which includes functionality for recognizing gestures made by a user within a vehicle. The mobile device operates by receiving image information that captures a scene including objects within an interaction space. The interaction space corresponds to a volume that projects out from the mobile device in a direction of the user. The mobile device then determines, based on the image information, whether the user has performed a recognizable gesture within the interaction space, without touching the mobile device. The mobile device can receive the image information from a camera device that is an internal component of the mobile device and/or a camera device that is a component of a mount which secures the mobile device within the vehicle. In some implementations, one or more projectors provided by the mobile device and/or the mount may illuminate the interaction space.

Claims

1. A method for recognizing gestures using a mobile device that is mounted in a vehicle, the mobile device functioning as a handheld mobile device when not mounted in the vehicle, comprising:

receiving image information from at least one camera device,

the image information capturing a scene that includes an interaction space as part thereof, the interaction space comprising a volume having prescribed dimensions that projects out from the mobile device in a direction of a user who is operating the vehicle; and

determining, using a gesture recognition module, whether the user has performed a recognizable gesture within the interaction space, based on the image information,

wherein the gesture comprises one or more of;

(a) a static pose made with at least one hand of the user without touching the mobile device; and

(b) a dynamic movement made with said at least one hand of the user without touching the mobile device.

2. The method of claim 1, wherein said determining comprises:

generating depth information based on the image information using a depth reconstruction technique; and

extracting a representation of said at least one hand that is positioned within the interaction space, based on the depth information.

3. The method of claim 1, wherein said determining comprises:

projecting one or more beams of electromagnetic radiation, said one or more beams defining a region of increased relative illumination; and

extracting a representation of said at least one hand that is positioned within the interaction space by detecting an object having increased relative brightness in the image information.

4. The method of claim 1, wherein said at least one camera is a component of the mobile device.

5. The method of claim 1, wherein said at least one camera is a component of a mount that secures the mobile device within the vehicle.

6. The method of claim 1, wherein said receiving of image information is performed in conjunction with irradiating the interaction space with electromagnetic radiation, using at least one projector.

7. The method of claim 6, wherein said at least one projector is a component of the mobile device.

8. The method of claim 6, wherein said at least one projector is a component of a mount that secures the mobile device within the vehicle.

9. The method of claim 1, wherein said at least one camera device produces the image information in response to receipt of infrared spectrum radiation.

10. The method of claim 1, wherein said at least one camera device contains a bandpass filter that diminishes visible spectrum radiation.

11. The method of claim 1, further comprising defining the interaction space in a calibration procedure prior to said determining of the recognizable gesture.

12. The method of claim 1, further comprising:

assessing performance of the gesture recognition module, to provide an assessed performance; and

dynamically adjusting at least one operational setting of the gesture recognition module based on the assessed performance.

13. The method of claim 12, wherein said at least one operational setting is selected from:

at least one parameter that affects projection of electromagnetic radiation into the interaction space by at least one projector;

at least one parameter that affects receipt of the image information by said at least one camera device; and

a mode of image capture used by the gesture recognition module to recognize gestures.

14. The method of claim 1, further comprising performing a control action in response to determining that the user has performed the gesture, the control action affecting a manner of operation of the mobile device.

15. The method of claim 14, wherein the gesture is associated with a voice recognition mode, and wherein said performing of the control action comprises activating the voice recognition mode in response to determining that the user has performed the gesture.

16. A mobile device for use within a vehicle, comprising:

input functionality configured to receive image information regarding objects within a scene, the scene including, as part thereof, an interaction space, the interaction space projecting out a prescribed distance from the mobile device within the vehicle,

the image information originating from one or more of;

an internal camera device that is an internal component of the mobile device; and

an external camera device that is a component of a mount which secures the mobile device within the vehicle; and

the input functionality also including a gesture recognition module configured to determine whether a user has made a gesture within the interaction space, based on one or more of;

depth information that is generated from the image information using a depth reconstruction technique; and

the image information itself without consideration of the depth information,

wherein the gesture comprises one or more of;

(a) a static pose made with at least one hand of the user without touching the mobile device; and

(b) a dynamic movement made with said at least one hand of the user without touching the mobile device.

17. A mount for holding a mobile device, comprising:

a cradle for securing the mobile device; and

an imaging member including external camera functionality, the external camera functionality comprising;

at least one external camera device for receiving image information, the image information capturing a scene that includes an interaction space as part thereof, the interaction space comprising a volume having prescribed dimensions that projects out from the mobile device; and

an interface for providing the image information to input functionality provided by the mobile device.

18. The mount of claim 17, further comprising at least one projector for projecting electromagnetic radiation into the interaction space.

19. The mount of claim 17, further comprising image processing functionality for processing the image information.

20. The mount of claim 19, wherein the image processing functionality is configured to generate depth information based on the image information using a depth reconstruction technique.