Vehicle operating system using motion capture

• US 10,281,989 B2
• Filed: 11/21/2016
• Issued: 05/07/2019
• Est. Priority Date: 03/16/2015
• Status: Active Grant

• Current Assignee: Thunder Power New Energy Vehicle Development Company Limited
• Original Assignee: Thunder Power New Energy Vehicle Development Company Limited
• Inventors: Hsiao, Jen-Chieh, Chen, Yong-Syuan
• Primary Examiner(s): Rahman, Mohammad J
• Application Number:
  US15/357,077
• Publication Number:
  US 20170235371A1
• Time in Patent Office: 897 Days
• Field of Search: None
• CPC Class Codes: B60K 2360/1438 Touch screens
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/21 using cameras
• CPC Class Codes: B60K 35/00 Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: B60K 35/22 Display screens
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/207 using multi-purpose displays, e.g. camera image and navigation or video on same display
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/8006 for monitoring and displaying scenes of vehicle interior, e.g. for monitoring passengers or cargo
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Vehicle operating systems for operating a vehicle having a driving seat for a vehicle driver and at least one passenger seat for passengers are described. The vehicle operating system may include one or more camera devices for shooting images of hand actions of the driver or images of hand actions of a passenger, and a storage device for storing operating signals corresponding to hand actions. A processing device may be configured to select the driver or the passengers as a gesture command operator, and to control the camera device to shoot hand action images of the selected gesture command operator. The command system may also be configured to convert the shot hand action images into corresponding operating signals according to hand action indicia stored in the storage device. The operating signals may be sent to execution devices, that execute the corresponding operations.

Claims

1. A vehicle operating system for operating a vehicle having multiple users inside the vehicle, the vehicle operating system comprising:

multiple camera devices for capturing at least one image for each of the multiple users including a first user and a second user, wherein the multiple camera devices include a first camera device corresponding to the first user and a second camera device corresponding to the second user;

a storage device for storing operating signals corresponding to gesture actions;

a processing device configured to

control the first and second camera devices to capture gesture action images for the multiple users simultaneously or substantially simultaneously, wherein the captured gesture action images include a first action image for the first user captured by the first camera device and a second action image for the second user captured by the second camera device, the first action image and the second action image being captured by the first and second camera devices simultaneously or substantially simultaneously,

convert the captured gesture action images into corresponding operating signals according to the operating signals corresponding to the gesture actions stored in the storage device,

determine the first user has performed a first gesture action based on the first action image and the second action image,

determine the first gesture action to be an active gesture action;

select a first operating signal from the first user based on the first gesture action by virtue of the first gesture action having been determined to be the active gesture action, and

send out the first operating signal; and

execution devices configured to receive the operating signals sent by the processing device, and to execute a first operation corresponding to the first operating signal.

2. The vehicle operating system of claim 1, wherein the processing device switches the gesture command operator from the driver to a passenger or from a passenger to the driver according to a switch instruction.

3. The vehicle operating system of claim 2, wherein the switch instruction is a hand action of the gesture command operator.

4. The vehicle operating system of claim 1, wherein the camera device captures an image from a body locating point of the gesture command operator to his hand, the processing device checks whether the gesture image and the body locating point image are image-continuous, and if so, the gesture actions are converted into the operating signals.

5. The vehicle operating system of claim 4, wherein the body locating point comprises an arm, head or breast of the gesture command operator.

6. The vehicle operating system of claim 1, wherein the camera device comprises a camera and a driving device for the camera, and the driving device configured for driving the camera to move or rotate.

7. The vehicle operating system of claim 6, wherein the driving device comprises a motor and a gear set.

8. The vehicle operating system of claim 1, wherein the camera device is arranged in front of the driving seat.

9. The vehicle operating system of claim 1, including a plurality of camera devices, at least one of the plurality of camera devices being arranged in front of the driving seat, and at least one camera device being arranged at a different location providing a view of the back seat.

10. The vehicle operating system of claim 1, wherein the execution devices comprise at least one of a multimedia system, a door switch, a refill opening switch, Bluetooth equipment, a light switch and an air-conditioning system.

11. The vehicle operating system of claim 1, wherein,

the execution devices comprise a center console provided with at least one touch screen, the touch screen at least extends from the front part between the driving seat and the front passenger seat to the front part of the front passenger seat,

a touch screen operating interface is displayed on the touch screen, and

the center console can control the touch screen operating interface to move on the touch screen according to an operating signal.

12. The vehicle operating system of claim 1, wherein,

each operating signal corresponds to a two-step hand action,

a first step of the hand action is used for selecting an execution device, and

a second step of the hand action is used for instructing the selected execution device to carry out an specific operation.

13. An electric vehicle, comprising:

a driving seat for a vehicle driver;

at least one passenger seat for passengers; and

a vehicle operating system including;

multiple camera devices for capturing at least one of images of gestures of the driver or images of gestures of a passenger, wherein the multiple camera devices include a first camera device corresponding to the first user and a second camera device corresponding to the second user;

a storage device for storing operating signals corresponding to gesture actions;

a processing device configured to

control first and second camera devices to capture gesture action images for the multiple users simultaneously or substantially simultaneously, wherein the captured gesture action images include a first action image for the first user captured by the first camera device and a second action image for the second user captured by the second camera device, the first action image and the second action image being captured by the first and second camera devices simultaneously or substantially simultaneously,

convert the captured gesture action images into corresponding operating signals according to the operating signals corresponding to the gesture actions stored in the storage device,

determine the first user has performed a first gesture action based on the first action image and the second action image,

determine the first gesture action to be an active gesture action;

select a first operating signal from the first user based on the first gesture action by virtue of the first gesture action having been determined to be the active gesture action, and

send out the first operating signal; and

an execution device configured to receive the operating signals sent by the processing device, and to execute a first operation corresponding to the first operating signal.