Neural network systems for vehicles

• US 7,660,437 B2
• Filed: 02/28/2008
• Issued: 02/09/2010
• Est. Priority Date: 05/05/1992
• Status: Expired due to Fees

• Current Assignee: American Vehicular Sciences LLC (Acacia Research Corporation)
• Original Assignee: Automotive Technologies International Incorporated
• Inventors: Breed, David S.
• Primary Examiner(s): Do; Anh Hong
• Application Number:
  US12/039,427
• Publication Number:
  US 20080144944A1
• Time in Patent Office: 712 Days
• Field of Search: 382/100, 382/312, 382/314, 382/315, 382/103, 382/104, 382/159, 709/210, 235/380, 701/45, 702/143, 702/149, 280/734, 280/735, 340/937, 340/933, 348/148, 180/274
• US Class Current: 382/104
• CPC Class Codes: G06V 20/59 inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Method for obtaining information about an occupying item in a space in a vehicle in accordance with the invention includes obtaining images of an area above a seat in the vehicle in which the occupying item is situated and classifying the occupying item by inputting signals derived from the images into a trained neural network form which is trained to output an indication of the class of occupying item from one of a predetermined number of possible classes. The method is applicable for various vehicles including automobiles, trucks, buses, airplanes and boats. The images may be pre-processed to remove background portions of the images and then converted into signals for input into the neural network form.

Claims

1. A method for obtaining information about an occupying item in a space in a vehicle, comprising:

arranging an image-obtaining device in the vehicle in a position in which it obtains images of the space when operating;

using the image-obtaining device, obtaining images of an area in the vehicle in which the occupying item may be situated, the area being part of the space in the vehicle in which the occupying item may be situated; and

using a processor coupled to the image-obtaining device, classifying the occupying item by inputting signals derived from the images obtained by the image-obtaining device into a trained neural network form embodied in the processor, the trained neural network form being trained to output an indication of a class of occupying item from one of a predetermined number of possible classes of occupying items, the class of the occupying item constituting information about the occupying item.

2. The method of claim 1, further comprising:

using a processor for pre-processing the images to remove background portions of the images to thereby obtained processed images; and

then

using a processor for converting the processed images into signals for input into the neural network form, whereby the images prior to pre-processing are not input into the neural network form.

3. The method of claim 2, wherein the step of using the processor for pre-processing the images comprises performing edge analysis of the images to identify a region of interest of the occupying item and edges of the region of interest of the occupying item and then removing portions of the images outside of the edges of the region of interest of the occupying item.

4. The method of claim 1, further comprising:

using a processor for analyzing multiple images obtained from the same image-obtaining device;

then

using a processor for pre-processing each of the images by removing portions of the images which are the same in multiple images to thereby obtain processed images; and

then

using a processor for converting the processed images into signals for input into the neural network form, whereby the images prior to pre-processing are not input into the neural network form.

5. The method of claim 1, wherein the neural network form is comprises a modular neural network that comprises a plurality of neural networks.

6. The method of claim 5, wherein at least one of the neural networks of the modular neural networks includes a plurality of submodular neural networks.

7. The method of claim 5, wherein the step of using the processor for classifying the occupying item comprises training each of the modular neural networks to determine whether the occupying item is one of a predetermined number of known occupying items.

8. The method of claim 1, wherein the vehicle includes a seat on which the occupying item may be present, the step of obtaining images of the area in the vehicle in which the occupying item may be situated comprising obtaining images of an area above the seat in which the occupying item may be situated, the area above the seat being part of the space in the vehicle in which the occupying item may be situated, the step of using the processor for classifying the occupying item comprising:

forming different subsystems of the neural network form, each of the different subsystems being embodied on computer-readable media accessible by the processor;

directing images obtained by the image-obtaining device to one of the different subsystems as a function of light conditions in the vehicle; and

in each subsystem, providing a first neural network to determine whether there is an occupying item on the seat, providing a second neural network to determine whether there is an infant or child seat on the seat and providing a third neural network to determine whether there is an adult on the seat.

9. The method of claim 1, wherein the neural network form is a combination neural network including a plurality of neural networks, each of the plurality of neural networks being arranged to classify the occupying item using different input or to provide a different output.

10. The method of claim 1, wherein the vehicle is selected from a group consisting of an automobile, a truck, a bus, an airplane and a boat.

11. The method of claim 1, wherein the occupying item is a human occupant, further comprising controlling actuation of an occupant protection or restraint device based on the classification of the occupant.

12. The method of claim 1, wherein the step of using the processor for classifying the occupying item comprises determining whether the occupying item is identifiable as one of a number of predetermined possible occupying items.

13. A method of classifying a vehicle occupant or a state of the vehicle occupant into one of a first category and a second category in a vehicle having an array of sensors and a controller connected to the sensors for embodying the method, comprising:

providing a processor for obtaining respective sensor signals from the sensors;

enabling the processor to interact with a neural network form embodied on computer-readable media and trained to provide, upon input of sensor signals, a first or second value as a decision about a classification of the vehicle occupant;

inputting the obtained sensor signals into the neural network form; and

using the processor for determining the first or second value via the neural network form, determination and output of the first value being indicative of the vehicle occupant or the state of the vehicle occupant being classified as being in the first category and a determination and output of the second value being indicative of the vehicle occupant or the state of the vehicle occupant being classified as being in the second category.

14. The method of claim 13, wherein the sensors are optical sensors arranged to obtain images of the vehicle occupant, further comprising using a processor for converting the obtained images into sensor signals to be input into the neural network form.

15. The method of claim 14, further comprising:

using a processor for pre-processing the images to remove background portions of the images to thereby obtain processed images; and

then

using a processor for converting the processed images into signals for input into the neural network form, whereby the images prior to pre-processing are not input into the neural network form.

16. The method of claim 14, wherein images of the vehicle occupant in a seat of the vehicle are obtained by the optical sensors, further comprising:

forming, in the processor, different subsystems of the neural network form, each of the different subsystems being embodied on computer-readable media accessible by the processor;

directing sensor signals derived from the images to one of the different subsystems as a function of light conditions in the vehicle; and

in each subsystem, providing a first neural network to determine whether there is an occupant on the seat, providing a second neural network to determine whether there is an infant or child seat on the seat and providing a third neural network to determine whether there is an adult on the seat.

17. The method of claim 13, wherein the neural network form comprises a modular neural network that comprises a plurality of neural networks.

18. The method of claim 17, wherein at least one of the neural networks of the modular neural networks includes a plurality of submodular neural networks.

19. The method of claim 17, further comprises training each of the modular neural networks to determine whether the occupant is one of a predetermined number of known occupant types and output either the first value or the second value depending on whether the occupant is of the occupant type for which the modular neural network is trained.

20. The method of claim 13, wherein the neural network form is a combination neural network including a plurality of neural networks, each of the plurality of neural networks being arranged to classify the occupying item using different input or to provide a different output.

21. The method of claim 13, further comprising controlling actuation of an occupant protection or restraint device based on the output of the first or second value from the neural network form.