Pedestrian behavior predicting device and pedestrian behavior predicting method

• US 9,904,846 B2
• Filed: 09/20/2011
• Issued: 02/27/2018
• Est. Priority Date: 09/20/2011
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Kindo, Toshiki, Ogawa, Masahiro, Funayama, Ryuji
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Summers, Geoffrey E
• Application Number:
  US14/346,015
• Publication Number:
  US 20140219505A1
• Time in Patent Office: 2,352 Days
• Field of Search: None
• CPC Class Codes: B60W 2420/40 Photo, light or radio wave sensitive means, e.g. infrared sensors
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: G08G 1/08 according to detected number or speed of vehicles
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

According to the present invention, a pedestrian is detected from an imaged image and a partial image including the pedestrian is extracted, shape information of the pedestrian acquired from the extracted partial image is accumulated and the shape information of a predetermined time before and the current shape information are compared using the accumulated shape information to detect change in the movement of the pedestrian, discontinuous movement estimating information indicating a discontinuous movement of the pedestrian that occurs following the change in the movement of the pedestrian is acquired from a storage means at the time the change in the movement of the pedestrian is detected, and a behavior of the pedestrian is predicted using the acquired discontinuous movement estimating information.

Claims

1. A pedestrian behavior predicting device comprising:

a processor including hardware; and

a storage unit including memory storing instructions executable by the processor and for storing shape information of a pedestrian, wherein the processor executes the instructions stored in the memory to cause the pedestrian behavior predicting device to;

detect a pedestrian from an imaged image;

acquired at a current time

extract a partial image including the detected pedestrian from the imaged image;

acquire shape information of the pedestrian from the extracted partial image, the shape information of the pedestrian including at least one of a luminance of the partial image, an edge of the partial image, and a color of the partial image;

accumulate the acquired shape information of the pedestrian in the storage unit;

detect a change in movement of the pedestrian by comparing shape information of the pedestrian from an image captured a predetermined time before the current time and the acquired shape information of the pedestrian using the accumulated shape information of the pedestrian,

store first images and pieces of discontinuous movement estimating information in the storage unit such that each first image is associated with each piece of discontinuous movement estimating information, the first image being an image that was acquired at a time when a first change in the movement of the pedestrian is detected, the piece of discontinuous movement estimating information being information indicating a discontinuous movement of the pedestrian that occurs following the first change in the movement of the pedestrian;

wherein when a second change in the movement of the pedestrian is detected, search, from the first images stored in the storage unit, a first image corresponding to a second image when the second change in the movement of the pedestrian is detected;

acquire a piece of discontinuous movement estimating information associated with the searched first image; and

predict a behavior of the pedestrian using the acquired piece of discontinuous movement estimating information.

2. The pedestrian behavior predicting device according to claim 1, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

detect the change in the movement of the pedestrian by acquiring a feature amount distribution representing the shape information of the pedestrian acquired from the extracted partial image, normalize the shape information represented by the acquired feature amount distribution, accumulate the normalized shape information, and compare the shape information of the image captured a predetermined time before the current time and the acquired shape information using the accumulated shape information.

3. The pedestrian behavior predicting device according to claim 2, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

calculate an optical flow of the imaged image and acquire the discontinuous movement estimating information using the optical flow at the time the first change in the movement of the pedestrian is detected.

4. The pedestrian behavior predicting device according to claim 3, wherein

the discontinuous movement estimating information includes at least one of a moving direction and a moving speed of the pedestrian.

5. The pedestrian behavior predicting device according to claim 2, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

acquire a position of the pedestrian on the imaged image,

generate continuous movement estimating information indicating a continuous movement of the pedestrian that occurs following movement of the position of the pedestrian based on a history of positions of the pedestrian acquired, and

predict the behavior of the pedestrian based on the continuous movement estimating information generated.

6. The pedestrian behavior predicting device according to claim 2, wherein

the discontinuous movement estimating information includes at least one of a moving direction and a moving speed of the pedestrian.

7. The pedestrian behavior predicting device according to claim 2, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

acquire the feature amount distribution of the pedestrian from the extracted partial image using a predetermined feature amount,

normalize the acquired feature amount distribution and acquire a probability distribution corresponding to the feature amount distribution,

accumulate the acquired probability distribution, and

calculate a difference between the probability distribution of an image captured a predetermined time before the current time and the acquired probability distribution using a predetermined scale, and detect the change in the movement of the pedestrian when the calculated difference is greater than a predetermined threshold value.

8. The pedestrian behavior predicting device according to claim 1, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

calculate an optical flow of the imaged image and acquire the discontinuous movement estimating information using the optical flow at the time the first change in the movement of the pedestrian is detected.

9. The pedestrian behavior predicting device according to claim 8, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

acquire a position of the pedestrian on the imaged image,

generate continuous movement estimating information indicating a continuous movement of the pedestrian that occurs following movement of the position of the pedestrian based on a history of positions of the pedestrian acquired, and

predict the behavior of the pedestrian based on the continuous movement estimating information generated.

10. The pedestrian behavior predicting device according to claim 8, wherein

the discontinuous movement estimating information includes at least one of a moving direction and a moving speed of the pedestrian.

11. The pedestrian behavior predicting device according to claim 1, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

acquire a position of the pedestrian on the imaged image,

generate continuous movement estimating information indicating a continuous movement of the pedestrian that occurs following movement of the position of the pedestrian based on a history of positions of the pedestrian acquired, and

predict the behavior of the pedestrian based on the continuous movement estimating information generated.

12. The pedestrian behavior predicting device according to claim 1, wherein

the discontinuous movement estimating information includes at least one of a moving direction and a moving speed of the pedestrian.

13. The pedestrian behavior predicting device according to claim 1, wherein

the processor further executes the instructions stored in the memory to cause the pedestrian behavior predicting device to:

acquire a feature amount distribution of the pedestrian from the extracted partial image using a predetermined feature amount,

normalize the acquired feature amount distribution and acquire a probability distribution corresponding to the feature amount distribution,

accumulate the acquired probability distribution, and

calculate a difference between the probability distribution of an image captured a predetermined time before the current time and the acquired probability distribution using a predetermined scale, and detect the change in the movement of the pedestrian when the calculated difference is greater than a predetermined threshold value.

14. A pedestrian behavior predicting method for a pedestrian behavior predicting device, the pedestrian behavior predicting device including a processor having hardware and a storage unit including memory storing instructions executable by the processor and for storing shape information of a pedestrian, wherein the processor executes the instructions stored in the memory of the pedestrian behavior predicting device, the method comprising the steps of:

detecting a pedestrian from an imaged image;

acquired at a current time

extracting a partial image including the detected pedestrian from the imaged image;

acquiring shape information of the pedestrian from the extracted partial image, the shape information of the pedestrian including at least one of a luminance of the partial image, an edge of the partial image, and a color of the partial image;

accumulating the acquired shape information of the pedestrian in the storage unit; and

detecting a change in movement of the pedestrian by comparing shape information of the pedestrian from an image captured a predetermined time before the current time and the acquired shape information of the pedestrian using the accumulated shape information of the pedestrian;

storing first images and pieces of discontinuous movement estimating information in the storage unit such that each first image is associated with each piece of discontinuous movement estimating information, the first image being an image that was acquired at a time when a first change in the movement of the pedestrian is detected, the piece of discontinuous movement estimating information being information indicating a discontinuous movement of the pedestrian that occurs following the first change in the movement of the pedestrian;

searching, when a second change in the movement of the pedestrian is detected, from the first images stored in the storage unit, a first image corresponding to a second image when the second change in the movement of the pedestrian is detected;

acquiring a piece of discontinuous movement estimating information associated with the searched first image; and

predicting a behavior of the pedestrian using the acquired piece of discontinuous movement estimating information.

15. The pedestrian behavior predicting method according to claim 14, the method further comprising steps of:

acquiring a feature amount distribution representing the shape information of the pedestrian acquired from the partial image extracted, normalizing the shape information represented with the acquired feature amount distribution, accumulating the normalized shape information and comparing the shape information of the image captured a predetermined time before the current time and the acquired shape information.

16. The pedestrian behavior predicting method according to claim 15, the method further comprising a step of:

calculating an optical flow of the imaged image, wherein the discontinuous movement estimating information is acquired using the optical flow at the time the first change in the movement of the pedestrian is detected.

17. The pedestrian behavior predicting method according to claim 15, the method further comprising steps of:

acquiring a position of the pedestrian on the imaged image, and

the pedestrian behavior predicting method further comprises generating continuous movement estimating information indicating a continuous movement of the pedestrian that occurs following movement of the position of the pedestrian based on a history of positions of the pedestrian acquired in the pedestrian detecting step, wherein

the behavior of the pedestrian is predicted based on the continuous movement estimating information generated.

18. The pedestrian behavior predicting method according to claim 14, the method further comprising steps of:

calculating an optical flow of the imaged image wherein estimating information is acquired using the optical flow at the time the first change in the movement of the pedestrian is detected.

19. The pedestrian behavior predicting method according to claim 18, the method further comprising steps of:

acquiring a position of the pedestrian on the imaged image, and

generating continuous movement estimating information indicating a continuous movement of the pedestrian that occurs following movement of the position of the pedestrian based on a history of positions of the pedestrian acquired, wherein

the behavior of the pedestrian is predicted based on the continuous movement estimating information generated.

20. The pedestrian behavior predicting method according to claim 14, the method further comprising steps of:

acquiring a position of the pedestrian on the imaged image, and

generating continuous movement estimating information indicating a continuous movement of the pedestrian that occurs following movement of the position of the pedestrian based on a history of positions of the pedestrian acquired, wherein

the behavior of the pedestrian is predicted based on the continuous movement estimating information generated.