Human detection system for construction machine

• US 10,465,362 B2
• Filed: 11/29/2016
• Issued: 11/05/2019
• Est. Priority Date: 06/03/2014
• Status: Active Grant

• Current Assignee: Sumitomo Heavy Industries Limited
• Original Assignee: Sumitomo Heavy Industries Limited
• Inventors: Kiyota, Yoshihisa, Aizawa, Susumu
• Primary Examiner(s): Beg, Samah A
• Application Number:
  US15/362,968
• Publication Number:
  US 20170073934A1
• Time in Patent Office: 1,071 Days
• Field of Search: None
• CPC Class Codes: B60R 1/23 with a predetermined field of view
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/8033 for pedestrian protection
• CPC Class Codes: E02F 3/32 working downwardly and towards the machine, e.g. with backhoes
• CPC Class Codes: E02F 9/24 Safety devices , e.g. for preventing overload E02F9/226 takes precedence
• CPC Class Codes: E02F 9/26 Indicating devices E02F5/145 takes precedence
• CPC Class Codes: E02F 9/261 Surveying the work-site to be treated
• CPC Class Codes: E02F 9/262 with follow-up actions to control the work tool, e.g. controller
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A human detection system may detect the presence of a human around a shovel by using an image captured by an imaging device attached to the shovel. Also, the human detection system may have an extracting part that extracts a part of the captured image as an identification process target image, and an identifying part that identifies whether an image included in the identification process target image is an image of a human. A region of the identification process target image is preliminarily associated with one of head image positions in the captured image. The extracting part finds out a helmet image in the captured image, and extracts the identification process target image by associating a representative position of the helmet image with one of the head image positions.

Claims

1. A human detection system for a construction machine that is configured to detect the presence of a human around the construction machine by using an image captured by an imaging device attached to the construction machine, the system comprising:

a memory; and

a processor coupled to the memory, and configured to

search digital data of the captured image for digital data of a feature image representing a human characteristic portion to identify the feature image in the captured image;

extract a part of the digital data of the captured image as a target image, the target image including an image corresponding in size to a whole human body that includes the identified feature image; and

identify whether the image included in the target image is a human image,

wherein the target image corresponds to a preset point most correlated with a position of the feature image in the captured image among a plurality of preset points that are assumed standing positions of the human in a real space.

2. The human detection system for the construction machine as claimed in claim 1,

wherein the feature image includes an image of a helmet, an image of a shoulder, an image of a head, or an image of a reflector or a marker attached to a human.

3. The human detection system for the construction machine-as claimed in claim 1,

wherein the processor is configured to extract the target image including the identified feature image in the captured image by utilizing a predetermined geometric relationship,

wherein the predetermined geometric relationship indicates a geometric relationship of a region of the target image in the captured image, a virtual plane region in the real space corresponding to the region of the target image, an assumed standing position of the human in the real space corresponding to the virtual plane region, a virtual feature position as a position of a characteristic portion of the human in the real space corresponding to the assumed standing position, and a predetermined image position in the captured image corresponding to the virtual feature position.

4. The human detection system for the construction machine as claimed in claim 3,

wherein the processor is configured to normalize the target image, and

wherein a head image portion, a trunk image portion, and a leg image portion are allocated equally in the normalized target image, independently of a distance between the virtual plane region and the imaging device.

5. The human detection system for the construction machine as claimed in claim 4,

wherein the normalized target image includes a first normalized target image and a second normalized target image, and

wherein a first head image portion in the first normalized target image is the same size as a second head image portion in the second normalized target image when the distance is equal to or greater than a predetermined value.

6. The human detection system for the construction machine as claimed in claim 1,

wherein a region of the target image in the captured image is associated with a grid point of a virtual grid placed on a horizontal surface in the real space.

7. The human detection system for the construction machine as claimed in claim 6,

wherein the processor is configured to derive the position of the feature image, and

wherein the position of the feature image is associated with a grid point of the virtual grid.

8. The human detection system for the construction machine as claimed in claim 1, wherein the processor is configured to search the digital data of the captured image by raster-scanning the captured image to identify the feature image.

9. The human detection system for the construction machine as claimed in claim 1, wherein the processor is configured to search the digital data of the captured image for digital data of an image whose projection image projected onto the captured image is nearly circular to identify the feature image.

10. The human detection system for the construction machine as claimed in claim 1, wherein

the processor is configured to search the digital data of a plurality of regions of the captured image region by region for the digital data of the feature image, and

a size of each of the plurality of regions and a size of the target image change in accordance with a distance from a position of the imaging device.

11. The human detection system for the construction machine as claimed in claim 1, wherein the target image is more inclined as the target image is more angled relative to the imaging device.

12. The human detection system for the construction machine as claimed in claim 1, wherein

the plurality of preset points are placed at corresponding grid points of a virtual grid on a virtual land surface,

a plurality of virtual plane regions are correlated with the corresponding preset points on the virtual land surface, and

the processor is configured to extract the digital data of the part of the captured image as the target image, the part of the captured image corresponding to one of the plurality of virtual plane regions.

13. The human detection system for the construction machine as claimed in claim 12, wherein the processor is configured to extract the digital data of the part of the captured image as the target image using a reference table or a geometric relationship of the plurality of preset points and the plurality of virtual plane regions based on a position of the identified feature image.

14. The human detection system for the construction machine as claimed in claim 1, wherein the processor is configured to mask a region of a machine body of the construction machine in the target image.

15. A shovel that is configured to detect the presence of a human around the shovel by using an image captured by an imaging device, the shovel comprising:

a memory; and

a processor coupled to the memory, and configured to

search digital data of the captured image for digital data of a feature image representing a human characteristic portion to identify the feature image in the captured image;

extract a part of the digital data of the captured image as a target image, the target image including an image corresponding in size to a whole human body that includes the identified feature image; and

identify whether the image included in the target image is a human image,

wherein the target image corresponds to a preset point most correlated with a position of the feature image in the captured image among a plurality of preset points that are assumed standing positions of the human in a real space.

16. A human detection system that is configured to detect the presence of a human by using an image captured by an imaging device of which an optical axis is directed diagonally downward, the system comprising:

a memory; and

a processor coupled to the memory, and configured to

search digital data of the captured image for digital data of a feature image representing a human characteristic portion to identify the feature image in the captured image;

extract a part of the digital data of the captured image as a target image, the target image including an image corresponding in size to a whole human body that includes the identified feature image; and

identify whether the image included in the target image is a human image,

wherein the target image corresponds to a preset point most correlated with a position of the feature image in the captured image among a plurality of preset points that are assumed standing positions of the human in a real space.

17. The human detection system as claimed in claim 16,

wherein the processor is configured to extract the target image including the identified feature image in the captured image by utilizing a predetermined geometric relationship,

wherein the predetermined geometric relationship indicates a geometric relationship of a region of the target image in the captured image, a virtual plane region in the real space corresponding to the region of the target image, an assumed standing position of the human in the real space corresponding to the virtual plane region, a virtual feature position as a position of a characteristic portion of the human in the real space corresponding to the assumed standing position, and a predetermined image position in the captured image corresponding to the virtual feature position.

18. The human detection system as claimed in claim 16,

wherein a region of the target image in the captured image is associated with a grid point of a virtual grid placed on a horizontal surface in the real space.