Vision System for Monitoring Humans in Dynamic Environments

• US 20110050878A1
• Filed: 08/28/2009
• Published: 03/03/2011
• Est. Priority Date: 08/28/2009
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations Incorporated (Motors Liquidation Co. GUC Trust)
• Inventors: Kim, Kyungnam, Chen, Yang, Owechko, Yuri, Medasani, Swarup, Wampler, Charles W. II, Menassa, Roland J., Wells, James W.
• Granted Patent: US 8,253,792 B2
• Time in Patent Office: Days
• US Class Current: 348/86
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A safety monitoring system for a workspace area. The workspace area related to a region having automated moveable equipment. A plurality of vision-based imaging devices capturing time-synchronized image data of the workspace area. Each vision-based imaging device repeatedly capturing a time synchronized image of the workspace area from a respective viewpoint that is substantially different from the other respective vision-based imaging devices. A visual processing unit for analyzing the time-synchronized image data. The visual processing unit processes the captured image data for identifying a human from a non-human object within the workspace area. The visual processing unit further determining potential interactions between a human and the automated moveable equipment. The visual processing unit further generating control signals for enabling dynamic reconfiguration of the automated moveable equipment based on the potential interactions between the human and the automated moveable equipment in the workspace area.

Claims

1. A safety monitoring system for a workspace area, the workspace area related to a region having automated moveable equipment, the system comprising:

a plurality of vision-based imaging devices capturing time-synchronized image data of the workspace area, each vision-based imaging device repeatedly capturing respective time synchronized image data of the workspace area from a respective viewpoint that is substantially different from the other respective vision-based imaging devices;

a visual processing unit for analyzing the time-synchronized image data generated by the plurality of vision-based imaging devices;

wherein the visual processing unit processes the captured time-synchronized image data for identifying a human from a non-human object within the workspace area by integrating one or more patterns in each of the time-synchronized image data, the visual processing unit further determining potential interactions between a human and the automated moveable equipment, the visual processing unit further generating control signals for enabling dynamic reconfiguration of the automated moveable equipment based on the potential interactions between the human and the automated moveable equipment in the workspace area.

2. The safety system of claim 1 wherein the visual processing unit determines a predicted movement of the human in the workspace area, wherein the visual processing unit determines the potential interactions with the automated moveable equipment based on the predicted movements of the human in the workspace area, the visual processing unit further generating control signals for enabling dynamic reconfiguration of the automated moveable equipment based on the potential interactions between the predicted movements of the human and the automated moveable equipment in the workspace area.

3. The safety system of claim 1 wherein a preprogrammed movement of the automated moveable equipment is provided to the visual processing unit, and wherein the visual processing unit determines the potential interactions with the automated moveable equipment based on the predicted movements of the human in the workspace area and the preprogrammed movement of the automated moveable equipment, the visual processing unit further generating control signals for enabling dynamic reconfiguration of the automated moveable equipment based on the potential interactions between the predicted movements of the human and the preprogrammed movement of the automated moveable equipment in the workspace area.

4. The safety system of claim 1 further comprising a computer system for simultaneously capturing and formatting time-synchronized image streams in real-time from the plurality of vision-based imaging devices.

5. The safety system of claim 1 wherein the plurality of vision-based imaging devices includes at least two overhead cameras for providing high angle views of the workspace area.

6. The safety system of claim 5 wherein the plurality of vision-based imaging devices includes at least two vision-based imaging devices positioned to sides of the workspace area for capturing mid angle views.

7. The safety system of claim 6 wherein the plurality of vision-based imaging devices includes at least two vision-based imaging devices positioned at the sides of the workspace area for capturing low angle views.

8. The safety system of claim 1 further comprising a human classifier for identifying the presence of the human in workspace area in response to the image data captured by the plurality of vision-based imaging devices.

9. The safety system of claim 8 wherein the human classifier collectively identifies the human in each of the captured images, wherein the human classifier uses a multiple body-axes routine to identify a corresponding body axis, and wherein a human detection score is determined and used to integrate the corresponding body-axes.

10. The safety system of claim 1 further comprising a tracking database for providing a best matching track of the human within the workspace area, the tracking database containing a plurality of tracks of respective humans in the workspace, wherein one of the plurality of tracks may be selected from the tracking database that best identifies the human within the workspace area.

11. The safety system of claim 1 further comprising a filter for predicting movements of the human in the workspace.

12. The safety system of claim 1 further comprising an alert notification system for alerting the human in response to the visual processing unit determining that a potential interaction is likely to occur between the human and automated moveable equipment within the workspace area.

13. The safety system of claim 1 wherein the visual processing unit enables dynamic reconfiguration of virtual intrusion zones in relation to the automated moveable equipment in the workspace area based on the predicted movements of the human, the virtual intrusion zone being warning zones encapsulating the automated moveable factory equipment.

14. The system of claim 1 further comprising a second plurality of vision based imaging devices and a second visual processing unit for capturing and processing captured time-synchronous images data from a second workspace area, wherein data relating to humans are shared between the processing units for integrating a transition of a respective human between the first workspace area and the second workspace area.

15. The system of claim 1 further comprising a dynamic integrity monitoring subsystem that monitors integrity of the plurality of vision based imaging devices and visual processing unit for component failures, wherein the processing unit enables a safe mode of operation in response to detecting a component failure.

16. The system of claim 15 wherein the dynamic integrity monitoring system detects degradation conditions, wherein the processing unit enables the safe mode of operation in response to detecting the conditions

17. A method for identifying a potential interaction between a human and automated moveable equipment in a workspace area, the system comprising:

capturing time synchronized images from a plurality of vision-based imaging devices in the workspace area, each vision-based imaging device capturing respective time synchronized images from a respective viewpoint that is substantially different from the other respective vision-based imaging devices;

identifying moveable objects in the captured data of each respective viewpoint;

filtering moveable object data for distinguishing humans from non-human objects;

collectively identifying the human in each viewpoint of the captured data;

localizing the human in the workspace area based on collective identification of the human;

determining a potential interaction with the automated moveable equipment; and

providing a warning in response to the potential interaction between the automated moveable factor equipment and the localized human within the workspace area.

18. The method of claim 17 further comprising the step of generating control signals for enabling dynamic reconfiguration of the automated moveable equipment based on the potential interactions between human and the automated moveable equipment in the workspace area.

19. The method of claim 18 further comprising the step of determining a predicted movement of the human in the workspace area, wherein the potential interactions between the human and the automated moveable equipment are further determined based on the predicted movements of the human in the workspace area, and wherein control signals generated for enabling dynamic reconfiguration of the automated moveable equipment are further based on the potential interactions between the predicted movements of the human and the automated moveable equipment in the workspace area.

20. The method of claim 19 wherein the step of determining potential interactions between the human and the automated moveable equipment are further determined based on preprogrammed movements of the automated moveable equipment, and wherein control signals generated for enabling dynamic reconfiguration of the automated moveable equipment are further based on the potential interactions between the predicted movements of the human and preprogrammed movements of automated moveable equipment in the workspace area.

21. The method of claim 19 wherein filtering is used to determine the next predicted movements of the human.

22. The method of claim 18 wherein enabling dynamic reconfiguration of the automated moveable equipment includes decreasing a speed of the automated moveable equipment to avoid the potential interaction between the human and the automated moveable equipment.

23. The method of claim 18 wherein enabling dynamic reconfiguration of the automated moveable equipment includes increasing a speed of the automated moveable equipment to avoid the potential interaction between the human and the automated moveable equipment.

24. The method of claim 18 wherein enabling dynamic reconfiguration of the automated moveable equipment includes dynamically determining and traversing a new path of travel for the automated movable equipment to avoid the potential interaction between the human and the automated moveable equipment.

25. The method of claim 18 wherein enabling dynamic reconfiguration of the automated moveable equipment includes stopping the automated moveable equipment to avoid the potential interaction between the human and the automated moveable equipment.

26. The method of claim 17 wherein the step of collectively identifying the human in each of the captured images includes color profiling.

27. The method of claim 17 wherein the step of collectively identifying the human in each of the captured images includes applying body-axis modeling.

28. The method of claim 17 wherein the step of localizing the human in the workspace area includes applying a multi-view integrated body-axes mapping.

29. The method of claim 17 wherein additional workspace areas are monitored by additional vision based capturing devices, wherein data relating to humans are shared between workspace areas for integrating a transition of a respective human from a first workspace area to an adjacent workspace area.

30. The method of claim 17 wherein a warning is actuated in response to a respective human disposed within the workspace area becoming undetected within the workspace area by the plurality of vision-based imaging devices prior to detecting the respective human exiting the workspace area.

31. The method of claim 17 wherein a warning is actuated in response to detecting a respective human within the workspace by the plurality of vision-based imaging devices prior to detecting the respective human entering the workspace area.

32. A method for dynamically monitoring an integrity of a vision based safety monitoring system of a workspace area for failures, the workspace area relating to a region having automated moveable equipment and at least one vision-based imaging device for capturing time-synchronized image data of the workspace area, the method comprising the steps of:

providing a fiduciary target within the workspace area;

capturing images of the fiduciary target by the at least one imaging device over a time interval;

determining whether visual changes in the fiduciary target in the captured data have occurred;

modifying the automated equipment to operate in a safe mode in response to no visual changes detected in the captured images of the fiduciary targets over the time interval;

otherwise maintaining the current operating conditions of the automated equipment.

33. The method of claim 32 wherein the step of determining visual changes in the fiduciary target includes determining whether the image data captured by the at least one imaging device changes over the predetermined period of time.

34. The method of claim 33 wherein the fiduciary target includes a flashing beacon, wherein determining visual changes in the fiduciary target includes determining whether flashes during an operation of the beacon are detected in the captured data over the time interval.

35. The method of claim 32 wherein the flashing beacon is flashed at a respective rate, and wherein determining visual changes in the flashing beacon in the captured data includes determining that the beacon is flashing at the respective rate from the capture image data.

36. The method of claim 32 further comprising the step of modifying the automated equipment to operate in a safe mode in response to the fiduciary target captured in the images within the workspace area becoming undetected within the workspace area prior to detecting the fiduciary target exiting the workspace area.

37. The method of claim 32 further comprising the step of modifying the automated equipment to operate in a safe mode in response to movement of the fiduciary target from a first location to a second location, wherein the second location is disposed a predetermined distance from the first location, wherein the fiduciary target appears at the second location within a predetermined time from being last detected at the first location.

38. The method of claim 32 wherein the monitoring system includes a plurality of vision-based imaging devices for capturing time-synchronized image data of the workspace area, the plurality of vision based imaging devices being disposed at various locations about the workspace area for capturing images of the fiduciary target;

wherein the captured images from the plurality of vision-based imaging devices are collectively time synchronized for preventing occlusion of the fiduciary target by at least one of the respective vision-based imaging devices.