Information technology asset type identification using a mobile vision-enabled robot

• US 9,346,168 B2
• Filed: 05/20/2014
• Issued: 05/24/2016
• Est. Priority Date: 05/20/2014
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Connell, Jonathan H. II, Kephart, Jeffrey O., Lenchner, Jonathan
• Primary Examiner(s): Dunphy, David F
• Application Number:
  US14/282,531
• Publication Number:
  US 20150336274A1
• Time in Patent Office: 735 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B25J 9/16 Programme controls programme controls in general G05B19/00, e.g. numerical pogramme controls G05B19/18; recording or playback systems G05B19/42
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39369 Host and robot controller and vision processing
• CPC Class Codes: G05B 2219/40005 Vision, analyse image at one station during manipulation at next station
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

Mechanisms are provided for classifying an obstacle as an asset type. The mechanisms receive a digital image of an obstacle from an image capture device of an automated robot. The mechanisms perform a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle. The mechanisms determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle. The mechanisms update a map data structure for a physical premises in which the obstacle is present based on the final asset type.

Claims

1. A method, in a data processing system comprising a processor and a memory, for classifying an obstacle as an asset type, comprising:

receiving, by the data processing system, a digital image of an obstacle from an image capture device of an automated robot;

performing, by the data processing system, a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle;

determining, by the data processing system, a final asset type for the obstacle based on the proposed asset type classification for the obstacle; and

updating, by the data processing system, a map data structure for a physical premises in which the obstacle is present based on the final asset type, wherein performing the classification operation further comprises;

gathering additional sensor information from one or more other sensors provided on either the robot or in the physical premises;

performing the classification operation based on a classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information; and

utilizing a plurality of classification algorithms that each classify the obstacle with regard to one or more different characteristics of the obstacle as identified in at least one of the digital image or the additional sensor information, wherein performing the classification operation based on the classification of the one or more different characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information comprises comparing conditions of the physical premises in proximity to the obstacle as identified by the additional sensor information to patterns of similar information acquired through training of the plurality of trained classification algorithms.

2. The method of claim 1, wherein determining a final asset type for the obstacle based on the proposed asset type classification for the obstacle comprises:

presenting, by the data processing system, the proposed asset type classification and the digital image to a human user via a user interface output on a computing device associated with the human user, wherein the user interface identifies the proposed asset type classification, the digital image, and provides a mechanism for user feedback input indicating whether or not the proposed asset type classification is correct or incorrect; and

determining, by the data processing system, the final asset type for the obstacle based on input from the human user via the user interface.

3. The method of claim 1, wherein the method is implemented automatically without human intervention.

4. The method of claim 1, wherein the additional sensor information comprises information indicative of conditions within a vicinity of the obstacle.

5. The method of claim 1, wherein the additional sensor information comprises at least one of temperature information in proximity to the obstacle, humidity information in proximity to the obstacle, infrared profile information of the obstacle, proximity information indicating a proximity of the obstacle to other known types of assets, or configuration information for other known types of nearby assets relative to the obstacle.

6. The method of claim 1, further comprising:

outputting, by the data processing system, a user interface representing the map of the physical premises in which assets in the physical premises are distinguished by asset type.

7. The method of claim 1, wherein the physical premises is a data center and the obstacle is a physical asset of the data center.

8. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive a digital image of an obstacle from an image capture device of an automated robot;

perform a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle;

determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle; and

update a map data structure for a physical premises in which the obstacle is present based on the final asset type, wherein the computer readable program further causes the computing device to perform the classification operation at least by;

gathering additional sensor information from one or more other sensors provided on either the robot or in the physical premises;

performing the classification operation based on a classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information; and

utilizing a plurality of classification algorithms that each classify the obstacle with regard to one or more different characteristics of the obstacle as identified in at least one of the digital image or the additional sensor information, wherein the computer readable program further causes the computing device to perform the classification operation based on the classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information at least by comparing conditions of the physical premises in close proximity to the obstacle as identified by the additional sensor information to patterns of similar information acquired through training of the plurality of trained classification algorithms.

9. The computer program product of claim 8, wherein the computer readable program further causes the computing device to determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle at least by:

presenting the proposed asset type classification and the digital image to a human user via a user interface output on a computing device associated with the human user, wherein the user interface identifies the proposed asset type classification, the digital image, and provides a mechanism for user feedback input indicating whether or not the proposed asset type classification is correct or incorrect; and

determining the final asset type for the obstacle based on input from the human user via the user interface.

10. The computer program product of claim 8, wherein the computer readable program further causes the computing device to execute the receive, perform, determine, and update operations automatically without human intervention.

11. The computer program product of claim 8, wherein the additional sensor information comprises information indicative of conditions within a vicinity of the obstacle.

12. The computer program product of claim 8, wherein the additional sensor information comprises at least one of temperature information in proximity to the obstacle, humidity information in proximity to the obstacle, infrared profile information of the obstacle, proximity information indicating a proximity of the obstacle to other known types of assets, or configuration information for other known types of nearby assets relative to the obstacle.

13. The computer program product of claim 8, wherein the computer readable program further causes the computing device to:

output a user interface representing the map of the physical premises in which assets in the physical premises are distinguished by asset type.

14. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive a digital image of an obstacle from an image capture device of an automated robot;

perform a classification operation on the digital image of the obstacle to identify a proposed asset type classification for the obstacle;

determine a final asset type for the obstacle based on the proposed asset type classification for the obstacle; and

update a map data structure for a physical premises in which the obstacle is present based on the final asset type, wherein the instructions further cause the computing processor to perform the classification operation at least by;

gathering additional sensor information from one or more other sensors provided on either the robot or in the physical premises;

performing the classification operation based on a classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information; and

utilizing a plurality of classification algorithms that each classify the obstacle with regard to one or more different characteristics of the obstacle as identified in at least one of the digital image or the additional sensor information, wherein the instructions further cause the processor to perform the classification operation based on the classification of characteristics of the obstacle obtained from analysis of the digital image and analysis of the additional sensor information at least by comparing conditions of the physical premises in close proximity to the obstacle as identified by the additional sensor information to patterns of similar information acquired through training of the plurality of trained classification algorithms.