Selecting a device with visual confirmation in the internet of things

• US 10,152,652 B2
• Filed: 09/24/2015
• Issued: 12/11/2018
• Est. Priority Date: 09/24/2015
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Hardikar, Dilip V., Mandalia, Baiju D.
• Primary Examiner(s): Greene, Joseph L
• Application Number:
  US14/864,822
• Publication Number:
  US 20170090840A1
• Time in Patent Office: 1,174 Days
• Field of Search: 709243
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 3/1204 resulting in reduced user or operator actions, e.g. presetting, automatic actions, using hardware token storing data
• CPC Class Codes: G06F 3/1226 Discovery of devices having required properties
• CPC Class Codes: G06F 3/1229 Printer resources management or printer maintenance, e.g. device status, power levels
• CPC Class Codes: G06F 3/126 Job scheduling, e.g. queuing, determine appropriate device
• CPC Class Codes: G06F 3/1292 Mobile client, e.g. wireless printing
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A determination is made that an instruction has to be sent for processing in the IoT environment. A set of devices is located in the IoT environment. The set of devices includes the device and each device in the set of devices can execute the instruction for a similar outcome. For each device in a subset of the set of devices, a set of factors is collected, including an image of a portion of each device. For the device, a first weight is assigned to a feature of the device extracted from an image of a portion of the device, and a second weight is assigned to a feature of another device extracted from another image of a portion of the other device. When the first weight is greater than the second weight, the device is selected from the subset as a target for sending the instruction.

Claims

1. A method for selecting a device in an Internet of Things (IoT) environment, the method comprising:

determining that an instruction has to be sent for processing in the IoT environment;

locating, relative to a mobile data processing system a set of devices in the IoT environment, wherein the set of devices includes the device and each device in the set of devices can execute the instruction for a similar outcome;

collecting, for each device in a subset of the set of devices, a set of factors, the set of factors including a first factor representative of an image of a portion of each device, a second factor including a distance of the device from the mobile data processing system, a third factor including a delay at the device due to a queue of instructions pending at the device, and a fourth factor including a status of the device as determined by configuration information supplied by the device;

performing image analysis on the image of the portion of the device for each device to extract a feature in the image of the portion of the device to determine the first factor, wherein the feature includes information indicative of an actual operating state of the device at a first time;

assigning, for the device, a first weight to the first factor of the device;

computing a first overall rank for the device by a weighted combination of the first factor, the second factor, the third factor, and the fourth factor of the device;

assigning, for another device, a second weight to the first factor of the another device;

computing a second overall rank for the another device by a weighted combination of the first factor, the second factor, the third factor, and the fourth factor of the other device;

selecting, responsive to the first overall rank being greater than the second overall rank, the device from the subset as a target for sending the instruction;

sending the instruction to the device;

sending a second instruction to the device, the second instruction causing the device to withhold executing the instruction;

detecting that the mobile data processing system is within a specified distance of the device at a second time; and

sending, responsive to the detecting that the mobile data processing system is within the specified distance of the device at the second time, a third instruction to the device, the third instruction causing the device to schedule the instruction for execution.

2. The method of claim 1, further comprising:

assigning, for the device, a third weight to the second factor in the set of factors of the device;

assigning, for the other device, a fourth weight to the second factor in the set of factors of the other device;

computing the first overall rank of the device relative to the second overall rank of the other device in the subset of devices, the computing using the first and the third weights for the device and the computing using the second and the fourth weights for the other device, wherein the selecting is responsive to the first overall rank of the device being higher than the second overall rank of the other device.

3. The method of claim 2, further comprising:

assigning, for the device, a fifth weight to the third factor in the set of factors of the device;

assigning, for the other device, a sixth weight to the third factor in the set of factors of the other device;

using, in computing the first overall rank of the device relative to the second overall rank of the other device in the subset of devices, the fifth weight for the device and the sixth weight for the other device.

4. The method of claim 2, further comprising:

assigning, for the device, a seventh weight to the fourth factor in the set of factors of the device;

assigning, for the other device, an eighth weight to the fourth factor in the set of factors of the other device;

using, in computing the first overall rank of the device relative to the second overall rank of the other device in the subset of devices, the seventh weight for the device and the eighth weight for the other device.

5. The method of claim 1, further comprising:

receiving a user input relative to the image of the portion of the device, the user input assigning a desirability rating to the feature in the image of the portion of the device.

6. The method of claim 1, further comprising:

selecting the subset of device responsive to determining that only the subset of devices from the set of devices is usable for the instruction.

7. The method of claim 1, further comprising:

configuring corresponding to each device in the set of devices, an imaging device, wherein the imaging device is oriented to capture an image of at least a portion of the corresponding device.

8. The method of claim 7, wherein the imaging device provides the image of the portion of the corresponding device in response to a request for the image.

9. The method of claim 7, wherein the imaging device provides the image of the portion of the corresponding device periodically.

10. The method of claim 7, wherein the imaging device is mounted within the corresponding device, wherein the portion of the device is within the device and unobservable from an external view of the corresponding device.

11. The method of claim 7, wherein the imaging device is dedicated to imaging the corresponding device.

12. The method of claim 7, wherein the imaging device is configured to image a geographical area, wherein the corresponding device is situated in the geographical area causing the imaging device to capture the image of the corresponding device.

13. The method of claim 1, further comprising:

detecting that the mobile data processing system is located at a location in the IoT environment at a given time, wherein the instruction has to be sent from the computing device;

setting a distance threshold from the computing device, wherein each device in the set of devices is located within the distance threshold from the location of the computing device.

14. The method of claim 13, wherein the threshold distance is a vertical distance spanning a plurality of floors in a building.

15. The method of claim 1, wherein the method is embodied in a computer program product comprising one or more computer-readable storage devices and computer-readable program instructions which are stored on the one or more computer-readable tangible storage devices and executed by one or more processors.

16. A computer program product for selecting a device in an Internet of Things (IoT) environment, the computer program product comprising one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices, the stored program instructions comprising:

program instructions to determine that an instruction has to be sent for processing in the IoT environment;

program instructions to locate, relative to a mobile data processing system a set of devices in the IoT environment, wherein the set of devices includes the device and each device in the set of devices can execute the instruction for a similar outcome;

program instructions to collect, for each device in a subset of the set of devices, a set of factors, the set of factors including a first factor representative of an image of a portion of each device, a second factor including a distance of the device from the mobile data processing system, a third factor including a delay at the device due to a queue of instructions pending at the device, and a fourth factor including a status of the device as determined by configuration information supplied by the device;

program instructions to perform image analysis on the image of the portion of the device for each device to extract a feature in the image of the portion of the device to determine the first factor, wherein the feature includes information indicative of an actual operating state of the device at a first time;

program instructions to assign, for the device, a first weight to the first factor of the device;

program instructions to compute a first overall rank for the device by a weighted combination of the first factor, the second factor, the third factor, and the fourth factor of the device;

program instructions to assign, for another device, a second weight to the first factor of the another device;

program instructions to compute a second overall rank for the another device by a weighted combination of the first factor, the second factor, the third factor, and the fourth factor of the other device;

program instructions to select, responsive to the first overall rank being greater than the second overall rank, the device from the subset as a target for sending the instruction;

program instructions to send the instruction to the device;

program instructions to send a second instruction to the device, the second instruction causing the device to withhold executing the instruction;

program instructions to detect that the mobile data processing system is within a specified distance of the device at a second time; and

program instructions to send, responsive to the detecting that the mobile data processing system is within the specified distance of the device at the second time, a third instruction to the device, the third instruction causing the device to schedule the instruction for execution.

17. A computer system for selecting a device in an Internet of Things (IoT) environment, the computer system comprising one or more processors, one or more computer-readable memories, and one or more computer-readable storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the stored program instructions comprising:

program instructions to determine that an instruction has to be sent for processing in the IoT environment;

program instructions to locate, relative to a mobile data processing system a set of devices in the IoT environment, wherein the set of devices includes the device and each device in the set of devices can execute the instruction for a similar outcome;

program instructions to collect, for each device in a subset of the set of devices, a set of factors, the set of factors including a first factor representative of an image of a portion of each device, a second factor including a distance of the device from the mobile data processing system, a third factor including a delay at the device due to a queue of instructions pending at the device, and a fourth factor including a status of the device as determined by configuration information supplied by the device;

program instructions to perform image analysis on the image of the portion of the device for each device to extract a feature in the image of the portion of the device to determine the first factor, wherein the feature includes information indicative of an actual operating state of the device at a first time;

program instructions to assign, for the device, a first weight to the first factor of the device;

program instructions to compute a first overall rank for the device by a weighted combination of the first factor, the second factor, the third factor, and the fourth factor of the device;

program instructions to assign, for another device, a second weight to the first factor of the another device;

program instructions to compute a second overall rank for the another device by a weighted combination of the first factor, the second factor, the third factor, and the fourth factor of the other device;

program instructions to select, responsive to the first overall rank being greater than the second overall rank, the device from the subset as a target for sending the instruction;

program instructions to send the instruction to the device;

program instructions to send a second instruction to the device, the second instruction causing the device to withhold executing the instruction;

program instructions to detect that the mobile data processing system is within a specified distance of the device at a second time; and

program instructions to send, responsive to the detecting that the mobile data processing system is within the specified distance of the device at the second time, a third instruction to the device, the third instruction causing the device to schedule the instruction for execution.