Identifying IoT devices/objects/people using out-of-band signaling/metadata in conjunction with optical images

• US 9,400,943 B2
• Filed: 07/31/2014
• Issued: 07/26/2016
• Est. Priority Date: 08/02/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Sarkar, Nivedita, Sharma, Sandeep, Goel, Amit, Shuman, Mohammed Ataur Rahman
• Primary Examiner(s): STREGE, JOHN B
• Application Number:
  US14/448,880
• Publication Number:
  US 20150036881A1
• Time in Patent Office: 726 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: H04W 4/021 Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences
• CPC Class Codes: H04W 4/185 by embedding added-value information into content, e.g. geo-tagging
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

The disclosure relates to identifying an object associated with a nearby Internet of Things (IoT) device. In an aspect, a device receives identifying information associated with the nearby IoT device, detects a nearby object in a field of view of a camera application, determines whether or not the nearby object is associated with the nearby IoT device based on the received identifying information, and based on the nearby object being associated with the nearby IoT device, determines that the nearby object corresponds to the object associated with the nearby IoT device.

Claims

1. A method for identifying an object associated with a nearby Internet of Things (IoT) device, comprising:

receiving identifying information associated with the nearby IoT device;

detecting a nearby object in a field of view of a camera application;

determining whether or not the nearby object is associated with the nearby IoT device based on the received identifying information matching information associated with the nearby object detected by the camera application in the field of view of the camera application; and

based on the nearby object being associated with the nearby IoT device, correlating the nearby object with information associated with the object associated with the nearby IoT device.

2. The method of claim 1, further comprising:

creating an indication to identify the object associated with the nearby IoT device.

3. The method of claim 2, wherein the creating comprises:

transmitting a beacon signal to one or more nearby IoT devices, the one or more nearby IoT devices including the nearby IoT device.

4. The method of claim 1, wherein the information associated with the nearby object comprises a beacon signal.

5. The method of claim 4, wherein the beacon signal comprises a sound beacon or a light beacon, and wherein the received identifying information comprises an indication of a type of the beacon signal and a frequency or color of the beacon signal.

6. The method of claim 1, wherein the identifying information comprises temperature information of the object associated with the nearby IoT device, and wherein the determining whether or not the nearby object is associated with the nearby IoT device comprises:

determining a temperature of the nearby object; and

determining whether or not the temperature information is within a threshold of the temperature of the nearby object,

wherein the correlating comprises determining that the nearby object corresponds to the object associated with the nearby IoT device based on the temperature information being within the threshold of the temperature of the nearby object.

7. The method of claim 6, wherein the temperature of the nearby object is determined based on a position of the nearby IoT device relative to the nearby object.

8. The method of claim 1, wherein the identifying information comprises pattern information of the nearby IoT device, and wherein the determining whether or not the nearby object is associated with the nearby IoT device comprises:

determining a pattern of the nearby IoT device; and

determining whether or not the pattern information matches the pattern of the nearby IoT device,

wherein the correlating comprises determining that the nearby object corresponds to the object associated with the nearby IoT device based on the pattern information matching the pattern of the nearby IoT device.

9. The method of claim 1, wherein the identifying information comprises a spatial orientation of the nearby IoT device, and wherein the determining whether or not the nearby object is associated with the nearby IoT device comprises:

determining a spatial orientation of the nearby object; and

determining whether or not the spatial orientation of the nearby IoT device matches the spatial orientation of the nearby object,

wherein the correlating comprises determining that the nearby object corresponds to the object associated with the nearby IoT device based on the spatial orientation of the nearby IoT device matches the spatial orientation of the nearby object.

10. The method of claim 9, wherein the spatial orientation of the nearby object is determined based on a position of the nearby IoT device relative to the nearby object and/or gravity.

11. The method of claim 1, wherein the identifying information is received out-of-band.

12. The method of claim 1, further comprising:

receiving the information associated with the object associated with the nearby IoT device; and

correlating the nearby object to the information associated with the object.

13. The method of claim 12, wherein the correlating comprises tagging the nearby object with the information associated with the object in a picture or video of the object associated with the nearby IoT device.

14. The method of claim 12, wherein the object associated with the nearby IoT device is a human, and wherein the received identifying information comprises contact information of the human.

15. The method of claim 1, wherein the object associated with the nearby IoT device comprises the nearby IoT device.

16. The method of claim 1, further comprising:

based on the nearby object not being associated with the nearby IoT device, ignoring the identifying information associated with the nearby IoT device.

17. The method of claim 1, further comprising:

receiving identifying information associated with a second nearby IoT device, wherein the second nearby IoT device is associated with a second nearby object that is not in a field of view of the camera application; and

ignoring the identifying information associated with the second nearby IoT device.

18. The method of claim 17, wherein the camera application determines that the second nearby object is not in the field of view of the camera application based on not detecting a match between the identifying information associated with the second nearby IoT device and any nearby object detected in the field of view of the camera application.

19. The method of claim 1, wherein the camera application comprises a camera application of a user equipment.

20. The method of claim 19, wherein the nearby IoT device comprises an IoT device within peer-to-peer wireless communications range of the user equipment.

21. The method of claim 1, wherein the receiving comprises receiving a plurality of types of subject metadata associated with each of a plurality of nearby IoT devices associated with a plurality of objects, the plurality of objects including the object associated with the nearby IoT device, and wherein the plurality of types of subject metadata are the same for each of the plurality of nearby IoT devices.

22. The method of claim 21, further comprising:

determining a variance of each of the plurality of types of subject metadata across the plurality of nearby IoT devices;

ranking each of the plurality of types of subject metadata based on the determined variance; and

generating weightings of a plurality of subject identification methods corresponding to the plurality of types of subject metadata based on the ranking.

23. The method of claim 22, wherein a highest ranking is assigned to a type of subject metadata of the plurality of types of subject metadata with a highest variance.

24. The method of claim 23, wherein a highest weighting is assigned to a type of subject metadata of the plurality of types of subject metadata with the highest ranking.

25. The method of claim 22, further comprising:

identifying the plurality of objects using a subject identification method with a highest weighting.

26. The method of claim 22, further comprising:

transmitting the generated weightings to the plurality of nearby IoT devices and/or a server in communication with the plurality of nearby IoT devices.

27. The method of claim 26, further comprising:

receiving weightings of the plurality of subject identification methods from the plurality of nearby IoT devices and/or the server in communication with the plurality of nearby IoT devices, wherein the camera application incorporates the received weightings into the generated weightings.

28. The method of claim 21, wherein the plurality of types of subject metadata comprise two or more of a sound beacon, a light beacon, a heat signature, pattern information, and/or pose information associated with the plurality of objects.

29. The method of claim 22, wherein each of the plurality of subject identification methods is useable to identify the plurality of objects using a corresponding one of the plurality of types of subject metadata.

30. The method of claim 21, further comprising:

transmitting the plurality of types of subject metadata associated with each of the plurality of nearby IoT devices to a server,

wherein the server determines a variance of each of the plurality of types of subject metadata across the plurality of nearby IoT devices, ranks each of the plurality of types of subject metadata based on the determined variance, and generates weightings of a plurality of subject identification methods corresponding to the plurality of types of subject metadata based on the ranking.

31. The method of claim 30, further comprising:

receiving the weightings of the plurality of subject identification methods from the server.

32. The method of claim 30, wherein the server receives subject metadata from other IoT devices, and wherein the server generates the weightings of the plurality of subject identification methods based additionally on the subject metadata received from the other IoT devices.

33. The method of claim 1, further comprising:

transmitting a beacon signal to one or more IoT devices in a field of view of the camera application.

34. The method of claim 33, wherein the beacon signal is transmitted to the one or more IoT devices in the field of view of the camera application using directional speakers.

35. The method of claim 33, further comprising:

transmitting a discovery message to the one or more IoT devices, the discovery message including filtering criteria; and

receiving a beacon signal from each of a subset of the one or more IoT devices, wherein each IoT device of the subset of the one or more IoT devices has attributes that match the filtering criteria.

36. The method of claim 35, further comprising:

receiving the filtering criteria, wherein the filtering criteria is user defined and defines attribute values of the one or more IoT devices in the field of view of the camera application that should respond to the discovery message.

37. The method of claim 35, wherein each of the one or more IoT devices in the field of view of the camera application determines whether or not it has attribute values that match the filtering criteria.

38. The method of claim 37, wherein devices of the one or more IoT devices in the field of view of the camera application determine that they do not have attribute values that match the filtering criteria, and in response, do not respond to the received beacon signal or the discovery message.

39. The method of claim 37, wherein each of the subset of the one or more IoT devices in the field of view of the camera application determines that it has attribute values that match the filtering criteria, and in response, responds to the discovery message by emitting a beacon signal and transmitting a contact card.

40. An apparatus for identifying an object associated with a nearby Internet of Things (IoT) device, comprising:

a transceiver; and

at least one processor configured to;

receive identifying information associated with the nearby IoT device;

detect a nearby object in a field of view of a camera application;

determine whether or not the nearby object is associated with the nearby IoT device based on the received identifying information matching information associated with the nearby object detected by the camera application in the field of view of the camera application; and

correlate, based on the nearby object being associated with the nearby IoT device, the nearby object with information associated with the object associated with the nearby IoT device.

41. The apparatus of claim 40, wherein the at least one processor is further configured to:

create an indication to identify the object associated with the nearby IoT device.

42. The apparatus of claim 41, wherein the at least one processor being configured to create comprises the at least one processor being configured to:

cause the transceiver to transmit a beacon signal to one or more nearby IoT devices, the one or more nearby IoT devices including the nearby IoT device.

43. The apparatus of claim 40, wherein the information associated with the nearby object comprises a beacon signal.

44. The apparatus of claim 43, wherein the beacon signal comprises a sound beacon or a light beacon, and wherein the received identifying information comprises an indication of a type of the beacon signal and a frequency or color of the beacon signal.

45. The apparatus of claim 40, wherein the identifying information comprises temperature information of the object associated with the nearby IoT device, and wherein the at least one processor being configured to determine whether or not the nearby object is associated with the nearby IoT device comprises the at least one processor being configured to:

determine a temperature of the nearby object; and

determine whether or not the temperature information is within a threshold of the temperature of the nearby object,

wherein the at least one processor being configured to correlate comprises the at least one processor being configured to determine that the nearby object corresponds to the object associated with the nearby IoT device based on the temperature information being within the threshold of the temperature of the nearby object.

46. The apparatus of claim 45, wherein the temperature of the nearby object is determined based on a position of the nearby IoT device relative to the nearby object.

47. The apparatus of claim 40, wherein the identifying information comprises pattern information of the nearby IoT device, and wherein the at least one processor being configured to determine whether or not the nearby object is associated with the nearby IoT device comprises the at least one processor being configured to:

determine a pattern of the nearby IoT device; and

determine whether or not the pattern information matches the pattern of the nearby IoT device,

wherein the at least one processor being configured to correlate comprises the at least one processor being configured to determine that the nearby object corresponds to the object associated with the nearby IoT device based on the pattern information matching the pattern of the nearby IoT device.

48. The apparatus of claim 40, wherein the identifying information comprises a spatial orientation of the nearby IoT device, and wherein the at least one processor being configured to determine whether or not the nearby object is associated with the nearby IoT device comprises the at least one processor being configured to:

determine a spatial orientation of the nearby object; and

determine whether or not the spatial orientation of the nearby IoT device matches the spatial orientation of the nearby object,

wherein the at least one processor being configured to correlate comprises the at least one processor being configured to determine that the nearby object corresponds to the object associated with the nearby IoT device based on the spatial orientation of the nearby IoT device matches the spatial orientation of the nearby object.

49. The apparatus of claim 48, wherein the spatial orientation of the nearby object is determined based on a position of the nearby IoT device relative to the nearby object and/or gravity.

50. The apparatus of claim 40, wherein the identifying information is received out-of-band.

51. The apparatus of claim 40, further comprising:

logic configured to receive the information associated with the object associated with the nearby IoT device.

52. The apparatus of claim 51, wherein the at least one processor being configured to correlate comprises the at least one processor being configured to tag the nearby object with the information associated with the object in a picture or video of the object associated with the nearby IoT device.

53. The apparatus of claim 51, wherein the object associated with the nearby IoT device is a human, and wherein the received identifying information comprises contact information of the human.

54. The apparatus of claim 40, wherein the object associated with the nearby IoT device comprises the nearby IoT device.

55. The apparatus of claim 40, wherein the at least one processor is further configured to:

ignore the identifying information associated with the nearby IoT device based on the nearby object not being associated with the nearby IoT device.

56. The apparatus of claim 40, wherein the at least one processor is further configured to:

cause the transceiver to receive identifying information associated with a second nearby IoT device, wherein the second nearby IoT device is associated with a second nearby object that is not in a field of view of the camera application; and

ignore the identifying information associated with the second nearby IoT device.

57. The apparatus of claim 56, wherein the camera application determines that the second nearby object is not in the field of view of the camera application based on no detection of a match between the identifying information associated with the second nearby IoT device and any nearby object detected in the field of view of the camera application.

58. The apparatus of claim 40, wherein the camera application comprises a camera application of a user equipment.

59. The apparatus of claim 58, wherein the nearby IoT device comprises an IoT device within peer-to-peer wireless communications range of the user equipment.

60. The apparatus of claim 40, wherein the at least one processor being configured to receive comprises the at least one processor being configured to receive a plurality of types of subject metadata associated with each of a plurality of nearby IoT devices associated with a plurality of objects, the plurality of objects including the object associated with the nearby IoT device, and wherein the plurality of types of subject metadata are the same for each of the plurality of nearby IoT devices.

61. The apparatus of claim 60, wherein the at least one processor is further configured to:

determine a variance of each of the plurality of types of subject metadata across the plurality of nearby IoT devices;

rank each of the plurality of types of subject metadata based on the determined variance; and

generate weightings of a plurality of subject identification methods corresponding to the plurality of types of subject metadata based on the ranking.

62. The apparatus of claim 61, wherein a highest ranking is assigned to a type of subject metadata of the plurality of types of subject metadata with a highest variance.

63. The apparatus of claim 62, wherein a highest weighting is assigned to a type of subject metadata of the plurality of types of subject metadata with the highest ranking.

64. The apparatus of claim 61, wherein the at least one processor is further configured to:

identify the plurality of objects using a subject identification method with a highest weighting.

65. The apparatus of claim 61, wherein the at least one processor is further configured to:

cause the transceiver to transmit the generated weightings to the plurality of nearby IoT devices and/or a server in communication with the plurality of nearby IoT devices.

66. The apparatus of claim 65, wherein the transceiver is configured to:

receive weightings of the plurality of subject identification methods from the plurality of nearby IoT devices and/or the server in communication with the plurality of nearby IoT devices, wherein the camera application incorporates the received weightings into the generated weightings.

67. The apparatus of claim 60, wherein the plurality of types of subject metadata comprise two or more of a sound beacon, a light beacon, a heat signature, pattern information, and/or pose information associated with the plurality of objects.

68. The apparatus of claim 61, wherein each of the plurality of subject identification methods is useable to identify the plurality of objects using a corresponding one of the plurality of types of subject metadata.

69. The apparatus of claim 60, wherein the transceiver is configured to:

transmit the plurality of types of subject metadata associated with each of the plurality of nearby IoT devices to a server,

wherein the server determines a variance of each of the plurality of types of subject metadata across the plurality of nearby IoT devices, ranks each of the plurality of types of subject metadata based on the determined variance, and generates weightings of a plurality of subject identification methods corresponding to the plurality of types of subject metadata based on the ranking.

70. The apparatus of claim 69, wherein the transceiver is configured to:

receive the weightings of the plurality of subject identification methods from the server.

71. The apparatus of claim 69, wherein the server receives subject metadata from other IoT devices, and wherein the server generates the weightings of the plurality of subject identification methods based additionally on the subject metadata received from the other IoT devices.

72. The apparatus of claim 40, wherein the at least one processor is further configured to:

cause the transceiver to transmit a beacon signal to one or more IoT devices in a field of view of the camera application.

73. The apparatus of claim 72, wherein the beacon signal is transmitted to the one or more IoT devices in the field of view of the camera application using directional speakers.

74. The apparatus of claim 72, wherein the transceiver is configured to:

transmit a discovery message to the one or more IoT devices, the discovery message including filtering criteria; and

receive a beacon signal from each of a subset of the one or more IoT devices, wherein each IoT device of the subset of the one or more IoT devices has attributes that match the filtering criteria.

75. The apparatus of claim 74, wherein the at least one processor is further configured to:

receive the filtering criteria, wherein the filtering criteria is user defined and defines attribute values of the one or more IoT devices in the field of view of the camera application that should respond to the discovery message.

76. The apparatus of claim 74, wherein each of the one or more IoT devices in the field of view of the camera application determines whether or not it has attribute values that match the filtering criteria.

77. The apparatus of claim 76, wherein devices of the one or more IoT devices in the field of view of the camera application determine that they do not have attribute values that match the filtering criteria, and in response, do not respond to the received beacon signal or the discovery message.

78. The apparatus of claim 76, wherein each of the subset of the one or more IoT devices in the field of view of the camera application determine that it has attribute values that match the filtering criteria, and in response, responds to the discovery message by emitting a beacon signal and transmitting a contact card.

79. An apparatus for identifying an object associated with a nearby Internet of Things (IoT) device, comprising:

means for receiving identifying information associated with the nearby IoT device;

means for detecting a nearby object in a field of view of a camera application;

means for determining whether or not the nearby object is associated with the nearby IoT device based on the received identifying information matching information associated with the nearby object detected by the camera application in the field of view of the camera application; and

means for correlating the nearby object with information associated with the object associated with the nearby IoT device based on the nearby object being associated with the nearby IoT device.

80. A non-transitory computer-readable medium for identifying an object associated with a nearby Internet of Things (IoT) device, comprising:

at least one instruction to receive identifying information associated with the nearby IoT device;

at least one instruction to detect a nearby object in a field of view of a camera application;

at least one instruction to determine whether or not the nearby object is associated with the nearby IoT device based on the received identifying information matching information associated with the nearby object detected by the camera application in the field of view of the camera application; and

at least one instruction to correlate the nearby object with information associated with the object associated with the nearby IoT device based on the nearby object being associated with the nearby IoT device.