MONITORING AND COMPARING FEATURES ACROSS ENVIRONMENTS

• US 20190325351A1
• Filed: 04/20/2018
• Published: 10/24/2019
• Est. Priority Date: 04/20/2018
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Stein, David J., Wang, Ruoyang, Wu, Ke, Chen, Bee-Chung, Gariba, Priyanka
• Application Number:
  US15/958,999
• Publication Number:
  US 20190325351A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

The disclosed embodiments provide a system for processing data. During operation, the system selects a set of entity keys associated with reference feature values used with one or more machine learning models, wherein the reference feature values are generated in a first environment. Next, the system matches the set of entity keys to feature values from a second environment. The system then compares the feature values and the reference feature values to assess a consistency of a feature across the first and second environments. Finally, the system outputs a result of the assessed consistency for use in managing the feature in the first and second environments.

Claims

1. A method, comprising:

selecting, by one or more computer systems, a set of entity keys associated with reference feature values used with a machine learning model, wherein the reference feature values are generated in a first environment;

matching, by the one or more computer systems, the set of entity keys to feature values from a second environment;

comparing the feature values and the reference feature values to assess a consistency of a feature across the first and second environments; and

outputting a result of the assessed consistency for use in managing the feature in the first and second environments.

2. The method of claim 1, wherein selecting the set of entity keys associated with the reference feature values comprises:

sampling the set of entity keys from a data set comprising the reference feature values.

3. The method of claim 2, wherein sampling the set of entity keys from the data set comprises:

selecting the set of entity keys based on an entity domain associated with the entity keys.

4. The method of claim 3, wherein the entity domain is associated with at least one of:

a member;

a company;

a job; and

a location.

5. The method of claim 2, wherein selecting the set of entity keys associated with the reference feature values further comprises:

transmitting the set of entity keys in a stream of messages.

6. The method of claim 1, wherein matching the set of entity keys to values of the feature from the second environment further comprises:

obtaining an anchor comprising metadata for accessing the feature in the second environment; and

using the anchor and the set of entity keys to retrieve the feature values of the feature from the second environment.

7. The method of claim 1, wherein comparing the values and the reference feature values comprises:

obtaining a first data set comprising the feature values;

obtaining a second data set comprising the reference feature values; and

applying a comparison to records in the first and second data sets.

8. The method of claim 7, wherein applying the comparison to the first and second data sets comprises:

for each entity key in the set of entity keys, using the entity key to obtain a first record in the first data set and a second record in the second data set; and

comparing a reference feature value of the feature from the first record and a feature value of the feature from the second record.

9. The method of claim 1, wherein the result comprises at least one of:

a first proportion of entity keys with matching feature values in the first and second environments; and

a second proportion of entity keys with similar feature values in the first and second environments.

10. The method of claim 1, wherein comparing the feature values and the reference feature values comprises:

applying a hypothesis test to the feature values and the reference feature values to determine a distribution-level consistency in the feature.

11. The method of claim 10, wherein the result comprises a subset of the feature values that contribute to a lack of the distribution-level consistency in the feature.

12. The method of claim 1, wherein:

the first environment comprises an offline environment; and

the second environment comprises an online environment.

13. A system, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the system to;

select a set of entity keys associated with reference feature values used with one or more machine learning models, wherein the reference feature values are generated in a first environment;

match the set of entity keys to feature values from a second environment;

compare the feature values and the reference feature values to assess a consistency of a feature across the first and second environments; and

output a result of the assessed consistency for use in managing the feature in the first and second environments.

14. The system of claim 13, wherein selecting the set of entity keys associated with the reference feature values comprises:

sampling the set of entity keys from a data set comprising the reference feature values.

15. The system of claim 13, wherein matching the set of entity keys to values of the feature from the second environment further comprises:

obtaining an anchor comprising metadata for accessing the feature in the second environment; and

using the anchor and the set of entity keys to retrieve the feature values of the feature from the second environment.

16. The system of claim 13, wherein comparing the values and the reference feature values comprises:

obtaining a first data set comprising the feature values;

obtaining a second data set comprising the reference feature values; and

applying a comparison to records in the first and second data sets.

17. The system of claim 16, wherein applying the comparison to the first and second data sets comprises:

for each entity key in the set of entity keys, using the entity key to obtain a first record in the first data set and a second record in the second data set; and

comparing a reference feature value of the feature from the first record and a feature value of the feature from the second record.

18. The system of claim 13, wherein comparing the feature values and the reference feature values comprises:

applying a hypothesis test to the feature values and the reference feature values to determine a distribution-level consistency in the feature.

19. The system of claim 13, wherein:

the first environment comprises an offline environment; and

the second environment comprises an online environment.

20. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

selecting a set of entity keys associated with reference feature values used with one or more machine learning models, wherein the reference feature values are generated in a first environment;

matching the set of entity keys to feature values from a second environment;

comparing the feature values and the reference feature values to assess a consistency of a feature across the first and second environments; and

outputting a result of the assessed consistency for use in managing the feature in the first and second environments.