Native application testing

• US 10,032,186 B2
• Filed: 01/26/2016
• Issued: 07/24/2018
• Est. Priority Date: 07/23/2013
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Johnson, Michael Dudley, Tozer, Mathieu Benjamin, Roberto, Breno Pompeu
• Primary Examiner(s): Roumi, Mahran Abu
• Application Number:
  US15/006,477
• Publication Number:
  US 20160140600A1
• Time in Patent Office: 910 Days
• Field of Search: 709217
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0242 Determining effectiveness of advertisements
• CPC Class Codes: G06Q 30/0277 Online advertisement
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

In one embodiment, a method includes deploying an application to a plurality of client systems associated with a plurality of users, respectively, wherein the application is installed on each of the client systems, and wherein one or more treatments can be remotely activated on the application, receiving, from a first client system associated with a first user of the plurality of users, a request from the application installed on the first client system to access the application server, determining whether the first user belongs in a treatment group associated with a first treatment, remotely activating the first treatment on the first client system associated with the first user if the first user belongs in the treatment group, and synchronizing activation of the first treatment in response to subsequent access by the first client system.

Claims

1. A method comprising, by an application server:

deploying, by the application server, an application to a plurality of client systems associated with a plurality of users, respectively, wherein the application is installed on each of the client systems, and wherein one or more treatments can be remotely activated on the application;

receiving, from a first client system associated with a first user of the plurality of users, a request from the application installed on the first client system to access the application server;

determining, by the application server, whether the first user belongs in a treatment group associated with a first treatment;

remotely activating, by the application server, the first treatment on the first client system associated with the first user if the first user belongs in the treatment group, wherein remotely activating the first treatment comprises modifying the behavior of the application installed on the first client system associated with the first user; and

synchronizing, by the application server, remote activation of the first treatment on the application on the first client system associated with the first user in response to subsequent requests from the application installed on the first client system, wherein synchronizing remote activation of the first treatment on the application on the first client system associated with the first user comprises maintaining the first user in the treatment group and maintaining activation of the first treatment on the application on the first client system associated with the first user through one or more treatment-disabling events involving the first client system associated with the first user, wherein maintaining activation of the first treatment through a treatment-disabling event allows the first user to continue experiencing the modification of the application associated with the first treatment on the first client system after the treatment-disabling event.

2. The method of claim 1, further comprising:

receiving, from the first client system, exposure data identifying an exposure of the first user to the first treatment.

3. The method of claim 1, further comprising:

determining a treatment effect of the first treatment for the first user based on exposure data received from the first client system.

4. The method of claim 3, further comprising:

modifying the first treatment based on the determined treatment effect of the first treatment.

5. The method of claim 1, further comprising:

deactivating the first treatment on the first client system after a specified timeout period.

6. The method of claim 1, further comprising:

receiving, from the first client system, conversion data identifying user interactions with respect to the first treatment.

7. The method of claim 1, wherein synchronizing activation of the first treatment in response to subsequent access by the first client system comprises:

determining whether the first treatment is active on the first client system; and

remotely re-activating the first treatment on the first client system if the first treatment is determined to be not active on the first client system.

8. The method of claim 1, further comprising:

accessing a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, each of the edges between two of the nodes representing a single degree of separation between them, the plurality of nodes corresponding to a plurality of users associated with an online social network, respectively.

9. The method of claim 8, further comprising:

identifying a plurality of clusters in the social graph using graph cluster randomization, each cluster comprising a discrete set of nodes from the plurality of nodes, wherein a threshold number of nodes in each cluster is network exposed with respect to the other nodes in the cluster.

10. The method of claim 9, wherein the first user corresponds to a node in a first cluster of the plurality of clusters.

11. The method of claim 10, further comprising:

remotely activating the first treatment on one or more second client systems associated with one or more second users of the plurality of users, respectively, the one or more second client systems corresponding to one or more nodes in a second cluster of the plurality of clusters, the first cluster being discrete from the second cluster.

12. The method of claim 1, determining whether the first user belongs in the treatment group associated with a first treatment comprises determining whether the first user belongs in a treatment set of users or a control set of users.

13. The method of claim 1, wherein the first treatment is a particular advertisement.

14. The method of claim 1, wherein the first treatment is a particular product or feature associated with the application.

15. The method of claim 1, wherein the client systems are mobile client systems.

16. The method of claim 1, wherein the application server is a server associated with an online social network.

17. The method of claim 1, wherein the application is a mobile application for accessing an online social network.

18. The method of claim 1, wherein the one or more treatment-disabling events comprises one or more of:

a cache-clearing event;

a deletion event;

or

a deployment event.

19. The method of claim 1, wherein maintaining activation of the first treatment on the first client system through one or more treatment-disabling events comprises remotely re-activating the treatment of the application responsive to determining the treatment is inactive on the first client system following a treatment-disabling event.

20. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

deploy, by an application server, an application to a plurality of client systems associated with a plurality of users, respectively, wherein the application is installed on each of the client systems, and wherein one or more treatments can be remotely activated on the application;

receive, from a first client system associated with a first user of the plurality of users, a request from the application installed on the first client system to access the application server;

determine, by the application server, whether the first user belongs in a treatment group associated with a first treatment;

remotely activate, by the application server, the first treatment on the first client system associated with the first user if the first user belongs in the treatment group, wherein remotely activating the first treatment comprises modifying the behavior of the application installed on the first client system associated with the first user; and

synchronize, by the application server, remote activation of the first treatment on the application on the first client system associated with the first user in response to subsequent requests from the application installed on the first client system, wherein synchronizing remote activation of the first treatment on the application on the first client system associated with the first user comprises maintaining the first user in the treatment group and maintaining activation of the first treatment on the application on the first client system associated with the first user through one or more treatment-disabling events involving the first client system associated with the first user, wherein maintaining activation of the first treatment through a treatment-disabling event allows the first user to continue experiencing the modification of the application associated with the first treatment on the first client system after the treatment-disabling event.

21. A system comprising:

one or more processors; and

a memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to;

deploy, by an application server, an application to a plurality of client systems associated with a plurality of users, respectively, wherein the application is installed on each of the client systems, and wherein one or more treatments can be remotely activated on the application;

receive, from a first client system associated with a first user of the plurality of users, a request from the application installed on the first client system to access the application server;

determine, by the application server, whether the first user belongs in a treatment group associated with a first treatment;

remotely activate, by the application server, the first treatment on the first client system associated with the first user if the first user belongs in the treatment group, wherein remotely activating the first treatment comprises modifying the behavior of the application installed on the first client system associated with the first user; and

synchronize, by the application server, remote activation of the first treatment on the application on the first client system associated with the first user in response to subsequent requests from the application installed on the first client system, wherein synchronizing remote activation of the first treatment on the application on the first client system associated with the first user comprises maintaining the first user in the treatment group and maintaining activation of the first treatment on the application on the first client system associated with the first user through one or more treatment-disabling events involving the first client system associated with the first user, wherein maintaining activation of the first treatment through a treatment-disabling event allows the first user to continue experiencing the modification of the application associated with the first treatment on the first client system after the treatment-disabling event.

22. The system of claim 21, the processors being further operable when executing the instructions to:

receive, from the first client system, exposure data identifying an exposure of the first user to the first treatment.

23. The system of claim 21, the processors being further operable when executing the instructions to:

determine a treatment effect of the first treatment for the first user based on exposure data received from the first client system.

24. The system of claim 23, the processors being further operable when executing the instructions to:

modify the first treatment based on the determined treatment effect of the first treatment.

25. The system of claim 21, the processors being further operable when executing the instructions to:

deactivate the first treatment on the first client system after a specified timeout period.

26. The system of claim 21, the processors being further operable when executing the instructions to:

receive, from the first client system, conversion data identifying user interactions with respect to the first treatment.

27. The system of claim 21, wherein synchronizing activation of the first treatment in response to subsequent access by the first client system comprises:

determining whether the first treatment is active on the first client system; and

remotely re-activating the first treatment on the first client system if the first treatment is determined to be not active on the first client system.

28. The system of claim 21, the processors being further operable when executing the instructions to:

access a social graph comprising a plurality of nodes and a plurality of edges connecting the nodes, each of the edges between two of the nodes representing a single degree of separation between them, the plurality of nodes corresponding to a plurality of users associated with an online social network, respectively.

29. The system of claim 28, the processors being further operable when executing the instructions to:

identify a plurality of clusters in the social graph using graph cluster randomization, each cluster comprising a discrete set of nodes from the plurality of nodes, wherein a threshold number of nodes in each cluster is network exposed with respect to the other nodes in the cluster.

30. The system of claim 29, wherein the first user corresponds to a node in a first cluster of the plurality of clusters.

31. The system of claim 30, the processors being further operable when executing the instructions to:

remotely activate the first treatment on one or more second client systems associated with one or more second users of the plurality of users, respectively, the one or more second client systems corresponding to one or more nodes in a second cluster of the plurality of clusters, the first cluster being discrete from the second cluster.

32. The system of claim 21, wherein determining whether the first user belongs in the treatment group associated with a first treatment comprises determining whether the first user belongs in a treatment set of users or a control set of users.

33. The system of claim 21, wherein the first treatment is a particular advertisement.

34. The system of claim 21, wherein the first treatment is a particular product or feature associated with the application.

35. The system of claim 21, wherein the client systems are mobile client systems.

36. The system of claim 21, wherein the application server is a server associated with an online social network.

37. The system of claim 21, wherein the application is a mobile application for accessing an online social network.

38. The system of claim 21, wherein the one or more treatment-disabling events comprises one or more of:

a cache-clearing event;

a deletion event;

or

a deployment event.

39. The system of claim 21, wherein maintaining activation of the first treatment on the first client system through one or more treatment-disabling events comprises remotely re-activating the treatment of the application responsive to determining the treatment is inactive on the first client system following a treatment-disabling event.