Online asynchronous reinforcement learning from concurrent customer histories

• US 8,909,590 B2
• Filed: 09/28/2012
• Issued: 12/09/2014
• Est. Priority Date: 09/28/2011
• Status: Active Grant

• Current Assignee: NICE Systems Technologies UK Limited (Nice Ltd)
• Original Assignee: NICE Systems Technologies UK Limited (Nice Ltd)
• Inventors: Newnham, Leonard Michael, McFall, Jason Derek, Silver, David, Barker, David J
• Primary Examiner(s): Chang, Li-Wu
• Application Number:
  US13/631,032
• Publication Number:
  US 20130080377A1
• Time in Patent Office: 802 Days
• Field of Search: None
• US Class Current: 706/52
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

In one embodiment, an indication of a Decision Request or an Update Request may be received, where the Update Request is activated independent of user activity. A user state pertaining to at least one user may be received, obtained, accessed or constructed. For the Decision Request, one or more actions may be scored according to one or more value functions associated with a computing device, a policy associated with the computing device may be applied to identify one of the scored actions as a decision, and an indication of the decision may be provided or applied. For the Update Request, the one or more value functions and/or the policy may be updated. An indication of updates to the one or more value functions and/or an indication of updates to the policy may be provided.

Claims

1. A computer implemented method, comprising:

obtaining an indication that a decision has been requested or selected with respect to one or more users;

determining whether to schedule, request, or perform a set of one or more activities, the set of one or more activities including performing one or more updates and selecting one or more decisions, wherein the one or more updates are performed with respect to a value function approximating an expected reward over time for the one or more users and a policy for selecting additional decisions, and wherein the one or more decisions pertain to the one or more users; and

scheduling, requesting, or performing the set of one or more activities according to a result of the determining step, wherein scheduling, requesting, or performing the set of one or more activities comprises;

generating a sequence of requests, wherein the sequence of requests includes one or more Update Requests and one or more Decision Requests, wherein each request in the sequence of requests pertains to the one or more users; and

providing or transmitting each request in the sequence of requests or indication thereof according to a particular schedule, wherein each of the one or more Decision Requests indicates a request to select an additional decision with respect to the at least one user,

wherein each of the Update Requests indicates at least one of;

a request to update a value function approximating an expected reward over time for the one or more users and a request to update a policy for selecting additional decisions.

2. The computer implemented method of claim 1, wherein determining is performed in response to at least one of:

the indication that the decision has been requested or selected, providing or obtaining an Update Request pertaining to the one or more users, customer input and other input.

3. The computer implemented method of claim 1, wherein determining is performed based, at least in part, upon the decision that has been selected or an outcome of an update.

4. The computer implemented method of claim 1, wherein scheduling, requesting, or performing the set of one or more activities according to the result of the determining step comprises:

generating a schedule according to which the set of one or more activities are to be requested or performed.

5. A computer implemented method, comprising:

obtaining an indication that a decision has been requested or selected with respect to one or more users;

after obtaining the indication of the decision that has been requested or selected, requesting or performing a sequence of one or more activities according to a schedule, the sequence of one or more activities including performing one or more updates and selecting one or more decisions, wherein the one or more updates are performed with respect to a value function approximating an expected reward over time for the one or more users and a policy for selecting additional decisions, and wherein the one or more decisions pertain to the one or more users;

prior to requesting or performing the sequence of activities, generating the schedule according to which the sequence of one or more activities are to be requested or performed, wherein generating the schedule is performed in response to at least one of;

the indication that the decision was requested or selected, an indication that an update to the value function pertaining to the one or more users was requested, an indication that an update to the policy pertaining to the one or more users was requested, customer input and other input.

6. The computer implemented method of claim 5, wherein the schedule indicates one or more future times at which one or more activities in the sequence of activities is to be requested or performed.

7. The computer implemented method of claim 5, wherein a time at which each of the one or more activities is requested or performed is independent of activity of the one or more users.

8. The computer implemented of claim 5, wherein requesting or performing the sequence of one or more activities is performed in response to at least one of:

the indication that the decision has been requested or selected, an Update Request pertaining to the one or more users, customer input and other input.

9. The computer implemented method of claim 5, wherein requesting or performing the sequence of one or more activities is performed based, at least in part, upon the decision that has been selected or an outcome of an update.

10. The computer implemented method of claim 5, wherein a time at which each of the one or more activities is requested or performed is independent of activity of any of a plurality of users.

11. The computer implemented method of claim 5, the method further comprising;

receiving, obtaining, accessing or constructing a user state pertaining to the at least one user; and

prior to requesting or performing the sequence of activities, determining the schedule based, at least in part, upon the user state.

12. The computer implemented method of claim 5, the method further comprising:

prior to requesting or performing the sequence of activities, determining the schedule based, at least in part, upon a type of the one or more decisions to be selected.

13. The computer implemented method of claim 5, further comprising:

storing or providing the schedule.

15. The computer implemented method of claim 5, wherein the schedule is generated independent of activity of the one or more users.

16. The computer implemented method of claim 6, wherein the schedule indicates a future time that each of the one or more activities in the sequence is to be requested or performed, a type of activities in the sequence, a number of activities in the sequence and any combination thereof.

17. The computer implemented method of claim 5, wherein the indication that the decision has been requested or selected pertains to a first type of action and wherein the sequence of activities comprises selecting one or more decisions, wherein each of the decisions pertains to a corresponding type of action.

14. A computer implemented method comprising:

obtaining an indication that a decision has been requested or selected with respect to one or more users;

after obtaining the indication of the decision that has been requested or selected, requesting or performing a sequence of one or more activities according to a schedule, the sequence of one or more activities including performing one or more updates and selecting one or more decisions, wherein the one or more updates are performed with respect to a value function approximating an expected reward over time for the one or more users and a policy for selecting additional decisions, and wherein the one or more decisions pertain to the one or more users;

prior to requesting or performing the sequence of activities, generating the schedule according to which the sequence of one or more activities are to be requested or performed, wherein generating the schedule is performed based, at least in part, upon a decision that has been selected with respect to the one or more users or an outcome of an update pertaining to the one or more users.

18. A computer implemented method comprising:

obtaining an indication that a decision has been requested or selected with respect to one or more users;

after obtaining the indication of the decision that has been requested or selected, requesting or performing a sequence of one or more activities according to a schedule, the sequence of one or more activities including performing one or more updates and selecting one or more decisions, wherein the one or more updates are performed with respect to a value function approximating an expected reward over time for the one or more users and a policy for selecting additional decisions, and wherein the one or more decisions pertain to the one or more users,

wherein the sequence of activities comprises a plurality of decisions to be selected, wherein each of the plurality of decisions pertains to a corresponding type of action, wherein the type of action for each of the plurality of decisions to be selected is identified in the schedule, each of the plurality of decisions including one or more actions.

19. An apparatus, comprising:

a processor; and

a memory, at least one of the processor or the memory being adapted for;

obtaining an indication that a decision has been requested or selected with respect to one or more users;

after obtaining the indication of the decision that has been requested or selected, requesting or performing a sequence of one or more activities according to a schedule, the sequence of one or more activities including performing one or more updates and selecting one or more decisions, wherein the one or more updates are performed with respect to a value function approximating an expected reward over time for the one or more users and a policy for selecting additional decisions, and wherein the one or more decisions pertain to the one or more users,

wherein requesting or performing the sequence of activities comprises providing or transmitting a sequence of one or more requests,

wherein each of the requests in the sequence of requests is one of two types of requests, the two types of requests include a Decision Request and an Update Request and the schedule indicates a type of each of the requests in the sequence of requests and a number of each type of requests in the sequence of requests.

20. The apparatus of claim 19, wherein the schedule indicates one or more future times at which one or more activities in the sequence of activities is to be requested or performed.

21. The apparatus of claim 19, further comprising:

receiving, obtaining, accessing or constructing a user state pertaining to the one or more users; and

prior to providing or transmitting the sequence of requests, determining the particular schedule based, at least in part, upon the user state.

26. The apparatus of claim 19, wherein the sequence of requests comprises one or more Decision Requests generated to provide one or more of the following:

targeted content for a website, targeted advertisements for sending via email, targeted offers at a point of sale, or targeted messages for providing to the at least one user via a call center.

27. The apparatus of claim 19, wherein the sequence of requests comprises a sequence of one or more Update Requests, wherein each of the Update Requests indicates one or more requests to update at least one of a policy and a value function maintained by or associated with a network device or Decisioning Component.

28. The apparatus of claim 19, wherein providing or transmitting the sequence of requests comprises:

providing or transmitting each request in a sequence of requests by a Decisioning Component.

29. The apparatus of claim 19, wherein providing or transmitting the sequence of requests comprises:

providing or transmitting each request in the sequence of requests to a Decisioning Component.

22. An apparatus comprising:

a processor; and

a memory, at least one of the processor or the memory being adapted for;

obtaining an indication that a decision has been requested or selected with respect to one or more users;

after obtaining the indication of the decision that has been requested or selected, requesting or performing a sequence of one or more activities according to a schedule, the sequence of one or more activities including performing one or more updates and selecting one or more decisions, wherein the one or more updates are performed with respect to a value function approximating an expected reward over time for the one or more users and a policy for selecting additional decisions, and wherein the one or more decisions pertain to the one or more users, wherein requesting or performing the sequence of activities comprises providing or transmitting a sequence of one or more requests; and

at least one of the processor or the memory being further adapted for;

receiving, obtaining, accessing or constructing a first user state pertaining to the one or more users;

prior to generating a first portion of the sequence of requests, determining a first schedule based, at least in part, upon the first user state;

receiving, obtaining, accessing or constructing a second user state pertaining to the one or more users;

prior to generating a second portion of the sequence of requests, determining a second schedule based, at least in part, upon the second user state.

23. The apparatus of claim 22, wherein the schedule indicates a future time that each of the one or more requests in the sequence of requests is to he provided, generated, or transmitted.

24. The apparatus of claim 22, wherein the schedule indicates at least one of:

a type of requests in the sequence of requests and a number of requests in the sequence of requests.

25. The apparatus of claim 22, wherein the sequence of requests comprises one or more Decision Requests requesting selection of each of the one or more decisions.