Q-learning-based multi-rate transmission control (MRTC) scheme for RRC in WCDMA systems

• US 7,286,484 B2
• Filed: 01/10/2003
• Issued: 10/23/2007
• Est. Priority Date: 01/10/2003
• Status: Expired due to Fees

• Current Assignee: Chunghwa Telecom Co., Ltd.
• Original Assignee: Chunghwa Telecom Co., Ltd.
• Inventors: Chen, Yih-Shen, Chang, Chung-Ju, Chen, Jian-An, Ren, Fang-Chin
• Primary Examiner(s): Lee; Andrew C.
• Application Number:
  US10/339,600
• Publication Number:
  US 20040136321A1
• Time in Patent Office: 1,747 Days
• Field of Search: None
• US Class Current: 370/252
• CPC Class Codes: H04B 1/7103 the interference being multiple access interference
• CPC Class Codes: H04W 28/16 Central resource management; Negotiation of resources or communication parameters, e.g. negotiating bandwidth or QoS [Quality of Service]

Abstract

The present invention provides a method using Q-learning rule to accurately estimate the transmission cost for the multi-rate transmission control (MRTC) for RRC (radio resource control) in the WCDMA network. This method comprises the following iterative steps: first obtaining a system state and an action using an incoming request, a mean of interference and a variance of interference, then obtaining Q values based on the system state and the action. Afterwards, allocating an optimal action with the Q values, and when a new request with a new state arrives, going back to the first step and updating the Q values using the new state and a cost function based on the system state and the action.

Claims

1. A method for radio resource control (RRC) in the WCDMA system, comprising the following iterative steps:

obtaining a system state x and an action A(x) using an incoming request, a mean of interference I_m and a variance of interference I_v;

obtaining Q values based on x and A(x) by using I_m, I_v,a change of mean Δ

I_m and a change of variance Δ

I_v;

allocating an optimal action A*(x) using the Q values; and

updating the Q values using a state y and a cost function based on x and A*(x) when a new request with the state y arrives.

2. The method of claim 1, wherein the incoming request is a real-time request.

3. The method of claim 1, wherein the incoming request is a non-real-time request.

4. The method of claim 1, wherein the Q values are obtained by

obtaining x and A(x) as a function of (I_m+Δ

I_m) and (I_v+Δ

I_v); and

performing back-propagation learning rule to the function of (I_m+Δ

I_m) and (I_v+Δ

I_v).

5. The method of claim 1, wherein the Q values are further obtained by using an error signal as feedback to the Q values.

6. The method of claim 1, wherein the Q values are obtained with the following equations:

$Q<br><br>\left(x,A<br><br>\left(x\right)\right)=\{\begin{array}{cc}Q<br><br>\left(x,A<br><br>\left(x\right)\right)+\mathrm{\eta <br><br>}<br><br><br><br>\mathrm{\Delta <br><br>}<br><br><br><br>Q<br><br>\left(x,A<br><br>\left(x\right)\right),& \mathrm{if}<br><br><br><br>\left(x,a\right)<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{chosen}\\ Q<br><br>\left(x,A<br><br>\left(x\right)\right)& \mathrm{otherwise}.\end{array}$

wherein η

is a learning rate, 0≦

η

≦

1, and

$\mathrm{\Delta <br><br>}<br><br><br><br>Q<br><br>\left(x,A<br><br>\left(x\right)\right)=\left\{c<br><br>\left(x,A<br><br>\left(x\right)\right)+\mathrm{\gamma <br><br>}<br><br><br><br>\underset{B<br><br>\left(y\right)}{\mathrm{Min}}<br><br>\left[Q<br><br>\left(y,B<br><br>\left(y\right)\right)\right]\right\}-Q<br><br>\left(x,A<br><br>\left(x\right)\right)$

wherein c(x,A(x)) is a cost function, 0≦

γ

<

1 is a discounted factor and B(y) is an optimal action of y.

7. The method of claim 6, wherein the cost function c(x,A(x)) is a linear combination of performance measures of state x and A(x), each performance measure representing its corresponding QoS requirements of heterogeneous services.

8. The method of claim 1, wherein the obtaining Q values step further comprising assigning a set of initial Q values.

9. The method of claim 8, wherein the set of initial Q values are log-normally distributed.

14. The method of claim 1, wherein the Q values are obtained with the following equations:

$Q<br><br>\left(x,A<br><br>\left(x\right)\right)=\{\begin{array}{cc}Q<br><br>\left(x,A<br><br>\left(x\right)\right)+\mathrm{\eta <br><br>}<br><br><br><br>\mathrm{\Delta <br><br>}<br><br><br><br>Q<br><br>\left(x,A<br><br>\left(x\right)\right),& \mathrm{if}<br><br><br><br>\left(x,a\right)<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{chosen}\\ Q<br><br>\left(x,A<br><br>\left(x\right)\right)& \mathrm{otherwise}.\end{array}$

wherein η

is a learning rate, 0≦

η

≦

1, and

$\mathrm{\Delta <br><br>}<br><br><br><br>Q<br><br>\left(x,A<br><br>\left(x\right)\right)=\left\{c<br><br>\left(x,A<br><br>\left(x\right)\right)+\mathrm{\gamma <br><br>}<br><br><br><br>\underset{B<br><br>\left(y\right)}{\mathrm{Min}}<br><br>\left[Q<br><br>\left(y,B<br><br>\left(y\right)\right)\right]\right\}-Q<br><br>\left(x,A<br><br>\left(x\right)\right)$

wherein c(x,A(x)) is a cost function, 0≦

γ

<

1 is a discounted factor and B(y) is an optimal action of y.

15. The method of claim 14, wherein the cost function c(x,A(x)) is a linear combination of performance measures of state x and A(x), each performance measure representing its corresponding QoS requirements of heterogeneous services.

10. A method for radio resource control in the WCDMA system, comprising the following iterative steps:

obtaining a system state x and an action A(x) using an incoming request, a mean of interference I_m and a variance of interference I_v;

obtaining Q values using x and A(x);

allocating an optimal action A*(x) using the Q values; and

updating the Q values using a state y and a cost function based on x and A*(x) when a new request with the state y arrives.

11. The method of claim 10, wherein the incoming request is a real-time request.

12. The method of claim 10, wherein the incoming request is a non-real-time request.

13. The method of claim 10, wherein the Q values are further obtained by using an error signal as feedback to the Q values.

16. The method of claim 10, wherein the obtaining Q values step further comprising assigning a set of initial Q values.

17. The method of claim 16, wherein the set of initial Q values are log-normally distributed.

18. A method for radio resource control in the WCDMA system, comprising the following iterative steps:

obtaining a system state x and an action A(x) using an incoming request, a mean of interference I_m and a variance of interference I_v;

obtaining x and A(x) as a function of Im+Δ

I_m and I_v+Δ

I_v, wherein Δ

I_m and Δ

I_v representing the change of I_m and I_v respectively;

performing back-propagation learning rule to the function of I_m+Δ

I_m and I_v+Δ

I_v;

allocating an optimal action A*(x) using the Q values; and

updating the Q values using a state y and a cost function based on x and A*(x) when a new request with the state y arrives.

19. The method of claim 18, wherein the incoming request is a real-time request.

20. The method of claim 18, wherein the incoming request is a non-real-time request.

21. The method of claim 18, wherein the Q values are further obtained by using an error signal as feedback to the Q values.

22. The method of claim 18, wherein the obtaining Q values step further comprising assigning a set of initial Q values.

23. The method of claim 22, wherein the set of Q initial values are log-normally distributed.

31. The circuit of claim 23, wherein the Q-function computation circuit for obtaining Q values further comprising an input for assigning a set of initial Q values.

32. The circuit of claim 31, wherein the set of initial Q values are log-normally distributed.

24. A circuit for radio resource control in the WCDMA system, comprising:

a state construction circuit for obtaining a system state x and an action A(x) using an incoming request, a mean of interference I_m and a variance of interference I_v;

a Q-function computation circuit for obtaining Q values based on x and A(x) by using I_m, I_v,a change of mean Δ

I_m and a change of variance Δ

I_v;

a rate allocation circuit for allocating an optimal action A*(x) with the Q values; and

a Q-learning rule circuit for updating the Q values using state y and a cost function based on x and A*(x) when a new request with a state y arrives.

25. The circuit of claim 24, wherein the incoming request is a real-time request.

26. The circuit of claim 24, wherein the incoming request is a non-real-time request.

27. The circuit of claim 24, wherein the Q function computation circuit further comprising:

a resultant interference profile circuit for obtaining x and A(x) as a function of (I_m+Δ

I_m) and (I_v+Δ

I_v); and

a RBFN neural network for performing back-propagation learning rule to the function of (I_m+Δ

I_m) and (I_v+Δ

I_v).

28. The circuit of claim 24, wherein the Q values are further obtained by using an error signal as feedback to the Q values.

29. The circuit of claim 24, wherein the Q values are obtained with the following equations:

$Q<br><br>\left(x,A<br><br>\left(x\right)\right)=\{\begin{array}{cc}Q<br><br>\left(x,A<br><br>\left(x\right)\right)+\mathrm{\eta <br><br>}<br><br><br><br>\mathrm{\Delta <br><br>}<br><br><br><br>Q<br><br>\left(x,A<br><br>\left(x\right)\right),& \mathrm{if}<br><br><br><br>\left(x,a\right)<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{chosen}\\ Q<br><br>\left(x,A<br><br>\left(x\right)\right)& \mathrm{otherwise}.\end{array}$

wherein η

is a learning rate, 0≦

η

≦

1, and

$\mathrm{\Delta <br><br>}<br><br><br><br>Q<br><br>\left(x,A<br><br>\left(x\right)\right)=\left\{c<br><br>\left(x,A<br><br>\left(x\right)\right)+\mathrm{\gamma <br><br>}<br><br><br><br>\underset{B<br><br>\left(y\right)}{\mathrm{Min}}<br><br>\left[Q<br><br>\left(y,B<br><br>\left(y\right)\right)\right]\right\}-Q<br><br>\left(x,A<br><br>\left(x\right)\right)$

wherein c(x,A(x)) is a cost function, 0≦

γ

<

1 is a discounted factor and B(y) is an optimal action of y.

30. The circuit of claim 29, wherein the cost function c(x,A(x)) is a linear combination of performance measures of state x and A(x), each performance measure representing its corresponding QoS requirements of heterogeneous services.