Predictive action decision device and action decision method

• US 20060155664A1
• Filed: 01/30/2004
• Published: 07/13/2006
• Est. Priority Date: 01/31/2003
• Status: Active Grant

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Morikawa, Koji, Omori, Takashi, Oka, Natsuki, Ohigashi, Yu
• Granted Patent: US 7,107,107 B2
• Time in Patent Office: Days
• US Class Current: 706/47
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

In a predictive action determination apparatus (10), a state observation section (12) observes a state with respect to an environment (11) and obtains state data s(t). An environment prediction section (13) predicts, based on the state data s(t), a future state change in the environment. A target state determination section (15) determines, as a target state, a future state suitable for action determination with reference to a state value storage section (14). A prediction-based action determination section (16) determines an action based on a determined target state.

Claims

1. A predictive action determination apparatus comprising:

a state observation section for observing a state with respect to a predetermined environment and obtaining state data;

a state value storage section for storing a state value for each of states of the environment;

an environment prediction section for predicting a future state change in the environment, based on the state data obtained by the state observation section;

a target state determination section for determining, as a target state, a future state suitable for action determination among future states predicted by the environment prediction section, based on the state value for each of future states stored in the state value storage section; and

a first action determination section for determining an action of the apparatus, based on the target state determined by the target state determination section,

wherein the environment prediction section predicts a future state change in the environment, which is not influenced by actions of the apparatus.

3. The apparatus of claim 1, wherein the target state determination section determines, as a target state, a future state of which a state value is maximal.

4. The apparatus of claim 1, further comprising a state value update section for updating, by learning, the state value stored in the state value storage section,

wherein the target state determination section determines, as the target state, one of the future states of which the state value has been already updated by the state value update section.

6. The apparatus of claim 1,

wherein the state value storage section stores a state value for a state including the apparatus, and

wherein the apparatus further includes:

a value conversion section for obtaining, based on the state value stored in the state value storage section, a state value for a future state which is predicted by the environment prediction section and does not include the apparatus and giving the obtained state value to the target state determination section.

7. The apparatus of claim 1, further comprising:

a second action determination section for determining an action of the apparatus, based on a predetermined action rule; and

an action selection section for receiving actions determined by the first and second action determination section, respectively, as first and second action candidates and selecting one of the first and second action candidates as an actual action.

8. The apparatus of claim 7,

wherein the target state determination section gives a selection signal indicating whether or not a target state could be determined to the action selection section, and

wherein if the selection signal indicates that a target state is determined, the action selection section selects the first action candidate while if the selection signal indicates that a target state could not be determined, the action selection section selects the second action candidate.

9. The apparatus of claim 1,

wherein the first action determination section includes:

a state-change-with-action detection section for receiving the state data and detecting, from a current state indicated by the state data, a state and an action in a previous step;

a state-change-with-action storage section for storing, as a state change, a combination of the current state and the state and the action in the previous step detected by the state-change-with-action detection section; and

an action planning section for searching the state-change-with-action storage section for a history of a state change in a period between the current state and the target state and determining an action, based on a result of the search.

10. The apparatus of claim 9, wherein the action planning section performs a backward search in the direction from the target state to the current state when the state change storage section performs the search.

2. (canceled)

5. A predictive action determination apparatus comprising:

a state observation section for observing a state with respect to a predetermined environment and obtaining state data;

a state value storage section for storing a state value for each of states of the environment;

an environment prediction section for predicting a future state change in the environment, based on the state data obtained by the state observation section;

a target state determination section for determining, as a target state, a future state suitable for action determination among future states predicted by the environment prediction section, based on the state value for each of future states stored in the state value storage section; and

a first action determination section for determining an action of the apparatus, based on the target state determined by the target state determination section,

wherein the target state determination section discounts the state value obtained from the state value storage section according to the number of steps from a current step and uses the discounted state value.

11. A predictive action determination apparatus comprising:

a state observation section for observing a state with respect to a predetermined environment and obtaining state data;

a state value storage section for storing a state value for each of states of the environment;

an environment prediction section for predicting a future state change in the environment, based on the state data obtained by the state observation section;

a target state determination section for determining, as a target state, a future state suitable for action determination among future states predicted by the environment prediction section, based on the state value for each of future states stored in the state value storage section; and

a first action determination section for determining an action of the apparatus, based on the target state determined by the target state determination section,

wherein the environment prediction section includes;

a state change detection section for receiving the state data and detecting a state in a previous step from a current state indicated by the state data;

a state change storage section for storing, as a state change, a combination of the current state and the state in the previous step detected by the state change detection section; and

a state prediction section for predicting a state after the current state from the state change storage section.

12. A method of determining in a predictive action determination apparatus an action of the apparatus, comprising:

a first step of observing a state with respect to a predetermined environment and obtaining state data;

a second step of predicting a future state change in the environment, based on the obtained state data;

a third step of determining, as a target state, a future state suitable for action determination among predicted future states, with reference to the state value for each of the future states; and

a fourth step of determining the action of the apparatus, based on the determined target state,

wherein a predicted state change is a future state change in the environment, which is not influenced by actions of the apparatus.

14. The method of claim 12,

wherein in the third step,

one of the future states of which a state value is maximal is determined as the target state.

15. The method of claim 12,

wherein the predictive action determination apparatus updates a state value for each of states of the environment by learning, and

wherein in the third step, one of the future states of which a state value has been already updated is determined as a target state.

13. (canceled)