Algorithm flow

1. The main architecture can be broken down into the following parts to understand :

     
 * first DDPG Is characterized by actor Although it is PG Architecture of , however actor The output action value is not a probability distribution , It's a deterministic continuous action choice ( It is suitable for continuous action ), Its network is to give state as input , The network then gives an action as output , In this state , The best action value to take ,  And then this online Network update is based on critic The network provides the basis for updating , That is the following formula , Will take advantage of this s and a obtain critic Evaluated Q value , Then change the gradient to tell the network how to take the action value next time .【 That is, its optimized training online Network time , Need to be based on critic A description of the action provided Q Value evaluation for gradient adjustment 】Q yes critic Extracted ,Q The gradient of values is actually the same critic Network computing provides .

                        

        stay actor Network target The network part is entirely a segment   Time of soft update , that is  
Save the parameters after a period of time ( this target The main function of the network is to use  
critic Network target The real part selects the next action , That is to say, the behavior will be selected according to the actual network situation here ) 
In fact, there is a question here ,actor The input to the network is state , Output is the only action , that critic What about the Internet , It's input state and behavior , output Q Is it worth it ?  Or input status , Enter the Q value , And then choose from them ? Which of the two ways .....【 After searching, we found that , here critic The network should be the first form , The input is   State and action , The output is corresponding to this action Q value 】

        * stay critic part , It also includes two parts , Respectively online and target part ,
online Some of them are very similar to those before DDQN mode , That is, the network will be based on input behavior and state Q Value output ,  And then according to the utilization target Real value calculation provided by network TD-error Network training , And after a period of time target Network update .

       
* The third part is   Behavior exploration and memory renewal .  Behavioral exploration corresponds to 1 and 2 Two parts , The early stage tends to be a kind of random exploration with noise , This will increase their ability to explore , Every behavior exploration will produce state transition and immediate reward .

2. The use of memory corresponds to 3 and 4 Two parts , that is actor While exploring the network, some intuitive environmental feedback will be saved ( Prophase biased random , In the later stage, they become more intelligent ), And then online training   It is the continuous sampling of memory for two parts of network training , But remember that it should be a fragmented way of memory , There is no need for a strong correlation between the memories used by the two networks , 
  Only when updating , There will be some use of each other between networks .

Algorithm structure

 

Symbolic meaning

1: According to certain noisy behavior strategy β choice behavior . 

2: The environment is given according to the selected action   Rewards and new status responses .   

3: be similar to DDQN That kind of thing , Memory with or without intelligence will be stored .

4: Select batches from the memory and train the two networks in different forms and utilization .   

5: stay critic in ,Q The real network takes the next state and the next selected action as the network input ,r+Q(next) Computational acquisition Q Realistic value .

6:critic Network through computing td-error The network is updated after gradient .   

7:actor Network  
online When the network is updated , Focus on your current behavior , It is necessary to calculate and adjust the gradient , Make the network in the same state tend to generate better action choice ,  So rely on this action Of Q value ,Q Value, you need to critic Network computing results . 
 

8: It is the gradient result calculated by the optimizer to update the network parameters . 

9: Soft update between two network forms .

         

Technology
©2020 ioDraw All rights reserved
Swing actual combat Understanding closure about vue in el-date-picker type=daterange The problem of date not echoing Ubuntu 18.04 swap Partition expansion Snake code --c Language Edition visual c++6.0 open 【Golang Basic series 10 】Go language On conditional sentences ifString class —— summary ,String The nature of , Memory resolution , Commonly used APIAdobe Illustrator Publish native support Apple Silicon Test version of the software study java My first class antd table sort