Algorithm flow
1. The main architecture can be broken down into the following parts to understand ：
* first DDPG Is characterized by actor Although it is PG Architecture of , however actor The output action value is not a probability distribution , It's a deterministic continuous action choice （ It is suitable for continuous action ）, Its network is to give state as input , The network then gives an action as output , In this state , The best action value to take , And then this online Network update is based on critic The network provides the basis for updating , That is the following formula , Will take advantage of this s and a obtain critic Evaluated Q value , Then change the gradient to tell the network how to take the action value next time .【 That is, its optimized training online Network time , Need to be based on critic A description of the action provided Q Value evaluation for gradient adjustment 】Q yes critic Extracted ,Q The gradient of values is actually the same critic Network computing provides .
stay actor Network target The network part is entirely a segment Time of soft update , that is
Save the parameters after a period of time （ this target The main function of the network is to use
critic Network target The real part selects the next action , That is to say, the behavior will be selected according to the actual network situation here ）
In fact, there is a question here ,actor The input to the network is state , Output is the only action , that critic What about the Internet , It's input state and behavior , output Q Is it worth it ? Or input status , Enter the Q value , And then choose from them ? Which of the two ways .....【 After searching, we found that , here critic The network should be the first form , The input is State and action , The output is corresponding to this action Q value 】
* stay critic part , It also includes two parts , Respectively online and target part ,
online Some of them are very similar to those before DDQN mode , That is, the network will be based on input behavior and state Q Value output , And then according to the utilization target Real value calculation provided by network TD-error Network training , And after a period of time target Network update .
* The third part is Behavior exploration and memory renewal . Behavioral exploration corresponds to 1 and 2 Two parts , The early stage tends to be a kind of random exploration with noise , This will increase their ability to explore , Every behavior exploration will produce state transition and immediate reward .
2. The use of memory corresponds to 3 and 4 Two parts , that is actor While exploring the network, some intuitive environmental feedback will be saved （ Prophase biased random , In the later stage, they become more intelligent ）, And then online training It is the continuous sampling of memory for two parts of network training , But remember that it should be a fragmented way of memory , There is no need for a strong correlation between the memories used by the two networks ,
Only when updating , There will be some use of each other between networks .
Algorithm structure
Symbolic meaning
1： According to certain noisy behavior strategy β choice behavior .
2： The environment is given according to the selected action Rewards and new status responses .
3： be similar to DDQN That kind of thing , Memory with or without intelligence will be stored .
4： Select batches from the memory and train the two networks in different forms and utilization .
5： stay critic in ,Q The real network takes the next state and the next selected action as the network input ,r+Q（next） Computational acquisition Q Realistic value .
6:critic Network through computing td-error The network is updated after gradient .
7：actor Network
online When the network is updated , Focus on your current behavior , It is necessary to calculate and adjust the gradient , Make the network in the same state tend to generate better action choice , So rely on this action Of Q value ,Q Value, you need to critic Network computing results .
8： It is the gradient result calculated by the optimizer to update the network parameters .
9： Soft update between two network forms .
Technology