- 2020-10-19 13:55
*views 5*- Loss
- Information entropy
- machine learning
- CrossEntropy
- artificial intelligence

Supervised learning is mainly divided into two categories ：

* classification ： The target variable is discrete , For example, judge whether a watermelon is good or bad , Then the target variable can only be 1（ Good melon ）,0（ Bad melon ）

* regression ： The target variable is continuous , Such as predicting the sugar content of watermelon （0.00~1.00）

Classification is mainly divided into ：

* Dichotomy ： For example, judge whether a watermelon is good or bad

* Multiclassification ： Such as judging a watermelon variety , Black Beauty , Te Xiaofeng , Annong 2, etc

The cross entropy loss function is the most commonly used loss function in classification , Cross entropy is used to measure the difference between two probability distributions , It is used to measure the difference between the learned distribution and the real distribution .

<> Dichotomy

In the case of dichotomy , Only two values can be predicted for each category , Suppose the prediction is good melon （1） The probability is P P P, Bad melon （0） The probability is 1 − P 1-P 1−P：

be , The general form of cross entropy loss is , among y Label ：

How to understand this formula ?

It is predicted to be a good melon （1） The probability is P P P： P ( y = 1 ∣ x ) = P P(y=1|x)=P P(y=1∣x)=P,

Predicted as bad melon （0） The probability is 1 − P 1-P 1−P： P ( y = 0 ∣ x ) = 1 − P P(y=0|x)=1-P P(y=0

∣x)=1−P

be , P ( y ∣ x ) = P y ( 1 − P ) 1 − y P(y|x)=P^y(1-P)^{1-y} P(y∣x)=Py(1

−P)1−y, When y by 1 Time , P ( y ∣ x ) = P P(y|x)=P P(y∣x)=P, When y by 0 Time , P ( y ∣ x ) = 1 − P

P(y|x)=1-PP(y∣x)=1−P.

Principle of cross entropy function formula ：

Learn something , We need to get to know him , Otherwise, it will stay on the surface , We can't go any further , How is the cross entropy obtained , How to understand him ?

First, understand the following concepts ：

*

information content

The amount of information represents the degree to which an information eliminates uncertainty , For example, China's current high-speed rail technology ranks first in the world , The probability is zero 1, The sentence itself is certain , There is no elimination of any uncertainty . China's high-speed rail technology will always remain the first in the world , This sentence is an uncertain event , It contains a large amount of information .

The amount of information is inversely proportional to the probability of an event .

*

Information entropy

Information entropy is the expectation of the amount of information that may be produced before the results come out , Expectation can be understood as the probability of all possible results multiplied by the corresponding result .

Information entropy is used to measure the uncertainty of things . The bigger the information entropy is （ The more information you have ,P The smaller ）, The more uncertain things become , The more complex things are .

*

Relative entropy ( Namely KL divergence )

Relative entropy is also called mutual entropy , set up p ( x ) p(x) p(x) and q ( x ) q(x) q(x) Is the value of the two probability distributions , The relative entropy is used to express

The difference between two probability distributions , When two random distributions are the same , Their relative entropy is zero , When the difference between two random distributions increases , Their relative entropy also increases .：

（ You can think of it that way , For a binary classification ,p Either for 0 Either for 1, from log Image knowledge ,p by 0 The time value is 0, by 1 Time ,q The closer it gets 1, l o g p q log{p \over q} l

ogqp The smaller , The closer to 0）（ there log The base number is 2）

*

Cross entropy

By expanding the relative entropy, we can get the following results ：

It can be seen from the above formula , Relative entropy = Cross entropy - Information entropy ,H(p,q) It's cross entropy ：

Because in machine learning and deep learning , Samples and labels are known （ Namely p Known ）, So information entropy H（p） Equivalent to constant , here , Just fit the cross entropy , Make the cross entropy fit to 0 that will do .

<> Multiclassification

Multiclass is similar to dichotomy , The second category label is 1 and 0, And multi classification can be used one-hot Code , Now we need to predict watermelon varieties , There are black beauties , Te Xiaofeng , Annong 2 , If the real label is texiaofeng, i.e （0,1,0）, The forecast label is anlong-2 （0,0,1）, Bring in the probabilities of prediction tags , The case of multi classification is actually an extension of two classification ：

y i k y_{ik} yik Denotes the second i i i The real label of samples is k k k , share K K K Label values N N N Samples ,

p i , k p_{i,k}pi,k Denotes the second i i i The second sample is predicted to be the third k k k The probability of a tag value . By fitting the loss function , It also increases the distance between classes to a certain extent .

Technology

- Python149 blogs
- Java133 blogs
- Vue87 blogs
- Flow Chart79 blogs
- javascript44 blogs
- C++42 blogs
- programing language38 blogs
- MySQL37 blogs
- more...

Daily Recommendation

views 6

©2020-2021 ioDraw All rights reserved

vue3 reactive Implement responsive Appgamekit Making Xiaole games （ Code attached ）#3【 Graduation season 】 There is no access control after leaving , But remember to go home early front end — Every day 5 Pavement test questions （ twelve ）Error updating database. Cause: java.sql.SQLSyntaxErrorException: ORA-00947: Not enough values ForkJionGantt chartUVM Message printer system uvm_report ( one )python One line of code _ Baidu information search _python One line of code JavaScript study （ Eighty-eight ）— Summary of array knowledge points , Super detail !!!