Supervised learning is mainly divided into two categories :

* classification : The target variable is discrete , For example, judge whether a watermelon is good or bad , Then the target variable can only be 1( Good melon ),0( Bad melon )
* regression : The target variable is continuous , Such as predicting the sugar content of watermelon (0.00~1.00)
        Classification is mainly divided into :

* Dichotomy : For example, judge whether a watermelon is good or bad
* Multiclassification : Such as judging a watermelon variety , Black Beauty , Te Xiaofeng , Annong 2, etc
        The cross entropy loss function is the most commonly used loss function in classification , Cross entropy is used to measure the difference between two probability distributions , It is used to measure the difference between the learned distribution and the real distribution .

<> Dichotomy

        In the case of dichotomy , Only two values can be predicted for each category , Suppose the prediction is good melon (1) The probability is P P P, Bad melon (0) The probability is 1 − P 1-P 1−P:
        be , The general form of cross entropy loss is , among y Label :

How to understand this formula ?

        It is predicted to be a good melon (1) The probability is P P P: P ( y = 1 ∣ x ) = P P(y=1|x)=P P(y=1∣x)=P,
        Predicted as bad melon (0) The probability is 1 − P 1-P 1−P: P ( y = 0 ∣ x ) = 1 − P P(y=0|x)=1-P P(y=0
        be , P ( y ∣ x ) = P y ( 1 − P ) 1 − y P(y|x)=P^y(1-P)^{1-y} P(y∣x)=Py(1
−P)1−y, When y by 1 Time , P ( y ∣ x ) = P P(y|x)=P P(y∣x)=P, When y by 0 Time , P ( y ∣ x ) = 1 − P

Principle of cross entropy function formula :

        Learn something , We need to get to know him , Otherwise, it will stay on the surface , We can't go any further , How is the cross entropy obtained , How to understand him ?
First, understand the following concepts :

information content
The amount of information represents the degree to which an information eliminates uncertainty , For example, China's current high-speed rail technology ranks first in the world , The probability is zero 1, The sentence itself is certain , There is no elimination of any uncertainty . China's high-speed rail technology will always remain the first in the world , This sentence is an uncertain event , It contains a large amount of information .
The amount of information is inversely proportional to the probability of an event .

Information entropy
        Information entropy is the expectation of the amount of information that may be produced before the results come out , Expectation can be understood as the probability of all possible results multiplied by the corresponding result .

        Information entropy is used to measure the uncertainty of things . The bigger the information entropy is ( The more information you have ,P The smaller ), The more uncertain things become , The more complex things are .

Relative entropy ( Namely KL divergence )
        Relative entropy is also called mutual entropy , set up p ( x ) p(x) p(x) and q ( x ) q(x) q(x) Is the value of the two probability distributions , The relative entropy is used to express
The difference between two probability distributions , When two random distributions are the same , Their relative entropy is zero , When the difference between two random distributions increases , Their relative entropy also increases .:

( You can think of it that way , For a binary classification ,p Either for 0 Either for 1, from log Image knowledge ,p by 0 The time value is 0, by 1 Time ,q The closer it gets 1, l o g p q log{p \over q} l
ogqp​ The smaller , The closer to 0)( there log The base number is 2)

Cross entropy
        By expanding the relative entropy, we can get the following results :

        It can be seen from the above formula , Relative entropy = Cross entropy - Information entropy ,H(p,q) It's cross entropy :

        Because in machine learning and deep learning , Samples and labels are known ( Namely p Known ), So information entropy H(p) Equivalent to constant , here , Just fit the cross entropy , Make the cross entropy fit to 0 that will do .

<> Multiclassification

Multiclass is similar to dichotomy , The second category label is 1 and 0, And multi classification can be used one-hot Code , Now we need to predict watermelon varieties , There are black beauties , Te Xiaofeng , Annong 2 , If the real label is texiaofeng, i.e (0,1,0), The forecast label is anlong-2 (0,0,1), Bring in the probabilities of prediction tags , The case of multi classification is actually an extension of two classification :

        y i k y_{ik} yik​ Denotes the second i i i The real label of samples is k k k , share K K K Label values N N N Samples ,
p i , k p_{i,k}pi,k​ Denotes the second i i i The second sample is predicted to be the third k k k The probability of a tag value . By fitting the loss function , It also increases the distance between classes to a certain extent .