本系列基本不讲数学原理,只从代码角度去让读者们利用最简洁的Python代码实现机器学习方法。

集成模型发展到现在的XGboost,LightGBM,都是目前竞赛项目会采用的主流算法。是真正的具有做项目的价值。这两个方法都是具有很多GBM没有的特点,比如收敛快,精度好,速度快等等。但由于他们底层不是Python,没有进sklearn库,要自己单独安装,用法和sklearn库也不完全相同。

两种模型都有自己的原生用法和sklearn库接口的用法,下面把回归和分类问题案例都一一介绍。只是案例演示,因此使用都采用sklearn库的自带数据集,会方便一点。

模块安装

安装都很简单,在这里输入

 或者Win+R,打开CMD,输入下面两句
pip install xgboost pip install lightgbm
然后就等自动装了

参数详情

两个模型都要大量的超参数,后面可能用到。先了解一下,后面有不明白的再回头看

还有eta 表示学习率,默认值0.3;

gamma 表示叶子结点进一步分裂的阈值。即分裂这个节点让损失函数下降超过这个值才会进行分裂,默认值0;

max_leaves 表示最大叶子节点数,默认0;

max_bin 最大桶数量,默认值256;

min_child_weigh 表示子节点包含实例权重的最小和,防止过拟合用的,越大越不容易过拟合,默认值0;

subsmaple 表示训练样本的采样率,即划分多少去训练,若是训练前就划分了训练测试集就不用管,默认值1;

colsample_bytree 表示列采样率,默认值1;

colsample_bylevel 每一级每次分裂的采样率,默认值1;

scale_pos_weight 控制正负样本的权重平衡,取值应该设为负类样本量/正类样本量,默认值1;

predictor 预测器类型,默认值cpu_predictor, 还可以换显卡加速:gpu_predictor。

seed 随机数种子,默认0;

slient 打印运行的信息,默认打印,默认值0;

objective [默认值=reg:linear]
reg:linear– 线性回归

reg:logistic – 逻辑回归

binary:logistic – 二分类逻辑回归,输出为概率

binary:logitraw – 二分类逻辑回归,输出的结果为wTx

count:poisson –
计数问题的poisson回归,输出结果为poisson分布。在poisson回归中,max_delta_step的默认值为0.7 (used to
safeguard optimization)

multi:softmax – 设置 XGBoost 使用softmax目标函数做多分类,需要设置参数num_class(类别个数)

multi:softprob – 如同softmax,但是输出结果为ndata*nclass的向量,其中的值是每个数据分为每个类的概率。

eval_metric [默认值=取决于目标函数选择]
rmse: 均方根误差

mae: 平均绝对值误差

logloss: negative log-likelihood

error: 二分类错误率。其值通过错误分类数目与全部分类数目比值得到。对于预测,预测值大于0.5被认为是正类,其它归为负类。 error@t:
不同的划分阈值可以通过 ‘t’进行设置

merror: 多分类错误率

mlogloss: 多分类log损失

auc: 曲线下的面积

ndcg: Normalized Discounted Cumulative Gain

map: 平均正确率

一般来说,我们都会使用xgboost.train(params, dtrain)函数来训练我们的模型。这里的params指的是booster参数。

XGboost原生用法

分类
import numpy as np import pandas as pd #import pickle import xgboost as xgb
from sklearn.datasets import load_iris from sklearn.model_selection import
train_test_split #鸢尾花 iris=load_iris() X=iris.data y=iris.target X.shape,y.shape
最经典的3分类的鸢尾花数据集

划分训练测试集,将数据变为xgb需要的格式
# 做数据切分 X_train, X_test,y_train, y_test =
train_test_split(X,y,test_size=0.2,random_state=0) xgb_train =
xgb.DMatrix(X_train, y_train) xgb_test = xgb.DMatrix( X_test,y_test )
 设置参数
params =
{'objective':'multi:softmax','num_class':3,'booster':'gbtree','max_depth':5,
'eta':0.1, 'subsample':0.7, 'colsample_bytree':0.7}
训练
num_round=50 watchlist = [(xgb_train,'train'), (xgb_test,'test')] model =
xgb.train(params, xgb_train, num_round, watchlist)
 

 预测
pred = model.predict(xgb_test) pred

error_rate=np.sum(pred!=y_test)/y_test.shape[0] error_rate #错误率
 

回归问题
from sklearn.datasets import load_boston X,y=load_boston(return_X_y=True)
X.shape,y.shape
波士顿房价数据集,经典回归数据集
X_train,X_test,y_train,y_test=
train_test_split(X,y,test_size=0.2,random_state=0)
print(X_train.shape,X_test.shape,y_train.shape,y_test.shape) xgb_train =
xgb.DMatrix(X_train, y_train) xgb_test = xgb.DMatrix( X_test,y_test )
 设置参数
params = {'objective':'reg:squarederror','booster':'gbtree','max_depth':5,
'eta':0.1, 'min_child_weight':1} num_round=50 watchlist = [(xgb_train,'train'),
(xgb_test,'test')] model = xgb.train(params, xgb_train, num_round, watchlist)

pred = model.predict(xgb_test) pred,y_test
计算均方误差,拟合优度
from sklearn.linear_model import LinearRegression from sklearn.metrics import
mean_squared_error,r2_score reg=LinearRegression() reg.fit(y_test.reshape(-1,
1),pred.reshape(-1, 1)) reg.score(y_test.reshape(-1, 1),pred.reshape(-1, 1))
mean_squared_error(y_test,pred),r2_score(y_test,pred)
交叉验证
#交叉验证
result=xgb.cv(params=params,dtrain=xgb_train,nfold=10,metrics='rmse',#'auc'
num_boost_round=300,as_pandas=True,seed=123) result.shape result.head()
 

 画出交叉验证误差图
# Plot CV Errors import matplotlib.pyplot as plt plt.plot(range(1, 301),
result['train-rmse-mean'], 'k', label='Training Error') plt.plot(range(1, 301),
result['test-rmse-mean'], 'b', label='Test Error') plt.xlabel('Number of
Trees') plt.ylabel('RMSE') plt.axhline(0, linestyle='--', color='k',
linewidth=1) plt.legend() plt.title('CV Errors for XGBoost') plt.show()
 自定义目标函数和损失函数

XGboost还可以自定义损失函数和评价函数
from sklearn.datasets import load_breast_cancer
X,y=load_breast_cancer(return_X_y=True) X.shape,y.shape
乳腺癌数据集
X_train,X_test,y_train,y_test=
train_test_split(X,y,test_size=0.2,random_state=8)
print(X_train.shape,X_test.shape,y_train.shape,y_test.shape) xgb_train =
xgb.DMatrix(X_train, y_train) xgb_test = xgb.DMatrix( X_test,y_test )
 
params = {'booster':'gbtree','max_depth':5, 'eta':0.1} num_round=50 watchlist
= [(xgb_train,'train'), (xgb_test,'test')]
 定义损失函数和评价函数
def logregobj(preds,dtrain): labels=dtrain.get_label()
preds=1.0/(1.0+np.exp(-preds)) grad=preds-labels hess=preds*(1.0-preds) return
grad,hess def evalerror(preds,dtrain): labels=dtrain.get_label() return
'error',float(sum(labels!=(preds>0.0)))/len(labels)
训练
model = xgb.train(params, xgb_train, num_round,
watchlist,obj=logregobj,feval=evalerror)

 交叉验证也可以使用自定义函数
result=xgb.cv(params=params,dtrain=xgb_train,nfold=10,metrics='auc',#'auc'
num_boost_round=300,as_pandas=True,seed=123,obj=logregobj,feval=evalerror)
result.head()

# Plot CV Errors import matplotlib.pyplot as plt plt.plot(range(1, 301),
result['train-error-mean'], 'k', label='Training Error') plt.plot(range(1,
301), result['test-error-mean'], 'b', label='Test Error') plt.xlabel('Number of
Trees') plt.ylabel('AUC') plt.axhline(0, linestyle='--', color='k',
linewidth=1) plt.legend() plt.title('CV Errors for XGBoost') plt.show()
 

XGboost的sklearn库接口

sklearn库接口就好用很多,符合sklearn库的一些常用的函数,例如交叉验证,网格化搜参,变量筛选,都可以用。

回归
import xgboost as xgb import numpy as np import pandas as pd import
matplotlib.pyplot as plt from sklearn.model_selection import KFold,
train_test_split, GridSearchCV from sklearn.metrics import confusion_matrix,
mean_squared_error from sklearn.model_selection import train_test_split from
sklearn.datasets import load_iris, load_boston from sklearn.datasets import
load_breast_cancer X, y= load_boston(return_X_y=True)
拟合评价(经典sklearn用法)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
random_state=1) model = xgb.XGBRegressor(objective='reg:squarederror',
n_estimators=300, max_depth=6, subsample=0.6, colsample_bytree=0.8,
learning_rate=0.1, random_state=0) model.fit(X_train, y_train)
model.score(X_test, y_test)

pred = model.predict(X_test) rmse = np.sqrt(mean_squared_error(y_test, pred))
rmse
 

回归交叉验证 
rng = np.random.RandomState(123) kf = KFold(n_splits=3, shuffle=True,
random_state=rng) print("在3折数据上的交叉验证") for train_index, test_index in
kf.split(X): xgb_model = xgb.XGBRegressor(objective='reg:squarederror',
n_estimators=300, max_depth=6,subsample=0.6, colsample_bytree=0.8,
learning_rate=0.1, random_state=0).fit(X[train_index],y[train_index])
predictions = xgb_model.predict(X[test_index]) actuals = y[test_index]
print("均方根误差:") print(np.sqrt(mean_squared_error(actuals, predictions)))
print('拟合优度') print(xgb_model.score(X[test_index],y[test_index]))
 

回归网格化搜索最优超参数
# 回归网格化搜索最优超参数 model =
xgb.XGBRegressor(objective='reg:squarederror',subsample=0.6,
colsample_bytree=0.8, random_state=0,nthread=8) param_dict = {'max_depth':
[5,6,7,8],'n_estimators': [100,200,300],'learning_rate':[0.05,0.1,0.2]} clf =
GridSearchCV(model, param_dict, cv=10,verbose=1 , scoring='r2')
clf.fit(X_train, y_train) print(clf.best_score_) print(clf.best_params_)
 

分类交叉验证 
#二分类 rng = np.random.RandomState(123) X,y=load_breast_cancer(return_X_y=True)
print(X.shape,y.shape) kf = KFold(n_splits=3, shuffle=True, random_state=rng)
print("在3折数据上的交叉验证") for train_index, test_index in kf.split(X): xgb_model =
xgb.XGBClassifier(objective='binary:logistic',
n_estimators=300,random_state=0,eta=0.1,max_depth=6,
use_label_encoder=False,eval_metric=['logloss','auc','error']).fit(X[train_index],y[train_index])
predictions = xgb_model.predict(X[test_index]) actuals = y[test_index]
print("混淆矩阵:") print(confusion_matrix(actuals, predictions))

# 多分类:混淆矩阵 print("\nIris: 多分类") iris = load_iris() y = iris['target'] X =
iris['data'] kf = KFold(n_splits=5, shuffle=True, random_state=rng)
print("在5折数据上的交叉验证") for train_index, test_index in kf.split(X): xgb_model =
xgb.XGBClassifier(objective='multi:softmax',
n_estimators=300,random_state=0,eta=0.1,max_depth=6,
use_label_encoder=False,eval_metric=['logloss','auc','error']).fit(X[train_index],y[train_index])
predictions = xgb_model.predict(X[test_index]) actuals = y[test_index]
print("混淆矩阵:") print(confusion_matrix(actuals, predictions))

 分类网格化搜索最优超参数
# 网格化搜索最优超参数 print("参数最优化:") X,y=load_breast_cancer(return_X_y=True) xgb_model
=
xgb.XGBClassifier(objective='binary:logistic',random_state=0,use_label_encoder=False,eval_metric=['logloss','auc','error'])
param_dict = {'max_depth': [2,4,6],'n_estimators':
[50,100,200],'eta':[0.05,0.1,0.2]} clf = GridSearchCV(xgb_model, param_dict,
verbose=1) clf.fit(X,y) print(clf.best_score_) print(clf.best_params_)

早停 

和神经网络一样,可以使用早停防止过拟合
#早停 X_train, X_val, y_train, y_val = train_test_split(X, y,test_size=0.2,
random_state=0) clf =
xgb.XGBClassifier(objective='binary:logistic',use_label_encoder=False,random_state=0)
clf.fit(X_train, y_train, early_stopping_rounds=10,
eval_metric="auc",eval_set=[(X_val, y_val)])

变量重要性 

xgb包自带的画图用法
#变量重要性
xgb.plot_importance(clf,height=0.5,importance_type='gain',max_num_features=10)
 

 sklearn库用法
clf.feature_importances_

cancer=load_breast_cancer() cancer.feature_names sorted_index =
clf.feature_importances_.argsort() plt.figure(figsize=(10,5))
plt.barh(range(len(cancer.feature_names)),
clf.feature_importances_[sorted_index])
plt.yticks(np.arange(len(cancer.feature_names)),cancer.feature_names[sorted_index])
plt.xlabel('Feature Importance') plt.ylabel('Feature')
plt.title('GradientBoosting') #plt.savefig('梯度提升特征排序.png') plt.tight_layout()

变量筛选 

 根据变量重要性,小于阈值的变量就扔掉
from sklearn.feature_selection import SelectFromModel selection
=SelectFromModel(clf,threshold=0.05,prefit=True)
select_X_train=selection.transform(X_train) select_X_train.shape
threshold=0.05,表示变量重要性小于0.05就扔掉,最后只留下了四个变量(和上图也一致)

将测试集也筛选一下
select_X_val=selection.transform(X_val) select_X_val.shape

查看一些筛出来了那些变量
print(selection.get_support()) print(selection.get_support(True))
[cancer.feature_names[i] for i in selection.get_support(True)]
 

 xgb到这了

lightgbm用起来其实和xgboost差不多,就是参数有细微的差别,用sklearn库会更加一致,当然也展示一下原生用法。 

LightGBM原生用法
from sklearn.datasets import load_iris import lightgbm as lgb from lightgbm
import plot_importance import matplotlib.pyplot as plt from
sklearn.model_selection import train_test_split from sklearn.metrics import
accuracy_score # 加载鸢尾花数据集 iris = load_iris() X,y = iris.data,iris.target #
数据集分割 X_train,X_test,y_train,y_test =
train_test_split(X,y,test_size=0.2,random_state=123457) # 参数 params = {
'booster': 'gbtree', 'objective': 'multiclass', #回归:'objective': 'regression'
'num_class': 3, 'num_leaves': 31, 'subsample': 0.8, 'bagging_freq': 1,
'feature_fraction ': 0.8, 'slient': 1, 'learning_rate ': 0.01, 'seed': 0 } #
构造训练集 dtrain = lgb.Dataset(X_train,y_train) dtest = lgb.Dataset(X_test,y_test)
num_rounds = 500 model = lgb.train(params,dtrain, num_rounds,
valid_sets=[dtrain, dtest], verbose_eval=100, early_stopping_rounds=10)
 
# 对测试集进行预测 y_pred = model.predict(X_test) # 计算准确率 accuracy =
accuracy_score(y_test, np.argmax(y_pred, axis=1))
print('accuarcy:%.2f%%'%(accuracy*100)) # 显示重要特征 plot_importance(model)
plt.show() # 模型保存 #gbm.save_model('model.txt') # 模型加载 #gbm =
lgb.Booster(model_file='model.txt')

LightGBM的sklearn接口 

回归
from lightgbm import LGBMRegressor from lightgbm import plot_importance import
matplotlib.pyplot as plt from sklearn.model_selection import train_test_split
from sklearn.datasets import load_boston from sklearn.metrics import
mean_squared_error # 导入数据集 boston = load_boston() X ,y =
boston.data,boston.target X_train,X_test,y_train,y_test =
train_test_split(X,y,test_size=0.2,random_state=0) model = LGBMRegressor(
boosting_type='gbdt', num_leaves=31, max_depth=-1, learning_rate=0.1,
n_estimators=100, objective='regression', # 默认是二分类 min_split_gain=0.0,
min_child_samples=20, subsample=1.0, subsample_freq=0, colsample_bytree=1.0,
reg_alpha=0.0, reg_lambda=0.0, random_state=None, silent=True )
model.fit(X_train,y_train, eval_set=[(X_train, y_train), (X_test, y_test)],
verbose=100, early_stopping_rounds=50) # 对测试集进行预测 y_pred =
model.predict(X_test) mse = mean_squared_error(y_test,y_pred) print('mse', mse)
# 显示重要特征 plot_importance(model) plt.show()

 分类
from lightgbm import LGBMClassifier from sklearn.datasets import load_iris
from lightgbm import plot_importance import matplotlib.pyplot as plt from
sklearn.model_selection import train_test_split from sklearn.metrics import
accuracy_score # 加载样本数据集 iris = load_iris() X,y = iris.data,iris.target
X_train,X_test,y_train,y_test =
train_test_split(X,y,test_size=0.2,random_state=12343) model = LGBMClassifier(
max_depth=3, learning_rate=0.1, n_estimators=200, # 使用多少个弱分类器
objective='multiclass', num_class=3, booster='gbtree', min_child_weight=2,
subsample=0.8, colsample_bytree=0.8, reg_alpha=0, reg_lambda=1, seed=0 # 随机数种子
) model.fit(X_train,y_train, eval_set=[(X_train, y_train), (X_test, y_test)],
verbose=100, early_stopping_rounds=50) # 对测试集进行预测 y_pred =
model.predict(X_test) model.predict_proba #计算准确率 accuracy =
accuracy_score(y_test,y_pred) print('accuracy:%3.f%%'%(accuracy*100)) # 显示重要特征
plot_importance(model) plt.show()

 当然,上面使用验证集去早停和评估什么的比较麻烦,模型定义好后,训练可以简单一点
model.fit(X_train,y_train)

 评价预测什么的和sklearn库完全一样,可以参考我以前的文章。原生用法也可以参考xgb。
print(model.score(X_test,y_test)) model.predict(X_test)

 

更新

LGBM的参数其实很多,再放一张图

调参思路其实是:

1.先选取较大的学习率,加速收敛

2.对决策树等参数进行调整:max_depth,num_leaves,subsample,colsample_bytree

3.然后再对正则化参数调整,min_child_weight,lambda...

4.降低学习率,配合估计器个数,进行最后的调整。

必要的时候还可以使用早停机制。

技术
下载桌面版
GitHub
百度网盘(提取码:draw)
Gitee
云服务器优惠
阿里云优惠券
腾讯云优惠券
华为云优惠券
站点信息
问题反馈
邮箱:ixiaoyang8@qq.com
QQ群:766591547
关注微信