LR模型实践

1、DEMO（可获取函数）
https://blog.youkuaiyun.com/Trisyp/article/details/89318333
https://blog.youkuaiyun.com/weixin_42527994/article/details/110930583（sklearn）
2、predict和predict_proba区别
https://www.easck.com/cos/2020/0628/552525.shtml
3、DEMO实践

import numpy as np
from sklearn.linear_model import LogisticRegression
x_fearures = np.array([[-1, -2], [-2, -1], [-3, -2], [1, 3], [2, 1], [3, 2]])
y_label = np.array([0, 0, 0, 1, 1, 1])
lr_clf = LogisticRegression()
lr_clf = lr_clf.fit(x_fearures, y_label)
print(lr_clf)
coef=lr_clf.coef_
print(coef)

x_fearures_new = np.array([[1, -1], [2, 3], [1, 1]])
score=lr_clf.predict(x_fearures_new)
print(score)

4、案例

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import os  
import sys  
from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.sql import HiveContext
from pyspark.sql import SparkSession
from pyspark.sql.functions import concat, concat_ws, collect_list
from pyspark.sql.types import StructField,StringType,StructType
import numpy as np
from sklearn.linear_model import LogisticRegression

print(sys.version)

tx_date = sys.argv[1]
tx_date = tx_date.strip()
print('tx_date: %s' % (tx_date))

sparkConf = SparkConf()

os.environ["PYSPARK_PYTHON"] = "py3.6/bin/python"
os.environ["PYSPARK_DRIVER_PYTHON"] = "py3.6/bin/python"
sparkConf.set("spark.pyspark.python", "py3.6/bin/python")
sparkConf.set("spark.yarn.dist.archives", "hdfs://**/py3.6.zip#py3.6")


sparkConf.set('spark.driver.memory', '20G')
sparkConf.set('spark.driver.cores', '20')
sparkConf.set("spark.kryoserializer.buffer.max", "128m")
sparkConf.set("spark.executor.instances", "100")
sparkConf.set("spark.driver.maxResultSize", '10G')
sparkConf.set("spark.executor.cores", "2")
sparkConf.set("spark.executor.memory", "10G")
sparkConf.set("spark.app.name", 'user_feature_merge')

spark = SparkSession.builder.config(conf=sparkConf).enableHiveSupport().getOrCreate()
tag = []
feature = []
spark.sql('use dmc_ll')
user_feat = spark.sql("""
select b.clicked,a.q3imp,a.q3click,a.q3ctr,a.d3imp,a.d3click,a.d3ctr,a.d15imp,a.d15click,a.d15ctr
from(select split(skuid,':')[1] as item,
if(q3imp is null,0,q3imp) as q3imp,
if(q3click is null,0,q3click) as q3click,
if(q3ctr is null,0,q3ctr) as q3ctr,
if(d3imp is null,0,d3imp) as  d3imp,
if(d3click is null,0,d3click) as d3click,
if(d3ctr is null,0,d3ctr) as d3ctr,
if(d15imp is null,0,d15imp) as d15imp,
if(d15click is null,0,d15click) as d15click,
if(d15ctr is null,0,d15ctr) as d15ctr from dmc_ll.tablea where dt='2021-03-22') a
join(select skuid,if(clicked,1,0) as clicked from dmc_ll.tableb  where dt='2021-03-22') b
on a.item=b.skuid""")

features = []
label = []
for row in user_feat.collect():
    new_row = row.asDict()
    label.append(new_row.pop('clicked'))
    feature = []
    for (k,v) in new_row.items():
        feature.append(v)
    features.append(feature)
lr_clf = LogisticRegression()
lr_clf = lr_clf.fit(features,label)
coef=lr_clf.coef_
for s in coef[0]:
    print('{:.10f}'.format(s))

x_fearures_new = np.array([[3659, 215, 0.05875922383164799, 11, 2, 0.0, 1105, 52, 0.04705882352941176]])
score=lr_clf.predict_proba(x_fearures_new)
print(score)

小知识点：
（1）科学记数法如何转换成数字
‘{:.10f}’.format(s))->10代表小数点后几位
（2）enableHiveSupport()是支持hive函数
（3） Logistic Regression 中的 coef_ 和 intercept_ 的解释
https://blog.youkuaiyun.com/qq_38890412/article/details/108222544
（4）Spark-SQL之DataFrame操作大全
https://blog.youkuaiyun.com/dabokele/article/details/52802150
（5）pyspark.sql.types.row
https://blog.youkuaiyun.com/qq_34615112/article/details/107007391
5、dateframe和RDD区别
https://blog.youkuaiyun.com/linxiyimeng007/article/details/98629640
6、debug小技巧
如果想知道返回数据的类型，用print(type(var))来查看输出的数据类型

7、机器学习入门书籍
https://dataminingguide.books.yourtion.com