using namespace 数学工具构造器;

这个月，笔者复盘了2020年做的一些AutoML项目，并在多个优秀的开源项目的基础上，博采众长，写了一个超参优化库：UltraOpt。这个库包含一个笔者自研的贝叶斯优化算法：ETPE，其在基准测试中比HyperOpt的TPE算法表现更为出色。UltraOpt对分布式计算有更强的适应性，支持MapReduce和异步通信两种并行策略，并且可以扩展到各种计算环境中。除此之外，UltraOpt对与新手也特别友好，笔者特地花了3周的时间写中文文档，就是为了让小白也能0基础看懂AutoML（自动机器学习）是在做什么。

models.search_cnn.SearchCNNControllern_ops = 8, n_nodes = 4之所有有个i + 2是因为前置节点有两个for i in range(n_nodes): self.alpha_normal.append(nn.Parameter(1e-3*torch.randn (i+2, n_ops))) self.alpha_reduce.append(nn.Parameter(1e-3*torch.randn (i+2, n_

https://github.com/carpedm20/ENAS-pytorch虽然该项目声称可以对CNN(cifar, mnist)和RNN(ptb, wikitest)进行NAS, 但经过实测, CNN部分的代码根本没写完,估计还是要看原版tensorflow的代码In ENAS, there are two sets of learnable parameters: the parameters of the controller LSTM, denoted by θ\thetaθ, and t

激活函数choices: [sigmoid,tanh,relu,elu,leaky_relu][sigmoid, tanh, relu, elu, leaky\_relu][sigmoid,tanh,relu,elu,leaky_relu]隐层神经元choices: [25,50,100,150,200][25,50,100,150,200][25,50,100,150,200]一共3个隐层学习率choices: [10−4,10−3,10−2,0.1,0.2][10^{-4},10^{-3},10^

采样import hyperopt.pyll.stochasticprint hyperopt.pyll.stochastic.sample(space)

In [31]: arr=np.random.rand(10000,10000) In [32]: arr.__sizeof__() / 1024 /1024

low budget 表现好， high budget反而凉了{"estimating:__choice__": "random_forest", "preprocessing:cat->normed:__choice__": "encode.one_hot", "preprocessing:combined->normed:__choice__": "encode.cat_boost", "preprocessing:highC_cat->combined:__choice__": .

如果用Titanic数据的话，会推荐以下几个最近的数据集：self.kNDOut[9]: ['openml_166905', 'openml_146578', 'openml_146602', 'openml_166906', 'openml_279', 'openml_3054']ND2repo['openml_166905'][0][0]['process_sequence']Out[11]: 'cat->normed;highC_cat->combined;combi

因为数模的缘故，我花了3400大洋买了台红米book（没错，小米不仅有手机电视更有电脑，性价比很香），现在开始安装matlab。考虑到我平时用python，matlab只是为了数模用用，所以我只装了我认为重要的库（库是干啥的全靠瞎猜）...

按照论文的说法，landmarking方法因为过于耗时所以不算在元特征中EXCLUDE_META_FEATURES_CLASSIFICATIONOut[5]: {'Landmark1NN', 'LandmarkDecisionNodeLearner', 'LandmarkDecisionTree', 'LandmarkLDA', 'LandmarkNaiveBayes', 'PCA', 'PCAFractionOfComponentsFor95PercentVariance', 'PCA

if splitter is None: if self.n_samples > self.max_n_samples_for_CV: self.logger.info( f"TrainSet has {self.n_samples} samples," f" greater than max n_samples for Cross-Validatio...

askl2_training_data = json.loads(Path("askl2_training_data.json").read_text()) In [10]: askl2_training_data['metafeatures'].keys() .

lgbm=LGBMClassifier(n_estimators=256, num_leaves=60, learning_rate=0.01)leaf.shapeOut[18]: (4177, 7168)ohe.shapeOut[21]: (4177, 199284)max_iter=20的训练时间：66slgbm=LGBMClassifier(n_estimators=128, num_leaves=20, learning_rate=0.1)lgbm.fit(X, y)leaf =

standardizelgbm score 0.9310618224253633lr score 0.8508692055869721rf score 0.9074903045815534et score 0.8914293518321599nonelgbm score 0.930806442906943lr score 0.7266447624392611rf score 0.9075442389649911et score 0.8914293518321599robust (0.25

主要代码在autofeat.autofeat.AutoFeatModel#fit_transform， 在做完特征生成后， 调用 autofeat.featsel.select_features函数做特征筛选。在select_features函数， 会做featsel_runs次特征筛选 selected_columns = [] for i in range(featsel_runs): selected_columns.e

鉴于最近看的东西太杂了，所以单独用一个文档记录一些平时看到的机器学习项目。NNI的https://nni.readthedocs.io/en/latest/FeatureEngineering/Overview.htmlTreeBasedClassifier指的是ExtraTrees

DataTypes_Auto_inferbool类型转category查int64和float64中的id列处理这种情况：pandas把int的列转成了float，因为nan对于int64的的变量，视基数情况转成float64或类别变量float64，如果基数=2，转OHE # if there are inf or -inf then replace them with NaN data.replace([np.inf,-np.inf],np.NaN,inplace=True)

#!/usr/bin/env python# -*- coding: utf-8 -*-# @Author : qichun tang# @Contact : tqichun@gmail.comimport numpy as npfrom category_encoders import OneHotEncoder as OriginOneHotEncoderfrom category_encoders.utils import convert_inputfrom sklearn.ba

参考资料"lightgbm": { "n_estimators": 2000, "boosting_type": "gbdt", "num_leaves": {"_type": "int_quniform", "_value": [10,150],"_default": 31}, "max_depth": {"_type": "int_quniform", "_value": [1,100],"_default": 31}, "learning_rate": {"_type": "

文章目录什么是实体嵌入(Entity Embeddings)独热编码标签编码实体嵌入用PyTorch重构MLBox的EntityEmbedding计算相应的嵌入维度计算MLP隐层的神经元数构造各个类别的线性Embed构造中间两个隐层构造最后的输出层权重初始化实验结论什么是实体嵌入(Entity Embeddings)在处理机器学习问题时，有一个问题必须思考：如何编码数据集中的类别变量？类别编码的两个基本方法是独热编码（onehot encoding）和标签编码（label encoding）。独热编

hyperparams = {'NN': {'num_epochs': 10, 'activation': 'relu', 'dropout_prob': ag.Real(0.0,0.5)}, 'GBM': {'num_boost_round': 1000, 'learning_rate': ag.Real(0.01,0.1,log=True)} }进入autogluon.task.tabular_prediction.tabular_prediction.Tabular

def recursion_origin(self, hdl: Dict, path=()) -> ConfigurationSpace: cs = ConfigurationSpace() # 检测一下这个dict是否在直接描述超参 key_list = list(hdl.keys()) if len(key_list) == 0: cs.add_hyperparameter(Constant("pla...

#!/usr/bin/env python# -*- coding: utf-8 -*-# @Author : qichun tang# @Contact : tqichun@gmail.comfrom ConfigSpace import CategoricalHyperparameter, Constantfrom ConfigSpace import ConfigurationSpacefrom ConfigSpace import ForbiddenInClause, Forbi

To mitigate this risk, for algorithms that can be trained iteratively (e.g.,gradient boosting and linear models trained with stochastic gradient descent)we implemented two measures.Firstly,we allow a pipeline to stop training based on a h

auto-sklearn对于类别不平衡的处理策略有2类，我们分别讨论。文章目录调整sample_weights调整class_weights调整sample_weights首先列出有哪些算法采用sample_weights策略：clf_ = ['adaboost', 'random_forest', 'extra_trees', 'sgd', 'passive_aggressive']pre_ = []为什么用这些算法呢？比如说'adaboost', 'random_forest', 'ext

最近rayteams出了个新项目，Tune-Sklearn。Tune-sklearn is a package that integrates Ray Tune’s hyperparameter tuning and scikit-learn’s models, allowing users to optimize hyerparameter searching for sklearn using Tune’s schedulers2月份我还没仔细研究贝叶斯优化的时候就像手撸一个这样的项目的，现在开

0 [0.02037037, 0.01296296, 0.02777778, 0.51555556, 0.90074074,0.43259259]1 [0.01296296, 0.01592593, 0.02851852, 0.09333333, 0.90148148, 0.90740741] [0.00740741, 0.01481481, 0.02592593, 0.16666667, 0.89259259,0.90740741]np.argsort([0.01296296, 0.015925

论文提到预先准备一些模型，并且按一个手设的priority进行训练# Higher values indicate higher priority, priority dictates the order models are trained for a given level.DEFAULT_MODEL_PRIORITY = dict( RF=100, XT=90, KNN=80, GBM=70, CAT=60, NN=50, LR=40,

class ChoicesKDE: def __init__(self, data): self.data = data self.prob = {} self.bw = 0.2 for number, cnt in zip(*np.unique(data, return_counts=True)): self.prob[int(number)] = cnt / data.size def pdf(se

自研贝叶斯优化算法，如何判断算法能拟合？我目前是在一个tiny的数据集上跑一下，看算法能否收敛到正确的的局部最小值。这里要有两个关键词：收敛。算法是需要收敛的。黑盒优化的本质就是增加在优势样本附近的采样率。如果算法如同随机搜索不收敛，那么是有问题的。正确。收敛点是正确的，如果收敛到错误的点，那还不如随机搜索。文章目录自研SMAC代理模型获益函数遇到过的坑TPE遇到的坑自研SMAC代理模型SMAC的本质是用随机森林作为代理模型。这个代理模型调包就好了（前提是你熟读开源代码千百遍，知道调什么）

文章目录第四范式第四范式树模型对于高维离散特征效果不好逐域网络 Field-wise network跨域网络 Across field补充：FM融合网络逐域网络数量多（1000个域有1000个DNN），但比较轻量拼起来丢进去算，连续的做一次，离散的做一次聚类：tSNE域内相似度更搞，域与域距离更大LR 线性组合FM 两两内积Attention 有选择性地学习用什么没有定论，需要大量实验把好的operator放进来，用不用当做超参数，用数据驱动的方式选择operator。上

GP-MCMC专有采集函数的计算robo.acquisition_functions.marginalization.MarginalizationGPMCMC#computeself.estimatorsOut[10]: [<robo.acquisition_functions.log_ei.LogEI at 0x7f3067cdeeb8>, <robo.acquisition_functions.log_ei.LogEI at 0x7f3067b0b320>, &l

实例化BOHB或其他Master需要提供ConfigSpaceclass BOHB(Master): def __init__( self, configspace=None, eta=3, min_budget=0.01, max_budget=1, min_points_in_model=None, top_n_percent=

hpbandster.core.result.Result#__init__self.data[0].keys()Out[32]: dict_keys([(0, 0, 0), (0, 0, 1), (0, 0, 2), (0, 0, 3), (0, 0, 4), (0, 0, 5), (0, 0, 6), (0, 0, 7), (0, 0, 8), (0, 0, 9), (0, 0, 10), (0, 0, 11), (0, 0, 12), (0, 0, 13), (0, 0, 14), (0, 0,

行方向上拼接两个数据框pandas=1.0.1 work,pandas= 0.25.3 不workdf = pd.concat(Xs, axis=0)df.sort_index(inplace=True)df = pd.concat(Xs, axis=0, sort=False)df.sort_index(inplace=True)

pmf-automl是2018NIPS论文中提出的一种新的automl方法，他构造了一个离散的pipeline空间，并用概率矩阵分解作为概率模型来实现贝叶斯优化。

example/example.pyfrom atm import ATMatm = ATM()results = atm.run(train_path="/home/tqc/PycharmProjects/automl/ATM/demos/pollution_1.csv")results.describe()atm.worker.Worker#select_hyperpartition调试打印的信息和论文描述的一致，超划分hyperpartition表示条件参数树(conditiona

论文：（2018ICML）https://ml.informatik.uni-freiburg.de/papers/18-AUTOML-AutoChallenge.pdf代码：http://ml.informatik.uni-freiburg.de/downloads/automl_competition_2018.zip数据：(codalab平台，需要注册)https://competitions.codalab.org/competitions/17767#participate-get_da