手写基于Logistic Regression的文章分类器（AG News）

A Text Classifier based on Log-Linear Model

A simple model from scratch.

code repository

Dataset

AG News

Implementation

Preprocess

• Sample data to reduce dataset size
• Merge the Title and Content
• With the help of nltk:
  • Remove punctuations and numbers
  • Remove URLs
  • Split the Content into words
  • Filter stopwords
  • Stem and Lemmarize

Feature Extraction

Use TF-IDF as the Feature

• Keep only the most frequent words for a reasonable feature size
• Calculate TF-IDF
  $$TF(t,d)=\frac{\text{count}(t,d)}{{\sum }_k\text{count}(k,d)}$$
  , where count($t$, $d$) means the count of term $t$ in document $d$.
  $$IDF(t,D)=\log \frac{N+1}{num(t,D)+1}+1$$
  , where num($t$,