2.1 When should we use Feature Hashing?

Feature hashing is useful when the user does not easy to know the dimension of the feature vector. For example, the bag-of-word representation in document classification problem requires scanning entire dataset to know how many words we have, i.e. the dimension of the feature vector.

In general, feature hashing is useful in the following environment:

• Streaming Environment
• Distirbuted Environment

Because it is expensive or impossible to know the real dimension of the feature vector.

3.1 Logistic Regression with glmnet

# The following script assumes that the data.frame
# of the training dataset and testing dataset are 
# assigned to variable `ipinyou.train` and `ipinyou.test`
# respectively

library(FeatureHashing)

# Checking version.
stopifnot(packageVersion("FeatureHashing") >= package_version("0.9"))

data(ipinyou)
f <- ~ IP + Region + City + AdExchange + Domain +
  URL + AdSlotId + AdSlotWidth + AdSlotHeight +
  AdSlotVisibility + AdSlotFormat + CreativeID +
  Adid + split(UserTag, delim = ",")
# if the version of FeatureHashing is 0.8, please use the following command:
# m.train <- as(hashed.model.matrix(f, ipinyou.train, 2^16, transpose = FALSE), "dgCMatrix")
m.train <- hashed.model.matrix(f, ipinyou.train, 2^16)
m.test <- hashed.model.matrix(f, ipinyou.test, 2^16)

# logistic regression with glmnet

library(glmnet)

## Loading required package: Matrix

## Loaded glmnet 4.1-8

cv.g.lr <- cv.glmnet(m.train, ipinyou.train$IsClick,
  family = "binomial")#, type.measure = "auc")
p.lr <- predict(cv.g.lr, m.test, s="lambda.min")

library(pROC)

## Type 'citation("pROC")' for a citation.

## 
## Attaching package: 'pROC'

## The following objects are masked from 'package:stats':
## 
##     cov, smooth, var

auc(ipinyou.test$IsClick, c(p.lr))

## Setting levels: control = FALSE, case = TRUE

## Setting direction: controls < cases

## Area under the curve: 0.5187

3.2 Gradient Boosted Decision Tree with xgboost

Following the script above,

# GBDT with xgboost
if(require("xgboost")){
  cv.g.gdbt <- xgboost(m.train, ipinyou.train$IsClick, max.depth=7, eta=0.1, subsample = 0.7, colsample_bytree = 0.7,
    nround = 10, objective = "binary:logistic", verbose = ifelse(interactive(), 1, 0))
  p.lm <- predict(cv.g.gdbt, m.test)
  auc(ipinyou.test$IsClick, p.lm)  
}

## Loading required package: xgboost

## Setting levels: control = FALSE, case = TRUE

## Setting direction: controls < cases

## Area under the curve: 0.6412

3.3 Per-Coordinate FTRL-Proximal with \(L_1\) and \(L_2\) Regularization for Logistic Regression

The following scripts use an implementation of the FTRL-Proximal for Logistic Regresion, which is published in McMahan et al. (2013), to predict the probability (1-step prediction) and update the model simultaneously.

source(system.file("ftprl.R", package = "FeatureHashing"))

m.train <- hashed.model.matrix(f, ipinyou.train, 2^16, transpose = TRUE)
ftprl <- initialize.ftprl(0.1, 1, 0.1, 0.1, 2^16)
ftprl <- update.ftprl(ftprl, m.train, ipinyou.train$IsClick, predict = TRUE)
auc(ipinyou.train$IsClick, attr(ftprl, "predict"))

## Setting levels: control = FALSE, case = TRUE

## Setting direction: controls < cases

## Area under the curve: 0.5993

If we use the same algorithm to predict the click through rate of the 3rd season of iPinYou, the overall AUC will be 0.77 which is comparable to the overall AUC of the 3rd season 0.76 reported in Zhang et al. (2014).