title: "EncodingExample"

author: "Win-Vector LLC"

date: "4/15/2017"

output:

md_document:

variant: markdown_github

[`R`](https://cran.r-project.org) has "one-hot" encoding hidden in most of its modeling paths. Asking an `R`

user where one-hot encoding is used is like asking a fish where there is water; they can't point to it as it is everywhere.

For example we can see evidence of one-hot encoding in the variable names chosen by a linear regression:

```{r modelenc}

Much of the encoding in `R` is essentially based on "contrasts" implemented in

`stats::model.matrix()` Note: do not use `base::data.matrix()` or use [hashing](http://www.win-vector.com/blog/2014/12/a-comment-on-preparing-data-for-classifiers/) before

modeling- you might get away with them (especially with tree based methods), but they are [not in general

good technique](http://www.win-vector.com/blog/2014/12/a-comment-on-preparing-data-for-classifiers/) as we show below:

```{r datamatrix}

`stats::model.matrix()` does not store its one-hot plan in a convenient manner (it

can be inferred by pulling the "`contrasts`" attribute plus examining

the column names of the first encoding, but the levels identified are not conveniently

represented).

When directly applying `stats::model.matrix()` you can not

safely assume the same formula applied to two different data sets (say train

and application or test) are using the same encoding! We demonstrate this below:

```{r modelmatrixexample}

The above mal-coding can be a critical flaw when you are building a model and then later using the model on new

data (be it cross-validation data, test data, or future application data). Many

`R` users are not familiar with the above issue as encoding is hidden in model training,

and how to encode new data is stored as part of the model. `Python` `scikit-learn` users coming

to `R` often ask "where is the one-hot encoder" (as it isn't discussed as much in `R` as it

is in `scikit-learn`) and even supply a number of (low quality) one-off packages "porting one-hot encoding to `R`."

The main place an `R` user needs a proper encoder (and that is an encoder that stores its encoding

plan in a conveniently re-usable form, which many of the "one-off ported from `Python`" packages actually fail to do) is

when using a machine learning implementation that isn't completely `R`-centric. One such system is

[`xgboost`](https://github.com/dmlc/xgboost) which requires (as is typical of machine learning in `scikit-learn`)

data to already be encoded as a numeric matrix (instead of a heterogeneous structure such as a `data.frame`).

This requires explicit conversion on the part of the `R` user, and many `R` users get it wrong (fail to store the

encoding plan somewhere). To make this concrete let's work a simple example.

Let's try the Titanic data set to see encoding in action. Note: we are not working hard on this example (as in adding extra variables derived from cabin layout, commonality of names, and other sophisticated feature transforms)- just plugging the obvious variable into `xgboost`. As we said: `xgboost` requires a numeric matrix for its input, so unlike many `R` modeling methods we must manage the data encoding ourselves (instead of leaving that to `R` which often hides the encoding plan in the trained model). Also note: differences observed in performance that are below the the sampling noise level should not be considered significant (e.g., all the methods demonstrated here performed about the same).

We bring in our data:

```{r modelingexample}

And design our cross-validated modeling experiment:

```{r setup}

Our preferred way to encode data is to use the [`vtreat`](https://CRAN.R-project.org/package=vtreat) package in the "no variables mode" shown below (differing from the powerful "y aware" modes we usually teach).

```{r vtreatZ}

Model matrix can perform similar encoding *when* we only have a single data set.

```{r modelmatrix}

The `caret` package also supplies an encoding functionality properly split between training (`caret::dummyVars()`) and application (called `predict()`).

```{r caret}

We usually forget to teach `vtreat::designTreatmentsZ()` as it is often dominated by the

more powerful y-aware methods `vtreat` supplies (though not for this simple example).

`vtreat::designTreatmentsZ` has a number of useful properties:

* Does not look at the outcome values, so does not require extra care in cross-validation.

* Saves its encoding, so can be used correctly on new data.

The above two properties are shared with `caret::dummyVars()`. Additional features

of `vtreat::designTreatmentsZ` (that differ from `caret::dummyVars()`'s choices) include:

* No `NA` values are passed through by `vtreat::prepare()`.

* `NA` presence is added as an additional informative column.

* A few derived columns (such as pooling of rare levels are made available).

* Rare dummy variables are pruned (under a user-controlled threshold) to prevent encoding explosion.

* Novel levels (levels that occur during test or application, but not during training) are deliberately passed through as "no training level activated" by `vtreat::prepare()` (`caret::dummyVars()` considers this an error).

The `vtreat` y-aware methods include proper nested modeling and y-aware dimension reduction.

`vtreat` is designed "to always work" (always return a pure numeric data frame with no missing values). It also excels in "big data" situations where the statistics it can collect on high cardinality categorical variables can have a huge positive impact in modeling performance. In many cases `vtreat` works around problems that kill the analysis pipeline (such as discovering new variable levels during test or application). We teach `vtreat` sore of "bimodally" in both a ["fire and forget" mode](http://www.win-vector.com/blog/2017/03/vtreat-prepare-data/) and a ["all the details on deck" mode](https://arxiv.org/abs/1611.09477) (suitable for formal citation). Either way `vtreat` can make your modeling procedures stronger, more reliable, and easier.

##################

You could also try y-aware encoding, but it isn't adding anything positive in this situation as

we have not introduced any high-cardinality categorical variables into this modeling example

(the main place y-aware encoding helps).

```{r vtreatC}