Issue
I am doing target encoding for my column, using nested cross validation approach (to avoid leakage) as mentioned here, here and here.
If I had to include my target encoding (by a custom transformer), in the sklearn pipeline, I need different transform function from the train set and the test set. This is because, for the train folds, the encoding is calculated using a further kfold split of the train data. Whereas for the test fold, encoding is mean of the train.
I know the sklearn pipeline will apply the same transformation for train and test split in the cv, is there a way to apply separate transformations for train and test splits using a sklearn pipeline and custom transformer
Solution
The category_encoders
package implements some target encoders. It gets around the issue of different training and testing dataset behavior by implementing a fit_transform
method that is not equivalent to just fit
followed by transform
: fit_transform
performs the training-set transformation, while transform
performs the test/production set transformation.
Answered By - Ben Reiniger
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.