.. _keep_cross_validation_predictions:

``keep_cross_validation_predictions``
-------------------------------------

- Available in: GBM, DRF, Deep Learning, GLM, GAM, Naïve-Bayes, K-Means, XGBoost, AutoML
- Hyperparameter: no

Description
~~~~~~~~~~~

N-fold cross-validation is used to validate a model internally, i.e., to estimate the model performance without having to sacrifice a validation split. When building cross-validated models, H2O builds ``nfolds+1`` models: ``nfolds`` cross-validated models and 1 overarching model over all of the training data. For example, if you specify ``nfolds=5``, then 6 models are built. The first 5 models are the cross-validation models and are built on 80% of the training data. Each cross-validated model produces a prediction frame pertaining to its fold. You can save each of these prediction frames by enabling the ``keep_cross_validation_predictions`` option. Note that this option is disabled by default.

More information is available in the `Cross-Validation <../../cross-validation.html>`__ section. 

Related Parameters
~~~~~~~~~~~~~~~~~~

- `keep_cross_validation_fold_assignment <keep_cross_validation_fold_assignment.html>`__
- `keep_cross_validation_models <keep_cross_validation_models.html>`__
- `nfolds <nfolds.html>`__


Example
~~~~~~~

.. tabs::
   .. code-tab:: r R

		library(h2o)
		h2o.init()

		# import the cars dataset:
		# this dataset is used to classify whether or not a car is economical based on
		# the car's displacement, power, weight, and acceleration, and the year it was made
		cars <- h2o.importFile("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")

		# convert response column to a factor
		cars["economy_20mpg"] <- as.factor(cars["economy_20mpg"])

		# set the predictor names and the response column name
		predictors <- c("displacement", "power", "weight", "acceleration", "year")
		response <- "economy_20mpg"

		# split into train and validation sets
		cars_split <- h2o.splitFrame(data = cars, ratios = 0.8, seed = 1234)
		train <- cars_split[[1]]
		valid <- cars_split[[2]]

		# try using the `keep_cross_validation_predictions` (boolean parameter):
		# train your model, set nfolds parameter
		cars_gbm <- h2o.gbm(x = predictors, y = response, training_frame = train,
		                    nfolds = 5, keep_cross_validation_predictions= TRUE, seed = 1234)

		# print the cross-validation predictions
		h2o.cross_validation_predictions(cars_gbm)

   .. code-tab:: python

		import h2o
		from h2o.estimators.gbm import H2OGradientBoostingEstimator
		h2o.init()

		# import the cars dataset:
		# this dataset is used to classify whether or not a car is economical based on
		# the car's displacement, power, weight, and acceleration, and the year it was made
		cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")

		# convert response column to a factor
		cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()

		# set the predictor names and the response column name
		predictors = ["displacement","power","weight","acceleration","year"]
		response = "economy_20mpg"

		# split into train and validation sets
		train, valid = cars.split_frame(ratios = [.8], seed = 1234)

		# try using the `keep_cross_validation_predictions` (boolean parameter):
		# first initialize your estimator, set nfolds parameter
		cars_gbm = H2OGradientBoostingEstimator(keep_cross_validation_predictions = True, nfolds = 5, seed = 1234)

		# then train your model
		cars_gbm.train(x = predictors, y = response, training_frame = train)

		# print the cross-validation predictions
		cars_gbm.cross_validation_predictions()