Medial Code Documentation
Loading...
Searching...
No Matches
Data Structures | Functions | Variables
quantile_data_iterator Namespace Reference

Data Structures

class  IterForDMatrixDemo
 

Functions

 main ()
 

Variables

int COLS = 64
 
int ROWS_PER_BATCH = 1000
 
int BATCHES = 32
 

Detailed Description

Demo for using data iterator with Quantile DMatrix
==================================================

    .. versionadded:: 1.2.0

The demo that defines a customized iterator for passing batches of data into
:py:class:`xgboost.QuantileDMatrix` and use this ``QuantileDMatrix`` for
training.  The feature is used primarily designed to reduce the required GPU
memory for training on distributed environment.

Aftering going through the demo, one might ask why don't we use more native
Python iterator?  That's because XGBoost requires a `reset` function, while
using `itertools.tee` might incur significant memory usage according to:

  https://docs.python.org/3/library/itertools.html#itertools.tee.