
# Code generation for client APIs

For each target language `LANG`, the client code is generated by a Python script names `gen_LANG.py` in the current folder.

## Custom code generation

For our main client APIs in `R` and `Python`, various customizations had to be applied for several algorithms.

Therefore, for various reasons – readability/reviews, maintainability, documentation, clear separation of logic for each algo – those customizations are loaded by the code generator through a template mechanism.
   
For each algorithm `ALGO`, we can then find a Python template script `gen_ALGO.py` under `custom/python` or `custom/R`
                  
To avoid some code duplications (hooks that are common to many algorithms but not to all of them), it's also possible to put the commonly shared logic in a file named `gen_defaults.py`.  
For example, in `R` the validation of `x, y, training_frame` args is shared by the majority of algorithms, so it resides in `custom/R/gen_defaults.py` template.
This also means that for algorithms not using those defaults, there could be some small code duplication, but it's at the benefit of a clearer structure and readability.
This `gen_defaults.py` has the same structure as other `gen_ALGO.py`, it is just a fallback used if and only if its properties haven't been overridden in the dedicated `gen_ALGO.py` file.

File structure:
```text
h2o-bindings/bin
├── custom
│   ├── R
│   │   ├── gen_aggregator.py
│   |   ├── ...
│   │   ├── gen_defaults.py
│   |   ├── ...
│   │   └── gen_xgboost.py
│   └── python
│       ├── gen_aggregator.py
│       ├── ...
│       ├── gen_defaults.py
│       ├── ...
│       └── gen_xgboost.py
├── gen_R.py
├── gen_python.py
```

### Templates structure

Except for Python as we'll see below, the `gen_ALGO.py` files are mainly template files, so we can't directly write interpretable code there.
Instead, custom code is just wrapped in a `Python` triple-quoted string.
For readability, it is recommended to start the code block in the line following the first `"""`, and to left-align the code without additional indentation: the code generator will ensure that the block is correctly indented relatively to previous code.

The same applies for documentation.

Example in R:
```python
extensions = dict(
    required_params=['x', 'y', 'training_frame'],
    frame_params=['training_frame', 'validation_frame'],
    validate_required_params="""
# If x is missing, then assume user wants to use all columns as features.
if (missing(x)) {
   if (is.numeric(y)) {
       x <- setdiff(col(training_frame), y)
   } else {
       x <- setdiff(colnames(training_frame), y)
   }
}
""",
)

doc = dict(
    params=dict(
        x="""
(Optional) A vector containing the names or indices of the predictor variables to use in building the model.
If x is missing, then all columns except y are used.
""",
        y="""
The name or column index of the response variable in the data. 
The response must be either a numeric or a categorical/factor variable. 
If the response is numeric, then a regression model will be trained, otherwise it will train a classification model.
""",
    )
)
```

#### Python

for Python the `gen_ALGO.py` files have the following possible entries (none of them being required):

```python
rest_api_version = 99   # if algo doesn't use the default 3

def update_params(name, param):
    # a hook to modify schema params on the fly: mainly used to remove some enum values...
    pass

extensions = dict(
    __imports__="a string with additional imports",
    __class__="a string or a function defining additional methods for the default algo class",
    __module__="a string or a function defining additional methods for the algo module",
    __init__validation="a string defining additional code for validation phase of __init__: used only for generic",
    __init__setparams="a string defining additional code for set_params phase of __init__: used only for word2vec",
)

overrides = dict(
    foo = dict(
         getter="string defining code overriding the getter for property foo",
         setter="string defining code overriding the setter for property foo",
    ),
)

doc = dict(
    __class__ = "additional documentation added at class level",
    foo = "additional documentation added to the getter of property foo"
)

examples = dict(
    __class__ = "doc examples added at class level",
    foo = "doc examples added to the getter of property foo"
)
```

Given that `Python` = `Python`, we can directly write custom in Python functions instead of strings.
This provides better color syntax and code readability in general, but those functions are still usually not directly runnable due to difficulties in importing the right dependencies from the template file (playing with sys.path doesn't help in IDEs either). 
 
Here is an example of custom code in `gen_deeplearning.py`:

```python
def module_extensions():
    class H2OAutoEncoderEstimator(H2ODeepLearningEstimator):
        """
        :examples:

        >>> import h2o as ml
        >>> from h2o.estimators.deeplearning import H2OAutoEncoderEstimator
        >>> ml.init()
        >>> rows = [[1,2,3,4,0]*50, [2,1,2,4,1]*50, [2,1,4,2,1]*50, [0,1,2,34,1]*50, [2,3,4,1,0]*50]
        >>> fr = ml.H2OFrame(rows)
        >>> fr[4] = fr[4].asfactor()
        >>> model = H2OAutoEncoderEstimator()
        >>> model.train(x=range(4), training_frame=fr)
        """
        def __init__(self, **kwargs):
            super(H2OAutoEncoderEstimator, self).__init__(**kwargs)
            self._parms['autoencoder'] = True


extensions = dict(
    __module__=module_extensions
)

doc = dict(
    __class__="""
Build a Deep Neural Network model using CPUs
Builds a feed-forward multilayer artificial neural network on an H2OFrame
"""
)

examples = dict(
    __class__="""
>>> import h2o
>>> from h2o.estimators.deeplearning import H2ODeepLearningEstimator
>>> h2o.connect()
>>> rows = [[1,2,3,4,0], [2,1,2,4,1], [2,1,4,2,1], [0,1,2,34,1], [2,3,4,1,0]] * 50
>>> fr = h2o.H2OFrame(rows)
>>> fr[4] = fr[4].asfactor()
>>> model = H2ODeepLearningEstimator()
>>> model.train(x=range(4), y=4, training_frame=fr)
""",
    training_frame="""
>>> cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")
>>> cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
>>> predictors = ["displacement","power","weight","acceleration","year"]
>>> response = "economy_20mpg"
>>> train, valid = cars.split_frame(ratios = [.8], seed = 1234)
>>> cars_gbm = H2ODeepLearningEstimator(seed = 1234)
>>> cars_gbm.train(x = predictors,
...                y = response,
...                training_frame = train,
...                validation_frame = valid)
>>> cars_gbm.auc(valid=True)
""",
)


```
the name of the function doesn't matter, it is just loaded by the `gen_python.py` generator, which extracts it's source code, and removes the first line, which is the function signature.


#### R

for R the `gen_ALGO.py` template files have the following possible entries (none of them being required):

```python
rest_api_version = 99   # if algo doesn't use the default 3

def update_params(name, param):
    # a hook to modify schema params on the fly: mainly used to remove some enum values...
    pass

extensions = dict(
    required_params=['x', 'y', 'training_frame'], #required params on top of signature: list of names or tuple(name, default_value)
    frame_params=['training_frame', 'validation_frame'], #by default (have special validation handling)
    extra_params=[('verbose', 'FALSE')],  #additional params not in schema: list of tuple(name, default_value)
    validate_required_params="specific code to validated required params",
    validate_params="specific code to validate params",
    set_required_params="specific code to set required params to `parms` object",
    skip_default_set_params_for=['foo'], # list of params for which we need to remove default set param mechanism because it will be customized below
    set_params="specific code to set other params to `parms` object",
    with_model="code to postprocess the model",
    module="additional R functions added to the module",
)

doc = dict(
    preamble="doc preamble",
    params=dict(
        foo="doc for param foo (without tag): used for params not in schema, otherwise overrides help doc from schema",
        bar="doc for param bar",
    ),
    returns="@return doc (without tag)",
    seealso="@seealso doc (without tag)",
    references="@references doc (without tag)",
    examples="@examples doc (code only)",
)
```