.. _data-transformers:

Data Transformers
=================

Before a Vega-Lite or Vega specification can be passed to a renderer, it typically
has to be transformed in a number of ways:

* pandas Dataframe has to be sanitized and serialized to JSON.
* The rows of a Dataframe might need to be sampled or limited to a maximum number.
* The Dataframe might be written to a ``.csv`` of ``.json`` file for performance
  reasons.

These data transformations are managed by the data transformation API of Altair.

.. note::

    The data transformation API of Altair should not be confused with the ``transform``
    API of Vega and Vega-Lite.

A data transformer is a Python function that takes a Vega-Lite data ``dict`` or
pandas ``DataFrame`` and returns a transformed version of either of these types::

    from typing import Union
    Data = Union[dict, pd.DataFrame]

    def data_transformer(data: Data) -> Data:
        # Transform and return the data
        return transformed_data

Dataset Consolidation
~~~~~~~~~~~~~~~~~~~~~
Datasets passed as pandas dataframes can be represented in the chart in two
ways:

- As literal dataset values in the ``data`` attribute at any level of the
  specification
- As a named dataset in the ``datasets`` attribute of the top-level
  specification.

The former is a bit more simple, but common patterns of usage in Altair can
often lead to full datasets being listed multiple times in their entirety
within a single specification.

For this reason, Altair 2.2 and newer will by default move all
directly-specified datasets into the top-level ``datasets`` entry, and
reference them by a unique name determined from the hash of the data
representation. The benefit of using a hash-based name is that even if the
user specifies a dataset in multiple places when building the chart, the
specification will only include one copy.

This behavior can be modified by setting the ``consolidate_datasets`` attribute
of the data transformer.

For example, consider this simple layered chart:

.. altair-plot::
   :chart-var-name: chart
		    
   import altair as alt
   import pandas as pd

   df = pd.DataFrame({'x': range(5),
                      'y': [1, 3, 4, 3, 5]})

   line = alt.Chart(df).mark_line().encode(x='x', y='y')
   points = alt.Chart(df).mark_point().encode(x='x', y='y')
   chart = line + points

If we look at the resulting specification, we see that although the dataset
was specified twice, only one copy of it is output in the spec:

.. altair-plot::
   :output: stdout

   from pprint import pprint
   pprint(chart.to_dict())

This consolidation of datasets is an extra bit of processing that is turned on
by default in all renderers.

If you would like to disable this dataset consolidation for any reason, you can
do so by setting ``alt.data_transformers.consolidate_datasets = False``, or
by using the ``enable()`` context manager to do it only temporarily:

.. altair-plot::
   :output: stdout

   with alt.data_transformers.enable(consolidate_datasets=False):
       pprint(chart.to_dict())
   
Notice that now the dataset is not specified within the top-level ``datasets``
attribute, but rather as values within the ``data`` attribute of each
individual layer. This duplication of data is the reason that dataset
consolidation is set to ``True`` by default.


Built-in Data Transformers
~~~~~~~~~~~~~~~~~~~~~~~~~~

Altair includes a default set of data transformers with the following signatures.

Raise a ``MaxRowsError`` if a Dataframe has more than ``max_rows`` rows::

    limit_rows(data, max_rows=5000)

Randomly sample a DataFrame (without replacement) before visualizing::

    sample(data, n=None, frac=None)

Convert a Dataframe to a separate ``.json`` file before visualization::

    to_json(data, prefix='altair-data'):

Convert a Dataframe to a separate ``.csv`` file before visualization::

    to_csv(data, prefix='altair-data'):

Convert a Dataframe to inline JSON values before visualization::

    to_values(data):

Piping
~~~~~~

Multiple data transformers can be piped together using ``pipe``::

    from altair import limit_rows, to_values
    from toolz.curried import pipe
    pipe(data, limit_rows(10000), to_values)

Managing Data Transformers
~~~~~~~~~~~~~~~~~~~~~~~~~~

Altair maintains a registry of data transformers, which includes a default
data transformer that is automatically applied to all Dataframes before rendering.

To see the registered transformers::

    >>> import altair as alt
    >>> alt.data_transformers.names()
    ['default', 'json', 'csv']

The default data transformer is the following::

    def default_data_transformer(data):
        return pipe(data, limit_rows, to_values)

The ``json`` and ``csv`` data transformers will save a Dataframe to a temporary
``.json`` or ``.csv`` file before rendering. There are a number of performance
advantages to these two data transformers:

* The full dataset will not be saved in the notebook document.
* The performance of the Vega-Lite/Vega JavaScript appears to be better
  for standalone JSON/CSV files than for inline values.

There are disadvantages of the JSON/CSV data transformers:

* The Dataframe will be exported to a temporary ``.json`` or ``.csv``
  file that sits next to the notebook.
* That notebook will not be able to re-render the visualization without
  that temporary file (or re-running the cell).

In our experience, the performance improvement is significant enough that
we recommend using the ``json`` data transformer for any large datasets::

    alt.data_transformers.enable('json')

We hope that others will write additional data transformers - imagine a
transformer which saves the dataset to a JSON file on S3, which could
be registered and enabled as::

    alt.data_transformers.register('s3', lambda data: pipe(data, to_s3('mybucket')))
    alt.data_transformers.enable('s3')


Storing JSON Data in a Separate Directory
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When creating many charts with ``alt.data_transformers.enable('json')`` the
working directory can get a bit cluttered. To avoid this we can build a simple
custom data transformer that stores all JSON files in separate directory.::

    import os
    import altair as alt
    from toolz.curried import pipe


    def json_dir(data, data_dir='altairdata'):
        os.makedirs(data_dir, exist_ok=True)
        return pipe(data, alt.to_json(filename=data_dir + '/{prefix}-{hash}.{extension}') )


    alt.data_transformers.register('json_dir', json_dir)
    alt.data_transformers.enable('json_dir', data_dir='mydata')

After enabling this data transformer, the JSON files will be stored in what ``data_dir``
was set to when enabling the transformer or 'altairdata' by default. All we had to do
was to prefix the ``filename`` argument of the ``alt.to_json`` function with our
desired directory and make sure that the directory actually exists.
