{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/IPython/core/magics/pylab.py:160: UserWarning: pylab import has clobbered these variables: ['specgram']\n",
      "`%matplotlib` prevents importing * from pylab and numpy\n",
      "  \"\\n`%matplotlib` prevents importing * from pylab and numpy\"\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "import matplotlib.pylab as pylab\n",
    "pylab.rcParams['figure.figsize'] = 16, 8  # that's default image size for this interactive session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TimeSide API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Timeside* API is based on different core processing unit called *processors* :\n",
    "\n",
    "- Decoders (*timeside.api.IDecoder*) that enables to decode a giving audio source and split it up into frames for further processing\n",
    "- Analyzers (*timeside.api.IAnalyzer*) that provides some signal processing module to analyze incoming audio frames\n",
    "- Encoders (*timeside.api.IEncoder*) that can encode incoming frames back into an audio object\n",
    "- Graphers (*timeside.api.IGrapher*) that can display some representations of the signal or corresponding extracted features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IDecoder\n",
      "========\n",
      "  * array_decoder 1.0: Decoder taking Numpy array as input\n",
      "  * aubio_decoder 1.0: File decoder based on aubio \n",
      "  * file_decoder 1.0: File Decoder based on Gstreamer\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "import timeside.core\n",
    "\n",
    "from timeside.core import list_processors\n",
    "\n",
    "print(list_processors(timeside.core.api.IDecoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IAnalyzer\n",
      "=========\n",
      "  * aubio_melenergy 0.4.6: Aubio Mel Energy analyzer\n",
      "  * aubio_mfcc 0.4.6: Aubio MFCC analyzer\n",
      "  * aubio_pitch 0.4.6: Aubio Pitch estimation analyzer\n",
      "  * aubio_silence 0.4.6: Aubio Silence detection analyzer\n",
      "  * aubio_specdesc 0.4.6: Aubio Spectral Descriptors collection analyzer\n",
      "  * aubio_temporal 0.4.6: Aubio Temporal analyzer\n",
      "  * essentia_dissonance 2.1b5.dev416: Dissonance from Essentia\n",
      "  * vamp_constantq 1.1.0: Constant Q transform from QMUL vamp plugins\n",
      "  * vamp_crest 1.0.0: Extract the spectral crest measure of an audio spectrum\n",
      "  * vamp_flatness 1.0.0: Extract the flatness of an audio spectrum\n",
      "  * vamp_kurtosis 1.0.0: Extract the kurtosis of a range of values\n",
      "  * vamp_simple_host 1.1.0: Vamp plugins library interface analyzer\n",
      "  * vamp_spectral_centroid 1.0.0: Calculate the centroid frequency of the spectrum of the input signal\n",
      "  * vamp_spectral_kurtosis 1.0.0: Extract the kurtosis of an audio spectrum\n",
      "  * vamp_spectral_slope 1.0.0: Extract the spectral slope of an audio spectrum\n",
      "  * loudness_itu 1.0: Measure of audio loudness using standard ITU-R BS.1770-3\n",
      "  * spectrogram_analyzer 1.0: Spectrogram image builder with an extensible buffer based on tables\n",
      "  * onset_detection_function 1.0: Onset Detection Function analyzer\n",
      "  * spectrogram_analyzer_buffer 1.0: Spectrogram image builder with an extensible buffer based on tables\n",
      "  * waveform_analyzer 1.0: Waveform analyzer\n",
      "\tIValueAnalyzer\n",
      "\t--------------\n",
      "\t  * mean_dc_shift 1.0: Mean DC shift analyzer\n",
      "\t  * essentia_dissonance_value 2.1b5.dev416: Mean Dissonance Value from Essentia\n",
      "\t  * vamp_tempo 1.1.0: Tempo from QMUL vamp plugins\n",
      "\t  * vamp_tuning 1.1.0: Tuning from NNLS Chroma vamp plugins\n",
      "\t  * level 1.0: Audio level analyzer\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(list_processors(timeside.core.api.IAnalyzer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IEncoder\n",
      "========\n",
      "  * flac_aubio_encoder 1.0: FLAC encoder based on aubio\n",
      "  * vorbis_aubio_encoder 1.0: OGG Vorbis encoder based on aubio\n",
      "  * wav_aubio_encoder 1.0: Wav encoder based on aubio\n",
      "  * live_encoder 1.0: Gstreamer-based Audio Sink\n",
      "  * flac_encoder 1.0: FLAC encoder based on Gstreamer\n",
      "  * aac_encoder 1.0: AAC encoder based on Gstreamer\n",
      "  * mp3_encoder 1.0: MP3 encoder based on Gstreamer\n",
      "  * vorbis_encoder 1.0: OGG Vorbis encoder based on Gstreamer\n",
      "  * opus_encoder 1.0: Opus encoder based on Gstreamer\n",
      "  * wav_encoder 1.0: WAV encoder based on Gstreamer\n",
      "  * webm_encoder 1.0: WebM encoder based on Gstreamer\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(list_processors(timeside.core.api.IEncoder))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IGrapher\n",
      "========\n",
      "  * grapher_aubio_pitch 1.0: Image representing Pitch\n",
      "  * grapher_aubio_silence 1.0: Image representing Aubio Silence\n",
      "  * grapher_dissonance 1.0: Image representing Dissonance\n",
      "  * grapher_vamp_cqt 1.0: Image representing Constant Q Transform\n",
      "  * grapher_loudness_itu 1.0: Image representing Loudness ITU\n",
      "  * spectrogram 1.0: Image representing Linear Spectrogram\n",
      "  * grapher_onset_detection_function 1.0: Image representing Onset detection\n",
      "  * grapher_waveform 1.0: Image representing Waveform from Analyzer\n",
      "  * spectrogram_log 1.0: Logarithmic scaled spectrogram (level vs. frequency vs. time).\n",
      "  * spectrogram_lin 1.0: Linear scaled spectrogram (level vs. frequency vs. time).\n",
      "  * waveform_simple 1.0: Simple monochrome waveform image.\n",
      "  * waveform_centroid 1.0: Waveform where peaks are colored relatively to the spectral centroids of each frame buffer.\n",
      "  * waveform_contour_black 1.0: Black amplitude contour waveform.\n",
      "  * waveform_contour_white 1.0: an white amplitude contour wavform.\n",
      "  * waveform_transparent 1.0: Transparent waveform.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(list_processors(timeside.core.api.IGrapher))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processors pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All these processors can be chained to form a process pipeline.\n",
    "\n",
    "Let first define a decoder that reads and decodes audio from a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from timeside.core import get_processor\n",
    "\n",
    "from timeside.core.tools.test_samples import samples\n",
    "file_decoder = get_processor('file_decoder')(samples['C4_scale.wav'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then some other processors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# analyzers\n",
    "pitch = get_processor('aubio_pitch')()\n",
    "level = get_processor('level')()\n",
    "\n",
    "# Encoder\n",
    "mp3 = get_processor('mp3_encoder')('/tmp/guitar.mp3', overwrite=True)\n",
    "\n",
    "# Graphers\n",
    "specgram = get_processor('spectrogram_lin')()\n",
    "waveform = get_processor('waveform_simple')()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now define a process pipeline with all these processors and run it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = (file_decoder | pitch | level | mp3 | specgram | waveform)\n",
    "pipe.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analyzers results are available through the pipe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['63921b5c-bb7a-457f-bed3-23143ecf0a13', 'c461920d-c32c-425d-a549-4504721115ae'])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.results.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or from the analyzer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['aubio_pitch.pitch', 'aubio_pitch.pitch_confidence'])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pitch.results.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['id_metadata', 'data_object', 'audio_metadata', 'parameters']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pitch.results['aubio_pitch.pitch'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AnalyzerResult(id_metadata=IdMetadata(id='aubio_pitch.pitch', name='F0 (aubio) pitch', unit='Hz', description='Aubio Pitch estimation analyzer', date='2022-12-23 14:27:36', version='1.0', author='TimeSide', proc_uuid='63921b5c-bb7a-457f-bed3-23143ecf0a13'), data_object=FrameValueObject(value=array([266.7347 , 263.19116, 263.19116, 263.1912 , 263.19116, 263.19116,\n",
       "       263.19107, 263.19116, 263.19116, 263.19116, 263.19125, 263.19116,\n",
       "       263.19116, 263.19116, 263.19104, 263.19113, 263.19116, 263.19116,\n",
       "       263.19125, 263.19116, 263.19113, 263.19116, 263.19107, 263.1912 ,\n",
       "       263.19107, 263.19125, 263.19107, 263.19104, 263.19116, 263.19125,\n",
       "       263.19128, 263.19116, 263.19116, 263.19116, 263.19122, 263.19116,\n",
       "       263.19125, 263.19125, 263.19116, 263.19107, 263.19107, 263.19122,\n",
       "       263.19116, 268.12445,   0.     ,   0.     ,   0.     ,   0.     ,\n",
       "         0.     ,   0.     ,   0.     , 319.86548, 296.1497 , 295.05573,\n",
       "       295.05582, 295.0559 , 295.05582, 295.0559 , 295.05582, 295.05582,\n",
       "       295.0559 , 295.0559 , 295.0559 , 295.05588, 295.05582, 295.0559 ,\n",
       "       295.05582, 295.05573, 295.05597, 295.05588, 295.0559 , 295.05606,\n",
       "       295.05566, 295.0559 , 295.05582, 295.05573, 295.05597, 295.0559 ,\n",
       "       295.0559 , 295.05597, 295.05597, 295.05582, 295.05597, 295.0559 ,\n",
       "       295.0559 , 295.0559 , 295.05573, 295.05588, 295.05582, 295.05582,\n",
       "       295.05597, 295.0559 , 295.05573, 295.05597, 295.16232,   0.     ,\n",
       "         0.     ,   0.     ,   0.     ,   0.     ,   0.     ,   0.     ,\n",
       "         0.     , 338.1412 , 330.84784, 330.7225 , 330.72263, 330.72266,\n",
       "       330.72263, 330.72275, 330.72263, 330.72256, 330.72263, 330.72266,\n",
       "       330.72256, 330.72266, 330.72266, 330.72266, 330.72247, 330.72272,\n",
       "       330.7225 , 330.72253, 330.72266, 330.7227 , 330.72253, 330.72266,\n",
       "       330.72266, 330.72253, 330.72253, 330.72263, 330.72263, 330.72253,\n",
       "       330.72266, 330.72263, 330.72256, 330.72253, 330.72266, 330.72253,\n",
       "       330.72247, 330.72266, 330.72247, 330.72266, 330.72266, 330.72256,\n",
       "       330.72256, 330.72272, 331.81577,   0.     ,   0.     ,   0.     ,\n",
       "         0.     ,   0.     ,   0.     ,   0.     ,   0.     , 353.317  ,\n",
       "       350.25754, 350.25748, 350.25757, 350.25754, 350.25757, 350.25757,\n",
       "       350.25748, 350.25754, 350.2575 , 350.25757, 350.25757, 350.25748,\n",
       "       350.2575 , 350.25757, 350.25754, 350.2576 , 350.25748, 350.25757,\n",
       "       350.2575 , 350.25748, 350.25754, 350.25748, 350.2575 , 350.25757,\n",
       "       350.25748, 350.25748, 350.25757, 350.2575 , 350.25757, 350.25745,\n",
       "       350.2575 , 350.2575 , 350.2575 , 350.25748, 350.25757, 350.25748,\n",
       "       350.25757, 350.25757, 350.25745, 350.25748, 350.25757, 350.25757,\n",
       "       355.18585,   0.     ,   0.     ,   0.     ,   0.     ,   0.     ,\n",
       "         0.     ,   0.     , 414.70154, 393.92392, 392.9126 , 392.9126 ,\n",
       "       392.91254, 392.91266, 392.9127 , 392.91266, 392.9126 , 392.9126 ,\n",
       "       392.91254, 392.91254, 392.9126 , 392.9126 , 392.9127 , 392.91257,\n",
       "       392.91257, 392.9126 , 392.91266, 392.91257, 392.91266, 392.91257,\n",
       "       392.91266, 392.91254, 392.91254, 392.91257, 392.91266, 392.9125 ,\n",
       "       392.9126 , 392.91266, 392.91275, 392.91257, 392.9127 , 392.9126 ,\n",
       "       392.91257, 392.91254, 392.91254, 392.91263, 392.9127 , 392.91266,\n",
       "       392.91257, 392.91263, 392.9126 , 392.92957,   0.     ,   0.     ,\n",
       "         0.     ,   0.     ,   0.     ,   0.     ,   0.     ,   0.     ,\n",
       "       447.19846, 440.90103, 440.76074, 440.7608 , 440.76074, 440.7608 ,\n",
       "       440.76074, 440.7608 , 440.76074, 440.76074, 440.7608 , 440.7608 ,\n",
       "       440.7608 , 440.7608 , 440.76077, 440.7608 , 440.7608 , 440.76074,\n",
       "       440.7608 , 440.7608 , 440.76086, 440.76077, 440.76077, 440.76074,\n",
       "       440.76074, 440.76086, 440.76074, 440.76074, 440.76074, 440.7608 ,\n",
       "       440.7608 , 440.76074, 440.76074, 440.7608 , 440.76077, 440.7608 ,\n",
       "       440.76086, 440.7608 , 440.76074, 440.7608 , 440.7608 , 440.76074,\n",
       "       440.76074, 441.13644,   0.     ,   0.     ,   0.     ,   0.     ,\n",
       "         0.     ,   0.     ,   0.     ,   0.     , 497.2589 , 494.56213,\n",
       "       494.5621 , 494.56207, 494.56213, 494.5621 , 494.56207, 494.56198,\n",
       "       494.5621 , 494.5621 , 494.562  , 494.56213, 494.562  , 494.56207,\n",
       "       494.56213, 494.56207, 494.56207, 494.5621 , 494.56207, 494.5621 ,\n",
       "       494.56213, 494.56213, 494.56213, 494.56198, 494.5621 , 494.5621 ,\n",
       "       494.56213, 494.562  , 494.56198, 494.5621 , 494.5621 , 494.56207,\n",
       "       494.5621 , 494.56207, 494.56207, 494.56207, 494.5621 , 494.56207,\n",
       "       494.5621 , 494.56213, 494.56213, 494.56207, 494.56207, 497.16904,\n",
       "         0.     ,   0.     ,   0.     ,   0.     ,   0.     ,   0.     ,\n",
       "         0.     , 544.3793 , 524.8503 , 523.86566, 523.86566, 523.8656 ,\n",
       "       523.86566, 523.8658 , 523.8657 , 523.8657 , 523.86566, 523.86566,\n",
       "       523.8657 , 523.8656 , 523.86566, 523.86584, 523.8656 , 523.86566,\n",
       "       523.86566, 523.8657 , 523.86566, 523.86566, 523.8656 , 523.8656 ,\n",
       "       523.86566, 523.8657 , 523.8657 , 523.86566, 523.86566, 523.8656 ,\n",
       "       523.86566, 523.86566, 523.8656 , 523.8657 , 523.86566, 523.86566,\n",
       "       523.86566, 523.86566, 523.86566, 523.8656 , 523.86566, 523.86566,\n",
       "       523.8656 , 523.8657 , 523.89   ], dtype=float32), y_value=array([], dtype=float64), frame_metadata=FrameMetadata(samplerate=44100, blocksize=2048, stepsize=1024)), audio_metadata=AudioMetadata(uri='file:///srv/lib/timeside/tests/samples/C4_scale.wav', start=0.0, duration=9.4, is_segment=False, sha1='a598e78d0b5c90da54a77e34c083abdcd38d42ba', channels=1, channelsManagement=''), parameters={'blocksize_s': None, 'stepsize_s': None})"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pitch.results['aubio_pitch.pitch']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grapher result can also be display or save into a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f7137101588>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 47532 characters]\n",
      "text/plain": [
       "<Figure size 1152x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "imshow(specgram.render(), origin='lower')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f7136ca4860>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 7108 characters]\n",
      "text/plain": [
       "<Figure size 1152x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "imshow(waveform.render(), origin='lower')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "waveform.render('/tmp/waveform.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## And TimeSide can be embedded into a web page dynamically. For example, in Telemeta:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=1300 height=260 frameborder=0 scrolling=no marginheight=0 marginwidth=0 src=http://demo.telemeta.org/archives/items/6/player/1200x170></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML('<iframe width=1300 height=260 frameborder=0 scrolling=no marginheight=0 marginwidth=0 src=http://demo.telemeta.org/archives/items/6/player/1200x170></iframe>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
