.. _advanced.sanitization:

Sanitization
============

Most feeds embed :abbr:`HTML (HyperText Markup Language)` markup within feed
elements.  Some feeds even embed other types of markup, such as :abbr:`SVG
(Scalable Vector Graphics)` or :abbr:`MathML (Mathematical Markup Language)`.
Since many feed aggregators use a web browser (or browser component) to display
content, :program:`Universal Feed Parser` sanitizes embedded markup to remove
things that could pose security risks.

These elements are sanitized by default:

* :ref:`reference.entry.content`
* :ref:`reference.entry.summary`
* :ref:`reference.entry.title`
* :ref:`reference.feed.info`
* :ref:`reference.feed.rights`
* :ref:`reference.feed.subtitle`
* :ref:`reference.feed.title`


..  note::

    If the content is declared to be (or is determined to be)
    :mimetype:`text/plain`, it will not be sanitized. This is to avoid data loss.
    It is recommended that you check the content type in e.g.
    ``entries[i].summary_detail.type``. If it is :mimetype:`text/plain` then
    it has not been sanitized (and you should perform HTML escaping before
    rendering the content).


.. _advanced.sanitization.html:

:abbr:`HTML (HyperText Markup Language)` Sanitization
-----------------------------------------------------

The following :abbr:`HTML (HyperText Markup Language)` elements are allowed by
default (all others are stripped):

..  hlist::
    :columns: 3

    * a
    * abbr
    * acronym
    * address
    * area
    * article
    * aside
    * audio
    * b
    * big
    * blockquote
    * br
    * button
    * canvas
    * caption
    * center
    * cite
    * code
    * col
    * colgroup
    * command
    * datagrid
    * datalist
    * dd
    * del
    * details
    * dfn
    * dialog
    * dir
    * div
    * dl
    * dt
    * em
    * event-source
    * fieldset
    * figure
    * font
    * footer
    * form
    * h1
    * h2
    * h3
    * h4
    * h5
    * h6
    * header
    * hr
    * i
    * img
    * input
    * ins
    * kbd
    * keygen
    * label
    * legend
    * li
    * m
    * map
    * menu
    * meter
    * multicol
    * nav
    * nextid
    * noscript
    * ol
    * optgroup
    * option
    * output
    * p
    * pre
    * progress
    * q
    * s
    * samp
    * section
    * select
    * small
    * sound
    * source
    * spacer
    * span
    * strike
    * strong
    * sub
    * sup
    * table
    * tbody
    * td
    * textarea
    * tfoot
    * th
    * thead
    * time
    * tr
    * tt
    * u
    * ul
    * var
    * video


The following :abbr:`HTML (HyperText Markup Language)` attributes are allowed
by default (all others are stripped):

..  hlist::
    :columns: 3

    * abbr
    * accept
    * accept-charset
    * accesskey
    * action
    * align
    * alt
    * autocomplete
    * autofocus
    * autoplay
    * axis
    * background
    * balance
    * bgcolor
    * bgproperties
    * border
    * bordercolor
    * bordercolordark
    * bordercolorlight
    * bottompadding
    * cellpadding
    * cellspacing
    * ch
    * challenge
    * char
    * charoff
    * charset
    * checked
    * choff
    * cite
    * class
    * clear
    * color
    * cols
    * colspan
    * compact
    * contenteditable
    * coords
    * data
    * datafld
    * datapagesize
    * datasrc
    * datetime
    * default
    * delay
    * dir
    * disabled
    * draggable
    * dynsrc
    * enctype
    * end
    * face
    * for
    * form
    * frame
    * galleryimg
    * gutter
    * headers
    * height
    * hidden
    * hidefocus
    * high
    * href
    * hreflang
    * hspace
    * icon
    * id
    * inputmode
    * ismap
    * keytype
    * label
    * lang
    * leftspacing
    * list
    * longdesc
    * loop
    * loopcount
    * loopend
    * loopstart
    * low
    * lowsrc
    * max
    * maxlength
    * media
    * method
    * min
    * multiple
    * name
    * nohref
    * noshade
    * nowrap
    * open
    * optimum
    * pattern
    * ping
    * point-size
    * poster
    * pqg
    * preload
    * prompt
    * radiogroup
    * readonly
    * rel
    * repeat-max
    * repeat-min
    * replace
    * required
    * rev
    * rightspacing
    * rows
    * rowspan
    * rules
    * scope
    * selected
    * shape
    * size
    * span
    * src
    * start
    * step
    * summary
    * suppress
    * tabindex
    * target
    * template
    * title
    * toppadding
    * type
    * unselectable
    * urn
    * usemap
    * valign
    * value
    * variable
    * volume
    * vrml
    * vspace
    * width
    * wrap
    * xml:lang


.. _advanced.sanitization.svg:

:abbr:`SVG (Scalable Vector Graphics)` Sanitization
---------------------------------------------------

The following SVG elements are allowed by default (all others are stripped):

..  hlist::
    :columns: 3

    * a
    * animate
    * animateColor
    * animateMotion
    * animateTransform
    * circle
    * defs
    * desc
    * ellipse
    * font-face
    * font-face-name
    * font-face-src
    * foreignObject
    * g
    * glyph
    * hkern
    * line
    * linearGradient
    * marker
    * metadata
    * missing-glyph
    * mpath
    * path
    * polygon
    * polyline
    * radialGradient
    * rect
    * set
    * stop
    * svg
    * switch
    * text
    * title
    * tspan
    * use


The following :abbr:`SVG (Scalable Vector Graphics)` attributes are allowed by
default (all others are stripped):

..  hlist::
    :columns: 3

    * accent-height
    * accumulate
    * additive
    * alphabetic
    * arabic-form
    * ascent
    * attributeName
    * attributeType
    * baseProfile
    * bbox
    * begin
    * by
    * calcMode
    * cap-height
    * class
    * color
    * color-rendering
    * content
    * cx
    * cy
    * d
    * descent
    * display
    * dur
    * dx
    * dy
    * end
    * fill
    * fill-opacity
    * fill-rule
    * font-family
    * font-size
    * font-stretch
    * font-style
    * font-variant
    * font-weight
    * from
    * fx
    * fy
    * g1
    * g2
    * glyph-name
    * gradientUnits
    * hanging
    * height
    * horiz-adv-x
    * horiz-origin-x
    * id
    * ideographic
    * k
    * keyPoints
    * keySplines
    * keyTimes
    * lang
    * marker-end
    * marker-mid
    * marker-start
    * markerHeight
    * markerUnits
    * markerWidth
    * mathematical
    * max
    * min
    * name
    * offset
    * opacity
    * orient
    * origin
    * overline-position
    * overline-thickness
    * panose-1
    * path
    * pathLength
    * points
    * preserveAspectRatio
    * r
    * refX
    * refY
    * repeatCount
    * repeatDur
    * requiredExtensions
    * requiredFeatures
    * restart
    * rotate
    * rx
    * ry
    * slope
    * stemh
    * stemv
    * stop-color
    * stop-opacity
    * strikethrough-position
    * strikethrough-thickness
    * stroke
    * stroke-dasharray
    * stroke-dashoffset
    * stroke-linecap
    * stroke-linejoin
    * stroke-miterlimit
    * stroke-opacity
    * stroke-width
    * systemLanguage
    * target
    * text-anchor
    * to
    * transform
    * type
    * u1
    * u2
    * underline-position
    * underline-thickness
    * unicode
    * unicode-range
    * units-per-em
    * values
    * version
    * viewBox
    * visibility
    * width
    * widths
    * x
    * x-height
    * x1
    * x2
    * xlink:actuate
    * xlink:arcrole
    * xlink:href
    * xlink:role
    * xlink:show
    * xlink:title
    * xlink:type
    * xml:base
    * xml:lang
    * xml:space
    * xmlns
    * xmlns:xlink
    * y
    * y1
    * y2
    * zoomAndPan


.. _advanced.sanitization.mathml:

:abbr:`MathML (Mathematical Markup Language)` Sanitization
----------------------------------------------------------

The following :abbr:`MathML (Mathematical Markup Language)` elements are
allowed by default (all others are stripped):

..  hlist::
    :columns: 3

    * annotation
    * annotation-xml
    * maction
    * maligngroup
    * malignmark
    * math
    * menclose
    * merror
    * mfenced
    * mfrac
    * mglyph
    * mi
    * mlabeledtr
    * mlongdiv
    * mmultiscripts
    * mn
    * mo
    * mover
    * mpadded
    * mphantom
    * mprescripts
    * mroot
    * mrow
    * ms
    * mscarries
    * mscarry
    * msgroup
    * msline
    * mspace
    * msqrt
    * msrow
    * mstack
    * mstyle
    * msub
    * msubsup
    * msup
    * mtable
    * mtd
    * mtext
    * mtr
    * munder
    * munderover
    * none
    * semantics


The following :abbr:`MathML (Mathematical Markup Language)` attributes are
allowed by default (all others are stripped):

..  hlist::
    :columns: 3

    * accent
    * accentunder
    * actiontype
    * align
    * alignmentscope
    * altimg
    * altimg-height
    * altimg-valign
    * altimg-width
    * alttext
    * bevelled
    * charalign
    * close
    * columnalign
    * columnlines
    * columnspacing
    * columnspan
    * columnwidth
    * crossout
    * decimalpoint
    * denomalign
    * depth
    * dir
    * display
    * displaystyle
    * edge
    * encoding
    * equalcolumns
    * equalrows
    * fence
    * fontstyle
    * fontweight
    * form
    * frame
    * framespacing
    * groupalign
    * height
    * href
    * id
    * indentalign
    * indentalignfirst
    * indentalignlast
    * indentshift
    * indentshiftfirst
    * indentshiftlast
    * indenttarget
    * infixlinebreakstyle
    * largeop
    * length
    * linebreak
    * linebreakmultchar
    * linebreakstyle
    * lineleading
    * linethickness
    * location
    * longdivstyle
    * lquote
    * lspace
    * mathbackground
    * mathcolor
    * mathsize
    * mathvariant
    * maxsize
    * minlabelspacing
    * minsize
    * movablelimits
    * notation
    * numalign
    * open
    * other
    * overflow
    * position
    * rowalign
    * rowlines
    * rowspacing
    * rowspan
    * rquote
    * rspace
    * scriptlevel
    * scriptminsize
    * scriptsizemultiplier
    * selection
    * separator
    * separators
    * shift
    * side
    * src
    * stackalign
    * stretchy
    * subscriptshift
    * superscriptshift
    * symmetric
    * voffset
    * width
    * xlink:href
    * xlink:show
    * xlink:type
    * xmlns
    * xmlns:xlink


.. _advanced.sanitization.css:

:abbr:`CSS (Cascading Style Sheets)` Sanitization
-------------------------------------------------

The following :abbr:`CSS (Cascading Style Sheets)` properties are allowed by
default in style attributes (all others are stripped):

..  hlist::
    :columns: 3

    * azimuth
    * background-color
    * border-bottom-color
    * border-collapse
    * border-color
    * border-left-color
    * border-right-color
    * border-top-color
    * clear
    * color
    * cursor
    * direction
    * display
    * elevation
    * float
    * font
    * font-family
    * font-size
    * font-style
    * font-variant
    * font-weight
    * height
    * letter-spacing
    * line-height
    * overflow
    * pause
    * pause-after
    * pause-before
    * pitch
    * pitch-range
    * richness
    * speak
    * speak-header
    * speak-numeral
    * speak-punctuation
    * speech-rate
    * stress
    * text-align
    * text-decoration
    * text-indent
    * unicode-bidi
    * vertical-align
    * voice-family
    * volume
    * white-space
    * width


..  note::

    Not all possible CSS values are allowed for these properties.  The
    allowable values are restricted by a whitelist and a regular expression that
    allows color values and lengths.  :abbr:`URI (Uniform Resource Identifier)`\s
    are not allowed, to prevent `platypus attacks <https://web.archive.org/web/20080826033749/http://diveintomark.org/archives/2003/06/12/how_to_consume_rss_safely>`_.
    See the _HTMLSanitizer class for more details.


Whitelist, Don't Blacklist
--------------------------

I am often asked why :program:`Universal Feed Parser` is so hard-assed about
:abbr:`HTML (HyperText Markup Language)` and :abbr:`CSS (Cascading Style
Sheets)` sanitizing.  To illustrate the problem, here is an incomplete list of
potentially dangerous :abbr:`HTML (HyperText Markup Language)` tags and
attributes:

*   script, which can contain malicious script
*   applet, embed, and object, which can automatically download and execute malicious code
*   meta, which can contain malicious redirects
*   onload, onunload, and all other on* attributes, which can contain malicious script
*   style, link, and the style attribute, which can contain malicious script

*style?* Yes, style. :abbr:`CSS (Cascading Style Sheets)` definitions can contain executable code.


Embedding Javascript in :abbr:`CSS (Cascading Style Sheets)`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This sample is taken from `$READTHEDOCS_CANONICAL_URL/examples/rss20.xml <$READTHEDOCS_CANONICAL_URL/examples/rss20.xml>`_:

..  sourcecode:: html

    <description>Watch out for
    &lt;span style="background: url(javascript:window.location='http://example.org/')"&gt;
    nasty tricks&lt;/span&gt;</description>


This sample is more advanced, and does not contain the keyword javascript: that
many naive :abbr:`HTML (HyperText Markup Language)` sanitizers scan for:

..  sourcecode:: html

    <description>Watch out for
    &lt;span style="any: expression(window.location='http://example.org/')"&gt;
    nasty tricks&lt;/span&gt;</description>


Internet Explorer for Windows will execute the Javascript in both of these examples.

Now consider that in :abbr:`HTML (HyperText Markup Language)`, attribute values may be entity-encoded in several different ways.


Embedding encoded Javascript in :abbr:`CSS (Cascading Style Sheets)`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To a browser, this:

..  sourcecode:: html

    <span style="any: expression(window.location='http://example.org/')">


is the same as this (without the line breaks):

..  sourcecode:: html

    <span style="&#97;&#110;&#121;&#58;&#32;&#101;&#120;&#112;&#114;&#101;
    &#115;&#115;&#105;&#111;&#110;&#40;&#119;&#105;&#110;&#100;&#111;&#119;
    &#46;&#108;&#111;&#99;&#97;&#116;&#105;&#111;&#110;&#61;&#39;&#104;
    &#116;&#116;&#112;&#58;&#47;&#47;&#101;&#120;&#97;&#109;&#112;&#108;
    &#101;&#46;&#111;&#114;&#103;&#47;&#39;&#41;">


which is the same as this (without the line breaks):

..  sourcecode:: html

    <span style="&#x61;&#x6e;&#x79;&#x3a;&#x20;&#x65;&#x78;&#x70;&#x72;
    &#x65;&#x73;&#x73;&#x69;&#x6f;&#x6e;&#x28;&#x77;&#x69;&#x6e;
    &#x64;&#x6f;&#x77;&#x2e;&#x6c;&#x6f;&#x63;&#x61;&#x74;&#x69;
    &#x6f;&#x6e;&#x3d;&#x27;&#x68;&#x74;&#x74;&#x70;&#x3a;&#x2f;
    &#x2f;&#x65;&#x78;&#x61;&#x6d;&#x70;&#x6c;&#x65;&#x2e;&#x6f;
    &#x72;&#x67;&#x2f;&#x27;&#x29;">


And so on, plus several other variations, plus every combination of every
variation.

The more I investigate, the more cases I find where Internet Explorer for
Windows will treat seemingly innocuous markup as code and blithely execute it.
This is why :program:`Universal Feed Parser` uses a whitelist and not a
blacklist. I am reasonably confident that none of the elements or attributes on
the whitelist are security risks. I am not at all confident about elements or
attributes that I have not explicitly investigated. And I have no confidence at
all in my ability to detect strings within attribute values that Internet
Explorer for Windows will treat as executable code.

Disabling HTML Sanitization
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Though not recommended, it is possible to disable :program:`Universal Feed Parser`\'s
HTML sanitization by passing ``sanitize_html=False`` to ``feedparser.parse()``.
When passing this flag you are responsible for manually sanitizing HTML from the feed.

..  seealso::

    `How to consume RSS safely <https://web.archive.org/web/20080826033749/http://diveintomark.org/archives/2003/06/12/how_to_consume_rss_safely>`_
        Explains the platypus attack.
