Normalization

Venus builds on, and extends, the Universal Feed Parser and html5lib to convert all feeds into Atom 1.0, with well formed XHTML, and encoded as UTF-8, meaning that you don't have to worry about funky feeds, tag soup, or character encoding.

Encoding

Input data in feeds may be encoded in a variety of formats, most commonly ASCII, ISO-8859-1, WIN-1252, AND UTF-8. Additionally, many feeds make use of the wide range of character entity references provided by HTML. Each is converted to UTF-8, an encoding which is a proper superset of ASCII, supports the entire range of Unicode characters, and is one of only two encodings required to be supported by all conformant XML processors.

Encoding problems are one of the more common feed errors, and every attempt is made to correct common errors, such as the inclusion of the so-called moronic versions of smart-quotes. In rare cases where individual characters can not be converted to valid UTF-8 or into characters allowed in XML 1.0 documents, such characters will be replaced with the Unicode Replacement character, with a title that describes the original character whenever possible.

In order to support the widest range of inputs, use of Python 2.3 or later, as well as the installation of the python iconvcodec, is recommended.

HTML

A number of different normalizations of HTML are performed. For starters, the HTML is sanitized, meaning that HTML tags and attributes that could introduce javascript or other security risks are removed.

Then, relative links are resolved within the HTML. This is also done for links in other areas in the feed too.

Finally, unmatched tags are closed. This is done with a knowledge of the semantics of HTML. Additionally, a large subset of MathML, as well as a tiny profile of SVG is also supported.

Atom 1.0

The Universal Feed Parser also normalizes the content of feeds. This involves a large number of elements; the best place to start is to look at annotated examples. Among other things a wide variety of date formats are converted into RFC 3339 formatted dates.

If no ids are found in entries, attempts are made to synthesize one using (in order):

If no updated dates are found in an entry, the updated date from the feed is used. If no updated date is found in either the feed or the entry, the current time is substituted.

Overrides

All of the above describes what Venus does automatically, either directly or through its dependencies. There are a number of errors which can not be corrected automatically, and for these, there are configuration parameters that can be used to help.