html entities to unicode text,python - Convert HTML entities to Unicode and vice versa - Stack Overf...

Update for Python 2.7 and BeautifulSoup4

Unescape -- Unicode HTML to unicode with htmlparser (Python 2.7 standard lib):

>>> escaped = u'Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood'

>>> from HTMLParser import HTMLParser

>>> htmlparser = HTMLParser()

>>> unescaped = htmlparser.unescape(escaped)

>>> unescaped

u'Monsieur le Cur\xe9 of the \xabNotre-Dame-de-Gr\xe2ce\xbb neighborhood'

>>> print unescaped

Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood

Unescape -- Unicode HTML to unicode with bs4 (BeautifulSoup4):

>>> html = '''

Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood

'''

>>> from bs4 import BeautifulSoup

>>> soup = BeautifulSoup(html)

>>> soup.text

u'Monsieur le Cur\xe9 of the \xabNotre-Dame-de-Gr\xe2ce\xbb neighborhood'

>>> print soup.text

Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood

Escape -- Unicode to unicode HTML with bs4 (BeautifulSoup4):

>>> unescaped = u'Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood'

>>> from bs4.dammit import EntitySubstitution

>>> escaper = EntitySubstitution()

>>> escaped = escaper.substitute_html(unescaped)

>>> escaped

u'Monsieur le Curé of the «Notre-Dame-de-Grâce» neighborhood'