Natural Language Processing With Python (2)

davidcqw

于 2014-08-26 11:34:25 发布

阅读量730

点赞数

CC 4.0 BY-SA版权

分类专栏： Reading 文章标签： natural language processing python 杂谈

本文链接：https://blog.youkuaiyun.com/davidcqw/article/details/38845363

Reading 专栏收录该内容

4 篇文章

订阅专栏

本篇博客详细介绍了如何从网络和磁盘获取原始文本，并通过Unicode编码进行内存处理。重点讲述了正则表达式的使用，包括搜索、查找、替换等功能。此外，还涉及了文本的规范化与分段，如词干提取、词形还原、句子和单词的分割等关键技术。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Chapter 3:

This chapter describes the skill to process raw text.

Some important point:

1. Access text from web and disk : api such as urlopen(), open(), read(), write() and some string operation . Also some tool to process text of html.

2. Text processing with Unicode : file/terminal(specific encoding) -> In-memory program including python processing(Unicode) -> file/terminal (specific encoding)

3. Regular expressions : re.search, find, findall, replace, splite and so on (remember to add r charater for raw text of regular expression).

Another api in nltk is nltk.regexp_tokenize() which is similar to findall.

Useful for finding word stems and searching tokenized text.

4. Normalizing Text and Segmentation : Stemmers, Lemmatization, Sentence Segmantation, Word Segmantation.