爬虫dns cache设置

设置代理服务器squid的dns

• positive_dns_ttl 6 hours #设置为6小时，这个值取dns返回的ttl最小值
• dns_nameservers 8.8.8.8 #设置google dns服务器

设置nscd

• enable-cache hosts yes #和squid设置差不多

python requests设置

from urllib3.util import connection

    _orig_create_connection = connection.create_connection

    def patched_create_connection(address, *args, **kwargs):
        """Wrap urllib3's create_connection to resolve the name elsewhere"""
        # resolve hostname to an ip address; use your own
        # resolver here, as otherwise the system resolver will be used.
        host, port = address
        hostname = your_dns_resolver(host)

        return _orig_create_connection((hostname, port), *args, **kwargs)

    connection.create_connection = patched_create_connection

这里是stackoverflow回答