sogou spider 抓取网站robots.txt 400问题？

最新推荐文章于 2025-06-06 19:23:24 发布

陈杰-攀基生物

最新推荐文章于 2025-06-06 19:23:24 发布

阅读量1.6k

点赞数

CC 4.0 BY-SA版权

分类专栏：网站

本文链接：https://blog.youkuaiyun.com/weixin_44252244/article/details/88080309

首先，我要说，网站正常访问是没问题的。而且，百度，360 spider都访问ok。
但sogou站长工具测试没问题，后台日志显示，抓取的时候，就是400.
不过，确实看不出来400的与其他有什么差别。由于采用了https访问，所以，做了301转向调整。另外 panjishengwu.com 转向了www.panjisheng.com的转向跳转。都是301.
浏览器测试都是正常。

对于400错误，我一定办法没有。而且，只有这个文件是400.
但这个文件影响了我的收录。我调整域名，调整nginx的robots.txt配置，都无用。
请求根本到不了后端。到目前为止问题依然没有解决。看到的200状态，都是我利用sogou的站长工具测试的。测试是没有问题的。
看了一些文章，有说是域名不对。我域名设置为所有。针对非我域名做301跳转。
但我收到还是400.

也有说是客户端问题。那这个我就无法验证了。具体怎么回事，如果有大拿清楚原因，还请赐教。

123.126.113.90 - - [02/Mar/2019:15:21:30 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.73 - - [02/Mar/2019:19:22:31 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [29/Jan/2019:08:58:34 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.82 - - [29/Jan/2019:23:21:35 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.82 - - [29/Jan/2019:23:22:57 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.90 - - [30/Jan/2019:00:21:36 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.90 - - [30/Jan/2019:00:21:51 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.131 - - [30/Jan/2019:03:22:19 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.131 - - [30/Jan/2019:03:24:23 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.158 - - [30/Jan/2019:07:20:45 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.158 - - [30/Jan/2019:07:21:54 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [30/Jan/2019:08:54:13 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [30/Jan/2019:21:06:15 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.132 - - [31/Jan/2019:00:22:14 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.132 - - [31/Jan/2019:00:22:14 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.38.241.121 - - [31/Jan/2019:03:24:24 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [31/Jan/2019:21:26:51 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [01/Feb/2019:09:14:04 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.133 - - [02/Feb/2019:00:17:31 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.133 - - [02/Feb/2019:00:18:01 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
111.202.101.250 - - [02/Feb/2019:08:30:09 +0800] “GET /robots.txt HTTP/1.1” 200 21 “-” “Mozilla/5.0 (Linux; Android 6.0.1) AppleWebKit/601.1 (KHTML,like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Sogou web spider/4.0; +http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [02/Feb/2019:09:37:55 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [02/Feb/2019:22:18:44 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
218.30.103.29 - - [03/Feb/2019:03:23:56 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [03/Feb/2019:17:53:33 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.104 - - [04/Feb/2019:00:22:41 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.120.173.104 - - [04/Feb/2019:00:23:02 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
106.38.241.111 - - [04/Feb/2019:03:21:36 +0800] “GET /robots.txt HTTP/1.1” 400 271 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.113.91 - - [04/Feb/2019:11:54:26 +0800] “GET /robots.txt HTTP/1.1” 301 185 “-” “Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)” “-”
123.126.11