对于上面的问题怀疑是和nss版本有关,以下将分别在华南vpc和华南灰度机验证curl请求
一、在华南灰度环境测试
1、curl s3内网新域名(报错)
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# curl -I https://s3-internal.cn-south-1.jdcloud-oss.com/a/a -v
* About to connect() to s3-internal.cn-south-1.jdcloud-oss.com port 443 (#0)
* Trying 100.65.254.35...
* Connected to s3-internal.cn-south-1.jdcloud-oss.com (100.65.254.35) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP)
* Cannot communicate securely with peer: no common encryption algorithm(s).
* Error in TLS handshake, trying SSLv3...
> HEAD /a/a HTTP/1.1
> User-Agent: curl/7.29.0
> Host: s3-internal.cn-south-1.jdcloud-oss.com
> Accept: */*
>
* Connection died, retrying a fresh connect
* Closing connection 0
* Issue another request to this URL: 'https://s3-internal.cn-south-1.jdcloud-oss.com/a/a'
* About to connect() to s3-internal.cn-south-1.jdcloud-oss.com port 443 (#1)
* Trying 100.65.254.35...
* Connected to s3-internal.cn-south-1.jdcloud-oss.com (100.65.254.35) port 443 (#1)
* TLS disabled due to previous handshake failure
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP)
* Cannot communicate securely with peer: no common encryption algorithm(s).
* Closing connection 1
curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).
2、测试443端口是否开放(全部ok)
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# telnet 127.0.0.1 443
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
^CConnection closed by foreign host.
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# telnet 100.65.254.3 443
Trying 100.65.254.3...
Connected to 100.65.254.3.
Escape character is '^]'.
3、curl s3外网新域名 (结果ok)
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# curl -I https://s3.cn-south-1.jdcloud-oss.com/a/a -v
* About to connect() to s3.cn-south-1.jdcloud-oss.com port 443 (#0)
* Trying 59.37.144.139...
* Connected to s3.cn-south-1.jdcloud-oss.com (59.37.144.139) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* subject: CN=*.s3.cn-north-1.jdcloud-oss.com,O="BEIJING JINGDONG SHANGKE INFORMATION TECHNOLOGY CO., LTD.",L=beijing,ST=beijing,C=CN
* start date: Jan 29 09:31:09 2019 GMT
* expire date: Jan 30 09:31:09 2020 GMT
* common name: *.s3.cn-north-1.jdcloud-oss.com
* issuer: CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE
> HEAD /a/a HTTP/1.1
> User-Agent: curl/7.29.0
> Host: s3.cn-south-1.jdcloud-oss.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Server: jfe
Server: jfe
< Date: Mon, 11 Mar 2019 04:53:31 GMT
Date: Mon, 11 Mar 2019 04:53:31 GMT
< Content-Length: 0
Content-Length: 0
< Connection: keep-alive
Connection: keep-alive
< x-req-id: B6E24BAF242EF989
x-req-id: B6E24BAF242EF989
<
* Connection #0 to host s3.cn-south-1.jdcloud-oss.com left intact
4、curl s3老域名(结果OK)
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# curl -I https://s3.cn-south-1.jcloudcs.com/a/a -v
* About to connect() to s3.cn-south-1.jcloudcs.com port 443 (#0)
* Trying 100.65.11.130...
* Connected to s3.cn-south-1.jcloudcs.com (100.65.11.130) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
* subject: CN=*.jdcloud.com,O="BEIJING JINGDONG SHANGKE INFORMATION TECHNOLOGY CO., LTD.",L=beijing,ST=beijing,C=CN
* start date: Nov 19 02:26:04 2018 GMT
* expire date: Feb 18 09:06:02 2020 GMT
* common name: *.jdcloud.com
* issuer: CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE
> HEAD /a/a HTTP/1.1
> User-Agent: curl/7.29.0
> Host: s3.cn-south-1.jcloudcs.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Server: JDCloudOSS
Server: JDCloudOSS
< Date: Mon, 11 Mar 2019 04:53:46 GMT
Date: Mon, 11 Mar 2019 04:53:46 GMT
< Content-Length: 0
Content-Length: 0
< Connection: close
Connection: close
< x-req-id: 8228B04708806DDF
x-req-id: 8228B04708806DDF
<
* Closing connection 0
二、华南vpc中测试
1、curl s3内网新域名(ok),测试其它域名也ok这里就不全贴了
[root@domain-cn-south-1 ~]# curl -I https://s3-internal.cn-south-1.jdcloud-oss.com/a/a -v
* About to connect() to s3-internal.cn-south-1.jdcloud-oss.com port 443 (#0)
* Trying 100.65.254.35...
* Connected to s3-internal.cn-south-1.jdcloud-oss.com (100.65.254.35) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=*.s3.cn-north-1.jdcloud-oss.com,O="BEIJING JINGDONG SHANGKE INFORMATION TECHNOLOGY CO., LTD.",L=beijing,ST=beijing,C=CN
* start date: Jan 29 09:31:09 2019 GMT
* expire date: Jan 30 09:31:09 2020 GMT
* common name: *.s3.cn-north-1.jdcloud-oss.com
* issuer: CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE
> HEAD /a/a HTTP/1.1
> User-Agent: curl/7.29.0
> Host: s3-internal.cn-south-1.jdcloud-oss.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Server: JDCloudOSS
Server: JDCloudOSS
< Date: Mon, 11 Mar 2019 05:13:33 GMT
Date: Mon, 11 Mar 2019 05:13:33 GMT
< Content-Length: 0
Content-Length: 0
< Connection: keep-alive
Connection: keep-alive
< x-req-id: 828EF8FED8952127
x-req-id: 828EF8FED8952127
<
* Connection #0 to host s3-internal.cn-south-1.jdcloud-oss.com left intact
三、通过以上测试怀疑是nss版本问题导致无法加载新的证书
1、检查nss版本(华南灰度机nss版本低)
华南灰度机
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.15.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# rpm -qa | grep nss
openssh-server-6.6.1p1-11.el7.x86_64
nss-softokn-freebl-3.16.2.3-9.el7.x86_64
nss-util-3.16.2.3-2.el7.x86_64
openssl-1.0.1e-42.el7.x86_64
openssh-clients-6.6.1p1-11.el7.x86_64
nss-tools-3.16.2.3-5.el7.x86_64
nss-3.16.2.3-5.el7.x86_64
nss-sysinit-3.16.2.3-5.el7.x86_64
jansson-2.4-6.el7.x86_64
openssh-6.6.1p1-11.el7.x86_64
nss-softokn-3.16.2.3-9.el7.x86_64
libsss_nss_idmap-1.12.2-58.el7.x86_64
openssl-devel-1.0.1e-42.el7.x86_64
openssl-libs-1.0.1e-42.el7.x86_64
华南vpc
[root@domain-cn-south-1 ~]# curl -V
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.34 zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets
[root@domain-cn-south-1 ~]# rpm -qa | grep nss
nss-3.36.0-5.el7_5.x86_64
jansson-2.10-1.el7.x86_64
nss-util-3.36.0-1.el7_5.x86_64
nss-pem-1.0.3-4.el7.x86_64
openssl-1.0.2k-12.el7.x86_64
nss-sysinit-3.36.0-5.el7_5.x86_64
nss-tools-3.36.0-5.el7_5.x86_64
openssh-clients-7.4p1-16.el7.x86_64
nss-softokn-freebl-3.36.0-5.el7_5.x86_64
nss-softokn-3.36.0-5.el7_5.x86_64
openssl-libs-1.0.2k-12.el7.x86_64
openssh-7.4p1-16.el7.x86_64
libsss_nss_idmap-1.16.0-19.el7_5.5.x86_64
openssh-server-7.4p1-16.el7.x86_64
四、更新华南灰度机nss版本
1、
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# yum update nss
Loaded plugins: fastestmirror, langpacks
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
base | 3.6 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/2): extras/7/x86_64/primary_db | 180 kB 00:00:00
(2/2): updates/7/x86_64/primary_db | 2.5 MB 00:00:00
Determining fastest mirrors
Resolving Dependencies
--> Running transaction check
---> Package nss.x86_64 0:3.16.2.3-5.el7 will be updated
--> Processing Dependency: nss = 3.16.2.3-5.el7 for package: nss-sysinit-3.16.2.3-5.el7.x86_64
--> Processing Dependency: nss(x86-64) = 3.16.2.3-5.el7 for package: nss-tools-3.16.2.3-5.el7.x86_64
---> Package nss.x86_64 0:3.36.0-7.1.el7_6 will be an update
--> Processing Dependency: nss-util >= 3.36.0-1.1 for package: nss-3.36.0-7.1.el7_6.x86_64
--> Processing Dependency: nss-softokn(x86-64) >= 3.36.0-1 for package: nss-3.36.0-7.1.el7_6.x86_64
--> Processing Dependency: nspr >= 4.19.0 for package: nss-3.36.0-7.1.el7_6.x86_64
--> Processing Dependency: nss-pem(x86-64) for package: nss-3.36.0-7.1.el7_6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.31)(64bit) for package: nss-3.36.0-7.1.el7_6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.24)(64bit) for package: nss-3.36.0-7.1.el7_6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.21)(64bit) for package: nss-3.36.0-7.1.el7_6.x86_64
--> Running transaction check
---> Package nspr.x86_64 0:4.10.6-3.el7 will be updated
---> Package nspr.x86_64 0:4.19.0-1.el7_5 will be an update
---> Package nss-pem.x86_64 0:1.0.3-5.el7 will be installed
---> Package nss-softokn.x86_64 0:3.16.2.3-9.el7 will be updated
---> Package nss-softokn.x86_64 0:3.36.0-5.el7_5 will be an update
--> Processing Dependency: nss-softokn-freebl(x86-64) >= 3.36.0-5.el7_5 for package: nss-softokn-3.36.0-5.el7_5.x86_64
---> Package nss-sysinit.x86_64 0:3.16.2.3-5.el7 will be updated
---> Package nss-sysinit.x86_64 0:3.36.0-7.1.el7_6 will be an update
---> Package nss-tools.x86_64 0:3.16.2.3-5.el7 will be updated
---> Package nss-tools.x86_64 0:3.36.0-7.1.el7_6 will be an update
---> Package nss-util.x86_64 0:3.16.2.3-2.el7 will be updated
---> Package nss-util.x86_64 0:3.36.0-1.1.el7_6 will be an update
--> Running transaction check
---> Package nss-softokn-freebl.x86_64 0:3.16.2.3-9.el7 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.36.0-5.el7_5 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================================
Updating:
nss x86_64 3.36.0-7.1.el7_6 updates 835 k
Installing for dependencies:
nss-pem x86_64 1.0.3-5.el7 base 74 k
Updating for dependencies:
nspr x86_64 4.19.0-1.el7_5 base 127 k
nss-softokn x86_64 3.36.0-5.el7_5 base 315 k
nss-softokn-freebl x86_64 3.36.0-5.el7_5 base 222 k
nss-sysinit x86_64 3.36.0-7.1.el7_6 updates 62 k
nss-tools x86_64 3.36.0-7.1.el7_6 updates 515 k
nss-util x86_64 3.36.0-1.1.el7_6 updates 78 k
Transaction Summary
=============================================================================================================================================================================================
Install ( 1 Dependent package)
Upgrade 1 Package (+6 Dependent packages)
Total download size: 2.2 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/8): nspr-4.19.0-1.el7_5.x86_64.rpm | 127 kB 00:00:00
(2/8): nss-pem-1.0.3-5.el7.x86_64.rpm | 74 kB 00:00:00
(3/8): nss-softokn-freebl-3.36.0-5.el7_5.x86_64.rpm | 222 kB 00:00:00
(4/8): nss-softokn-3.36.0-5.el7_5.x86_64.rpm | 315 kB 00:00:00
(5/8): nss-3.36.0-7.1.el7_6.x86_64.rpm | 835 kB 00:00:00
(6/8): nss-tools-3.36.0-7.1.el7_6.x86_64.rpm | 515 kB 00:00:00
(7/8): nss-sysinit-3.36.0-7.1.el7_6.x86_64.rpm | 62 kB 00:00:00
(8/8): nss-util-3.36.0-1.1.el7_6.x86_64.rpm | 78 kB 00:00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 4.6 MB/s | 2.2 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : nspr-4.19.0-1.el7_5.x86_64 1/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Updating : nss-util-3.36.0-1.1.el7_6.x86_64 2/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Updating : nss-softokn-freebl-3.36.0-5.el7_5.x86_64 3/15
Updating : nss-softokn-3.36.0-5.el7_5.x86_64 4/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Installing : nss-pem-1.0.3-5.el7.x86_64 5/15
Updating : nss-sysinit-3.36.0-7.1.el7_6.x86_64 6/15
Updating : nss-3.36.0-7.1.el7_6.x86_64 7/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Updating : nss-tools-3.36.0-7.1.el7_6.x86_64 8/15
Cleanup : nss-tools-3.16.2.3-5.el7.x86_64 9/15
Cleanup : nss-sysinit-3.16.2.3-5.el7.x86_64 10/15
Cleanup : nss-3.16.2.3-5.el7.x86_64 11/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Cleanup : nss-softokn-3.16.2.3-9.el7.x86_64 12/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Cleanup : nss-util-3.16.2.3-2.el7.x86_64 13/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Cleanup : nspr-4.10.6-3.el7.x86_64 14/15
/sbin/ldconfig: /usr/local/lib/libpcreposix.so.0 is not a symbolic link
Cleanup : nss-softokn-freebl-3.16.2.3-9.el7.x86_64 15/15
Verifying : nss-3.36.0-7.1.el7_6.x86_64 1/15
Verifying : nss-pem-1.0.3-5.el7.x86_64 2/15
Verifying : nss-tools-3.36.0-7.1.el7_6.x86_64 3/15
Verifying : nspr-4.19.0-1.el7_5.x86_64 4/15
Verifying : nss-sysinit-3.36.0-7.1.el7_6.x86_64 5/15
Verifying : nss-softokn-freebl-3.36.0-5.el7_5.x86_64 6/15
Verifying : nss-util-3.36.0-1.1.el7_6.x86_64 7/15
Verifying : nss-softokn-3.36.0-5.el7_5.x86_64 8/15
Verifying : nss-softokn-freebl-3.16.2.3-9.el7.x86_64 9/15
Verifying : nss-util-3.16.2.3-2.el7.x86_64 10/15
Verifying : nss-sysinit-3.16.2.3-5.el7.x86_64 11/15
Verifying : nss-tools-3.16.2.3-5.el7.x86_64 12/15
Verifying : nss-softokn-3.16.2.3-9.el7.x86_64 13/15
Verifying : nss-3.16.2.3-5.el7.x86_64 14/15
Verifying : nspr-4.10.6-3.el7.x86_64 15/15
Dependency Installed:
nss-pem.x86_64 0:1.0.3-5.el7
Updated:
nss.x86_64 0:3.36.0-7.1.el7_6
Dependency Updated:
nspr.x86_64 0:4.19.0-1.el7_5 nss-softokn.x86_64 0:3.36.0-5.el7_5 nss-softokn-freebl.x86_64 0:3.36.0-5.el7_5 nss-sysinit.x86_64 0:3.36.0-7.1.el7_6
nss-tools.x86_64 0:3.36.0-7.1.el7_6 nss-util.x86_64 0:3.36.0-1.1.el7_6
Complete!
2、重新测试华南内网新域名OK
[root@A06-R12-302F0714-I12-86 --PROD-- ~]# curl -I https://s3-internal.cn-south-1.jdcloud-oss.com/a/a -v
* About to connect() to s3-internal.cn-south-1.jdcloud-oss.com port 443 (#0)
* Trying 100.65.254.3...
* Connected to s3-internal.cn-south-1.jdcloud-oss.com (100.65.254.3) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
* subject: CN=*.s3.cn-north-1.jdcloud-oss.com,O="BEIJING JINGDONG SHANGKE INFORMATION TECHNOLOGY CO., LTD.",L=beijing,ST=beijing,C=CN
* start date: Jan 29 09:31:09 2019 GMT
* expire date: Jan 30 09:31:09 2020 GMT
* common name: *.s3.cn-north-1.jdcloud-oss.com
* issuer: CN=GlobalSign Organization Validation CA - SHA256 - G2,O=GlobalSign nv-sa,C=BE
> HEAD /a/a HTTP/1.1
> User-Agent: curl/7.29.0
> Host: s3-internal.cn-south-1.jdcloud-oss.com
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Server: JDCloudOSS
Server: JDCloudOSS
< Date: Mon, 11 Mar 2019 04:54:29 GMT
Date: Mon, 11 Mar 2019 04:54:29 GMT
< Content-Length: 0
Content-Length: 0
< Connection: keep-alive
Connection: keep-alive
< x-req-id: ABE7F53066B4B389
x-req-id: ABE7F53066B4B389
<
* Connection #0 to host s3-internal.cn-south-1.jdcloud-oss.com left intact
- 结论:nss版本问题导致curl报错
- 处理方法:yum update nss