利用 S3-tests 测试 S3 接口兼容性

最新推荐文章于 2024-06-20 22:53:35 发布

360技术

最新推荐文章于 2024-06-20 22:53:35 发布

阅读量2.7k

点赞数

文章标签： python docker java nginx 大数据

原文链接：https://mp.weixin.qq.com/s/FBYmVK51GVmxq6g0lY7r7g#rd

版权

奇技 · 指南

在对象存储迭代研发、测试过程中，为了方便、准确验证 S3 接口协议兼容性，本文作者对Ceph官方采用的兼容性测试工具s3-tests进行了调研，并对其配置以及使用做出了详细介绍，相信对于s3的使用者，会起到实质性的帮助，下来就跟随作者一探究竟吧。

简介

在对象存储迭代研发、测试过程中，为了方便、准确验证 S3 接口协议兼容性，我们内部配置了 Ceph 官方采用的兼容性测试工具 s3-tests ，该工具基于 Python Nose 测试框架和官方的 Boto 库（同时支持 boto 2 / boto 3），并通过结合 HTML 扩展可将测试结果输出到Web界面，通过页面查询相关测试统计、分析结果。

s3-tests

S3 compatibility tests - This is a set of unofficial Amazon AWS S3 compatibility tests, that can be useful to people implementing software that exposes an S3-like API. The tests use the Boto2 and Boto3 libraries.

仓库地址：https://github.com/ceph/s3-tests

Nose 框架

Nose 是一种流行的 Python 单元测试框架扩展，它可以方便地自动运行一批测试和插件，比如度量代码覆盖率。Nose 能支持基于函数的测试，也能支持与 unittest 相类似的基于测试类的测试方式。

nose-html-reporting 扩展

Nose plugin that generates a nice html test report with ability of using template based on jinja2 templates from any folder.

源自: https://github.com/ionelmc/nose-htmloutput ，增加了对 jinja2 模板支持。

配置 s3-tests 实践

下载源码

# git clone https://github.com/ceph/s3-tests.git

安装、部署

# sudo apt-get install python-virtualenv
# cd s3-tests

启动测试运行环境，安装依赖软件：

# ./bootstrap

创建配置文件

首先，需要通过 S3 管理接口提前创建好配置文件中的用户、access_key，secret_key 等必要配置信息。

创建 config.yaml 增加如下内容：

[DEFAULT]
## this p is just used as default for all the "s3 *"
## ps, you can place these variables also directly there
## replace with e.g. "localhost" to run against local software
host = test-shyc2.s3.360.local  # 也可用本机 S3 Gateway IP 地址
## uncomment the port to use something other than 80
port = 80
## say "no" to disable TLS
is_secure = no
[fixtures]
## all the buckets created will start with this prefix;
## {random} will be filled with random characters to pad
## the prefix to 30 characters long, and avoid collisions
bucket prefix = s3test-{random}-
[s3 main]
## the tests assume two accounts are defined, "main" and "alt".
## user_id is a 64-character hexstring
user_id = infra-s3
## display name typically looks more like a unix login, "jdoe" etc
display_name = infra-s3
## replace these with your access keys
access_key = infra-ak
secret_key = infra-sk
[s3 alt]
## another user account, used for ACL-related tests
user_id = infra-alt
display_name = infra-alt
## the "alt" user needs to have email set, too
email = infra-alt@qihoo.net
access_key = infra-alt-ak
secret_key = infra-alt-sk

查看测试项

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests -v --collect-only
s3tests.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_bad ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_none ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_mismatch ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_none ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_negative ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_above ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_none ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_authorization_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok

Nose Html 扩展

配置静态资源服务器

搭建一个 HTTP 资源服务器用于展示 nose 输出。

在线安装 nginx / httpd 即可，下文中直接将输出文件保存到默认路径 /var/www/html/ 下。

安装 nose-html-reporting

官方地址 : https://pypi.org/project/nose-html-reporting/#description。

安装 nose-html-reporting。

# ./virtualenv/bin/pip install nose-html-reporting

验证，有如下输出，代表扩展已安装，支持输出 html 文件。

~/s3-tests# ./virtualenv/bin/nosetests -h | grep html
  --with-html           Enable plugin HtmlReport:  Output test results as
                        pretty html.  [NOSE_WITH_HTML]
  --html-report=FILE    Path to html file to store the report in. Default is
                        nosetests.html in the working directory
  --html-report-template=FILE
                        Path to html template file in with jinja2
                        format.Default is report.html in the lib
  --cover-html          Produce HTML coverage information
  --cover-html-dir=DIR  Produce HTML coverage information in dir
~/s3-tests#

执行单个用例

单独运行 test_bucket_list_empty 用例，指定 html 模板文件为 report2.jinja2。

执行如下命令：

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests s3tests.functional.test_s3:test_bucket_list_empty -v --with-html --html-report=/var/www/html/index.html --html-report-template=virtualenv/lib/python2.7/site-packages/nose_html_reporting/templates/report2.jinja2。

示例：

~/s3-tests# S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests s3tests.functional.test_s3:test_bucket_list_empty  -v --with-html --html-report=/var/www/html/index.html --html-report-template=virtualenv/lib/python2.7/site-packages/nose_html_reporting/templates/report2.jinja2
s3tests.functional.test_s3.test_bucket_list_empty ... ok
----------------------------------------------------------------------
HTML: /var/www/html/index.html
----------------------------------------------------------------------
Ran 1 test in 0.322s
OK
~/s3-tests#

打开 web 端，查看测试结果：

部分测试

运行 atomic read/write 单元测试用例:

#!/bin/bash
cases="\
  s3tests.functional.test_s3:test_atomic_read_1mb \
  s3tests.functional.test_s3:test_atomic_read_4mb \
  s3tests.functional.test_s3:test_atomic_read_8mb \
  s3tests.functional.test_s3:test_atomic_write_1mb \
  s3tests.functional.test_s3:test_atomic_write_4mb \
  s3tests.functional.test_s3:test_atomic_write_8mb \
  s3tests.functional.test_s3:test_atomic_dual_write_1mb \
  s3tests.functional.test_s3:test_atomic_dual_write_4mb \
  s3tests.functional.test_s3:test_atomic_dual_write_8mb \
  s3tests.functional.test_s3:test_atomic_conditional_write_1mb"
S3_USE_SIGV4=1 S3TEST_CONF=config.yaml virtualenv/bin/nosetests $cases -a \
auth_aws4 -x --debug-log=./tests.log --cover-html --process-timeout=60 \
--processes=10

如上通过指定特定的 case 进行测试，配置 ENV 中 S3_USE_SIGV4=1 和参数 -a auth_aws4 使能 AWS V4 鉴权（默认为 V2）, --debug-log 用于记录测试过程中详细日志。

执行全量测试

执行如下命令：

S3_USE_SIGV4=1 S3TEST_CONF=./config.yaml ./virtualenv/bin/nosetests -a auth_aws4 -v --with-html --html-report=/var/www/html/index.html

示例输出：

s3tests.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_bad ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_md5_none ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_mismatch ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_none ... ok
s3tests.functional.test_headers.test_object_create_bad_expect_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_negative ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_unreadable ... ok
s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_above ... FAIL
s3tests.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_empty ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_none ... ok
s3tests.functional.test_headers.test_object_create_bad_contenttype_unreadable ... FAIL
s3tests.functional.test_headers.test_object_create_bad_authorization_unreadable ... FAIL
s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok
s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok
s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok

最后见到如下统计输出，并且可以看到 HTML 输出到 /var/www/html/index.html 中了，代表任务成功运行，现在可以通过界面查看 HTML 端统计结果。

----------------------------------------------------------------------
HTML: /var/www/html/index.html
----------------------------------------------------------------------
Ran 570 tests in 542.311s
FAILED (SKIP=85, errors=26, failures=22)

统计分析

概览统计

查询测试结果，点击 Detail 展开单元测试项运行状态。

详细信息

通过点击失败用例对应的 Fail 按钮，查询具体错误信息。

通常我们根据断言信息及请求、响应 header 就可以定位到具体问题了。

自定义测试项

对于我们自定义的类S3接口（如自定义Meta信息、私有管理接口等），也可以很方便的通过扩展现有单元测试函数来实现，下面是 HTTP Range 请求下载功能的测试函数，函数位置：s3-tests/s3tests/functional/test_s3.py 中 Http Range 函数。

示例：

@attr(resource='object')
@attr(method='get')
@attr(operation='range')
@attr(assertion='returns correct data, 206')
def test_ranged_big_request_response_code():
    bucket = get_new_bucket()
    key = bucket.new_key('testobj')
    string = os.urandom(8 * 1024 * 1024)
    # 上传对象
    key.set_contents_from_string(string)
    # 设置 header，指定 Range 数据段
    key.open('r', headers={'Range': 'bytes=3145728-5242880'})
    status = key.resp.status
    content_range = key.resp.getheader('Content-Range')
    fetched_content = ''
    for data in key:
        fetched_content += data;
    key.close()
    # 验证 Gateway 响应状态码，数据长度等
    eq(fetched_content, string[3145728:5242881])
    eq(status, 206)
    eq(content_range, 'bytes 3145728-5242880/8388608')

可以此为例，进行自定义用例扩展。

往期精彩回顾