自动化测试数据爬取

原创 于 2024-05-07 11:24:08 发布
· 561 阅读 · 2 ·
版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#python #开发语言

使⽤爬⾍技术为⾃动化测试⽣成测试数据是⼀种⾼效的⽅法，特别是在需要⼤量真实数据进⾏测试时。通过抓取现实世界的数据，可以为应⽤提供更加全⾯的测试覆盖，从⽽提⾼软件的质量和稳定性。

以下是如何使⽤Python和爬⾍技术为⾃动化测试⽣成测试数据的基本步骤：

        1. 确定数据需求 ：在开始之前，明确⾃动化测试需要哪些类型的数据。这可能包括⽤⼾信息、商品数据、⽂章内容等。 

         2. 选择数据源：根据数据需求，选择合适的⽹站或API作为数据源。确保有权抓取这些数据，并且遵守相关法律法规和⽹站的使⽤条款。

        3. 编写爬⾍脚本： 使⽤Python的 requests 库来获取数据， BeautifulSoup 或 lxml 来解析HTML数据，或者直接使⽤ pandas 的 read_html 函数快速转换表格数据。如果是API数据，直接解析JSON格式的响应即可。

⽰例代码 ：假设需要从⼀个公开的API获取⽤⼾信息作为测试数据：

import requests
import json

def fetch_test_data(api_url, num_users):
    response = requests.get(f"{api_url}?results={num_users}")
    if response.status_code == 200:
        data = response.json()
        # 假设API响应中的数据在'data'键下
        return data['results']
    else:
        print(f"请求失败，状态码: {response.status_code}")
        return []

# ⽰例：从随机⽤⼾信息API获取测试数据
test_data = fetch_test_data("https://randomuser.me/api/", 10)
for user in test_data:
    print(user)

        4.处理和保存测试数据 ：处理抓取到的数据，转换成适合⾃动化测试使⽤的格式。例如，可能需要将数据保存到JSON⽂件、CSV⽂件或直接导⼊到数据库中 ；将测试数据保存为JSON⽂件

with open('test_data.json', 'w', encoding='utf-8') as f:
    json.dump(test_data, f, ensure_ascii=False, indent=4)

        5.在⾃动化测试中使⽤数据 ：最后，根据测试框架的不同，将保存好的测试数据引⼊到⾃动化测试脚本中。例如，可以在测试准备阶段读取JSON⽂件，然后在测试⽤例中使⽤这些数据。 #

注意事项：

 • 在抓取数据时，考虑到⽬标⽹站或API的请求限制和负载，避免频繁抓取导致IP被封禁。

 • 确保抓取的数据只⽤于测试⽬的，遵守数据保护法规，尤其是处理个⼈信息时。

 • 定期更新测试数据，确保测试覆盖的场景与现实世界保持⼀致。

test_data.json

[
    {
        "gender": "male",
        "name": {
            "title": "Mr",
            "first": "Ivan",
            "last": "Borshchevskiy"
        },
        "location": {
            "street": {
                "number": 7836,
                "name": "Polyarna"
            },
            "city": "Pershotravensk",
            "state": "Hersonska",
            "country": "Ukraine",
            "postcode": 98943,
            "coordinates": {
                "latitude": "76.4199",
                "longitude": "-21.5039"
            },
            "timezone": {
                "offset": "+8:00",
                "description": "Beijing, Perth, Singapore, Hong Kong"
            }
        },
        "email": "ivan.borshchevskiy@example.com",
        "login": {
            "uuid": "61dcbadf-4114-4666-94ba-1f9e49b8500c",
            "username": "sadmouse176",
            "password": "doggy",
            "salt": "shfryAz1",
            "md5": "0ef3fe9cfc8ac4b94e46aba19fdb32bb",
            "sha1": "5879d7ed4a87bdc1f5a203936fcc674dfc366325",
            "sha256": "63beecc0bae55d6bbc6a98f8a8e17a958e4d4de41fa415a89fbe5a9881f229af"
        },
        "dob": {
            "date": "1999-07-21T01:45:19.167Z",
            "age": 24
        },
        "registered": {
            "date": "2014-09-01T11:20:50.950Z",
            "age": 9
        },
        "phone": "(068) P65-3007",
        "cell": "(098) J70-3404",
        "id": {
            "name": "",
            "value": null
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/men/90.jpg",
            "medium": "https://randomuser.me/api/portraits/med/men/90.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/men/90.jpg"
        },
        "nat": "UA"
    },
    {
        "gender": "female",
        "name": {
            "title": "Ms",
            "first": "Laura",
            "last": "Cox"
        },
        "location": {
            "street": {
                "number": 9914,
                "name": "The Green"
            },
            "city": "Arklow",
            "state": "Wicklow",
            "country": "Ireland",
            "postcode": 97984,
            "coordinates": {
                "latitude": "42.2092",
                "longitude": "-53.0637"
            },
            "timezone": {
                "offset": "+6:00",
                "description": "Almaty, Dhaka, Colombo"
            }
        },
        "email": "laura.cox@example.com",
        "login": {
            "uuid": "c057178d-92bc-4fe0-bf85-b37b81eeff1b",
            "username": "tinywolf954",
            "password": "taylor",
            "salt": "GxSmEUFh",
            "md5": "e92aec4949eb2f691ba88c91a8ad7b1a",
            "sha1": "dcec671ae2ee6eff7c34160423dd8eab64942099",
            "sha256": "782ecd4ff5adf86982bd90242c46fc89fd396c2301dadeeb14200985b2f2e9e2"
        },
        "dob": {
            "date": "1950-02-12T21:38:57.524Z",
            "age": 74
        },
        "registered": {
            "date": "2016-04-05T19:50:31.274Z",
            "age": 8
        },
        "phone": "071-619-9888",
        "cell": "081-999-4894",
        "id": {
            "name": "PPS",
            "value": "7665135T"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/women/26.jpg",
            "medium": "https://randomuser.me/api/portraits/med/women/26.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/women/26.jpg"
        },
        "nat": "IE"
    },
    {
        "gender": "female",
        "name": {
            "title": "Miss",
            "first": "Sophie",
            "last": "Pelletier"
        },
        "location": {
            "street": {
                "number": 2333,
                "name": "St. Catherine St"
            },
            "city": "New Glasgow",
            "state": "Manitoba",
            "country": "Canada",
            "postcode": "V8J 5V0",
            "coordinates": {
                "latitude": "-40.4064",
                "longitude": "164.4734"
            },
            "timezone": {
                "offset": "+4:00",
                "description": "Abu Dhabi, Muscat, Baku, Tbilisi"
            }
        },
        "email": "sophie.pelletier@example.com",
        "login": {
            "uuid": "93eecdef-ff7d-4f6b-9982-581cfc493d93",
            "username": "blackwolf348",
            "password": "stayout",
            "salt": "wHagoNWy",
            "md5": "64c333255b24028a420d33d8648b198d",
            "sha1": "341fab0db79a7962d306960addd4e407a6b24f5a",
            "sha256": "a049680f307c0047d0ecac4cc4bcf3bcc1322d17c59c3f7be84e5e7f5973b9a1"
        },
        "dob": {
            "date": "1993-07-10T03:13:54.254Z",
            "age": 30
        },
        "registered": {
            "date": "2014-10-13T16:24:37.233Z",
            "age": 9
        },
        "phone": "M85 L54-6394",
        "cell": "E53 L36-0551",
        "id": {
            "name": "SIN",
            "value": "544663529"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/women/37.jpg",
            "medium": "https://randomuser.me/api/portraits/med/women/37.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/women/37.jpg"
        },
        "nat": "CA"
    },
    {
        "gender": "female",
        "name": {
            "title": "Miss",
            "first": "Lucy",
            "last": "Gomez"
        },
        "location": {
            "street": {
                "number": 8420,
                "name": "Broadway"
            },
            "city": "Truro",
            "state": "West Yorkshire",
            "country": "United Kingdom",
            "postcode": "K7 5WN",
            "coordinates": {
                "latitude": "47.5564",
                "longitude": "-49.9847"
            },
            "timezone": {
                "offset": "-1:00",
                "description": "Azores, Cape Verde Islands"
            }
        },
        "email": "lucy.gomez@example.com",
        "login": {
            "uuid": "8013ac6d-2f4d-4967-9868-7a5e629575a1",
            "username": "brownmouse497",
            "password": "look",
            "salt": "KqRs0ugW",
            "md5": "e35c431b9efd48b25e12dd99c4d6abec",
            "sha1": "ec2acd71446fd282bb896e14e8c288a110d81794",
            "sha256": "00a6f7cb7ececa3245cb8d770542f759bff350dca740a164ba03c17c982336c4"
        },
        "dob": {
            "date": "1946-06-19T15:28:43.025Z",
            "age": 77
        },
        "registered": {
            "date": "2014-08-12T23:11:50.201Z",
            "age": 9
        },
        "phone": "0121 472 9046",
        "cell": "07703 926530",
        "id": {
            "name": "NINO",
            "value": "MM 46 81 01 E"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/women/54.jpg",
            "medium": "https://randomuser.me/api/portraits/med/women/54.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/women/54.jpg"
        },
        "nat": "GB"
    },
    {
        "gender": "male",
        "name": {
            "title": "Mr",
            "first": "Malthe",
            "last": "Poulsen"
        },
        "location": {
            "street": {
                "number": 8629,
                "name": "Skolebakken"
            },
            "city": "Argerskov",
            "state": "Sjælland",
            "country": "Denmark",
            "postcode": 25311,
            "coordinates": {
                "latitude": "13.8045",
                "longitude": "2.9061"
            },
            "timezone": {
                "offset": "+9:00",
                "description": "Tokyo, Seoul, Osaka, Sapporo, Yakutsk"
            }
        },
        "email": "malthe.poulsen@example.com",
        "login": {
            "uuid": "294e04f4-3e96-47aa-983f-cd59d07fa506",
            "username": "silverfish938",
            "password": "77777",
            "salt": "5Nj1Qsdu",
            "md5": "d5978bee2b2d7c6797404ae2999abcb5",
            "sha1": "0218020fa26443890373eaabed575256e224bdd0",
            "sha256": "abfb3e759baef9a1273f45269c124e43d59d08713eedb26e5a1355c553b118d0"
        },
        "dob": {
            "date": "1997-07-14T11:51:10.291Z",
            "age": 26
        },
        "registered": {
            "date": "2010-06-30T00:10:32.106Z",
            "age": 13
        },
        "phone": "26175795",
        "cell": "28005456",
        "id": {
            "name": "CPR",
            "value": "140797-7776"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/men/75.jpg",
            "medium": "https://randomuser.me/api/portraits/med/men/75.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/men/75.jpg"
        },
        "nat": "DK"
    },
    {
        "gender": "female",
        "name": {
            "title": "Madame",
            "first": "Maria",
            "last": "Chevalier"
        },
        "location": {
            "street": {
                "number": 8178,
                "name": "Rue des Cuirassiers"
            },
            "city": "Flaach",
            "state": "Zürich",
            "country": "Switzerland",
            "postcode": 8756,
            "coordinates": {
                "latitude": "61.5685",
                "longitude": "-163.9956"
            },
            "timezone": {
                "offset": "+5:00",
                "description": "Ekaterinburg, Islamabad, Karachi, Tashkent"
            }
        },
        "email": "maria.chevalier@example.com",
        "login": {
            "uuid": "eff94b4f-51c8-43a7-8984-8ff8a56df01c",
            "username": "beautifulladybug123",
            "password": "captain",
            "salt": "Hpc6xMeY",
            "md5": "47ffc1bb1aadd500124a3d8559637298",
            "sha1": "e8a9c2fc1e76fc64aea7d55ba8accafa98eac3d5",
            "sha256": "f4b24255b74176dd0eebdc28cbe1ab30bc1b876c1fa9a6e1dd8fa0ebc54446ed"
        },
        "dob": {
            "date": "1982-05-26T01:45:46.200Z",
            "age": 41
        },
        "registered": {
            "date": "2009-07-14T01:06:58.415Z",
            "age": 14
        },
        "phone": "076 101 52 47",
        "cell": "078 531 49 04",
        "id": {
            "name": "AVS",
            "value": "756.6515.0863.29"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/women/85.jpg",
            "medium": "https://randomuser.me/api/portraits/med/women/85.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/women/85.jpg"
        },
        "nat": "CH"
    },
    {
        "gender": "male",
        "name": {
            "title": "Mr",
            "first": "Rad",
            "last": "Bezotosniy"
        },
        "location": {
            "street": {
                "number": 5530,
                "name": "Vigurivska"
            },
            "city": "Rubizhne",
            "state": "Rivnenska",
            "country": "Ukraine",
            "postcode": 12968,
            "coordinates": {
                "latitude": "-24.8128",
                "longitude": "32.4404"
            },
            "timezone": {
                "offset": "+6:00",
                "description": "Almaty, Dhaka, Colombo"
            }
        },
        "email": "rad.bezotosniy@example.com",
        "login": {
            "uuid": "6cbb9c6f-c4d7-4e37-84f1-79a556a8f9a2",
            "username": "sadduck269",
            "password": "rogue",
            "salt": "5yoWVp6E",
            "md5": "328faca57dc9af734be60d55ae728f06",
            "sha1": "c6ff761775bdf5dd79791acc7b8de06529c3e1c3",
            "sha256": "254ec5d061d2408be8414a78a8d1922b4796e8cb050c649c46060f0f80e5d90a"
        },
        "dob": {
            "date": "1962-07-13T08:57:14.676Z",
            "age": 61
        },
        "registered": {
            "date": "2004-06-13T09:06:41.414Z",
            "age": 19
        },
        "phone": "(098) I99-3546",
        "cell": "(066) P34-1354",
        "id": {
            "name": "",
            "value": null
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/men/98.jpg",
            "medium": "https://randomuser.me/api/portraits/med/men/98.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/men/98.jpg"
        },
        "nat": "UA"
    },
    {
        "gender": "female",
        "name": {
            "title": "Madame",
            "first": "Mary",
            "last": "Schmitt"
        },
        "location": {
            "street": {
                "number": 5496,
                "name": "Rue Paul Bert"
            },
            "city": "Épendes (Vd)",
            "state": "Fribourg",
            "country": "Switzerland",
            "postcode": 8242,
            "coordinates": {
                "latitude": "0.1841",
                "longitude": "-68.5012"
            },
            "timezone": {
                "offset": "+5:45",
                "description": "Kathmandu"
            }
        },
        "email": "mary.schmitt@example.com",
        "login": {
            "uuid": "9dacc566-7ad4-41af-9617-798bd96b5766",
            "username": "reddog286",
            "password": "dogman",
            "salt": "RC4N1yFe",
            "md5": "3d9a4ca7a8403e97535cc778ad94615c",
            "sha1": "5311d2adbd67b8416e3758cad95ea3ac33ae2dd6",
            "sha256": "c05645fe04eacbec510b137ec58c80380787f89061962a252be3c5ec05bf9ed7"
        },
        "dob": {
            "date": "1966-09-24T05:28:52.206Z",
            "age": 57
        },
        "registered": {
            "date": "2019-03-23T06:04:49.818Z",
            "age": 5
        },
        "phone": "079 349 99 81",
        "cell": "075 351 13 18",
        "id": {
            "name": "AVS",
            "value": "756.6237.3356.93"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/women/57.jpg",
            "medium": "https://randomuser.me/api/portraits/med/women/57.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/women/57.jpg"
        },
        "nat": "CH"
    },
    {
        "gender": "male",
        "name": {
            "title": "Mr",
            "first": "Magnus",
            "last": "Olsen"
        },
        "location": {
            "street": {
                "number": 3152,
                "name": "Strandgade"
            },
            "city": "Sønder Stenderup",
            "state": "Syddanmark",
            "country": "Denmark",
            "postcode": 18823,
            "coordinates": {
                "latitude": "-36.9019",
                "longitude": "-91.9956"
            },
            "timezone": {
                "offset": "+3:30",
                "description": "Tehran"
            }
        },
        "email": "magnus.olsen@example.com",
        "login": {
            "uuid": "d6f1fe05-e0ec-4277-b097-ce192ebfb3c1",
            "username": "smallkoala298",
            "password": "midget",
            "salt": "rVTjSVoH",
            "md5": "6232c4012364f5537d015c879259ea57",
            "sha1": "b3685844d0976ac7552dd6b45f6f284a57a7d53c",
            "sha256": "1f71b27f2ea5d109e112216993cd197c05e6ecd514482392f7e132bda433ba24"
        },
        "dob": {
            "date": "1986-01-09T19:29:02.621Z",
            "age": 38
        },
        "registered": {
            "date": "2018-05-04T07:16:11.357Z",
            "age": 5
        },
        "phone": "64845324",
        "cell": "92622368",
        "id": {
            "name": "CPR",
            "value": "090186-1758"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/men/17.jpg",
            "medium": "https://randomuser.me/api/portraits/med/men/17.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/men/17.jpg"
        },
        "nat": "DK"
    },
    {
        "gender": "male",
        "name": {
            "title": "Monsieur",
            "first": "Tony",
            "last": "Andre"
        },
        "location": {
            "street": {
                "number": 8113,
                "name": "Rue de L'Abbé-Roger-Derry"
            },
            "city": "Zillis-Reischen",
            "state": "Appenzell Innerrhoden",
            "country": "Switzerland",
            "postcode": 1002,
            "coordinates": {
                "latitude": "-33.5145",
                "longitude": "148.1249"
            },
            "timezone": {
                "offset": "-11:00",
                "description": "Midway Island, Samoa"
            }
        },
        "email": "tony.andre@example.com",
        "login": {
            "uuid": "3a8606b8-d725-4197-b8f0-25d1dd5f8962",
            "username": "tinybird665",
            "password": "novifarm",
            "salt": "43vbctVa",
            "md5": "389b9356f6be29e96a749f0106bce512",
            "sha1": "051a03505c3acb4bdc132c4d6b24cbba8878e52c",
            "sha256": "2017205b21600b5d12ec393b03c52f78c2733ba3de687376adf0c17226c92e6a"
        },
        "dob": {
            "date": "1989-09-25T04:43:34.825Z",
            "age": 34
        },
        "registered": {
            "date": "2018-04-26T07:23:02.477Z",
            "age": 6
        },
        "phone": "079 707 49 79",
        "cell": "076 869 00 79",
        "id": {
            "name": "AVS",
            "value": "756.5022.8900.90"
        },
        "picture": {
            "large": "https://randomuser.me/api/portraits/men/71.jpg",
            "medium": "https://randomuser.me/api/portraits/med/men/71.jpg",
            "thumbnail": "https://randomuser.me/api/portraits/thumb/men/71.jpg"
        },
        "nat": "CH"
    }
]

评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值