Python 批量获取Google+用户动态（分页）

最新推荐文章于 2024-12-17 15:36:45 发布

原创

最新推荐文章于 2024-12-17 15:36:45 发布 · 3.9w 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#google+ #数据挖掘

本文介绍如何使用Python编程实现批量获取Google+用户的动态信息，详细讲解了分页处理的技巧，为数据挖掘提供有效数据源。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

CODE:

#!/usr/bin/python 
# -*- coding: utf-8 -*-

'''
Created on 2014-9-7
@author: guaguastd
@name: user_activity_loop.py
'''

import json
from login import google_api_request
from html import cleanHtml
import os

MAX_RESULTS = 40
while True:
    query = raw_input("Input query(None to quit): ")
    if query.strip() == '':
        break
    
    people_feed = google_api_request(0, action='search', query=query)
    
    for user in people_feed['items']:
        userId = user['id']
        activity_feed = google_api_request(1, action='list', collection='public', maxResults='20', userId=userId)
        #print json.dumps(activity_feed, indent=1)
        activity_results = []
        while activity_feed != None and len(activity_results) < MAX_RESULTS:
            if activity_feed['items'] is None:
                break
            if 'nextPageToken' in activity_feed:
                nextPageToken = activity_feed['nextPageToken']
                endFlag = 0
            else:
                endFlag = 1
                
            activities = activity_feed

            if 'items' in activities:
                for activity in activities['items']:
                    print 'activity: ', activity
                    if activity['object']['objectType'] == 'note' and activity['object']['content'] != '':
                        activity['title'] = cleanHtml(activity['title'])
                        activity['object']['content'] = cleanHtml(activity['object']['content'])
                        activity_results += [activity]

            # use nextPageToken to get the response objects
            if endFlag == 0:
                activity_feed = google_api_request(1, action='list', collection='public', maxResults='20', userId=userId, pageToken=nextPageToken)
            else:
                break
            
        # Write the output to a fie for convenience
        f = open(os.path.join(r"E:", "\\", "eclipse", "Google", "dfile", userId + '.json'), 'w')
        f.write(json.dumps(activity_results, indent=1))
        f.close()

        print str(len(activity_results)), "activities written to", f.name

RESULT:

activity:  {u'kind': u'plus#activity', u'provider': {u'title': u'Google+'}, u'title': u"This is a really great review of my friend +William Janeway's book, Doing Capitalism in the Innovation...", u'url': u'https://plus.google.com/107033731246200681024/posts/Y9xzjP27w4H', u'object': {u'resharers': {u'totalItems': 11, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z12fhlxhbxrvfdchu04chl3pvxfbszirt5o/people/resharers'}, u'attachments': [{u'displayName': u'The One Book on Wall Street You Haven\u2019t Read\u2014but Should', u'fullImage': {u'url': u'http://www.slate.com/content/dam/slate/articles/arts/books/2013/02/SBR/ILLOS/1302_SBR_DOINGCAPITALISM_IL.jpg/_jcr_content/renditions/cq5dam.web.1280.1280.jpeg', u'type': u'image/jpeg'}, u'url': u'http://www.slate.com/articles/business/books/2013/02/william_janeway_s_doing_capitalism_in_the_innovation_economy_reviewed.html', u'image': {u'url': u'https://lh6.googleusercontent.com/proxy/aq9sbrtgUd_VDXEMdLmCOf4Mf3aCXly3WXZ4_0jz2mgWgc9DLSHN3ptTUNiQ1mDCmBMsSby_b87hMQstwvelwB6ryJJ5G-FnKW9swDu4o4IrPih2tsTqNT13isFWq7M6V2osI2nb-djCTJtnFDVgy9L1oj5X3NPFiNcqxmM8ZUssNch4pQNRO4SeCp2MGDXlEaXMNTXV-XNbe5o7WxQNv6PYRwFx91hTYE44qPOK6A=w506-h303-p', u'width': 506, u'type': u'image/jpeg', u'height': 303}, u'content': u'Why Wall Street? What good does Beezlebubbian finance really provide? Wouldn\u2019t America be better off if we could magically transport ourselves back to the \u201950s\u2014when Wall Street was sleepy, the middle class was robust, and children dreamed of rocket ships? What is the essence of this activity known as finance?...', u'objectType': u'article'}], u'url': u'https://plus.google.com/107033731246200681024/posts/Y9xzjP27w4H', u'content': u'This is a really great review of my friend <span class="proflinkWrapper"><span class="proflinkPrefix">+</span><a class="proflink" href="https://plus.google.com/114912453748528069739" oid="114912453748528069739">William Janeway</a></span>'s book, <i>Doing Capitalism in the Innovation Economy</i>. \xa0It succinctly lays out the key premises of the book, and explains why they are so important to a proper understanding of the role of venture capital, government, and bubbles in the growth of our economy.\ufeff', u'plusoners': {u'totalItems': 39, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z12fhlxhbxrvfdchu04chl3pvxfbszirt5o/people/plusoners'}, u'replies': {u'totalItems': 4, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z12fhlxhbxrvfdchu04chl3pvxfbszirt5o/comments'}, u'objectType': u'note'}, u'updated': u'2014-04-29T19:33:10.240Z', u'actor': {u'url': u'https://plus.google.com/107033731246200681024', u'image': {u'url': u'https://lh4.googleusercontent.com/-J8nmMwIhpiA/AAAAAAAAAAI/AAAAAAACoLI/0MpUu2BMqTM/photo.jpg?sz=50'}, u'displayName': u"Tim O'Reilly", u'id': u'107033731246200681024'}, u'access': {u'items': [{u'type': u'public'}], u'kind': u'plus#acl', u'description': u'Public'}, u'verb': u'post', u'etag': u'"L2Xbn8bDuSErT6QA3PEQiwYKQxM/DMb7RzmIBBbIbdrrj-R0SGXDbTw"', u'published': u'2014-04-29T19:33:10.240Z', u'id': u'z12fhlxhbxrvfdchu04chl3pvxfbszirt5o'}
activity:  {u'kind': u'plus#activity', u'provider': {u'title': u'Google+'}, u'title': u'How to Raise Moral Children\n\nI thought this article on child-raising had a lot of good ideas in it. ...', u'url': u'https://plus.google.com/107033731246200681024/posts/NVZVmG1ct6C', u'object': {u'resharers': {u'totalItems': 218, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z12svla42ymsupjw004chl3pvxfbszirt5o/people/resharers'}, u'attachments': [{u'displayName': u'Raising a Moral Child', u'fullImage': {u'url': u'http://static01.nyt.com/images/2014/04/13/sunday-review/0413SRWcover/0413SRWcover-videoSixteenByNine1050.jpg', u'type': u'image/jpeg'}, u'url': u'http://www.nytimes.com/2014/04/12/opinion/sunday/raising-a-moral-child.html', u'image': {u'url': u'https://lh5.googleusercontent.com/proxy/JqSWHUxT4ScL-_AWmYbKI3OJWgAR1S0Jbi2uckUdPHJ6SVz51uFhIMw4gm4V_TF4C5OgfG4Yi8IUBQQq-wibKiG0VJlQxdCC-SjjQ1mj8gShKM40t4QQNg7DL06hYhLEqx-8bVNO1fhDNfHuIvaPir98bG1CHlYudqcMsA=w506-h303', u'width': 506, u'type': u'image/jpeg', u'height': 303}, u'content': u'The tactics are different from those used for encouraging achievement.', u'objectType': u'article'}], u'url': u'https://plus.google.com/107033731246200681024/posts/NVZVmG1ct6C', u'content': u'<b>How to Raise Moral Children</b><br /><br />I thought this article on child-raising had a lot of good ideas in it. \xa0There was stuff that was new to me, and stuff that was old, but the old stuff was worth reviewing, and the new research quite enlightening.<br /><br />Do read the article even if you don't have children.\xa0The principles it outlines apply to our lives as adults as well, to leadership in the workplace and in the world. \xa0And while I've reproduced many of the best bits below, it's well worth going to the article itself, because it is heavily linked to the original research.<br /><br /><b>What matters most is not achievement but caring</b><br /><br />The article opened with the familiar advice that\xa0"when parents praise effort rather than ability, children develop a stronger work ethic and become more motivated." \xa0But it went from there in a surprising direction:<br /><br />"When people in 50 countries were asked to report their guiding principles in life, the value that mattered most was not achievement, but caring."<br /><br /><b>Contrary to popular advice, praise of character is more effective than praise of behavior</b><br /><br />"Many parents believe it\u2019s important to compliment the behavior, not the child \u2014 that way, the child learns to repeat the behavior. Indeed, I know one couple who are careful to say, 'That was such a helpful thing to do,' instead of, 'You\u2019re a helpful person.'"<br /><br />But that's not what the research shows:<br /><br />"...A couple of weeks later, when faced with more opportunities to give and share, the children were much more generous after their character had been praised than after their actions had been. Praising their character helped them internalize it as part of their identities. The children learned who they were from observing their own actions: I am a helpful person. This dovetails with new research led by the psychologist Christopher J. Bryan, who finds that for moral behaviors, nouns work better than verbs. To get 3- to 6-year-olds to help with a task, rather than inviting them 'to help,' it was 22 to 29 percent more effective to encourage them to 'be a helper.' Cheating was cut in half when instead of, 'Please don\u2019t cheat,' participants were told, 'Please don\u2019t be a cheater.' When our actions become a reflection of our character, we lean more heavily toward the moral and generous choices. Over time it can become part of us."<br /><br />I think this is an important insight. \xa0Internalizing our experiences and making them part of our identity is a key element of growing up.\xa0<br /><br />Note: \xa0"by the time children turned 10, the differences between praising character and praising actions vanished: Both were effective. Tying generosity to character appears to matter most around age 8, when children may be starting to crystallize notions of identity."<br /><br />I will note that the advice for dealing with bad behavior (as opposed to good behavior) is just the opposite: \xa0to emphasize the behavior, and not the character. \xa0You don't want a kid to internalize the idea that he or she is a bad person! \xa0That was the next key point, the distinction between guilt and shame.<br /><br /><b>"If we want our children to care about others, we need to teach them to feel guilt rather than shame when they misbehave."</b><br /><br />This is a fascinating distinction: \xa0<br /><br />"When children cause harm, they typically feel one of two moral emotions: shame or guilt. Despite the common belief that these emotions are interchangeable, research led by the psychologist June Price Tangney reveals that they have very different causes and consequences.<br /><br />"Shame is the feeling that I am a bad person, whereas guilt is the feeling that I have done a bad thing. Shame is a negative judgment about the core self, which is devastating: Shame makes children feel small and worthless, and they respond either by lashing out at the target or escaping the situation altogether. In contrast, guilt is a negative judgment about an action, which can be repaired by good behavior. When children feel guilt, they tend to experience remorse and regret, empathize with the person they have harmed, and aim to make it right."<br /><br />...<br /><br />"The most effective response to bad behavior is to express disappointment. According to independent reviews by Professor Eisenberg and David R. Shaffer, parents raise caring children by expressing disappointment and explaining why the behavior was wrong, how it affected others, and how they can rectify the situation. This enables children to develop standards for judging their actions, feelings of empathy and responsibility for others, and a sense of moral identity, which are conducive to becoming a helpful person. The beauty of expressing disappointment is that it communicates disapproval of the bad behavior, coupled with high expectations and the potential for improvement: 'You\u2019re a good person, even if you did a bad thing, and I know you can do better.'"<br /><br /><b>Children learn generosity not by listening to what their role models say, but by observing what they do.</b><br /><br />"In a classic experiment, the psychologist J. Philippe Rushton gave 140 elementary- and middle-school-age children tokens for winning a game, which they could keep entirely or donate some to a child in poverty. They first watched a teacher figure play the game either selfishly or generously, and then preach to them the value of taking, giving or neither. The adult\u2019s influence was significant: Actions spoke louder than words. When the adult behaved selfishly, children followed suit. The words didn\u2019t make much difference \u2014 children gave fewer tokens after observing the adult\u2019s selfish actions, regardless of whether the adult verbally advocated selfishness or generosity. When the adult acted generously, students gave the same amount whether generosity was preached or not \u2014 they donated 85 percent more than the norm in both cases. When the adult preached selfishness, even after the adult acted generously, the students still gave 49 percent more than the norm. Children learn generosity not by listening to what their role models say, but by observing what they do."<br /><br />So true. \xa0As I said above, all of this applies to adults too.\xa0\ufeff', u'plusoners': {u'totalItems': 260, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z12svla42ymsupjw004chl3pvxfbszirt5o/people/plusoners'}, u'replies': {u'totalItems': 30, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z12svla42ymsupjw004chl3pvxfbszirt5o/comments'}, u'objectType': u'note'}, u'updated': u'2014-04-29T14:10:10.952Z', u'actor': {u'url': u'https://plus.google.com/107033731246200681024', u'image': {u'url': u'https://lh4.googleusercontent.com/-J8nmMwIhpiA/AAAAAAAAAAI/AAAAAAACoLI/0MpUu2BMqTM/photo.jpg?sz=50'}, u'displayName': u"Tim O'Reilly", u'id': u'107033731246200681024'}, u'access': {u'items': [{u'type': u'public'}], u'kind': u'plus#acl', u'description': u'Public'}, u'verb': u'post', u'etag': u'"L2Xbn8bDuSErT6QA3PEQiwYKQxM/5XQ1t1Ya3hi5MBEp5jfQGJPCkT0"', u'published': u'2014-04-29T14:10:10.952Z', u'id': u'z12svla42ymsupjw004chl3pvxfbszirt5o'}
activity:  {u'kind': u'plus#activity', u'provider': {u'title': u'Google+'}, u'title': u'The stories here are truly astonishing! \xa0In the Basque country, they lift giant stones for sport:\n\n"...', u'url': u'https://plus.google.com/107033731246200681024/posts/ayohopoxvj6', u'object': {u'resharers': {u'totalItems': 8, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z13eyxlrbwqvt54yf04chl3pvxfbszirt5o/people/resharers'}, u'attachments': [{u'displayName': u'Stone Lifting as Sport in the Basque Country', u'fullImage': {u'url': u'http://static01.nyt.com/images/2014/04/20/travel/20PURSUITS8/20PURSUITS8-videoSixteenByNine1050.jpg', u'type': u'image/jpeg'}, u'url': u'http://www.nytimes.com/2014/04/20/travel/stone-lifting-as-sport-in-the-basque-country.html', u'image': {u'url': u'https://lh3.googleusercontent.com/proxy/BpBsjCJKzIQDrY1teXMIiCsoCy04r6rFS-KwYWz-G2uOiAoeC7Spw5bOBGkmdyVCELtoHHgWhC7LxXEmTSLOeDMjH-wvja65GYNFnuaw07YK0Rx-MfUN0CyL2GBrYO0Fiyb1Fvkm_linUbUBg7RTTDVMDA=w506-h303', u'width': 506, u'type': u'image/jpeg', u'height': 303}, u'content': u'Rural sports like wood chopping, hay bale tossing and stone lifting are part of a cultural fabric that\u2019s still visible today.', u'objectType': u'article'}], u'url': u'https://plus.google.com/107033731246200681024/posts/ayohopoxvj6', u'content': u'The stories here are truly astonishing! \xa0In the Basque country, they lift giant stones for sport:<br /><br />"Mr. Perurena is a butcher, a soap opera star on Basque television and, now, a tour guide. But he\u2019s famous in these parts for his stone-lifting feats, and after watching a short film of his superhuman lifting exploits in the museum, it\u2019s easy to see why. We watched footage of him hoisting stones that weigh more than 700 pounds; more footage where he lifted a 589-pound stone with one hand; and yet more in which he rolled a 465-pound stone around his neck 36 times in one minute. He told us that once, to celebrate the airing of the 1,700th episode of \u201cGoenkale,\u201d the soap opera he stars in, he lifted a 212-pound stone 1,700 times in nine hours."\ufeff', u'plusoners': {u'totalItems': 57, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z13eyxlrbwqvt54yf04chl3pvxfbszirt5o/people/plusoners'}, u'replies': {u'totalItems': 8, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z13eyxlrbwqvt54yf04chl3pvxfbszirt5o/comments'}, u'objectType': u'note'}, u'updated': u'2014-04-29T13:15:20.293Z', u'actor': {u'url': u'https://plus.google.com/107033731246200681024', u'image': {u'url': u'https://lh4.googleusercontent.com/-J8nmMwIhpiA/AAAAAAAAAAI/AAAAAAACoLI/0MpUu2BMqTM/photo.jpg?sz=50'}, u'displayName': u"Tim O'Reilly", u'id': u'107033731246200681024'}, u'access': {u'items': [{u'type': u'public'}], u'kind': u'plus#acl', u'description': u'Public'}, u'verb': u'post', u'etag': u'"L2Xbn8bDuSErT6QA3PEQiwYKQxM/vI3nrEZlvsgDaL59E7JKOSHBbrc"', u'published': u'2014-04-29T13:15:20.293Z', u'id': u'z13eyxlrbwqvt54yf04chl3pvxfbszirt5o'}
activity:  {u'kind': u'plus#activity', u'provider': {u'title': u'Google+'}, u'title': u"I love Robert Shiller's idea of indexing tax rates to income inequality http://www.nytimes.com/2014/...", u'url': u'https://plus.google.com/107033731246200681024/posts/1tr2mWxMhR8', u'object': {u'resharers': {u'totalItems': 18, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z133zl05olyjcjf0523gcxizetybvpydh/people/resharers'}, u'attachments': [{u'displayName': u'Better Insurance Against Inequality', u'fullImage': {u'url': u'http://static01.nyt.com/images/2014/04/13/business/13-VIEW/13-VIEW-videoSixteenByNine1050.jpg', u'type': u'image/jpeg'}, u'url': u'http://www.nytimes.com/2014/04/13/business/better-insurance-against-inequality.html', u'image': {u'url': u'https://lh3.googleusercontent.com/proxy/S2NaOxJpbIuj-zV7SwP7rMO59WN4PluOoJHA5ADtHouAIhbkCjj1sYGHoNXqmY_FAixaWI8E754P0Zl0NsiYNr_3VPFCZuH2yWQJN8D09M1cm_aTmVZG9KFsUdIFUdd_CLONWKQt6RxkTIMG1w=w506-h303', u'width': 506, u'type': u'image/jpeg', u'height': 303}, u'content': u'The tax system could be tweaked to respond whenever the income gap widens.', u'objectType': u'article'}], u'url': u'https://plus.google.com/107033731246200681024/posts/1tr2mWxMhR8', u'content': u'I love Robert Shiller's idea of indexing tax rates to income inequality <a href="http://www.nytimes.com/2014/04/13/business/better-insurance-against-inequality.html">http://www.nytimes.com/2014/04/13/business/better-insurance-against-inequality.html</a>\ufeff', u'plusoners': {u'totalItems': 85, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z133zl05olyjcjf0523gcxizetybvpydh/people/plusoners'}, u'replies': {u'totalItems': 43, u'selfLink': u'https://www.googleapis.com/plus/v1/activities/z133zl05olyjcjf0523gcxizetybvpydh/comments'}, u'objectType': u'note'}, u'updated': u'2014-04-29T04:19:06.409Z', u'actor': {u'url': u'https://plus.google.com/107033731246200681024', u'image': {u'url': u'https://lh4.googleusercontent.com/-J8nmMwIhpiA/AAAAAAAAAAI/AAAAAAACoLI/0MpUu2BMqTM/photo.jpg?sz=50'}, u'displayName': u"Tim O'Reilly", u'id': u'107033731246200681024'}, u'access': {u'items': [{u'type': u'public'}], u'kind': u'plus#acl', u'description': u'Public'}, u'verb': u'post', u'etag': u'"L2Xbn8bDuSErT6QA3PEQiwYKQxM/vhJwPFZSOfkyQV2JO2iTTpyhDtI"', u'published': u'2014-04-29T04:19:06.409Z', u'id': u'z133zl05olyjcjf0523gcxizetybvpydh'}
activity:  {u'kind': u'plus#activity', u'provider': {u'title': u'Google+'}, u'title': u'The heart of a star in the heart of a plant.\xa0At the National Botanic Garden yesterday.', u'url': u'https