Python Monitor Water Falls(1)Ideas from Scrapy and Twisted

Python Monitor Water Falls(1)Ideas from Scrapy and Twisted

>python -V
Python 2.7.13

>pip --version
pip 9.0.1 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)

Make sure we have the Scrapy Installed
>pip install scrapy
>pip install scrapyd

>scrapy version
Scrapy 1.4.0

>scrapyd --version
twistd (the Twisted daemon) 17.5.0

Visit the Web Page https://hydromet.lcra.org/riverreport

Basic method to load the content open_url.py
from selenium import webdriver

path2phantom = '/opt/phantomjs/bin/phantomjs'
browser = webdriver.PhantomJS(path2phantom)
browser.get('https://hydromet.lcra.org/riverreport')

tables = browser.find_elements_by_css_selector('table.table-condensed')

tbody = tables[5].find_element_by_tag_name("tbody")
for row in tbody.find_elements_by_tag_name("tr"):
cells = row.find_elements_by_tag_name("td")
if(cells[0].text == 'Marble Falls (Starcke)'):
print cells[1].text

browser.quit()

By executing the command directly, it will tell you the content
>python open_url.py
Jan 21 2018: No releases

Twisted RESTful
download the latest version
>wget https://twistedmatrix.com/Releases/Twisted/17.9/Twisted-17.9.0.tar.bz2
Unzip and go into that directory
>python setup.py install

Check the version
>>>> import twisted
>>> print twisted.version
[Twisted, version 17.5.0]

Maybe long time ago, I already install 17.5.0
This RESTful library is a little old
https://github.com/iancmcc/txrestapi

RESTful with twisted
BookAPI.py
from txrestapi.methods import GET, POST, PUT, ALL, DELETE
from txrestapi.resource import APIResource
import json

class BookResource(APIResource):
@GET('^/book/(?P<id>[^/]+)')
def getBook(self, request, id):
return 'Pick up one book with id %s' % id

@GET('^/book/')
def books(self,request):
return "books"

@PUT('^/book/(?P<id>[^/]+)')
def updateBook(self,request, id):
book = json.loads(request.content.read())
book['id'] = id
return json.dumps(book)

@POST('^/book/')
def saveBook(self, request):
book = json.loads(request.content.read())
book['id'] = 1
return json.dumps(book)

@DELETE('^/book/(?P<id>[^/]+)')
def deleteBook(self,request,id):
return "Delete book with id %s" % id

@ALL('^/')
def default_view(self, request):
return "I fail to match other URLs."

RestAPI.py
from twisted.web.resource import Resource
from twisted.web.server import Site
from twisted.internet import reactor

from BookAPI import BookResource

bookResource = BookResource()

rootResource = Resource()
rootResource.putChild("book", bookResource)

site = Site(rootResource, timeout=None)
reactor.listenTCP(8888,site)
reactor.run()


References:
https://scrapy.org/
https://hydromet.lcra.org/riverreport

Twisted
http://sillycat.iteye.com/blog/2243749
http://sillycat.iteye.com/blog/2243775
http://sillycat.iteye.com/blog/2244167
http://sillycat.iteye.com/blog/2244169

python json
https://code.tutsplus.com/tutorials/how-to-work-with-json-data-using-python--cms-25758