Chapter 1. Gathering Open Source Intelligence

In this chapter, we will cover the following topics:

Gathering information using the Shodan APIScripting a Google+ API searchDownloading profile pictures using the Google+ APIHarvesting additional results using the Google+ API paginationGetting screenshots of websites using QtWebKitScreenshots based on port listsSpidering websites

Introduction

Open Source Intelligence (OSINT) is the process of gathering information from Open (overt) sources. When it comes to testing a web application,that might seem a strange thing to do. However, a great deal of information can be learned about a particular website before even touching it. You might be able to find out what server-side language the website is written in, the underpinning framework, or even its credentials.Learning to use APIs and scripting these tasks can make the bulk of the gathering phase a lot easier.

In this chapter, we will look at a few of the ways we can use Python to leverage the power of APIs to gain insight into our target.

Gathering information using the Shodan API

Shodan is essentially a vulnerability search engine. By providing it with a name, an IP address, or even a port, it returns all the systems in its databases that match. This makes it one of the most effective sources for intelligence when it comes to infrastructure. It's like Google for internet-connected devices. Shodan constantly scans the Internet and saves the results into a public database. Whilst this database is searchable from the Shodan website ( Application Programming Interface (API).

Getting ready

At the time of writing this, Shodan membership is $49, and this is needed to get an API key. If you're serious about security, access to Shodan is invaluable.

If you don't already have an API key for Shodan, visit www.shodan.io/store/member and sign up for it. Shodan has a really nice Python library, which is also well documented at https://shodan.readthedocs.org/en/latest/.To get your Python environment set up to work with Shodan, all you need to do is simply install the library using cheeseshop: