41、Scala Web Services and Database Interaction Guide

Scala Web Services and Database Interaction Guide

1. Handling Timeout and HTTP Requests

When dealing with HTTP requests, handling timeouts is crucial. One approach involves using the Apache HttpClient library. Here’s a code snippet for getting content from an HTTP entity and shutting down the connection manager:

if (entity != null) {
  val inputStream = entity.getContent
  content = io.Source.fromInputStream(inputStream).getLines.mkString
  inputStream.close
}
httpClient.getConnectionManager.shutdown
content

To build an HttpClient with custom timeouts, use the following method:

private def buildHttpClient(connectionTimeout: Int, socketTimeout: Int):
DefaultHttpClient = {
  val httpClient = new DefaultHttpClient
  val httpParams = httpClient.getParams
  HttpConnectionParams.setConnectionTimeout(httpParams, connectionTimeout)
  HttpConnectionParams.setSoTimeout(httpParams, socketTimeout)
  httpClient.setParams(httpParams)
  httpClient
}

This approach requires more code than Source.fromURL and depends on the HttpClient library. However, if you’re already using the Apache HttpClient library, it’s a viable alternative, especially when working with request headers.

There are other ways to handle the timeout problem:
- Use Akka Futures as a wrapper around the Source.fromURL method call.
- New libraries like Newman, released by StackMob, show promise. Newman’s DSL is inspired by the Dispatch library, uses method names instead of symbols, and provides separate methods for different request types (GET, POST, PUT, DELETE, HEAD).

Here is a table summarizing the different approaches:
| Approach | Description |
| — | — |
| Apache HttpClient | More code, useful with existing HttpClient usage, good for header handling |
| Akka Futures | Wrap Source.fromURL for timeout handling |
| Newman | New library, easy - to - use DSL |

The following mermaid flowchart shows the process of using the Apache HttpClient for a basic request:

graph TD;
    A[Create HttpClient] --> B[Set Timeouts];
    B --> C[Make Request];
    C --> D{Entity Exists?};
    D -- Yes --> E[Get Content];
    D -- No --> F[Skip Content Retrieval];
    E --> G[Close InputStream];
    G --> H[Shutdown Connection Manager];
    F --> H;

2. Sending JSON Data to a POST URL

If you want to send JSON data to a POST URL, you can use the Apache HttpClient library along with a JSON library like Gson. Here’s an example:

import java.io._
import org.apache.commons._
import org.apache.http._
import org.apache.http.client._
import org.apache.http.client.methods.HttpPost
import org.apache.http.impl.client.DefaultHttpClient
import java.util.ArrayList
import org.apache.http.message.BasicNameValuePair
import org.apache.http.client.entity.UrlEncodedFormEntity
import com.google.gson.Gson

case class Person(firstName: String, lastName: String, age: Int)

object HttpJsonPostTest extends App {
  // create our object as a json string
  val spock = new Person("Leonard", "Nimoy", 82)
  val spockAsJson = new Gson().toJson(spock)
  // add name value pairs to a post object
  val post = new HttpPost("http://localhost:8080/posttest")
  val nameValuePairs = new ArrayList[NameValuePair]()
  nameValuePairs.add(new BasicNameValuePair("JSON", spockAsJson))
  post.setEntity(new UrlEncodedFormEntity(nameValuePairs))
  // send the post request
  val client = new DefaultHttpClient
  val response = client.execute(post)
  println("--- HEADERS ---")
  response.getAllHeaders.foreach(arg => println(arg))
}

In this simple example, Gson is used to construct the JSON string. For more complex cases, the Lift - JSON library might be a better choice. After constructing the JSON string, the NameValuePair , BasicNameValuePair , and UrlEncodedFormEntity classes of the Apache HttpClient are used to set an entity on the HttpPost object. Finally, the POST request is sent, and the response headers are printed.

The steps to send JSON data using this method are as follows:
1. Create a Scala object.
2. Convert the object to a JSON string using a JSON library.
3. Create an HttpPost object with the target URL.
4. Add the JSON data as a NameValuePair to the HttpPost object.
5. Set the entity of the HttpPost object.
6. Execute the POST request using an HttpClient .
7. Optionally, print the response headers.

3. Getting URL Headers

To access the HTTP response headers after making an HTTP request, you can use the Apache HttpClient library. Here’s an example:

import org.apache.http.client.methods.HttpGet
import org.apache.http.impl.client.DefaultHttpClient

object FetchUrlHeaders extends App {
  val get = new HttpGet("http://alvinalexander.com/")
  val client = new DefaultHttpClient
  val response = client.execute(get)
  response.getAllHeaders.foreach(header => println(header))
}

Running this program will print the headers of the HTTP response. The HttpClient library simplifies the process of setting and reading header information, which can be useful in scenarios like working with Single Sign - On (SSO) systems.

4. Setting URL Headers When Sending a Request

If you need to set URL headers when making an HTTP request, use the Apache HttpClient library. Here’s an example:

import org.apache.http.client.methods.HttpGet
import org.apache.http.impl.client.DefaultHttpClient

object SetUrlHeaders extends App {
  val url = "http://localhost:9001/baz"
  val httpGet = new HttpGet(url)
  // set the desired header values
  httpGet.setHeader("KEY1", "VALUE1")
  httpGet.setHeader("KEY2", "VALUE2")
  // execute the request
  val client = new DefaultHttpClient
  client.execute(httpGet)
  client.getConnectionManager.shutdown
}

If you don’t have a web server to test against, you can use a tool like HttpTea. Start HttpTea at the command line to listen on port 9001:

$ java -jar HttpTea.jar -l 9001

When you run the client program, HttpTea will show the request headers.

The steps to set headers and make a request are:
1. Create an HttpGet object with the target URL.
2. Set the desired headers using the setHeader method.
3. Create an HttpClient and execute the request.
4. Shutdown the connection manager.

5. Creating a GET Request Web Service with the Play Framework

To create a GET request web service using the Play Framework, follow these steps:
1. Create a new Play project:

$ play new WebServiceDemo

Respond to the prompts to create a new Scala application, then move into the WebServiceDemo directory.
2. Assume you want to create a web service to return a Stock object as JSON when a client makes a GET request at the /getStock URI. Add the following line to your conf/routes file:

GET   /getStock     controllers.Application.getStock

3. Create a getStock method in the default Application controller ( apps/controllers/Application.scala ):

package controllers
import play.api._
import play.api.mvc._
import play.api.libs.json._
import models.Stock

object Application extends Controller {
  def index = Action {
    Ok(views.html.index("Your new application is ready."))
  }
  def getStock = Action {
    val stock = Stock("GOOG", 650.0)
    Ok(Json.toJson(stock))
  }
}

However, if you only use a simple case class for Stock , you’ll get an error when accessing the /getStock URI. To fix this, create a Stock.scala file in the app/models directory:

package models
case class Stock(symbol: String, price: Double)

object Stock {
  import play.api.libs.json._
  implicit object StockFormat extends Format[Stock] {
    // convert from JSON string to a Stock object (de - serializing from JSON)
    def reads(json: JsValue): JsResult[Stock] = {
      val symbol = (json \ "symbol").as[String]
      val price = (json \ "price").as[Double]
      JsSuccess(Stock(symbol, price))
    }
    // convert from Stock object to JSON (serializing to JSON)
    def writes(s: Stock): JsValue = {
      // JsObject requires Seq[(String, play.api.libs.json.JsValue)]
      val stockAsList = Seq("symbol" -> JsString(s.symbol),
                            "price" -> JsNumber(s.price))
      JsObject(stockAsList)
    }
  }
}

4. Start the Play console and run the application:

$ play
[WebServiceDemo] $ run 8080

Access the http://localhost:8080/getStock URL from a web browser, and you should see the JSON representation of the Stock object.

The following mermaid flowchart shows the process of creating a GET request web service in the Play Framework:

graph TD;
    A[Create Play Project] --> B[Configure Routes];
    B --> C[Create Controller Method];
    C --> D{Case Class Error?};
    D -- Yes --> E[Create Format Object];
    D -- No --> F[Skip Format Creation];
    E --> G[Start Play Server];
    F --> G;
    G --> H[Access Web Service];

6. POSTing JSON Data to a Play Framework Web Service

To create a web service that accepts JSON data via a POST request using the Play Framework:
1. Follow the steps from the previous section to create a new Play project, controller, and model.
2. Use the reads method of the Format object in the model to convert the received JSON string to a Stock object:

def reads(json: JsValue): JsResult[Stock] = {
  val symbol = (json \ "symbol").as[String]
  val price = (json \ "price").as[Double]
  JsSuccess(Stock(symbol, price))
}

3. Create a saveStock method in the Application controller:

import play.api._
import play.api.mvc._

object Application extends Controller {
  import play.api.libs.json.Json
  def saveStock = Action { request =>
    val json = request.body.asJson.get
    val stock = json.as[Stock]
    println(stock)
    Ok
  }
}

4. Add a route to the conf/routes file:

POST   /saveStock     controllers.Application.saveStock

5. Start the Play console and run the application:

$ play
[WebServiceDemo] $ run 8080

6. Use the following Unix curl command to POST a sample JSON string to the saveStock web service:

curl \
  --header "Content - type: application/json" \
  --request POST \
  --data '{"symbol":"GOOG", "price":900.00}' \
  http://localhost:8080/saveStock

If everything works, you’ll see the Stock object printed in the Play console.

The steps can be summarized in the following table:
| Step | Action |
| — | — |
| 1 | Create Play project, controller, and model |
| 2 | Implement reads method in model |
| 3 | Create saveStock method in controller |
| 4 | Configure route |
| 5 | Start Play server |
| 6 | Send POST request with curl |

7. Connecting to MySQL with JDBC

You can connect to a MySQL database from a Scala application using “plain old JDBC” just like in Java. Here’s an example:

package tests
import java.sql.{Connection,DriverManager}

object ScalaJdbcConnectSelect extends App {
  // connect to the database named "mysql" on port 8889 of localhost
  val url = "jdbc:mysql://localhost:8889/mysql"
  val driver = "com.mysql.jdbc.Driver"
  val username = "root"
  val password = "root"
  var connection:Connection = _

  try {
    Class.forName(driver)
    connection = DriverManager.getConnection(url, username, password)
    val statement = connection.createStatement
    val rs = statement.executeQuery("SELECT host, user FROM user")
    while (rs.next) {
      val host = rs.getString("host")
      val user = rs.getString("user")
      println("host = %s, user = %s".format(host,user))
    }
  } catch {
    case e: Exception => e.printStackTrace
  }
  connection.close
}

The format of the MySQL JDBC URL string is jdbc:mysql://HOST:PORT/DATABASE . If MySQL is running on the standard port (3306), you can drop the port from the URL string.

To run this example, use the Simple Build Tool (SBT). Create an SBT directory structure and add the MySQL JDBC dependency to the build.sbt file:

libraryDependencies += "mysql" % "mysql - connector - java" % "5.1.24"

Copy the code into a file named Test1.scala in the root directory of your project and run the program:

$ sbt run

This recipe is suitable for small applications where you need a single database connection and don’t mind running simple JDBC SQL queries. For larger applications, you may want to use a more advanced library like the Spring Framework.

The process of connecting to MySQL with JDBC can be visualized in the following mermaid flowchart:

graph TD;
    A[Set URL, Driver, Username, Password] --> B[Load Driver];
    B --> C[Get Connection];
    C --> D[Create Statement];
    D --> E[Execute Query];
    E --> F{More Rows?};
    F -- Yes --> G[Get Data and Print];
    F -- No --> H[Close Connection];
    G --> F;

Scala Web Services and Database Interaction Guide

8. Database Interaction Libraries

In the realm of database interaction, several libraries offer unique advantages for Scala developers.

• Squeryl : Squeryl is a “Scala ORM and DSL”. It allows developers to write database code in a way similar to Hibernate. For example, you can insert and update records as follows:

// insert
val bill = people.insert(new Person("Bill"))
val candy = people.insert(new Person("Candy"))
// update
stock.price = 500.00
stocks.update(stock)

With its DSL, you can also perform more complex operations like conditional updates:

update(stocks)(s =>
  where(s.symbol === "AAPL")
  set(s.price := 500.00)
)

• Slick : Slick isn’t an object - relational mapping (ORM) tool, but its type - safe approach enables developers to write database access code almost as if they’re working with a collection.

The following table compares Squeryl and Slick:
| Library | Type | Advantage | Example |
| — | — | — | — |
| Squeryl | ORM and DSL | Similar to Hibernate, easy to write database operations | Inserting and updating records as shown above |
| Slick | Type - safe access | Write code like working with collections | Not shown in this text but has a unique approach |

9. MongoDB Interaction with Casbah

For “big data” projects, Scala can interact with MongoDB using the Casbah driver. Although the detailed code for insert, update, read, and delete operations isn’t provided here, the general process can be outlined as follows:

1. Insert : Create a MongoDB collection object and use the insert method to add documents.
2. Update : Locate the document to be updated and use the update method with the appropriate query and update operators.
3. Read : Use query methods to retrieve documents based on specific criteria.
4. Delete : Identify the document to be deleted and use the remove method.

The mermaid flowchart below shows the general process of interacting with MongoDB using Casbah:

graph LR;
    A[Connect to MongoDB] --> B[Select Collection];
    B --> C{Operation Type};
    C -- Insert --> D[Insert Document];
    C -- Update --> E[Update Document];
    C -- Read --> F[Read Document];
    C -- Delete --> G[Delete Document];

10. Scalding for Hadoop MapReduce

Twitter’s Scalding project makes it easy to specify Hadoop MapReduce jobs. It is analogous to the Apache Pig project but is tightly integrated with Scala. Although Scalding and Hadoop aren’t covered in great detail here, the Scalding source code tutorials can help developers quickly get up and running.

The steps to start using Scalding might be:
1. Set up the Scalding project environment.
2. Define the data sources and sinks.
3. Write the MapReduce logic using Scalding’s API.
4. Run the Hadoop job.

11. Summary of Key Points

• HTTP Requests :
  • Use the Apache HttpClient library for handling timeouts, sending JSON data, getting and setting headers.
  • Other alternatives like Akka Futures and Newman can also be considered for handling timeouts.
• Web Services with Play Framework :
  • For GET requests, create a Play project, configure routes, and implement the necessary controller methods. Ensure proper JSON serialization and deserialization using Format objects.
  • For POST requests, follow similar steps as GET requests but use the reads method to convert received JSON data to objects.
• Database Interaction :
  • Use “plain old JDBC” for simple MySQL connections in small applications.
  • Consider libraries like Squeryl, Slick for more advanced database operations in Scala.
  • Use Casbah for interacting with MongoDB in “big data” projects.
  • Explore Scalding for Hadoop MapReduce jobs.

The following table summarizes the key libraries and their use cases:
| Library | Use Case |
| — | — |
| Apache HttpClient | HTTP requests (timeout handling, JSON data sending, header management) |
| Play Framework | Creating web services (GET and POST requests) |
| Squeryl | Database operations in a type - safe and ORM - like manner |
| Slick | Type - safe database access |
| Casbah | MongoDB interaction |
| Scalding | Hadoop MapReduce jobs |

In conclusion, Scala offers a wide range of options for web services and database interaction. Developers can choose the appropriate tools and libraries based on the requirements of their projects, whether it’s a small - scale application or a large - scale “big data” project.