Saturday, 22 October 2016

scala play json traversal

Another night on the startup, I learned something about json traversal using play json library.

Play Json api is cool, for example I can convert the json string to Json using Json.parse("json string").

The fun part is when I had to traverse the Json object. For example I want to make sure that the json response I received from my HTTP server has the proper values.

So, here is the example of traversing the given list of json objects,

scala> import play.api.libs.functional.syntax._
import play.api.libs.functional.syntax._

scala> import play.api.libs.json._
import play.api.libs.json._

scala> implicit val productReader = (
     |         (__ \ "id").read[Long] and
     |       (__ \ "name").read[String] and
     |         (__ \ "brand").read[String] and
     |         (__ \ "price").read[Double] and
     |         (__ \ "category").read[String]and
     |         (__ \ "marketplace").read[String] and
     |         (__ \ "date").read[Long]
     |       ) tupled;
warning: there were 1 feature warning(s); re-run with -feature for details
productReader: play.api.libs.json.Reads[(Long, String, String, Double, String, String, Long)] = play.api.libs.json.Reads$$anon$8@711cbc4c

scala> val x = """[{"0":{"price":100.0,"releasedOn":1419783511,"brand":"Palace Skate Boards","category":"Men","name":"Tops","marketplaceName":"Oodni boutique","id":1}}, {"1":{"price":200.0,"releasedOn":1419783519,"brand":"Steven Wilson","category":"Metal","name":"Hands.Can not. Erase","marketplaceName":"Kscope","id":2}}, {"2":{"price":200.0,"releasedOn":1419783519,"brand":"sleepmakeswaves","category":"Rock","name":"And So We Destroyed Everything","marketplaceName":"Bird's Robe Records","id":3}}]"""
x: String = [{"0":{"price":100.0,"releasedOn":1419783511,"brand":"Palace Skate Boards","category":"Men","name":"Tops","marketplaceName":"Oodni boutique","id":1}}, {"1":{"price":200.0,"releasedOn":1419783519,"brand":"Steven Wilson","category":"Metal","name":"Hands.Can not. Erase","marketplaceName":"Kscope","id":2}}, {"2":{"price":200.0,"releasedOn":1419783519,"brand":"sleepmakeswaves","category":"Rock","name":"And So We Destroyed Everything","marketplaceName":"Bird's Robe Records","id":3}}]
scala> val jsonArray = Json.parse(x).as[JsArray]
jsonArray: play.api.libs.json.JsArray = [{"0":{"price":100.0,"releasedOn":1419783511,"brand":"Palace Skate Boards","category":"Men","name":"Tops","marketplaceName":"Oodni boutique","id":1}},{"1":{"price":200.0,"releasedOn":1419783519,"brand":"Steven Wilson","category":"Metal","name":"Hands.Can not. Erase","marketplaceName":"Kscope","id":2}},{"2":{"price":200.0,"releasedOn":1419783519,"brand":"sleepmakeswaves","category":"Rock","name":"And So We Destroyed Everything","marketplaceName":"Bird's Robe Records","id":3}}]

scala> jsonArray \\ "0"
res14: Seq[play.api.libs.json.JsValue] = ListBuffer({"price":100.0,"releasedOn":1419783511,"brand":"Palace Skate Boards","category":"Men","name":"Tops","marketplaceName":"Oodni boutique","id":1})
scala> jsonArray(0)
res3: play.api.libs.json.JsValue = {"0":{"price":100.0,"releasedOn":1419783511,"brand":"Palace Skate Boards","category":"Men","name":"Tops","marketplaceName":"Oodni boutique","id":1}}

scala> jsonArray(0).\("0")
res4: play.api.libs.json.JsValue = {"price":100.0,"releasedOn":1419783511,"brand":"Palace Skate Boards","category":"Men","name":"Tops","marketplaceName":"Oodni boutique","id":1}

scala> jsonArray(0).\\("0")
res5: Seq[play.api.libs.json.JsValue] = List({"price":100.0,"releasedOn":1419783511,"brand":"Palace Skate Boards","category":"Men","name":"Tops","marketplaceName":"Oodni boutique","id":1})
scala> jsonArray(0).\("0").\("price")
res11: play.api.libs.json.JsValue = 100.0

scala> jsonArray(0).\("0").\("name")
res12: play.api.libs.json.JsValue = "Tops"

scala> jsonArray(0).\("0").\("brand")
res13: play.api.libs.json.JsValue = "Palace Skate Boards"
scala> ((jsonArray \\ "0").map(on => on \ "name").map(_.as[String])).asInstanceOf[Seq[String]](0)
res33: String = Tops

Reference

https://www.playframework.com/documentation/2.5.x/ScalaJson


Thursday, 20 October 2016

consuming scala play Enumerator/Stream


I was working on a startup using scala play app and was unit testing the response from Play Middlerware which is basically returningFuture[SimpleResult] with body as Enumerator[Array[Byte]].
If the response has json a body, Enumerator won't give me the body as flat string, rather a list of chunks. So, Enumerator is a simply a async collection aka Stream which needs to be consumed by someone else( Iteratee).

Stream source


For example, song is a collection of buffered chunks of audio clips.

SBT_SCALA_VERSION=2.10.4 play console

scala> import play.api.libs.iteratee.{Enumerator, Iteratee}
import play.api.libs.iteratee.{Enumerator, Iteratee}

scala> val songStream: Enumerator[String] = Enumerator("first 5 minutes ", "second 5 minutes ", "last 5 minutes")
songStream: play.a.l.i.Enumerator[String] = play.api.libs.i.Enumerator$$anon$19@24142c37

 // Enumerator can also be created using the stream
scala> val songStream: Enumerator[String] = Concurrent.unicast[String](onStart = stream => {
  stream.push("first 5 minutes ")
  stream.push("second 5 minutes ")
  stream.push("last 5 minutes")
})


Stream Consumer - I

To get the song, we first need to consume the Enumerator, and then flatMap the async response.
scala> val consumeSong = songStream(Iteratee.consume[String]())
consumeSong: s.c.Future[p.a.l.i.Iteratee[String,String]] = s.c.i.Promise$DefaultPromise@6854ee8b

flatMap the consumed song which is a Future[X]

scala> import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.ExecutionContext.Implicits.global

scala> import scala.concurrent.Future
import scala.concurrent.Future

scala> val songThatCanBeListened :Future[String] = consumeSong.flatMap(chunk => chunk.run)
songThatCanBeListened: s.c.Future[String] = s.c.i.Promise$DefaultPromise@2a7fc3c5

scala> songThatCanBeListened.onSuccess {case song => println(s"playing a song : $song")}

scala> playing a song : first 5 minutessecond 5 minuteslast 5 minutes


Stream Consumer - II

The other way of consuming the stream(Enumerator) is with piping/sinking operator(I don't exactly know what Play apis calls it),
scala> val consumeSongOtherWay = Iteratee.flatten(songStream |>> Iteratee.consume[String]()).run
consumeSongOtherWay: s.c.Future[String] = s.c.i.Promise$DefaultPromise@6c9f484a

scala> consumeSongOtherWay.onSuccess{case wholeSong => println(wholeSong)}

Thursday, 13 October 2016

playing with docker native 1.12 container for mac

The following post might look like running kafka cluster in MacOS but it basically shows how to get started with Docker and basic docker commands.

Docker is widely being used to simply spawn a service I need as a separate container just like in production. And It can easily be guessed same container can be run the production server with the same config.

For example, if my application uses MongoDB, traditionally I used to download and install MongoDB myself and test it. I remember in one of my analytics project, I had weird issue with running different version of Elasticsearch because each Software Engineer does his own stuff in a large team.

With docker, I can create an Elasticsearch container that my application would connect to and share the same container with other members in a team. And the same Elasticsearch container is spawn inside the Production server, so that configuration everywhere is the same.

Download and install docker native for mac


wget https://download.docker.com/mac/stable/Docker.dmg
hdiutil mount Docker.dmg ##pops up installer to cp into /Applications
sudo cp -r /Volumes/Docker/Docker.app /Applications/
hdiutil unmount /Volumes/Docker

I will see following files in /Applications/

$ ls -l /Applications/Docker.app/Contents/
total 32
drwxr-xr-x  21 a1353612  admin   714 Oct  6 04:45 Frameworks
-rw-r--r--   1 a1353612  admin  2460 Oct  6 04:48 Info.plist
drwxr-xr-x   4 a1353612  admin   136 Oct  6 04:45 Library
drwxr-xr-x  11 a1353612  admin   374 Oct  6 04:48 MacOS
-rw-r--r--   1 a1353612  admin     8 Oct  6 04:45 PkgInfo
drwxr-xr-x  26 a1353612  admin   884 Oct  6 04:48 Resources
drwxr-xr-x   3 a1353612  admin   102 Oct  6 04:45 _CodeSignature
-rw-r--r--   1 a1353612  admin  7579 Oct  6 04:45 embedded.provisionprofile

Check its properly installed
docker --version
Docker version 1.12.1, build 6f9534c

docker-compose --version
docker-compose version 1.8.0, build f3628c7

docker-machine --version
docker-machine version 0.8.1, build 41b3b25

Test a webserver container is accessible from local.

docker run -it --rm -p 8888:8080 tomcat:8.0

Then send a GET request to localhost:8888


$ curl -X GET localhost:8888 -i
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=UTF-8
Transfer-Encoding: chunked
Date: Wed, 22 Mar 2017 10:44:35 GMT

running a docker container
To run a docker container, I am cloning a kafka container, Kafka is a distributed messaging/streaming platform.
git clone https://github.com/prayagupd/docker-kafka-zk.git
cd docker-kafka-zk
docker-compose up -d

docker-compose is a tool for defining and running multi-container Docker apps.

Check the running containers
docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
4369c261f6d4        wurstmeister/zookeeper   "/bin/sh -c '/usr/sbi"   48 seconds ago      Up 45 seconds       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   dockerkafkazk_zookeeper_1
6e83021e6c97        dockerkafkazk_kafka      "start-kafka.sh"         48 seconds ago      Up 46 seconds       0.0.0.0:32768->9092/tcp                              dockerkafkazk_kafka_1
Get the IP addresses for the containers
docker exec dockerkafkazk_kafka_1 cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.2 6e83021e6c97

or 
docker inspect dockerkafkazk_kafka_1 | grep -w "IPAddress"
            "IPAddress": "",
                    "IPAddress": "172.18.0.2",


and for zookeeper, 
docker exec dockerkafkazk_zookeeper_1 cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.18.0.3 4369c261f6d4


SecureSHell to Kafka container from local machine

./start-kafka-shell.sh 172.18.0.2 172.18.0.3:2181
bash-4.3# echo $KAFKA_HOME/
/opt/kafka_2.11-0.10.0.1/

Fire the following command to stop all containers,

docker stop $(docker ps -a -q)

In next post, I will probably write about streaming to this kafka container using nodejs. I'm currently doing some stuffs on nodejs during my Masters period, (off office work). So, thinking of streaming via nodejs app which scala app would consume on the other side.

Resources
--------------

https://www.viget.com/articles/how-to-use-docker-on-os-x-the-missing-guide

https://docs.docker.com/docker-for-mac/networking/