Mac OS - command line hacks

processor bits

"processor bits" means that how much data a microprocessor will process within one instruction cycle i.e. fetch-decode-execute

32-bit means that a microprocessor can execute 4 bytes of data in one instruction cycle

while 64-bit means that a microprocessor executes can execute 8 bytes of data in one instruction cycle.

$ sysctl hw | grep cpu #sysctl hw.cpu64bit_capable
hw.ncpu: 8
hw.activecpu: 8
hw.physicalcpu: 4
hw.physicalcpu_max: 4
hw.logicalcpu: 8
hw.logicalcpu_max: 8
hw.cputype: 7
hw.cpusubtype: 8
hw.cpu64bit_capable: 1
hw.cpufamily: 280134364
hw.cpufrequency: 2800000000
hw.cpufrequency_min: 2800000000
hw.cpufrequency_max: 2800000000
hw.cputhreadtype: 1

$ getconf LONG_BIT
64

The number of bits in a processor refers to the size of the data types that it handles and the size of its registry. Simply put, a 64-bit processor is more capable than a 32-bit processor because it can handle more data at once.

CPU info

sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz

$ system_profiler | grep Processor
    Apple: AUDynamicsProcessor:
    Apple: AUVoiceProcessor:
      Processor Name: Intel Core i7
      Processor Speed: 2.8 GHz
      Number of Processors: 1
    <key>BootCampProcessorPstates</key>
    |         "BootCampProcessorPstates" = <0a00>

no of cores

$ sysctl -n hw.ncpu
8

IP address of Machine

$ ipconfig getifaddr en0
10.11.208.214

or

$ ifconfig | grep inet
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet6 fe80::c6b3:1ff:fed3:d771%en0 prefixlen 64 scopeid 0x4
inet 10.11.208.214 netmask 0xfffffe00 broadcast 10.11.209.255
inet6 fe80::7023:7bff:fee3:61e2%awdl0 prefixlen 64 scopeid 0x9

Change password using DsCl (Directory Service CLI)

dscl . -passwd /Users/prayagupd
New Password: dreams
passwd: DS error: eDSAuthPasswordTooShort

<dscl_cmd> DS Error: -14170 (eDSAuthPasswordTooShort)

Note

$ man dscl | grep "passwd"

     Usage: passwd user_path [new_pasword | old_password new_pasword]

Uninstall application

mdfind -name "Skype"
/Applications/Skype.app

sudo rm -rf /Applications/Skype.app

add a printer (lpadmin)

find the IP address of the printer, and then add it to macos printers.

http://apple.stackexchange.com/q/126834/76240


printer stat

lpstat -p
printer _10_10_10_113 is idle.  enabled since Thu Oct 20 17:23:31 2016
 Waiting for printer to finish.
printer _10_20_50_100 is idle.  enabled since Fri Sep  9 15:39:45 2016

RAM Memory usage

$ vm_stat 
Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                              264375.
Pages active:                           2027888.
Pages inactive:                          651659.
Pages speculative:                        23565.
Pages throttled:                              0.
Pages wired down:                        852647.
Pages purgeable:                          39328.
"Translation faults":                 880109291.
Pages copy-on-write:                   44871249.
Pages zero filled:                    433518664.
Pages reactivated:                      5346396.
Pages purged:                           1423915.
File-backed pages:                       379668.
Anonymous pages:                        2323444.
Pages stored in compressor:             2870604.
Pages occupied by compressor:            372004.
Decompressions:                        33640670.
Compressions:                          43459639.
Pageins:                               10427527.
Pageouts:                                 34549.
Swapins:                               32669611.
Swapouts:                              34330610.

install *.dmg from command line in OSX

Installing

STEP 1: mount the apple disk image file as /Volumes

hdiutil mount ideaIC-15.0.2-custom-jdk-bundled.dmg 
----
----
/dev/disk2          GUID_partition_scheme          
/dev/disk2s1        Apple_HFS                      /Volumes/IntelliJ IDEA 15 CE

STEP 2: copy the mounted *.app to /Apps

sudo cp -r "/Volumes/IntelliJ IDEA 15 CE/IntelliJ IDEA 15 CE.app" /Applications

If the mounted file is *.pkg instead of *.app, then 

$ sudo installer -pkg /Applications/chefdk-1.0.3-1.pkg -target /
installer: Package name is Chef Development Kit
installer: Upgrading at base path /
installer: The upgrade was successful.

STEP 3: unmount the file from /Volumes

hdiutil unmount "/Volumes/IntelliJ IDEA 15 CE"
"/Volumes/IntelliJ IDEA 15 CE" unmounted successfully.

OR

sudo cp -r ggts-bundle/ggts-3.6.4.RELEASE/GGTS.app/ /Applications/

Uninstalling

remove from /Apps

$ ll /Applications/IntelliJ\ IDEA*
/Applications/IntelliJ IDEA CE.app:
total 0
drwxr-xr-x@ 12 prayagupd  admin  408 Jan 21 00:33 Contents

/Applications/IntelliJ IDEA.app:
total 0
drwxr-xr-x@ 13 prayagupd  admin  442 Dec 25 09:21 Contents

rm -rf /Applications/IntelliJ\ IDEA\ CE.app/

Scala Future with play framework

Async controller
----------------------------

package beard.controllers

import play.api.mvc._
import play.api.libs.json._
import play.api.libs.functional.syntax._
import play.libs.Json._
import reactivemongo.bson._
import service.BeardService
import play.api.libs.concurrent.Execution.Implicits.defaultContext

import scala.collection.mutable
import scala.collection.mutable.ListBuffer
import scala.concurrent.Future
import scala.util.{Failure, Success, Random}

class AsyncBeardController extends Controller {

 def sayAsyncBeard = Action.async { request =>
    val futureResult = Future {
      intensiveComputation()
    }
    futureResult.map(result =>
      Ok(result)
    )
  }


def intensiveComputation(): JsObject = {
      Thread.sleep(Random.nextInt(5000))
        Json.obj("value" -> "beard")
  }
}

Test
-------

$ curl -XGET http://localhost:9000/sayAsyncBeard
{"value":"beard"}

You will get the response after ~5seconds.

CPU cores in OSX

Seems OSX sucks in command line support as Linux does. However, found few commands to get hw information in OSX.

sysctl -n hw.ncpu
8

sysctl -n hw.physicalcpu
4

sysctl -n hw.logicalcpu
8


Ref : https://coolaj86.com/articles/get-a-count-of-cpu-cores-on-linux-and-os-x/

Kafka Events Streaming - Spark Consumer Example

* Apache Kafka supports a wide range of use cases as a general-purpose messaging system for scenarios where high throughput, reliable delivery, and horizontal scalability are important.
Use cases include:
* Stream Processing
* Website Activity Tracking
* Metrics Collection and Monitoring
* Log Aggs

* Apache Storm and Apache Spark both work very well in combination with Kafka.

Here's an example how Kafka can be used with apache spark for consuming events stream,

DEVELOPMENT
STEP 1 : ADD lib dependency to decoupled-invocation/build.sbt 

name := "decoupled-invocation"                                                                                                                        

version := "1.0"                                                                                    

scalaVersion := "2.10.4"                                                                            

val sparkVersion = "1.2.0"                                                                          

libraryDependencies ++= Seq(                                                                        
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",                              
  "org.apache.spark" %% "spark-streaming-kafka" % sparkVersion                                      
)   

STEP 2 : Create kafka stream in scala file decoupled-invocationsrc/main/scala/EventsConsumerApp.scala

object EventsConsumerApp {                                                                          

  def persist() = {                                                                                 

  }                                                                                                 

  def main(args : Array[String]): Unit = {                                                          
   val sparkConf = new SparkConf().setMaster("spark://prayagupd:7077").setAppName("EventsConsumerApp")
   val sparkStreamingContext = new StreamingContext(sparkConf, Seconds(10))                         

   val kafkaConf = Map("metadata.broker.list" -> "localhost:9092",                                  
                       "zookeeper.connect" -> "localhost:2181",                                     
                       "group.id" -> "events-topic-consumer-group",                                 
                       "zookeeper.connection.timeout.ms" -> "1000")                                 

    //http://spark.apache.org/docs/latest/streaming-programming-guide.html#discretized-streams-dstreams
   val kafkaDiscretizedStream = KafkaUtils.createStream[Array[Byte], String, DefaultDecoder, StringDecoder](sparkStreamingContext,
                                             kafkaConf,                                             
                                             Map("events-topic" -> 1), StorageLevel.MEMORY_ONLY_SER)
   //persist()                                                                                      
   kafkaDiscretizedStream.print()                                                                   
   sparkStreamingContext.start()                                                                    
   sparkStreamingContext.awaitTermination()                                                         
 }                                                                                                  
}   

STEP 3 : build application

sbt assembly ## doing sbt package wont find kafka jar while submitting job to spark







DEPLOYMENT
STEP 4 : start kafka broker with default config

cd /usr/local/kafka
bin/zookeeper-server-start.sh config/zookeeper.properties

#terminal 2
bin/kafka-server-start.sh config/server.properties

STEP 5 : Create kafka topic events-topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic events-topic
Created topic "events-topic".

verify topic is created

bin/kafka-topics.sh --list --zookeeper localhost:2181
events-topic

STEP 6 : Produce stream using kafka to topic events-topic

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic events-topic
{'message' : 'user is logging in'}
{'message' : 'User logged in'}

STEP 7 : submit spark job that will consume events produced by above kafka producer

/usr/local/spark-1.2.0/bin/spark-submit --class EventsConsumerApp --master spark://prayagupd:7077 target/scala-2.10/decoupled-invocation-assembly-1.0.jar

(null,{'message' : 'user is logging in'})
(null,{'message' : 'User logged in'})

15/09/20 17:04:12 INFO scheduler.JobScheduler: Finished job streaming job 1442786650000 ms.0 from job set of time 1442786650000 ms
15/09/20 17:04:12 INFO scheduler.JobScheduler: Total delay: 2.294 s for time 1442786650000 ms (execution: 2.289 s)
15/09/20 17:04:12 INFO rdd.BlockRDD: Removing RDD 5 from persistence list
15/09/20 17:04:12 INFO storage.BlockManager: Removing RDD 5
15/09/20 17:04:12 INFO kafka.KafkaInputDStream: Removing blocks of RDD BlockRDD[5] at createStream at EventsConsumerApp.scala:39 of time 1442786650000 ms
15/09/20 17:04:12 INFO scheduler.ReceivedBlockTracker: Deleting batches ArrayBuffer(1442786630000 ms)
15/09/20 17:04:12 INFO scheduler.ReceivedBlockTracker: Deleting batches ArrayBuffer()
15/09/20 17:04:20 INFO scheduler.JobScheduler: Added jobs for time 1442786660000 ms

As seen in console events produced using kafka-producer can be consumed in apache spark + kafka application, and can be persited to NoSQL databases or wherever thereafter.

Source code
scalability-patterns/decoupled-invocation

Resources
Spark Streaming + Kafka Integration Guide

git branching model

Git Branching most of the time seemed like a maze to me. Specially while sending a Pull Request for sprint development.

When there are three features in a sprint,

1) One idea could be  - maintain three local branches for each feature and send three parallel PRs separately.
After feature gets merged, delete local and remote feature branches.

2) The other could be - maintain three local branches for each feature, but when finished merge them to one single branch and send a PR one by one or ?.

I think the second one restricts from having number of unwanted branches to be deleted later, which looks OK to me while working in a team. 

Here's how I implement this approach; assuming sprint/develop is an upto-date branch created with HEAD of master,

STEP 1 - create feature branch

git checkout -b module-1/feature-1 sprint/develop

STEP 2 - after feature completion

Merge it to single sprint/develop branch

git checkout sprint/develop

git merge --no-ff module-1/feature-1 # create a new commit object.

## delete the feature branch

git branch -d module-1/feature-1

STEP 3 - rebase sprint/develop branch with master

git rebase master ## if any conflicts occur, resolve them manually, git add conflicted files, and

                            ## git rebase --continue

STEP 4 - push develop changes

git push origin sprint/develop  ## or git push --force origin sprint/develop

STEP 5 - Send PR and get reviewed
Send PR for merge on master. what?? few implementation not aligned with architecture of codebase?? FIX them on same branch and push.

A nerd's java practices, might not be universal though

Here's what a java nerd thinks on practices while coding. 
Purpose could be to 
* make beautiful code,
* improve perf
* improve readability etc

1. class members vs passing args between internal methods

Prefer class members or passing arguments between internal methods?

Do methods in class instances take a place in memory?

2. Temp variable in foreach iteration of a large loop

Temporary variable used for each iteration of a large loop, strings are immutable so what should I use?

3. Immutability habit (an affair with final) 
Immutability are to simplify a program, here's what the veterans have to say;

"Classes should be immutable unless there's a very good reason to make them mutable....If a class cannot be made immutable, limit its mutability as much as possible." - Joshua Bloch

"Immutability Fosters Concurrent Programming." - clojure

If immutable objects are good, why do people keep creating mutable objects?

Excessive use “final” keyword in Java

Wikileaks To Leak 5000 Open Source Java Projects With All That Private/Final Bullshit Removed