Virtualization 101

What is virtualization?

The idea of virtualization is to to be able to run virtual computers on existing piece of physical machine. One computer will get one IP address, with virtualization I can have multiple IPs on same hardware.

I first knew of virtualization while working on software for banks. The "IT department" of bank would create a VM for me to install software :)

https://www.redhat.com/en/topics/virtualization/what-is-virtualization

Setting up VM

Install VirtualBox which is a full virtualizer https://www.virtualbox.org/wiki/Downloads

Then download the ISO for OS you want to have a VM for and attach it to VirtualBox storage as shown in screenshot below(In MacOS). Ubuntu ISO can be downloaded from here, https://www.ubuntu.com/download/desktop

http://askubuntu.com/questions/825437/ubuntu-16-in-vm-does-not-have-access-to-internet

Regarding the Network setup, it really depends on the nature of the network being used. In following example, in one case wireless works for me.

while in wired environment ethernet adaptor works.

Resizing the VM
In case you need to increase the VM RAM or Disk size, after you created your VM, then follow the process as below :

STEP 1 : shut down VM

STEP 2 : resize the VM vdi using host terminal (windows in mycase)
VBoxManage modifyhd "C:\Users\AS18\VirtualBox VMs\WM_Bridge\WM_Bridge.vdi" --resize 100000

STEP 3 : 
Then, using gparted resize the /dev/sda1

Original config

Delete linux-swap

Reassign the size to linux-swap

http://askubuntu.com/a/558215/37643

http://askubuntu.com/a/95026/37643

Share folder
---------------------

1) Install guest edition

followed by sudo /media/prayagupd/VBOXADDITIONS_5.1.8_111374/VBoxLinuxAdditions.run and then reboot.

How to Access Folders on Your Host Machine from an Ubuntu Virtual Machine in VirtualBox

2) add current guest user to the vboxsf usergroup

sudo adduser $USER vboxsf
[sudo] password for prayagupd:
Adding user `prayagupd' to group `vboxsf' ...
Adding user prayagupd to group vboxsf
Done.

verify that its working, 

id $USER

uid=1000(prayagupd) gid=1000(prayagupd) groups=1000(prayagupd),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),128(sambashare),999(vboxsf)

3) mount shared folder

Create a mount folder on host machine with Auto-mount Yes, and Access Full.

use flume to stream from avro source and publish to logger sink

flume = a deep narrow channel or ravine with a stream running through it. 

PART 1 - setup flume (can be dockerized)

wget --proxy=off http://apache.claz.org/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz
tar -zxvf apache-flume-1.5.0-bin.tar.gz -C /opt/flume --strip-components=1
rm apache-flume-1.6.0-bin.tar
sudo chmod -R 777 /opt/flume

PART 2 start the flume node (in a docker container)

conf/supply_chain_flume.conf

# Define a memory channel called ch1 on supply_agent                                                        
supply_agent.channels.logEventStream.type = memory

# Define an Avro source called avro-source1 on agent1 and tell it                                     
# to bind to 0.0.0.0:41414. Connect it to channel ch1.                                               
supply_agent.sources.avro-source1.channels = logEventStream
supply_agent.sources.avro-source1.type = avro
supply_agent.sources.avro-source1.bind = 0.0.0.0
supply_agent.sources.avro-source1.port = 41414

# Define a logger sink that simply logs all events it receives                                       
# and connect it to the other end of the same channel.                                               
supply_agent.sinks.log-sink1.channel = logEventStream
supply_agent.sinks.log-sink1.type = logger

# Finally, now that we've defined all of our components, tell                                         
# agent1 which ones we want to activate.                                                           
supply_agent.channels = logEventStream
supply_agent.sources = avro-source1
supply_agent.sinks = log-sink1

start flume agent

bin/flume-ng agent --conf ./conf/ -f conf/supply_chain_flume.conf -Dflume.root.logger=DEBUG,console -n supply_agent

Info: Sourcing environment configuration script /usr/local/apache-flume-1.6.0-bin/conf/flume-env.sh
+ exec /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home//bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/usr/local/apache-flume-1.6.0-bin/conf:/usr/local/apache-flume-1.6.0-bin/lib/*' -Djava.library.path= org.apache.flume.node.Application -f conf/flume.conf -n supply_agent
2016-09-22 00:08:49,909 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:61)] Configuration provider starting
2016-09-22 00:08:49,912 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:78)] Configuration provider started
2016-09-22 00:08:49,913 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume.conf for changes
2016-09-22 00:08:49,914 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:133)] Reloading configuration file:conf/flume.conf
2016-09-22 00:08:49,918 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:log-sink1
2016-09-22 00:08:49,918 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1021)] Created context for log-sink1: channel
2016-09-22 00:08:49,918 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:931)] Added sinks: log-sink1 Agent: supply_agent
2016-09-22 00:08:49,918 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1017)] Processing:log-sink1
2016-09-22 00:08:49,919 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:314)] Starting validation of configuration for agent: supply_agent, initial-configuration: AgentConfiguration[supply_agent]
SOURCES: {avro-source1={ parameters:{bind=0.0.0.0, channels=ch1, port=41414, type=avro} }}
CHANNELS: {ch1={ parameters:{type=memory} }}
SINKS: {log-sink1={ parameters:{channel=ch1, type=logger} }}

2016-09-22 00:08:49,922 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:469)] Created channel ch1
2016-09-22 00:08:49,926 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:675)] Creating sink: log-sink1 using LOGGER
2016-09-22 00:08:49,927 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:372)] Post validation configuration for supply_agent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[supply_agent]
SOURCES: {avro-source1={ parameters:{bind=0.0.0.0, channels=ch1, port=41414, type=avro} }}
CHANNELS: {ch1={ parameters:{type=memory} }}
AgentConfiguration created with Configuration stubs for which full validation was performed[supply_agent]
SINKS: {log-sink1=ComponentConfiguration[log-sink1]
  CONFIG:
    CHANNEL:ch1
}

2016-09-22 00:08:49,927 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:136)] Channels:ch1

2016-09-22 00:08:49,927 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:137)] Sinks log-sink1

2016-09-22 00:08:49,927 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:138)] Sources avro-source1

2016-09-22 00:08:49,927 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:141)] Post-validation flume configuration contains configuration for agents: [supply_agent]
2016-09-22 00:08:49,928 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:145)] Creating channels
2016-09-22 00:08:49,933 (conf-file-poller-0) [INFO - org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)] Creating instance of channel ch1 type memory
2016-09-22 00:08:49,936 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:200)] Created channel ch1
2016-09-22 00:08:49,937 (conf-file-poller-0) [INFO - org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)] Creating instance of source avro-source1, type avro
2016-09-22 00:08:49,953 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)] Creating instance of sink: log-sink1, type: logger
2016-09-22 00:08:49,955 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel ch1 connected to [avro-source1, log-sink1]
2016-09-22 00:08:49,960 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:138)] Starting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:Avro source avro-source1: { bindAddress: 0.0.0.0, port: 41414 } }} sinkRunners:{log-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2d9e85b0 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} }
2016-09-22 00:08:49,969 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:145)] Starting Channel ch1
2016-09-22 00:08:50,021 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean.
2016-09-22 00:08:50,021 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: CHANNEL, name: ch1 started
2016-09-22 00:08:50,021 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:173)] Starting Sink log-sink1
2016-09-22 00:08:50,022 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:184)] Starting Source avro-source1
2016-09-22 00:08:50,022 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:228)] Starting Avro source avro-source1: { bindAddress: 0.0.0.0, port: 41414 }...
2016-09-22 00:08:50,023 (SinkRunner-PollingRunner-DefaultSinkProcessor) [DEBUG - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:143)] Polling sink runner starting
2016-09-22 00:08:50,254 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean.
2016-09-22 00:08:50,254 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SOURCE, name: avro-source1 started
2016-09-22 00:08:50,254 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.AvroSource.start(AvroSource.java:253)] Avro source avro-source1 started.


2016-09-22 00:09:50,023 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/flume.conf for changes




PART 3 - publish to avro source listening on port 41414

Running an Avro client, sends either a file or data from stdin to a specified host and port where a Flume NG Avro Source is listening.

/var/log/supply_source.log

{"timemillis" : 2876873673, "correlation" : 1, item : "pants"}
{"timemillis" : 8347583748, "correlation" : 2, item : "shirts"}

bin/flume-ng avro-client --conf conf -H localhost -p 41414 -F /var/log/supply_source.log -Dflume.root.logger=DEBUG,console

Now, the sink receives two events.

2016-09-22 00:36:36,610 (New I/O server boss #1 ([id: 0x4ca942fa, /0:0:0:0:0:0:0:0:41414])) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x49d18056, /127.0.0.1:50833 => /127.0.0.1:41414] OPEN
2016-09-22 00:36:36,610 (New I/O  worker #4) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x49d18056, /127.0.0.1:50833 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
2016-09-22 00:36:36,610 (New I/O  worker #4) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x49d18056, /127.0.0.1:50833 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:50833
2016-09-22 00:36:36,802 (New I/O  worker #4) [DEBUG - org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:371)] Avro source avro-source1: Received avro event batch of 2 events.

2016-09-22 00:36:36,802 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 7B 22 63 6F 72 72 65 6C 61 74 69 6F 6E 22 20 3A {"correlation" : }
2016-09-22 00:36:36,802 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 7B 22 63 6F 72 72 65 6C 61 74 69 6F 6E 22 20 3A {"correlation" : }
2016-09-22 00:36:36,817 (New I/O  worker #4) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x49d18056, /127.0.0.1:50833 :> /127.0.0.1:41414] DISCONNECTED
2016-09-22 00:36:36,817 (New I/O  worker #4) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x49d18056, /127.0.0.1:50833 :> /127.0.0.1:41414] UNBOUND
2016-09-22 00:36:36,817 (New I/O  worker #4) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x49d18056, /127.0.0.1:50833 :> /127.0.0.1:41414] CLOSED

2016-09-22 00:36:36,818 (New I/O  worker #4) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed(NettyServer.java:209)] Connection to /127.0.0.1:50833 disconnected.

Reference

docker flume example, https://github.com/prayagupd/docker-flume

Parallel application using OpenMP/ MacOS

brew install clang-omp
xcode-select --install

$ clang-omp --version
clang version 3.5.0
Target: x86_64-apple-darwin15.4.0
Thread model: posix

$ xcode-select --version
xcode-select version 2343.

application

#include <omp.h>
#include <stdio.h>

#define n 20                                                                                          

omp_set_num_threads(3);                                                                              
#pragma omp parallel for private(tid) schedule(static,1)                                              

for (i=0; i<n; i++) {                                                                                
 tid = omp_get_thread_num();                                                                          
 printf("Thread %d executing iteration %d\n", tid, i);                                                
}

$ clang-omp -fopenmp parallel.c 

Mac OS - command line hacks

processor bits

"processor bits" means that how much data a microprocessor will process within one instruction cycle i.e. fetch-decode-execute

32-bit means that a microprocessor can execute 4 bytes of data in one instruction cycle

while 64-bit means that a microprocessor executes can execute 8 bytes of data in one instruction cycle.

$ sysctl hw | grep cpu #sysctl hw.cpu64bit_capable
hw.ncpu: 8
hw.activecpu: 8
hw.physicalcpu: 4
hw.physicalcpu_max: 4
hw.logicalcpu: 8
hw.logicalcpu_max: 8
hw.cputype: 7
hw.cpusubtype: 8
hw.cpu64bit_capable: 1
hw.cpufamily: 280134364
hw.cpufrequency: 2800000000
hw.cpufrequency_min: 2800000000
hw.cpufrequency_max: 2800000000
hw.cputhreadtype: 1

$ getconf LONG_BIT
64

The number of bits in a processor refers to the size of the data types that it handles and the size of its registry. Simply put, a 64-bit processor is more capable than a 32-bit processor because it can handle more data at once.

CPU info

sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz

$ system_profiler | grep Processor
    Apple: AUDynamicsProcessor:
    Apple: AUVoiceProcessor:
      Processor Name: Intel Core i7
      Processor Speed: 2.8 GHz
      Number of Processors: 1
    <key>BootCampProcessorPstates</key>
    |         "BootCampProcessorPstates" = <0a00>

no of cores

$ sysctl -n hw.ncpu
8

IP address of Machine

$ ipconfig getifaddr en0
10.11.208.214

or

$ ifconfig | grep inet
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet6 fe80::c6b3:1ff:fed3:d771%en0 prefixlen 64 scopeid 0x4
inet 10.11.208.214 netmask 0xfffffe00 broadcast 10.11.209.255
inet6 fe80::7023:7bff:fee3:61e2%awdl0 prefixlen 64 scopeid 0x9

Change password using DsCl (Directory Service CLI)

dscl . -passwd /Users/prayagupd
New Password: dreams
passwd: DS error: eDSAuthPasswordTooShort

<dscl_cmd> DS Error: -14170 (eDSAuthPasswordTooShort)

Note

$ man dscl | grep "passwd"

     Usage: passwd user_path [new_pasword | old_password new_pasword]

Uninstall application

mdfind -name "Skype"
/Applications/Skype.app

sudo rm -rf /Applications/Skype.app

add a printer (lpadmin)

find the IP address of the printer, and then add it to macos printers.

http://apple.stackexchange.com/q/126834/76240


printer stat

lpstat -p
printer _10_10_10_113 is idle.  enabled since Thu Oct 20 17:23:31 2016
 Waiting for printer to finish.
printer _10_20_50_100 is idle.  enabled since Fri Sep  9 15:39:45 2016

RAM Memory usage

$ vm_stat 
Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                              264375.
Pages active:                           2027888.
Pages inactive:                          651659.
Pages speculative:                        23565.
Pages throttled:                              0.
Pages wired down:                        852647.
Pages purgeable:                          39328.
"Translation faults":                 880109291.
Pages copy-on-write:                   44871249.
Pages zero filled:                    433518664.
Pages reactivated:                      5346396.
Pages purged:                           1423915.
File-backed pages:                       379668.
Anonymous pages:                        2323444.
Pages stored in compressor:             2870604.
Pages occupied by compressor:            372004.
Decompressions:                        33640670.
Compressions:                          43459639.
Pageins:                               10427527.
Pageouts:                                 34549.
Swapins:                               32669611.
Swapouts:                              34330610.

install *.dmg from command line in OSX

Installing

STEP 1: mount the apple disk image file as /Volumes

hdiutil mount ideaIC-15.0.2-custom-jdk-bundled.dmg 
----
----
/dev/disk2          GUID_partition_scheme          
/dev/disk2s1        Apple_HFS                      /Volumes/IntelliJ IDEA 15 CE

STEP 2: copy the mounted *.app to /Apps

sudo cp -r "/Volumes/IntelliJ IDEA 15 CE/IntelliJ IDEA 15 CE.app" /Applications

If the mounted file is *.pkg instead of *.app, then 

$ sudo installer -pkg /Applications/chefdk-1.0.3-1.pkg -target /
installer: Package name is Chef Development Kit
installer: Upgrading at base path /
installer: The upgrade was successful.

STEP 3: unmount the file from /Volumes

hdiutil unmount "/Volumes/IntelliJ IDEA 15 CE"
"/Volumes/IntelliJ IDEA 15 CE" unmounted successfully.

OR

sudo cp -r ggts-bundle/ggts-3.6.4.RELEASE/GGTS.app/ /Applications/

Uninstalling

remove from /Apps

$ ll /Applications/IntelliJ\ IDEA*
/Applications/IntelliJ IDEA CE.app:
total 0
drwxr-xr-x@ 12 prayagupd  admin  408 Jan 21 00:33 Contents

/Applications/IntelliJ IDEA.app:
total 0
drwxr-xr-x@ 13 prayagupd  admin  442 Dec 25 09:21 Contents

rm -rf /Applications/IntelliJ\ IDEA\ CE.app/

Scala Future with play framework

Async controller
----------------------------

package beard.controllers

import play.api.mvc._
import play.api.libs.json._
import play.api.libs.functional.syntax._
import play.libs.Json._
import reactivemongo.bson._
import service.BeardService
import play.api.libs.concurrent.Execution.Implicits.defaultContext

import scala.collection.mutable
import scala.collection.mutable.ListBuffer
import scala.concurrent.Future
import scala.util.{Failure, Success, Random}

class AsyncBeardController extends Controller {

 def sayAsyncBeard = Action.async { request =>
    val futureResult = Future {
      intensiveComputation()
    }
    futureResult.map(result =>
      Ok(result)
    )
  }


def intensiveComputation(): JsObject = {
      Thread.sleep(Random.nextInt(5000))
        Json.obj("value" -> "beard")
  }
}

Test
-------

$ curl -XGET http://localhost:9000/sayAsyncBeard
{"value":"beard"}

You will get the response after ~5seconds.

CPU cores in OSX

Seems OSX sucks in command line support as Linux does. However, found few commands to get hw information in OSX.

sysctl -n hw.ncpu
8

sysctl -n hw.physicalcpu
4

sysctl -n hw.logicalcpu
8


Ref : https://coolaj86.com/articles/get-a-count-of-cpu-cores-on-linux-and-os-x/