Understanding Unix - Part 2 (for programmers)

This post is a sequel of previous post Linux Hacks Part 1 mostly important for programmers.


hack 1 => copy text to the clip(board)
Copy text from file Service.groovy,

$ xclip -selection clipboard grails-app/domain/eccount/Service.groovy

Paste to a new file ServiceNew.groovy

$ vi grails-app/domain/eccount/ServiceNew.groovy

Go INSERT mode and Ctrl+Shift+V

OR

gg"*yG

Reference  :
xclip Does Copy-and-Paste on the Linux Command Line, http://www.linuxplanet.com/linuxplanet/tips/6788/1
VI editor: Copy all the lines to clipboard,  http://stackoverflow.com/a/1620029/432903

hack 2 => find files with a name inside a directory
find searches in the real system.
is slower but always up-to-date and has more options (size, modification time,...)

eg. 1

loohcs$ find . -name "*Service*"
./test/unit/eccount/StudentServiceTests.groovy
./test/unit/eccount/UserServiceTests.groovy
./grails-app/domain/eccount/Service.groovy
./grails-app/domain/eccount/ServiceTransaction.groovy
./grails-app/services/eccount/TransactionService.groovy
./grails-app/services/eccount/UserService.groovy

eg. 2

$ find /packup/repo.softwares/ -name "*cassandra*"
/packup/repo.softwares/JVM/SolrLuceneES-BigData/apache-cassandra-2.0.6-bin.tar.gz
/packup/repo.softwares/JVM/SolrLuceneES-BigData/apache-cassandra-2.0.7-bin.tar.gz

Reference 10:
10 find command in unix examples basic, http://javarevisited.blogspot.com/2011/03/10-find-command-in-unix-examples-basic.html

What is the different between 'locate' and 'find' in Linux?, http://superuser.com/a/199473

hack 3 => awk (intrepreted prog lang)
find the total number of lines in a file (without using NR)
prayag@prayag:~/hacker_/loohcs$ awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' grails-app/controllers/LoginController.groovy 
134

hack 4 => SED (Stream EDitor) 
4.1) replace the word "ServiceTransaction" with "Transaction" in file


Stream editor is Stream editor not file editor.

root@0cad383c1249:/# sed --version
sed (GNU sed) 4.4
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Jay Fenlason, Tom Lord, Ken Pizzini,
and Paolo Bonzini.
GNU sed home page: <http://www.gnu.org/software/sed/>.
General help using GNU software: <http://www.gnu.org/gethelp/>.
E-mail bug reports to: <bug-sed@gnu.org>.

prayag@prayag:~/hacker_/loohcs$ sed s/ServiceTransaction/Transaction/ < TransactionService.groovy

## OR 
$ echo "Mbr Hlth Screening Mammogram" | sed s/Hlth/Health/ | sed s/Mbr/Member/
Member Health Screening Mammogram


## OR replace a word with special character containing $
sed --in-place 's/shaharma/\$ORGANISATION_NAME/' deployment.sh

4.2) delete empty lines

$ sed '/^$/d' src/main/resources/MoneyReply.xml > src/main/resources/MoneyReplyOut.xml

Practice more at Sed Interview Questions

4.3) remove certain chars

## Following is a script to remove character ) 
$ sed 's/[),]//g' Diagnosis.html

## remove \ character in serialised json,
sed 's/[\\]//g' < data

4.4) replace a line with certain content
Syntax -
sed -i '/TEXT_OF_A_LINE_TO_BE_REPLACED/c\prayagupd just changed this line.' FILE_NAME

eg.

sed --in-place '/Dvbox/c\   "-Dvbox.home=/usr/lib/virtualbox"' project.clj

on macos,

sed -i.bu 's/Dvbox/yo/' project.clj

grep using sed
sed -n '/Chrome/p' < apple.log | tee newApple.log

hack 5 => ssh connect

$ ssh -v uatestinguser@10.13.222.123

Copy file from remote linux server to local machine

$ scp -v uatestinguser@10.13.222.250:/tmp/analytics.csv /home/prayag/

Reference reading : 
Section 1.2: How do SSH, Telnet and Rlogin differ?,  http://cc-ipcp.icp.ac.ru/Section1.2.html
The SSH (Secure Shell) vs RemoteLogin Protocol, http://www.slideshare.net/sourav894/ssh-and-rlogin
3.3 Setting up permissions for rsh, rlogin and rcp(.rhosts), http://rmni.iqfr.csic.es/guide/man/lynux/chap3-3.htm

hack 6 > ls 
list files sorted by date in desc order
$ ls -tr

hack 7 => length of a string

(length of Long.MAX_VALUE in java)

echo "9223372036854775807" | wc -c

      20

$ expr length '$2a$10$osjPAbnn9ogMsY7z.Q7rWe2FLTx/C9Yi6.mj74aTbXWC3iyR1p/0q'
60

hack 8 => find a file with particular text 
The grep syntax looks as below.

$ grep -r "<SEARCH_TEXT>" <DIRECTORY_PATH>

$ grep -r "navbar-merchant" nioc2egdelonk/
nioc2egdelonk/web-app/less/nav.less:.navbar-admin {

hack 9 : know gtk version

$ dpkg -l libgtk[0-9]* | grep ^i
ii  libgtk2-perl                           2:1.223-1build3                         Perl interface to the 2.x series of the Gimp Toolkit library
ii  libgtk2.0-0                            2.24.10-0ubuntu6                        GTK+ graphical user interface library
ii  libgtk2.0-bin                          2.24.10-0ubuntu6                        programs for the GTK+ graphical user interface library
ii  libgtk2.0-common                       2.24.10-0ubuntu6                        common files for the GTK+ graphical user interface library

hack 10 : install gtk-recordmydesktop

$ sudo apt-get install gtk-recordmydesktop

Configure laptop settings as mentioned at recordMyDesktop output video shows *massive* artifacts, else plays too fast lowering resolution to to 1360x768.
$ xrandr -s 1360x768

Start recordmydesktop 

$ gtk-recordmydesktop -o /home/prayag/Videos/gwt_screencast.ogv

Convert *.ogv to *.flv with same-quality (size will increase though)

$ ffmpeg -sameq -i /home/prayag/Videos/gwt_screencast.ogv /home/prayag/Videos/gwt_screencast.flv

hack 11 : Zip a file

$  zip -r eccount-hg *

References
How can I create launchers on my desktop?, http://askubuntu.com/a/64237/37643
Csv to Html Table, https://gist.github.com/prayagupd/8050493

Understanding MySQL Joins

References
What's the difference between INNER JOIN, LEFT JOIN, RIGHT JOIN and FULL JOIN, http://stackoverflow.com/a/6188334/432903

map reduce in ruby

Install ruby

$ irb
.map()
Invokes the given {block} once for each element of self, creating a new array with the values returned by the block.
Documentation here.

1.9.3-p448 :007 > patients = ['Prayag', 'Paras', 'James', 'Angelina', 'Scarlette', 'Christian']
 => ["Prayag", "Paras", "James", "Angelina", "Scarlette", "Christian"] 

1.9.3-p448 :008 > patients.map {|patient| patient.upcase}
 => ["PRAYAG", "PARAS", "JAMES", "ANGELINA", "SCARLETTE", "CHRISTIAN"] 

1.9.3-p448 :010 > patients.map {|patient| patient.reverse}
 => ["gayarP", "saraP", "semaJ", "anilegnA", "ettelracS", "naitsirhC"] 

1.9.3-p448 :011 > patients.map {|patient| patient.size}
 => [6, 5, 5, 8, 9, 9]


.reduce()
Combines all elements of enum by applying a binary operation specified by a {block}.
Documentation here.

1.9.3-p448 :012 > rainbow.reduce(0) {|acc, n| acc += 1}
 => 30 

1.9.3-p448 :014 > patients.reduce(0) {|acc, n| acc += n.length}
 => 42 

1.9.3-p448 :016 > patients.reduce([]) {|acc, n| acc.push(n.upcase)}
 => ["PRAYAG", "PARAS", "JAMES", "ANGELINA", "SCARLETTE", "CHRISTIAN"] 


References
Understanding Ruby’s Select, Map, and Reduce, http://feynmanliang.com/?p=319
Understanding map and reduce, http://railspikes.com/2008/8/11/understanding-map-and-reduce

Hibernate Madness

Hibernate is db agnostic(=db generalised) Java ORM invented by Gavin King being frustrated working with EJB 2 style entity beans and brittle handwritten persistence layers. (Interview with Gavin King, founder of Hibernate, 2009)


automatic dirty checking
a feature of hibernate that saves time and effort to update the database when states of objects are modified inside a transaction.
All persistent objects are monitored by hibernate. It detects which objects have been modified and then calls update statements on all updated objects.

Example at Automatic Dirty Checking

[1] hql join @CollectionTable

For an @Entity Service having a property tags (as a @CollectionTable), 

@Entity

public class Service extends AbstractEntity<Long> {

            private static final long serialVersionUID = 9116959642944725990L;

        @ElementCollection(fetch = FetchType.EAGER, targetClass = java.lang.String.class)

        @CollectionTable(name = "service_tags", joinColumns = @JoinColumn(name = "s_id"))

        @Column(name = "tag")

        private Set<String> tags;

    }

To get Services with tags.tag="Telecome", 

select s from Service s INNER JOIN s.tags t where s.status=0 and and VALUE(s.tags) in ('Telecome'))

[2] hibernate spring query_cache

STEP 1 ADD EhCache 3.3.1 dependency as a cache provider
Add repository to artifactory.

Then add dependency to build.gradle as below : 
description 'eccount-core'


configurations {
all*.exclude module: 'commons-logging'
all*.exclude module: 'log4j'
all*.exclude module: 'icu4j'
all*.exclude module: 'catalina'

}
dependencies {

       compile 'org.hibernate:hibernate-ehcache:3.3.1.GA'
}

STEP 2 ADD hibernate query caching conf to resources/hibernate.cfg.xml


<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
        "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
        "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
    <session-factory>
        <property name="hibernate.default_batch_fetch_size">16</property>
        <property name="hibernate.max_fetch_depth">5</property>

        <property name="hibernate.cache.use_query_cache">true</property>
        <property name="hibernate.cache.use_second_level_cache">true</property>
        <property name="hibernate.generate_statistics">true</property>
        <property name="hibernate.cache.provider_class">org.hibernate.cache.EhCacheProvider</property> 
<property name="hibernate.cache.provider_configuration_file_resource_path">ehcache.xml</property>
        <property name="hibernate.c3p0.timeout">300</property>

    </session-factory>
</hibernate-configuration>

STEP 3 ADD resources/ehcache.xml


<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">

<diskStore path="/home/prayag/cache_"/>
<defaultCache
        eternal="false"
        maxElementsInMemory="1000"
        overflowToDisk="true"
        diskPersistent="true"
        timeToLiveSeconds="300"
        />
</ehcache>



STEP 4 ADD Spring's CacheManager conf to WEB-INF/jpa-context.xml


<?xml version="1.0" encoding="UTF-8"?>
<beans
    xmlns:p="http://www.springframework.org/schema/p" 
    xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:cache="http://www.springframework.org/schema/cache" 
xmlns:mvc="http://www.springframework.org/schema/mvc"
xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.0.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx.xsd
http://www.springframework.org/schema/cache
        http://www.springframework.org/schema/cache/spring-cache.xsd"
default-autowire="byName">

<!-- Scans within the base package of the application for @Components to 
configure as beans -->

<bean id="entityManagerFactory"
class="com.zazzercode.server.jpa.monitor.JavamelodyContainerEntityManagerFactoryBean">
<property name="jpaVendorAdapter">
<bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="showSql" value="${db.showSql}" />
<property name="generateDdl" value="${db.generateDdl}"/>
<property name="databasePlatform" value="${db.dialect}" />
</bean>
</property>
<property name="persistenceUnitName" value="${db.persistenceUnit}"/>
</bean>

<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
<property name="driverClass" value="${db.driver}"/>
<property name="jdbcUrl" value="${db.url}"/>
<property name="user" value="${db.username}"/>
<property name="password" value="${db.password}"/>
<property name="minPoolSize" value="${db.poolsize.min}"/>
<property name="maxPoolSize" value="${db.poolsize.max}"/>
<property name="maxStatementsPerConnection" value="15"/>
<property name="idleConnectionTestPeriod" value="3000"/>
</bean>

<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactory"></property>
</bean>

<tx:annotation-driven />
<cache:annotation-driven />

<bean
class="org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor" />

     <!-- generic cache manager -->
<bean id="cacheManager" class="org.springframework.cache.support.SimpleCacheManager">
 <property name="caches">
    <set>
     <bean class="org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean" p:name="merchantServices"/>
   </set>
      </property>
</bean>

</beans>


STEP 5 ADD Spring's @Cacheable to the Springdata Repository method 


@Cacheable("merchantServices")
@Query("select s from Service s JOIN s.statusList sas where s.status=?1 and s.priviligedUser.priviligedUserType IN (2,4) and s.id IN (?2) and sas.active=true and sas.transactorType=?3 ORDER BY s.name")
List<Service> getAllMerchantServicesByStatusAndServiceId(ServiceStatus status, List<Long> services, TransactorType transactor);


STEP 6 Check diskStore
Cache elements are stored with a key.
In diskStore, I see following files

prayag@prayag:~/cache_$ ls -l
total 0
-rw-rw-r-- 1 prayag prayag 0 Sep 20 13:31 org.hibernate.cache.StandardQueryCache.data
-rw-rw-r-- 1 prayag prayag 0 Sep 20 13:31 org.hibernate.cache.StandardQueryCache.index
-rw-rw-r-- 1 prayag prayag 0 Sep 20 13:31 org.hibernate.cache.UpdateTimestampsCache.data
-rw-rw-r-- 1 prayag prayag 0 Sep 20 13:31 org.hibernate.cache.UpdateTimestampsCache.index


Reference
Spring Data Repository caching results, http://stackoverflow.com/q/17896118/432903

How to configure Hibernate statistics in Spring 3.0 application, http://stackoverflow.com/a/6708644/432903

http://raibledesigns.com/wiki/Wiki.jsp?page=HibernateJMX

Caching with Hibernate + Spring - some Questions!,  http://gregluck.com/blog/archives/2007/05/comparing-memcached-and-ehcache-performance/

Caching with Hibernate + Spring - some Questions!, http://stackoverflow.com/a/5405971/432903

HQL Where IN for empty list crashes, http://stackoverflow.com/a/8453646/432903

http://stackoverflow.com/q/17410703/432903

mongodb hacks

connect to mongod server using ssh, 

use mongo client to connect to server, choose database;

> use recommendation;
switched to db recommendation

> db.getCollectionNames()

[ "movies", "system.indexes" ]

or

> show collections;

movies

system.indexes

Indexing
---------------------------------------------

> db.movies.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "recommendation.movies"
}
]

> db.movies.stats()
{
"ns" : "recommendation.movies",
"count" : 967696,   //100K
"size" : 232274176, // 232.274 M
"avgObjSize" : 240,
"storageSize" : 335,900672,
"numExtents" : 14,
"nindexes" : 1,
"lastExtentSize" : 92585984,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 32,818464,
"indexSizes" : {
"_id_" : 32,818464
},
"ok" : 1
}

[1.1] filter/ projection

db.movies.find({"title":"Apollo 13 (1995)"});

{

"_id" : ObjectId("55599a61f84a9472b69e2a42"),

"movieId" : 28,

"title" : "Apollo 13 (1995)",

"similarId" : 729,

"similarTitle" : "Nell (1994)",

"regularizedCorRelation" : 0.20162457556082056

}

{

"_id" : ObjectId("55599a69f84a9472b69e39f8"),

"movieId" : 28,

"title" : "Apollo 13 (1995)",

"similarId" : 809,

"similarTitle" : "Rising Sun (1993)",

"regularizedCorRelation" : 0.17383996759460163

}

db.movies.find({"title":"Apollo 13 (1995)", 

                "similarMovies.regularizedCorRelation" : 
                           {$gte : 0.09}

               });

{

"_id" : ObjectId("55599a61f84a9472b69e2a42"),

"movieId" : 28,

"title" : "Apollo 13 (1995)",
 "similarMovies" : 
 {

"similarId" : 729,

"similarTitle" : "Nell (1994)",

"regularizedCorRelation" : 0.20162457556082056

  },

  {

"similarId" : 809,

"similarTitle" : "Rising Sun (1993)",

"regularizedCorRelation" : 0.17383996759460163
  }

}

[1.2] sort (_id) in descending order
> db.payment_transactios.find().sort({_id:-1});

[1.3] limit 
> db.payment_transactios.find().sort({_id:-1}).limit(1);

[1.4] prettify mongodb query result

> db.payment_transactios.find().sort({_id:-1}).limit(1).pretty();

OR

> db.payment_transactios.find().sort({_id:-1}).limit(1).forEach(printjson);

OR

> db.payment_transactios.find().sort({_id:-1}).limit(1).toArray();

OR add following conf to ~/.mongorc.js

DBQuery.prototype._prettyShell = true

Aggregation
----------------------------------------

//get DISTINCT movie id
> db.movies.aggregate( [ { $group : { _id : "$movieId" } } ] )
{ "_id" : 1596 }
{ "_id" : 1309 }
{ "_id" : 1672 }
{ "_id" : 1679 }
{ "_id" : 1463 }
{ "_id" : 1510 }
{ "_id" : 1650 }
{ "_id" : 1523 }
{ "_id" : 1430 }
{ "_id" : 1482 }
{ "_id" : 1500 }
{ "_id" : 1432 }
{ "_id" : 1504 }
{ "_id" : 711 }
{ "_id" : 1662 }
{ "_id" : 1156 }
{ "_id" : 1420 }
{ "_id" : 1130 }
{ "_id" : 1577 }
{ "_id" : 1403 }

// sort the DISTINCT movie ids
> db.movies.aggregate( [ { $group : { _id : "$movieId" } }, {$sort :{_id:1}} ] )
{ "_id" : 1 }
{ "_id" : 2 }
{ "_id" : 3 }
{ "_id" : 4 }
{ "_id" : 5 }
{ "_id" : 6 }
{ "_id" : 7 }
{ "_id" : 8 }
{ "_id" : 9 }
{ "_id" : 10 }
{ "_id" : 11 }
{ "_id" : 12 }
{ "_id" : 13 }
{ "_id" : 14 }
{ "_id" : 15 }
{ "_id" : 16 }
{ "_id" : 17 }
{ "_id" : 18 }
{ "_id" : 19 }

{ "_id" : 20 }

// count by moviedId
db.movies.aggregate([{
              $group : {
                 _id : "$movieId", 
                 count: { $sum:1 }
              }}]);

{ "_id" : 1596, "count" : 1 }
{ "_id" : 1309, "count" : 1 }
{ "_id" : 1672, "count" : 1 }
{ "_id" : 1679, "count" : 1 }
{ "_id" : 1463, "count" : 1 }
{ "_id" : 1510, "count" : 1 }
{ "_id" : 1650, "count" : 1 }
{ "_id" : 1523, "count" : 1 }
{ "_id" : 1430, "count" : 4 }
{ "_id" : 1482, "count" : 1 }
{ "_id" : 1500, "count" : 1 }
{ "_id" : 1432, "count" : 4 }
{ "_id" : 1504, "count" : 2 }

...

References
How do I drop a MongoDB database, from the command line?, http://stackoverflow.com/a/8857323/432903

Troubleshoot the Map Function, http://docs.mongodb.org/manual/tutorial/troubleshoot-map-function/

Mongo - get occurrence of lastnames, http://stackoverflow.com/a/5544226/432903

Javascript and MapReduce, http://jcla1.com/blog/2013/05/11/javascript-mapreduce/

http://www.mongovue.com/2010/11/03/yet-another-mongodb-map-reduce-tutorial/

Windows CLI Hacks

[1] know OS  version : 32 bit or 64 bit
C:/>wmic os get osarchitecture
OSArchitecture
32bit

OR simply check the PROCESSOR_ARCHITECTURE environment variable
C:\>echo %PROCESSOR_ARCHITECTURE%


References

http://superuser.com/a/359655/107419

Making Ncell Connect work for linux

Making Ncell Connect work for linux

STEP 1 : install usb-modeswitch
USB_ModeSwitch is a mode switching tool for controlling "flip flop" (multiple device) USB gear.

$ sudo -s
$ apt-get install usb-modeswitch

STEP 2 : cat rules for ncell-connect

root@prayag:~# cat > /etc/udev/rules.d/99-ncell-connect.rules <<EOT
 # Ncell Connect switch mode to usbserial (E1550 device id => 12d1:1446)
 SUBSYSTEM=="usb", ATTRS{idProduct}=="1446", ATTRS{idVendor}=="12d1", RUN+="/lib/udev/modem-modeswitch --vendor 0x12d1 --product 0x1446 --type option-zerocd"
 EOT

STEP 3 configure network provider

Insert the device and goto Network settings at right corner of ubuntu 12.04.

Create a broadband connection under Mobile Broadband.

Then select country as Nepal, network provider as Mero Mobile, and billing plan as default.

REFERENCES
[1] Mobile Broadband Set-up In Ubuntu 11.04
http://www.muktware.com/1063/mobile-broadband-set-ubuntu-1104

[2] How to auto connect Ubuntu 12.04 USB GSM Mobile Broadband Connection on Boot as startup service without user login using Network Manager CLi,
http://www.thefanclub.co.za/how-to/how-auto-connect-ubuntu-1204-gsm-mobile-broadband-connection-on-boot-startup-service

[3] SMS From Ncell Connect Data SIM Card And Ubuntu How To, (September 7, 2011)
http://techspalace.blogspot.com/2011/09/sms-from-ncell-connect-data-sim-card.html