hash functions/ cryptographic hash functions

hash functions/ cryptographic hash functions

a mathematical algorithm that maps data of arbitrary size to a bit string of a fixed size (a hash function)
which is designed to also be a one-way function

secure hash(s-ha)

SHA-1 produces a 160-bit (20-byte) hash value known as a message digest.
A SHA-1 hash value is typically rendered as a hexadecimal number, 40 digits long = 40 digits/2 bytes.

$ echo "1" | shasum -a 1
e5fa 44f2 b31c 1fb5 53b6 021e 7360 d07d 5d91 ff5e  -

$ echo "11" | shasum -a 1
dd71 038f 3463 f511 ee74 03db cbc8 7195 302d 891c  -

$ printf 'dd71038f3463f511ee7403dbcbc87195302d891c' | wc -c
      40

$ shasum downloadData.log -a 1
f500ddd45af385b3bbdffdc3457701bf5b9a37a1  downloadData.log

Java security API

scala> val hash = java.security.MessageDigest.getInstance("SHA-1").digest("1".getBytes())
x: Array[Byte] = Array(53, 106, 25, 43, 121, 19, -80, 76, 84, 87, 77, 24, -62, -115, 70, -26, 57, 84, 40, -85)

scala> new String(hash)
res5: String = 5j?+y?�LTWM?�F�9T(�

scala> import java.math.BigInteger
import java.math.BigInteger

scala> new BigInteger(java.security.MessageDigest.getInstance("SHA-1").digest("1".getBytes()))
res8: java.math.BigInteger = 304942582444936629325699363757435820077590259883

//distributing in a cluster of 10 nodes

scala> new BigInteger(java.security.MessageDigest.getInstance("SHA-1").digest("1".getBytes())).mod(BigInteger.valueOf(10))
res19: java.math.BigInteger = 3

40 bytes Hex representation

scala> new BigInteger(java.security.MessageDigest.getInstance("SHA-1").digest("1".getBytes())).toString(16)
res9: String = 356a192b7913b04c54574d18c28d46e6395428ab

Merkle–Damgård 5

The MD5 algorithm is a widely used hash function producing a 128-bit hash value.
Used for checksum match against a file, but is very vulnerable with collisions within seconds.

Following is BSD example, Linux has md5sum.

$ echo -n '1' | openssl md5
c4ca4238a0b923820dcc509a6f75849b

$ echo -n "1" | md5
c4ca4238a0b923820dcc509a6f75849b

$ echo "1" | md5 # don't do this, as includes \n
b026324c 6904b2a9 cb4b88d6 d61c81d1 ##is a [hex number, compressed to base16](http://stackoverflow.com/q/43556742/432903)

#hash bytes
printf "%s" "c4ca4238a0b923820dcc509a6f75849b" | wc -c
      32
# hash bits = (32/2=16bytes)*8 = 128bits

## another way
$ md5 <<<"1"
b026324c6904b2a9cb4b88d6d61c81d1

$ md5 build.sbt 
MD5 (build.sbt) = d10c 6aff 431a 61c5 b3bd 1a03 8519 900c

Java MD5 API

scala> val hash = MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8"))
hash: Array[Byte] = Array(-60, -54, 66, 56, -96, -71, 35, -126, 13, -52, 80, -102, 111, 117, -124, -101)

scala> val hash = MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8")).length
hash: Int = 16

scala> import java.math.BigInteger
import java.math.BigInteger

scala> new BigInteger(1, MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8")))
res5: java.math.BigInteger = 261578874264819908609102035485573088411 //length 39

scala> new BigInteger(1, MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8"))).toString(16)
res6: String = c4ca4238a0b923820dcc509a6f75849b //length 32

scala> String.format("%032x", new BigInteger(1, MessageDigest.getInstance("MD5").digest("1".getBytes("UTF-8"))))
res7: String = c4ca4238a0b923820dcc509a6f75849b

Usage

1) data distribution/partitioning
http://docs.aws.amazon.com/streams/latest/dev/key-concepts.html

2) How do I ensure data integrity of objects uploaded to or downloaded from Amazon S3? - eTag/ Content-MD5

References

online app - http://www.xorbin.com/tools/sha1-hash-calculator