Erasure Code

In Distributed File Systems, the Erasure code is the engine behind achieving redundancy to tolerate against node failures.

It split the block into k units and using some coding (Reed Solomon) it generates more m units. to illustrate let’s assume we want to tolerate two nodes failures and have a total of 6 nodes. This means we split data into four units ( k units ) and do a mathematical operation on these four units to have more two units ( m units), sometimes we call it parity units.

so ,original data is ( k1,  k2, k3, k4) and parity data are (m1 , m2 ) here m1=F(d1,d2,d3,d4) and m2=F(k1,k2,k3,k4). following this scenario here we have six nodes each node will store one unit

node1 stores (k1) , node2 stores (k2)……….. node5 stores(m1), and node6 stores(m2).

if node1 became unavailable,we use remaining nodes to generate lost data by encoding one of node5 or node6 data plus surviving data nodes (node 1 to 4). This is cpu expensive , Disk I/O, and also high network bandwidth is utilized. CPU is used for encoding , Disk I/O is used to read data, while network is used to download data to the new node

enhancement is being done to modify algorithms, like Clay( Couple Layered) Reed Solomon and also converting traditional MDS to MRS ( Minimum Storage Regenerated Code) which will use fewer nodes to generate lost data.

Hadoop DFS will use replication for hot data ( more frequently accessed) and erasure code for cold data( less-frequently accessed). an operation called MapReduce is used to convert from replication-state to erasure-code state and then delete replicated data.

read this research from Berkley university that is really informative.

 

Click to access EECS-2016-155.pdf

 

ACID , CAP , and BASE

Quick summary :

Relational DBs(Oracle , MySQL ,) depend on ACID .

NoSQL DBs (AWS DynamoDB, Cassandra, and Google BigTable)depends on the updated version of CAP, known as BASE.

More Details:

In DB world old days, as the relational database was the dominance one. All RDBMs followed the ACID model as following :

Atomicity: a transaction is treated as a single unit, all statements within a transaction are executed, if any statement fails, all are rolled back and DB is left unchanged

Consistency: all data written to DB is validated against defined rules ( integer fileds, decimal fields, positive numbers..etc )

Isolation: it executes concurrent transactions as if they are done in sequential. Two transactions ( T1,T2),if T1 fails, will not affect T2

Durability: a committed transaction is protected against system failures. If you commit and receives success,it is done. The DB system must maintain the protection of data against any failure ( power failures, disk failure, crash..etc).

With evolving of the distributed computing system, the ACID model is no longer valid. Eric Brewer made his theory and named it CAP (Consistency, Availability, Partition Tolerance ), in which no single system can achieve all three at the same time

Consistency: every read will receive recent data

Availability: every request receives a response.

Partition Tolerance: the system continues to operate despite dropped messages between nodes

The new NoSQL DB systems ( Amazon DynamoDB ,Cassandra, Google BigTable) will achieve  (availability and Partition Tolerance) but will not achieve Consistency.

I can see they name these models as BASE ( Basically Available, Soft state, Eventually consistent).

Basically Available: system guarantees availability according to CAP theorem.

Soft state: the state of the system continues to change even with no input.

Eventual Consistency: the system will become eventual consistent once it stops receiving input. As to write data to concerned  nodes takes some time

Links for more details:

http://www.dataversity.net/acid-vs-base-the-shifting-ph-of-database-transaction-processing/

https://people.eecs.berkeley.edu/~brewer/cs262b-2004/PODC-keynote.pdf

What is iSCSI and How to Configure in Solaris 11 and Linux.

Before we define iSCSI,we must know what is SCSI itself.

SCSI: Small Computer Systems Interface, is a family of protocols for communicating with I/O devices( especially storage).

SCSI protocol is being used by servers to communicate with storages over various types of transports like SAS , SAS2 ,SATA, Parallel SSCSI , FC…etc.

for SAS or SATA we have the limitation in the distance that these cables support,while in Fibre Channel (FC) it can go for long distances ,but it is very costly.

 

Hint:

1- SCSI is a client-server architecture, client is “initiator” while storage is “target”

2- initiator mostly is OS ( Windows, Linux, Unix) . Target  mostly is storage like (NetApp,EMC,ZFS..etc ).

So, SCSI packets is being transported mostly by SAS ,SATA,and FC cables.

iSCSI : is transporting SCSI packets over TCP/IP, providing  block-level access to storage devices over TCP/IP network.

So, initiators will have its own IP and targets as well ,all packets are being transported over normal IP networks

Initiators issue SCSI “commands” to request services from targets.

But how initiator(operating system ) can discover targets (storag)?

iSCSI discovery mechanism:

1- Send Targets  : the target IP and Port is known to the initiator, a discovery session is established , then initiator sends SendTargets commands to get available iscsi targets from storage.

2- Static Configuration: the IP,Port and IQN of target are available for the initiator.

3- Zero-Configuration : the initiator doesn’t have any information about targets, a discovery messages are sent to Storage Name Server ( iSNS , SLP…etc), where all information regarding initiators and targets and access list.

iSNS: Internet Storage Name Server , is centeralized server that has iSCSI configurations of initiators and targets.

SLP : Service Location Protocol, is not widely implemented, but anyway it helps computers to find iSCSI services across network.

in iSCSI network, each component (initiator or target ) has its unique name,Lets have a look at naming types

iSCSI Naming: Three naming are supported by iSCSI

1- iqn ( iSCSI Qualified Name )format : iqn.yyyy-mm.ReversedDomain:String

                        iqn.2001.04.com.example:storage:diskarrays-sn-a8675309

2- eui :eui.{EUI-64 bit address} (e.g. eui.02004567A425678D)

3- NAA Network Address Authority: naa.{NAA 64 or 128 bit identifier} (e.g. naa.52004567BA64678D)

SCSI Transport Protocol	EUI-64	NAA	IQN
iSCSI	X	X	X
FCP ( Fibre Channel)		X
SAS		X

iSCSI Session Types: 

the initiator and target, both creates a session for communication as following

1- Discovery Session: opened to discover target only

2- Normal Operation Session: unrestricted session for I/O operations

iSCSI Security:

iSCSI supports two separate security mechanisms:

1-in-band authentication between initiator and target at the iSCSI connection level.(like CHAP authentication), occurs during login into storage

2- packet protection by IPsec at the IP level.( all packets are secured )

iSCSI will do a login to storgae first(here we implement in-band authentication if needed ),after successfull login it will start exchanging  packets ( here we can implement IPSec if needed)

Lets see how to configure Solaris and Linux as a client fro storages :

1- Solaris Configuration as a client to connect to iSCSI storage:

Assuming Storage IP is 10.10.10.151

Here is the sequence of commands to use:

# svcadm enable network/iscsi/initiator

# iscsiadm modify discovery –static enable

# iscsiadm modify discovery –sendtargets enable

# iscsiadm add discovery-address 10.10.10.151

# iscsiadm list discovery-address -v 10.10.10.151   (get iqn of target and use in next command)

# iscsiadm add static-config  iqn.1986-03.com.sun:02:1859adc5-0f56-c7a4-a08a-dfad6332b09a,10.10.10.151

#devfsadm -i iscsi

#iscsiadm list target -vS ( list iscsi devices and OS disk path)

2- Linux 7 Configuration as a client to connect to iSCSI storage:

# systemctl start iscsi iscsid    ( enable iscsi daemon )

# cat /etc/iscsi/initiatorname.iscsi  ( get you host iqn ,share it with storage admin)

# iscsiadm -m discovery -t sendtargets -p 10.10.10.151  ( query iqn of target ,it will be used in next command )

#  iscsiadm -m node -T  iqn.1986-03.com.sun:02:1859adc5-0f56-c7a4-a08a-dfad6332b09a  -p 10.10.10.151 –login

# iscsiadm -m node -o update -n node.startup -v automatic ( enable auto startup )

# iscsiadm -m node -o update -n node.conn[0].startup -v automatic ( enable auto connection startup )

# iscsiadm –mode node -l all  ( create all needed devices )

# iscsiadm -m session -P 2

# fdisk /dev/sdX

# add line in /etc/fstab ,like: ( be sure you add _netdev ,it tells kernel the device is network device ,otherwise linux wll run into single user-mode if you reboot the machine )

/dev/sdb1   /ahmed                       xfs     defaults,_netdev        0 0

References :

1- RFC 3721,7143 (IETF Internet Engineering Task Force)

2- NetApp Guide.

3- Tecmint.

4- Redhat Documentation

5- Oracle Documentations

Solaris 11 Link Aggregation

Definition : consists of several interfaces on a system that are configured together to form a single,logical unit.

So ,if i have two interfaces with 10Gbps speed,by aggregating them ,we ‘ll get a single interface with 20 Gbps speed and so on.

The Solaris Link Aggregation  follows the standard of IEEE 802.3ad

• all interfaces must have the same speed, and also the eeprom local-mac-address? must be set to true.

Three main parts of link aggregation:

model ( dlmp or trunk) :Datalink multipathing or standard link-aggregation(trunk)

Aggregation Mode (LCAP mode): off ,active,passive

LCAPDU is Link Aggregation Control Protocol Packets

off: LACPDUS are not generated

active:The system generates LACPDUs at regular intervals

passive:The system generates an LACPDU only when it receives an LACPDU from the switch

Policy for load balancing: L2 ,L3 , L4

L2: Determines the outgoing link by hashing the MAC (layer 2) header of each packet

L3: Determines the outgoing link by hashing the IP (layer 3 ) header of each packet

L4(default): Determines the outgoing link by hashing the TCP, UDP, or other ULP (L4) header of each packet

Typical commands to configure link aggregation ( assuming that we have two free interfaces net3 and net4)

# ipadm show-if     <– check for net3 and net4 are free

# dladm create-aggr -l net3 -l net4 aggr0

# dladm modify-aggr -m trunk aggr0

# dladm modify-aggr -L passive aggr0

# dladm modify-aggr -P L4 aggr0

#ipadm create-ip aggr0

#ipadm create-addr -T static -a 192.168.100.120/24 aggr0/v4

# dladm show-aggr -x <— to get status of link aggregation

LINK     PORT      SPEED    DUPLEX  STATE   ADDRESS           PORTSTATE
aggre0    --        1000Mb  full    up      0:11:1c:3d:91:4d   --
         net3       1000Mb  full    up      0:11:1c:3d:91:4d   attached
         net4       1000Mb  full    up      0:11:1c:19:8d:4f   attached

 

Adding and removing interface from link aggregation :

Lets add an interface (net5) and remove an interface (net4):

# dladm add-aggr -l net5 aggr0

# dladm remove-aggr -l net4 aggr0

Configuring Solaris 11 as iSCSI server

Sometimes you need an extra disks for your virtual machines or inside test servers,here we will set up a simple iscsi server and configure one client to access it

Assumption: Server is Solaris 11 , client is Solaris 10

Here is  commands to be used:

############### At server side #######################
# pkg install storage-server    <---- install the iscsi storgae server
# svcadm enable stmf
# zfs create -V .5g ziscsi/disk1  <--- create one volume with size of half gigabyte on zpool named ziscsi
# stmfadm create-lu /dev/zvol/rdsk/ziscsi/disk1    <--- create LUN on the raw volume
Logical unit created: 600144F04ABA0300000056E454F50001
# stmfadm add-view 600144F04ABA0300000056E454F50001   <---Allow access to this iscsi disk
# stmfadm list-lu    <--- to list current lus
Now , enable and Create storage server interface ( Target ) with two commands :
# svcadm enable -r svc:/network/iscsi/target:default
# itadm create-target
Target iqn.1986-03.com.sun:02:121dc29a-7ac6-6342-c131-fc9a8b4756b2 successfully created
# itadm list-target -v   <---- to list targets(storage interface) you have created
 
####### At client Side ###############

# svcadm enable network/iscsi/initiator
The following command to add target that we just created ,we must have two informations: iqn and IP ,from previous commands have a copy of iqn and ip
# iscsiadm add static-config iqn.1986-03.com.sun:02:121dc29a-7ac6-6342-c131-fc9a8b4756b2,10.10.10.23:3260
# iscsiadm list static-config  <--- list static config staus
# iscsiadm modify discovery --static enable
Scan for new devices and list them :
# devfsadm -Cv
# echo|format
Output should contain a word (comstart ) in the disk description

 4. c2t600144F04ABA0300000056E454F50001d0 <SUN -COMSTAR -1.0 cyl 509 alt 2 hd 64 sec 32>
 /scsi_vhci/disk@g600144f04aba0300000056e454f50001

You can directly use the disk now

Migrating OVMM ( Oracle VM Manager)

It was a painful task for me to migrate OVMM from one machine to another ,as no clear document with descriptive steps.

So ,I have rendered some documents and some googled points  into the following steps which works fine with me

To accomplish this task, we need these steps to be done

A-  Backup configuration file (as it will have UUID of VM Manager)

/u01/app/oracle/ovm-manager-3/.config

  Hint: /etc/sysconfig/ovmm will have uuid of vm manager also

Just ,we need   the UUID only to use it during fresh installation of OVMM

B- Use Automatic backups or Create your own manual backup now      

Directory (/u01/app/oracle/mysql/dbbackup) contains automated backups.

To do manual backup ( this step to run on old VM manager):

OVMM services must be running: service ovmm status; service ovmm_mysql status;

              # /u01/app/oracle/ovm-manager-3/ovm_tools/bin/BackupDatabase -w  

1.                                                                     (-w) to wait till backup is finished
2.                                                                A directory named Manualxxxxxx is created ,tar this file

# cd /u01/app/oracle/mysql/dbbackup

# tar cvf mydb.tar Manualxxxxxx.

D-     Install OVM manager in new OVM Manager:

Hint : there are multiple accounts ovm will use,some accounts are mysql users and others are for weblogic,during installation ,all these accounts will have the same password, so try using same password of old ovm that we are migrating from,other wise after migration to new server you need to reset all these accounts again.

 

– Run ( ./createOracle.sh )

– Run  ./runInstaller.sh –uuid 0004FB000000100002CH7Y2DFFA8D8  –noprereq

– Choose (1) then (1) again (once prompted for password ,use same admin password as old OVM manager)

– finish installation and check if it is done successfully.

C -Steps For Restoring DB backup into newly installed OVM manager :   

Hint: open another ssh sesiosn and tail  ovm manager log file as following :

# cd /u01/app/oracle/ovm-manager-3/ovm_tools/bin

# python OvmLogTool.py –t  

  keep this session aside as it will tell what is going on.

1- stop services of OVM:

# service ovmcli stop; service ovmm stop; service ovmm_mysql stop;

2- before starting restore ,ensure no database files already exist on the Oracle VM Manager host

           # cd /u01/app/oracle/mysql/data/  

      # rm -rf appfw ibdata1 ib_logfile0 ib_logfile1 mysql ovs performance_schema

3- copy tar file  to location  (/u01/app/oracle/mysql/dbbackup) and untar it

# cd /u01/app/oracle/mysql/dbbackup

# tar xvf mydb.tar

4- switch to oracle user:

# su – oracle

# cd /u01/app/oracle/mysql/dbbackup

5- run restore script :

# bash /u01/app/oracle/ovm-manager-3/ovm_tools/bin/RestoreDatabase.sh ManualBackup-20140324_102412

6- start services of OVM:

service ovmm_mysql  start; service ovmm start; service ovmcli  start;

7- since certificates have been regenerated during new installation,re-configure db to point to newly ones :

# export MW_HOME=/u01/app/oracle/Middleware

# /u01/app/oracle/ovm-manager-3/ovm_upgrade/bin/ovmkeytool.sh setupWebLogic

# /sbin/service ovmm restart.

Rebuild client certificate

# cd /u01/app/oracle/ovm-manager-3/bin/

# ./configure_client_cert_login.sh

8- Now ,login to OVM console BUI (https://IP:7002/ovm/console)

–  Do RefreshAll :

           On the Servers and VMs tab click on the Server Pools folder in the navigation pane.

Select the Server Pools perspective from the drop-down selector.

           Click Refresh All icon the perspective toolbar.

Replace disks on veritas volume manager and increase filesystem online

It is usual to get a request to increase volume sizes or replace old disk with a new one,with no downtime for the activity.

Task: a mount point (/u03) needs more  200GB space

the (/u03) is mounted on vxvm volume (/dev/vx/oracle_dg/u03_vol)

 

Steps: create a new LUN from storage and map to server.

First, create a new LUN from storage and map to server.

Now at command line ,do these steps

# vxdisk -oalldgs list <—– list all available  disks
DEVICE TYPE DISK GROUP STATUS
emc_clariion0_20 auto:cdsdisk u01 oracle_dg     online
emc_clariion0_21 auto:cdsdisk u02 oracle_dg     online
emc_clariion0_22 auto:cdsdisk u03 oracle_dg    online <—– old disk
emc_clariion0_23 auto:none       –             –              online  <—– new disk

# /opt/VRTS/bin/vxdisksetup -i emc_clariion0_23    <———— initialize the new disk
# vxdg -g oracle_dg adddisk u03-new=emc_clariion0_23 <——-add to disk group and name it u03-new
# /opt/VRTS/bin/vxevac -g oracle_dg u03  u03-new<——-move data from source (u03) disk to target (u03-new)  disk
# vxtask list         <———wait till it finishes and no tasks are running )
# vxdg -g oracle_dg rmdisk u03   <——–remove source or old disk from disk group
# vxassist -g oracle_dg maxsize u03-new  <——– get exact available space on new disk
# vxresize -g oracle_dg u03_vol +204794m  <— grow /u03 online
# vxedit -g oracle_dg rename u03-new u03  <—–(optional step) rename the new disk with old name so you will maintain same names and no need to change documents

Now , the file system has been increased with 200 GB.

Generating Certificate for iLOM and XSCF

 

To get a certificate, we do these steps:

1- generate private key (myprivkay.key)

2- generate  Certificate Signing Request (CSR ) from private key. (mycsr.csr)

3- Generate a SSL certificate from CSR file (mypubkey.cert)

4- import both private key (myprivkey.key) and public key (mypubkey.cert) into your

 

Now lets apply for iLOM and XSCF

Part I: iLOM

iLOM doesn’t have tools to generate keys,so we use any third party tool like openssl as following :

openssl genrsa -out myprivkey.key  2048

openssl req -new -key myprivkey.key  -out  mycsr.csr

Now you have CSR , you can send it to Certificate Authority CA to get signed certificate ,or use openssl to create your own certificate as following

openssl x509 -req -days 365 -in mycsr.csr -signkey myprivkey.key -out mypubkey.cert

In the iLOM SSL server page, perform the following;

Load  Private Key  file(myprivkey.key)

Load Certifcate key file (mypubkey.cert)

Click Save to apply the changes.

Part II: XSCF

XSCF has its built-in command (sethttps) to generate (private key ,CSR, and certifcate ).

Option 1: create your own certifcate by single command

XSCF> sethttps -c selfsign EG Hassaan  Mansoura Example DevDept
scf-host abc@example.com
CA key and CA cert already exist. Do you still wish to update? [y|n]
:y
Enter passphrase: xxxxxxxx (any word i.e.   sarvoor)
Verifying – Enter passphrase: xxxxxxx (the same word entered before : sarvoor )

XSCF> sethttps -c enable

XSCF> rebootxscf 

Option 2: use Certifcate Authority to get signed certificate

Step A (generate private key )

XSCF> sethttps -c genserverkey

Server key already exists. Do you still wish to update? [y|n] :y

Enter passphrase: xxxxxxxx

Verifying – Enter passphrase: xxxxxxxx

 

Step B : create CSR

XSCF> sethttps -c gencsr EG sarvoor hassaan  Example DevDept  scf_host abc@example.com

 

Step C

Send the copied CSR to the CA and request the web server certificate

 

Step D : import certificate (after copy and paste , Please press Enter and press the “Ctrl” and “D” keys)

XSCF> sethttps -c importca

Please import a certificate:

—–BEGIN CERTIFICATE—–

MIIDdTCCAt6gAwIBAgIBATANBgkqhkiG9w0BAQQFADCBgTELMAkGA1UEBhMCamox: R+OpXAVQvb2tjIn3kO99dq+begECo4mwknW1t7QI7A1BkcW2/MkOolIRa6iP1ZwgJ oPmwAbrGyAvGUtdzUoyIH0jl7dRQrVIRA==

—–END CERTIFICATE—–

 

Step F

XSCF> sethttps -c enable

XSCF> rebootxscf

 

References :

1- ILOM Security Guide Firmware Releases 3.0, 3.1, and 3.2.

2- XSCF User’s Guide ( E25381-01).

Adding standard(non-cloned-disk) to a disk group

We have a new LUN created and mapped to a solaris machine with Veritas volume manager.

we tried to add the disk to a disk group, but unfortunately we received this error after  the adding disk command (# vxdg -g my_dg adddisk disk_07=emcpower7 )

VxVM vxdg ERROR V-5-1-0 Disk Group my_dg has only cloned disks and tyring to add standard disk to diskgroup. Mix of standard and cloned disks in a diskgroup is not allowed. Please follow the vxdg (1M) man page

If we issue the (vxdisk -o alldgs -e list) , the output is telling disk group(my_dg) has non-cloned disk at all.

So ,the work around is to set disk property (clone=on) and add to disk group ,then again set property(clone=off).

# vxdisk  set emcpower7 clone=on

# vxdg -g   my_dg adddisk   disk_07=emcpower7

# vxdisk  -g  my_dg set disk_07  clone=off

Red Hat Enterprise Linux to Oracle Solaris 11 Comparison

Linux administrators can kick start their learning experience when planning Oracle Solaris deployments by reviewing the following summary between Oracle Solaris 11 features and Red Hat Enterprise Linux 7 features. For more information about the Oracle Solaris 11 features, be sure to check out the “Key Links”

Full Article On Oracle http://www.oracle.com/technetwork/server-storage/solaris11/overview/redhat-mapping-guide-1555721.html