Linux
------
Linux serial Number--#dmidecode -t system
Linux Cluster -- #rpm -q redhat-config-cluster
Linux os level -- #uname -a and cat /etc/*-release
----------------------------------------------------------------------------------------------------------------------------------------------------
user
---------------------
useradd -g primary_group -G group1,group2,... -d /home/user_name -m -s shell user_name----syntax
useradd -g olympus -G wheel -d /home/lmarc -m -s /usr/bin/ksh lmarc ---to create user
chfn lmarc --- for change the gecos
chage -E "2013-11-05" lmarc ------to set account expiry
useradd -c 'Chandran Kuselan' -g wheel ckuselan
---------------------------------------------------------------------------------------------------------------------------------------------------
Password reset for hp unix server.
First set the password using password command
passwd vijay
then type the below command for to promt the user to reset the password
passwd -f vijay
----------------------------------------------------------------------------------------------------------------------------------------------------
Disk scanning on linux
echo " - - - " >
/sys/class/scsi_host/host0/scan
echo 1 >
/sys/class/scsi_host/host0/scan
[root@dmnp23cis1 scsi_host]# ls -l
total 0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host1
drwxr-xr-x 2 root root 0 Aug 22 23:36 host2
drwxr-xr-x 2 root root 0 Aug 22 23:36 host3
drwxr-xr-x 2 root root 0 Aug 22 23:36 host4
drwxr-xr-x 2 root root 0 Aug 22 23:36 host5
drwxr-xr-x 2 root root 0 Jul 9 03:41 host6
drwxr-xr-x 2 root root 0 Jul 9 03:41 host7
------------------------------------------------------------------------------------------------------------------------------------------------
Service or daemon on Linux
---------------------------
To check the daemon service on linux
/etc/init.d/<daemon name> start | restart | stop | status
chkconfig -- will show more details per runlevel abt the daemons
chkconfig | grep vsftpd --> will show you whether vsftps configured for auto start on any runlevel.
runlevel -- to know the current run level
chkconfig --level 3 vsftpd on ----> whenever system reboot happen vsftpd will autoon on the runlevel 3.
chkconfig --level 35 vsftpd on ----> whenever system reboot happen vsftpd will autoon on the runlevel 3 and 5.
cd /etc/rc.d/rc3.d --> Inside of this folder we have the vsftpd file.
chkconfig --del vsftpd --> vsftpd service removed from the startup.If you check inside of /etc/rc.d/rc3.d vsftpd wont be there
ntsysv -- Through this Command we can chekc and uncheck enabling the service.
----------------------------------------------------------------------------------------------------------------------------------------------------
To set or verify the ulimit values on Linux:
a.Log in as the root user.
b.Edit the /etc/security/limits.conf file and specify the following values:
admin_user_ID soft nofile 32768
admin_user_ID hard nofile 65536
admin_user_ID soft fsize unlimited
admin_user_ID hard fsize unlimited
admin_user_ID soft data unlimited
admin_user_ID hard data unlimited
admin_user_ID soft stack unlimited
admin_user_ID hard stack unlimited
admin_user_ID soft rss unlimited
admin_user_ID hard rss unlimited
admin_user_ID soft core unlimited
admin_user_ID soft core unlimited
----------------------------------------------------------------------------------------------------------------------------------------------------
"defunct" processes
--------------------
A "defunct" processes is also known as a "zombie" processes. A Zombie process is referred as dead process which is receding on your system though its completed executing. In one shot we can say its a dead processes which is still in RAM. This process will be in your process table and consuming your memory. Having more defunct process will consume your memory which intern slows your system. We have to kill the defunct process in order to free RAM and make system stable.
Why defunct process are created?
Ans : When ever a process ends all the memory used by that process are cleared and assigned to new process but due to programming errors/bugs some processes are still left in process table. These are created when there is no proper communication between parent process and child process.
Some FAQ?
1. How to find a defunct process?
And : Grep defunct value in ps -ef output
#ps -ef | grep defunct
2. How can I kill a defunct process?
And : Just use kill command
#kill defunct-pid
3. Still not able to kill?
Ans : Then use kill -9 to force kill that process
#kill -9 defunct-pid
4. Still have an issue in killing it?
Ans : Then try to kill its parent id and then defunct.
#kill parent-id-of-defunct-pid
Then
#kill -9 parent-id-of-defunct-pid
5. Still having defunct?
Ans : If you still find defunct process eating up RAM then last and final solution is to reboot your machine(This is not preferred on production boxes).
6.What is orphan process?
Ans : An orphan process is said to be a process which runs though parent process is terminated, these process do not know what to do and when to terminate.
7. What is difference between orphan and defunct processes?
Ans : A defunct process is a dead process where there is no execution happening where as orphan process is a live process which is still in execution state but don't have parent process
I am having a system which daily creates defunct process, I cannot sit and kill these process on daily basis.
How to get rid of this problem?
Ans : Just write a shell script to grep defunct process and kill them by putting this script in corntab.
----------------------------------------------------------------------------------------------------------------------------------------------------
How to find wwn in Redhat Linux?
a. Below is the simple command to find WWN on Redhat servers.
systool -c fc_host –v
# systool -c fc_host -v | grep "port_name"
port_name = "0x5001438001347fdc"
port_name = "0x5001438001347fde"
port_name = "0x50014380013471f0"
port_name = "0x50014380013471f2"
#
b. Below is the simple steps to find WWN in Linux.
Step 1: cd /sys/class/fc_host
Step 2: cd to the host# directory.
Step 3: cat node_name
Useful command to find to list the FC Adapter
lspci | grep -i Fibre
c. To find WWN on linux with Emulex:
If you find difficult to find WWN on linux with Emulex HBA use below hbacmd to list the HBAs. This only available if hbanyware installed.
cd /usr/sbin/hbanyware
[root@tilak_phy01 hbanyware]# ./hbacmd ListHBAs
Manageable HBA List
Port WWN : 11:00:00:00:d9:58:bf:92
Node WWN : 10:00:00:00:b9:58:ba:92
Fabric Name: 20:3f:01:04:51:a1:71:00
Flags : 1100a980
Host Name : tilak_phy01
Mfg : Emulex Corporation
----------------------------------------------------------------------------------------------------------------------------------------------------
Disk scanning on linux
echo " - - - " >
/sys/class/scsi_host/host0/scan
echo 1 >
/sys/class/scsi_host/host0/scan
[root@dmnp23cis1 scsi_host]# ls -l
total 0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host1
drwxr-xr-x 2 root root 0 Aug 22 23:36 host2
drwxr-xr-x 2 root root 0 Aug 22 23:36 host3
drwxr-xr-x 2 root root 0 Aug 22 23:36 host4
drwxr-xr-x 2 root root 0 Aug 22 23:36 host5
drwxr-xr-x 2 root root 0 Jul 9 03:41 host6
drwxr-xr-x 2 root root 0 Jul 9 03:41 host7
----------------------------------------------------------------------------------------------------------------------------------------------------
Filesystem increase Linux
--------------------
[root@compdb1rx ~]# df -m /u02
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/vg_datavg-lv_u02 35278 33470 17 100% /u02
[root@compdb1rx ~]# lvextend -L +10G /dev/mapper/vg_datavg-lv_u02
Extending logical volume lv_u02 to 45.00 GiB
Logical volume lv_u02 successfully resized
[root@compdb1rx ~]# resize2fs /dev/mapper/vg_datavg-lv_u02
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_datavg-lv_u02 is mounted on /u02; on-line resizing required
old desc_blocks = 3, new_desc_blocks = 3
Performing an on-line resize of /dev/mapper/vg_datavg-lv_u02 to 11796480 (4k) blocks.
The filesystem on /dev/mapper/vg_datavg-lv_u02 is now 11796480 blocks long.
[root@compdb1rx ~]# df -m /u02
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/vg_datavg-lv_u02 45357 33474 9580 78% /u02
For RHEL4 and requires different command to extend the file system. Please see below command what I ran.
----------------
[root@csclabdb01 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sdbi2 VG00 lvm2 a- 84.09G 8.31G
[root@csclabdb01 ~]# lvextend -L +1G /dev/VG00/lvol2
Extending logical volume lvol2 to 4.50 GB
Logical volume lvol2 successfully resized
[root@csclabdb01 ~]# ext2online /usr
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
[root@csclabdb01 ~]# df -h /usr
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VG00-lvol2
4.5G 2.4G 1.9G 55% /usr
----------------------------------------------------------------------------------------------------------------------------------------------------
Cluser on Linux
----------------
Lets have everyone see this so they know what to do when they run to similar issue.
Seems like this a 3 node redhat cluster. look below please . first you need to make sure the file system you are trying to extend is part of the cluster or not part of the cluster.
Here the path to redhat cluster configuration, /etc/cluster/cluster.conf. in this case you are trying to extend the /ora_backup file system
Do a “grep /ora_backup /etc/cluster/cluster.conf “ if you don’t find it then the file system is not part of the cluster. from looking thru the cluster.conf , it seems /svr_home is the only file system which is clustered. And this file system /svr_home is a gfs type file system..
so /ora_backup you are trying to extend is a stand alone file system on csclab01 server ( not clustered ) . also there is no Linux multipathing on these servers. This cluster is running EMC powetpath.
Please make sure Storage folk are presenting this lun ( 50 gig ) just to csclab01 server ( this is not a cluster lun ). After the LUN is provisioned/presented to csclab01 server, do the steps below:
here the steps you need to do:
1- # fdisk -l |grep sd >/tmp/fdisk.out.before
2- # ls –al /sys/class/fc_host ( to see your HBAs )
3- # echo 1 >/sys/class/fc_host/host5/issue_lip ( to probe the first HBA to discover the new lun )
4- # echo 1 >/sys/class/fc_host/host6/issue_lip ( to probe the second HBA to discover the new lun )
5- # fdisk -l |grep sd >/tmp/fdisk.out.after ( now you can compare the disk.out.before and after to see the new luns OS sees )
6- At this point OS see it but powerpath don’t know anything about the new disk, so you have the powerpath to discover the new lun
7- # powermt display dev=all >/tmp/powermt.out.before
8- # powermt config ( to discover the new lun via powerpath )
9- # powermt display dev=all >/tmp/powermt.out.after
10- # vi /tmp/powermt.out.after ( and look for new powerpath lun. Look for the disk that you discover in fdisk and cross-reference that to emcpower<?>
11- # pvcreate /dev/emcpower<?>
12- # pvs ( you should be able to see it in your list and not being part of any VG )
[root@csclab01 host5]# pvs
PV VG Fmt Attr PSize PFree
/dev/emcpowera VG12 lvm2 a- 60.00G 1020.00M
/dev/emcpowerb VG01 lvm2 a- 120.00G 0
/dev/emcpowerc VG11 lvm2 a- 30.00G 1020.00M
/dev/emcpowere VG01 lvm2 a- 100.00G 1016.00M
/dev/emcpowerf lvm2 a- 50.00G 50.00G
dev/sda2 VG00 lvm2 a- 66.41G 17.53G
13- # vgextend /dev/<vgname> /dev/emcpower<?>
14- # Now you should be able to extdend you file system
15- # lvextend -L +50G /dev/<vgname>/<lv_name
16- # resie2fs /dev/<vgname>/<lv_name
[root@csclab01 cluster]# clustat
Cluster Status for hlab_cluster @ Tue Dec 16 01:49:39 2014
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
csclab01-int 1 Online, Local, rgmanager
csclab02-int 2 Online, rgmanager
csclab03-int 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:hlab_gfs_svc csclab03-int started
service:hlab_prod csclab01-int started
service:hlab_test csclab01-int started
service:hlab_upgrade (none) disabled
-------------------------------------------------------------------------------------------------------------------------------------------------------
------
Linux serial Number--#dmidecode -t system
Linux Cluster -- #rpm -q redhat-config-cluster
Linux os level -- #uname -a and cat /etc/*-release
----------------------------------------------------------------------------------------------------------------------------------------------------
user
---------------------
useradd -g primary_group -G group1,group2,... -d /home/user_name -m -s shell user_name----syntax
useradd -g olympus -G wheel -d /home/lmarc -m -s /usr/bin/ksh lmarc ---to create user
chfn lmarc --- for change the gecos
chage -E "2013-11-05" lmarc ------to set account expiry
useradd -c 'Chandran Kuselan' -g wheel ckuselan
---------------------------------------------------------------------------------------------------------------------------------------------------
Password reset for hp unix server.
First set the password using password command
passwd vijay
then type the below command for to promt the user to reset the password
passwd -f vijay
----------------------------------------------------------------------------------------------------------------------------------------------------
Disk scanning on linux
echo " - - - " >
/sys/class/scsi_host/host0/scan
echo 1 >
/sys/class/scsi_host/host0/scan
[root@dmnp23cis1 scsi_host]# ls -l
total 0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host1
drwxr-xr-x 2 root root 0 Aug 22 23:36 host2
drwxr-xr-x 2 root root 0 Aug 22 23:36 host3
drwxr-xr-x 2 root root 0 Aug 22 23:36 host4
drwxr-xr-x 2 root root 0 Aug 22 23:36 host5
drwxr-xr-x 2 root root 0 Jul 9 03:41 host6
drwxr-xr-x 2 root root 0 Jul 9 03:41 host7
------------------------------------------------------------------------------------------------------------------------------------------------
Service or daemon on Linux
---------------------------
To check the daemon service on linux
/etc/init.d/<daemon name> start | restart | stop | status
chkconfig -- will show more details per runlevel abt the daemons
chkconfig | grep vsftpd --> will show you whether vsftps configured for auto start on any runlevel.
runlevel -- to know the current run level
chkconfig --level 3 vsftpd on ----> whenever system reboot happen vsftpd will autoon on the runlevel 3.
chkconfig --level 35 vsftpd on ----> whenever system reboot happen vsftpd will autoon on the runlevel 3 and 5.
cd /etc/rc.d/rc3.d --> Inside of this folder we have the vsftpd file.
chkconfig --del vsftpd --> vsftpd service removed from the startup.If you check inside of /etc/rc.d/rc3.d vsftpd wont be there
ntsysv -- Through this Command we can chekc and uncheck enabling the service.
----------------------------------------------------------------------------------------------------------------------------------------------------
To set or verify the ulimit values on Linux:
a.Log in as the root user.
b.Edit the /etc/security/limits.conf file and specify the following values:
admin_user_ID soft nofile 32768
admin_user_ID hard nofile 65536
admin_user_ID soft fsize unlimited
admin_user_ID hard fsize unlimited
admin_user_ID soft data unlimited
admin_user_ID hard data unlimited
admin_user_ID soft stack unlimited
admin_user_ID hard stack unlimited
admin_user_ID soft rss unlimited
admin_user_ID hard rss unlimited
admin_user_ID soft core unlimited
admin_user_ID soft core unlimited
----------------------------------------------------------------------------------------------------------------------------------------------------
"defunct" processes
--------------------
A "defunct" processes is also known as a "zombie" processes. A Zombie process is referred as dead process which is receding on your system though its completed executing. In one shot we can say its a dead processes which is still in RAM. This process will be in your process table and consuming your memory. Having more defunct process will consume your memory which intern slows your system. We have to kill the defunct process in order to free RAM and make system stable.
Why defunct process are created?
Ans : When ever a process ends all the memory used by that process are cleared and assigned to new process but due to programming errors/bugs some processes are still left in process table. These are created when there is no proper communication between parent process and child process.
Some FAQ?
1. How to find a defunct process?
And : Grep defunct value in ps -ef output
#ps -ef | grep defunct
2. How can I kill a defunct process?
And : Just use kill command
#kill defunct-pid
3. Still not able to kill?
Ans : Then use kill -9 to force kill that process
#kill -9 defunct-pid
4. Still have an issue in killing it?
Ans : Then try to kill its parent id and then defunct.
#kill parent-id-of-defunct-pid
Then
#kill -9 parent-id-of-defunct-pid
5. Still having defunct?
Ans : If you still find defunct process eating up RAM then last and final solution is to reboot your machine(This is not preferred on production boxes).
6.What is orphan process?
Ans : An orphan process is said to be a process which runs though parent process is terminated, these process do not know what to do and when to terminate.
7. What is difference between orphan and defunct processes?
Ans : A defunct process is a dead process where there is no execution happening where as orphan process is a live process which is still in execution state but don't have parent process
I am having a system which daily creates defunct process, I cannot sit and kill these process on daily basis.
How to get rid of this problem?
Ans : Just write a shell script to grep defunct process and kill them by putting this script in corntab.
----------------------------------------------------------------------------------------------------------------------------------------------------
How to find wwn in Redhat Linux?
a. Below is the simple command to find WWN on Redhat servers.
systool -c fc_host –v
# systool -c fc_host -v | grep "port_name"
port_name = "0x5001438001347fdc"
port_name = "0x5001438001347fde"
port_name = "0x50014380013471f0"
port_name = "0x50014380013471f2"
#
b. Below is the simple steps to find WWN in Linux.
Step 1: cd /sys/class/fc_host
Step 2: cd to the host# directory.
Step 3: cat node_name
Useful command to find to list the FC Adapter
lspci | grep -i Fibre
c. To find WWN on linux with Emulex:
If you find difficult to find WWN on linux with Emulex HBA use below hbacmd to list the HBAs. This only available if hbanyware installed.
cd /usr/sbin/hbanyware
[root@tilak_phy01 hbanyware]# ./hbacmd ListHBAs
Manageable HBA List
Port WWN : 11:00:00:00:d9:58:bf:92
Node WWN : 10:00:00:00:b9:58:ba:92
Fabric Name: 20:3f:01:04:51:a1:71:00
Flags : 1100a980
Host Name : tilak_phy01
Mfg : Emulex Corporation
----------------------------------------------------------------------------------------------------------------------------------------------------
Disk scanning on linux
echo " - - - " >
/sys/class/scsi_host/host0/scan
echo 1 >
/sys/class/scsi_host/host0/scan
[root@dmnp23cis1 scsi_host]# ls -l
total 0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host0
drwxr-xr-x 2 root root 0 Jul 9 03:41 host1
drwxr-xr-x 2 root root 0 Aug 22 23:36 host2
drwxr-xr-x 2 root root 0 Aug 22 23:36 host3
drwxr-xr-x 2 root root 0 Aug 22 23:36 host4
drwxr-xr-x 2 root root 0 Aug 22 23:36 host5
drwxr-xr-x 2 root root 0 Jul 9 03:41 host6
drwxr-xr-x 2 root root 0 Jul 9 03:41 host7
----------------------------------------------------------------------------------------------------------------------------------------------------
Filesystem increase Linux
--------------------
[root@compdb1rx ~]# df -m /u02
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/vg_datavg-lv_u02 35278 33470 17 100% /u02
[root@compdb1rx ~]# lvextend -L +10G /dev/mapper/vg_datavg-lv_u02
Extending logical volume lv_u02 to 45.00 GiB
Logical volume lv_u02 successfully resized
[root@compdb1rx ~]# resize2fs /dev/mapper/vg_datavg-lv_u02
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_datavg-lv_u02 is mounted on /u02; on-line resizing required
old desc_blocks = 3, new_desc_blocks = 3
Performing an on-line resize of /dev/mapper/vg_datavg-lv_u02 to 11796480 (4k) blocks.
The filesystem on /dev/mapper/vg_datavg-lv_u02 is now 11796480 blocks long.
[root@compdb1rx ~]# df -m /u02
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/vg_datavg-lv_u02 45357 33474 9580 78% /u02
For RHEL4 and requires different command to extend the file system. Please see below command what I ran.
----------------
[root@csclabdb01 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sdbi2 VG00 lvm2 a- 84.09G 8.31G
[root@csclabdb01 ~]# lvextend -L +1G /dev/VG00/lvol2
Extending logical volume lvol2 to 4.50 GB
Logical volume lvol2 successfully resized
[root@csclabdb01 ~]# ext2online /usr
ext2online v1.1.18 - 2001/03/18 for EXT2FS 0.5b
[root@csclabdb01 ~]# df -h /usr
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VG00-lvol2
4.5G 2.4G 1.9G 55% /usr
----------------------------------------------------------------------------------------------------------------------------------------------------
Cluser on Linux
----------------
Lets have everyone see this so they know what to do when they run to similar issue.
Seems like this a 3 node redhat cluster. look below please . first you need to make sure the file system you are trying to extend is part of the cluster or not part of the cluster.
Here the path to redhat cluster configuration, /etc/cluster/cluster.conf. in this case you are trying to extend the /ora_backup file system
Do a “grep /ora_backup /etc/cluster/cluster.conf “ if you don’t find it then the file system is not part of the cluster. from looking thru the cluster.conf , it seems /svr_home is the only file system which is clustered. And this file system /svr_home is a gfs type file system..
so /ora_backup you are trying to extend is a stand alone file system on csclab01 server ( not clustered ) . also there is no Linux multipathing on these servers. This cluster is running EMC powetpath.
Please make sure Storage folk are presenting this lun ( 50 gig ) just to csclab01 server ( this is not a cluster lun ). After the LUN is provisioned/presented to csclab01 server, do the steps below:
here the steps you need to do:
1- # fdisk -l |grep sd >/tmp/fdisk.out.before
2- # ls –al /sys/class/fc_host ( to see your HBAs )
3- # echo 1 >/sys/class/fc_host/host5/issue_lip ( to probe the first HBA to discover the new lun )
4- # echo 1 >/sys/class/fc_host/host6/issue_lip ( to probe the second HBA to discover the new lun )
5- # fdisk -l |grep sd >/tmp/fdisk.out.after ( now you can compare the disk.out.before and after to see the new luns OS sees )
6- At this point OS see it but powerpath don’t know anything about the new disk, so you have the powerpath to discover the new lun
7- # powermt display dev=all >/tmp/powermt.out.before
8- # powermt config ( to discover the new lun via powerpath )
9- # powermt display dev=all >/tmp/powermt.out.after
10- # vi /tmp/powermt.out.after ( and look for new powerpath lun. Look for the disk that you discover in fdisk and cross-reference that to emcpower<?>
11- # pvcreate /dev/emcpower<?>
12- # pvs ( you should be able to see it in your list and not being part of any VG )
[root@csclab01 host5]# pvs
PV VG Fmt Attr PSize PFree
/dev/emcpowera VG12 lvm2 a- 60.00G 1020.00M
/dev/emcpowerb VG01 lvm2 a- 120.00G 0
/dev/emcpowerc VG11 lvm2 a- 30.00G 1020.00M
/dev/emcpowere VG01 lvm2 a- 100.00G 1016.00M
/dev/emcpowerf lvm2 a- 50.00G 50.00G
dev/sda2 VG00 lvm2 a- 66.41G 17.53G
13- # vgextend /dev/<vgname> /dev/emcpower<?>
14- # Now you should be able to extdend you file system
15- # lvextend -L +50G /dev/<vgname>/<lv_name
16- # resie2fs /dev/<vgname>/<lv_name
[root@csclab01 cluster]# clustat
Cluster Status for hlab_cluster @ Tue Dec 16 01:49:39 2014
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
csclab01-int 1 Online, Local, rgmanager
csclab02-int 2 Online, rgmanager
csclab03-int 3 Online, rgmanager
Service Name Owner (Last) State
------- ---- ----- ------ -----
service:hlab_gfs_svc csclab03-int started
service:hlab_prod csclab01-int started
service:hlab_test csclab01-int started
service:hlab_upgrade (none) disabled
-------------------------------------------------------------------------------------------------------------------------------------------------------
