Saturday, September 2, 2017

Parallel rsync

Following steps did the job for me:
  1. Run the rsync --dry-run first in order to get the list of files those would be affected.
rsync -avzm --stats --safe-links --ignore-existing --dry-run --human-readable /data/projects REMOTE-HOST:/data/ > /tmp/transfer.log
  1. I fed the output of cat transfer.log to parallel in order to run 5 rsyncs in parallel, as follows:
cat /tmp/transfer.log | parallel --will-cite -j 5 rsync -avzm --relative --stats --safe-links --ignore-existing --human-readable {} REMOTE-HOST:/data/ > result.log
Here, --relative option (link) ensured that the directory structure for the affected files, at the source and destination, remains the same (inside /data/ directory), so the command must be run in the source folder (in example, /data/projects).
Foreman Smart-Proxy installation

This is the layman's installation for smart-proxy.

I've installed smart-proxy on both CentOS(6/7) and Ubuntu 14.04. This article will outline how to install Smart-Proxy with Integration to FreeIPA, DHCP, TFTP. The Foreman documentation can be a bit cryptic about certain steps, so I will attempt to lay it out in a sensible format, and explain along the way.

Download and install Smart-Proxy.
http://theforeman.org/manuals/1.9/index.html#2.1Installation

I selected Ubuntu 14.04. Using sudo or as the root user, install the following packages:

# grab the latest puppet
cd /tmp
apt-get -y install ca-certificates
wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
dpkg -i puppetlabs-release-trusty.deb

# install foreman sources (mind the version number)
echo "deb http://deb.theforeman.org/ trusty 1.9" > /etc/apt/sources.list.d/foreman.list
echo "deb http://deb.theforeman.org/ plugins 1.9" >> /etc/apt/sources.list.d/foreman.list
wget -q http://deb.theforeman.org/pubkey.gpg -O- | apt-key add -

# download the installer
apt-get update && apt-get -y install foreman-installer

Depending on how you installed your Ubuntu system, you may need to set the hostname and IP information for your server.

# edit my_fqdn
my_fqdn=smart-proxy1.flamed.us

# using eth0
my_ip=`ifconfig eth0 | awk '{ print $2}' | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}"`

# truncate host from fqdn
host=$(cut -f 1 -d . $my_hostname)

# set hostname in /etc/hosts
echo "$my_ip $my_fqdn $host" >> /etc/hosts

# set hostname variable to my_fqdn
hostname $my_fqdn


Download and install IPA client and join the system.
# install client
apt-get install freeipa-client

# join realm
ipa-client-install



Parallel rsync

Following steps did the job for me: Run the  rsync --dry-run  first in order to get the list of files those would be affected. rsync -...