I used rsync command to copy a particular user directory from one storage device to another. The operation completed successfully without throwing any warning/error. But to my surprise, the data to be copied was around 167GB and the copied data was of just 1.4GB! I used ‘du’ command to calculate the disk size of source and destination folders and ensured that ‘du’ is not showing erratic results by following this tutorial – how to fix erratic disk usage statistics from du. Below is snapshot explaining the scenario.
Scenario: Let us assume – the old storage mount point is ‘/sata1/home/ramya’ and the new storage mount point is ‘/tmp/home/ramya’. As told earlier, I used rsync to copy the directory as shown below:
usr/bin/rsync -avzlh /sata1/home/ramya/* /tmp/home/ramya/ | tee /tmp/$(date+"%F_%R")-backup.log
Lookout for the ‘asterisk’ in the above command (that was the issue and I’ll be explaining the reason below the fold)
du command discrepancy findings:
du command output of the source directory (/sata1/home/ramya/):
# du -chs /sata1/home/ramya/ 167G /sata1/home/ramya/ 167G total
du command output of the destination directory (/tmp/home/ramya):
# du -chs /tmp/home/ramya/ 1.4G /tmp/home/ramya/ 1.4G total
From the above snapshots, you can see that the source directory is of size 167GB and the copied directory is of size 1.4GB. So where’s the remaining data? Let me explain how I fixed the issue.
Solution:
To debug the issue, I used du command to check the size of each file inside the source and destination folders as shown below: By listing the files and comparing the file size.
bash-3.2#du -h -x /sata1/home/ramya 15G /sata1/home/ramya/.g4 48M /sata1/home/ramya/1SVC 13G /sata1/home/ramya/.g8 120M /sata1/home/ramya/techg/.techla 90G /sata1/home/ramya/.openVAS.tar-gz 16G /sata1/home/ramya/.VSL.zip ... ... | bash-3.2#du -h -x /tmp/home/ramya 1G /tmp/home/ramya/samplejobtoec 48M /tmp/home/ramya/1SVC 41M /tmp/home/ramya/M15 12K /tmp/home/ramya/techglimpse/openvas.txt 134M /tmp/home/ramya/etc/pki 264K /tmp/home/ramya/NAMD_CV ... ... |
From the above output, I understood that the hidden files were not copied and it was due to “*" (asterisk) used in the rsync command. The asterisk will expand all files in the current working directory except the files whose name starts with a dot (hidden files). Thus, rsync never receives the hidden files as arguments. So the solution is to use entire directory name (instead of asterisk) as argument to rsync command.
rsync -avzlh --ignore-existing /sata1/home/ramya/ /tmp/home/ramya/ | tee /tmp/$(date+"%F_%R")-backup.log
Note: The trailing slashes at the end of both paths. Any other syntax may lead to unexpected results!
Voila! It Worked.
