[Solved]: Why rsync didn’t copy Hidden files/directories and Why asterisk does not include hidden(dot) files in Linux?

Updated on September 1, 2017

I used rsync command to copy a particular user directory from one storage device to another. The operation completed successfully without throwing any warning/error. But to my surprise, the data to be copied was around 167GB and the copied data was of just 1.4GB! I used ‘du’ command to calculate the disk size of source and destination folders and ensured that ‘du’ is not showing erratic results by following this tutorial – how to fix erratic disk usage statistics from du. Below is snapshot explaining the scenario.

Scenario: Let us assume – the old storage mount point is ‘/sata1/home/ramya’ and the new storage mount point is ‘/tmp/home/ramya’. As told earlier, I used rsync to copy the directory as shown below:

usr/bin/rsync -avzlh /sata1/home/ramya/* /tmp/home/ramya/ | tee /tmp/$(date+"%F_%R")-backup.log

Lookout for the ‘asterisk’ in the above command (that was the issue and I’ll be explaining the reason below the fold)

du command discrepancy findings:

du command output of the source directory (/sata1/home/ramya/):

# du -chs /sata1/home/ramya/
167G    /sata1/home/ramya/
167G    total

du command output of the destination directory (/tmp/home/ramya):

# du -chs /tmp/home/ramya/
1.4G /tmp/home/ramya/
1.4G total

From the above snapshots, you can see that the source directory is of size 167GB and the copied directory is of size 1.4GB. So where’s the remaining data? Let me explain how I fixed the issue.

Solution:

To debug the issue, I used du command to check the size of each file inside the source and destination folders as shown below: By listing the files and comparing the file size.

bash-3.2#du -h -x /sata1/home/ramya 
15G /sata1/home/ramya/.g4
48M /sata1/home/ramya/1SVC
13G /sata1/home/ramya/.g8
120M /sata1/home/ramya/techg/.techla
90G /sata1/home/ramya/.openVAS.tar-gz
16G /sata1/home/ramya/.VSL.zip
...
...
bash-3.2#du -h -x /tmp/home/ramya 
1G /tmp/home/ramya/samplejobtoec
48M /tmp/home/ramya/1SVC
41M /tmp/home/ramya/M15
12K /tmp/home/ramya/techglimpse/openvas.txt
134M /tmp/home/ramya/etc/pki
264K /tmp/home/ramya/NAMD_CV
...
...

From the above output, I understood that the hidden files were not copied and it was due to “*" (asterisk) used in the rsync command. The asterisk will expand all files in the current working directory except the files whose name starts with a dot (hidden files). Thus, rsync never receives the hidden files as arguments. So the solution is to use entire directory name (instead of asterisk) as argument to rsync command.

rsync -avzlh --ignore-existing /sata1/home/ramya/ /tmp/home/ramya/ | tee /tmp/$(date+"%F_%R")-backup.log

Note: The trailing slashes at the end of both paths. Any other syntax may lead to unexpected results!

Voila! It Worked.

Was this article helpful?

Related Articles

Leave a Comment