1. RSYNC Introduce

RSYNC It's an open source , fast , Multifunctional , It is an excellent tool to realize full and incremental local or remote data synchronous backup .Rsync The software is suitable for unix/linux/windows And so on .

Remote data backup tool , Full backup and incremental backup can be realized , It can also be backed up locally .

The full English name is Remote synchronization.

It has the following three functions :

  1. long-range copy The function of : amount to ssh Self contained scp command , But better than scp command ,scp Every time it's a full copy .rsync High in incremental copy ,scp Every time it's full copy.
  2. Local copy function : Equivalent to cp command , But better than cp command , because cp Every time it's full copy.
  3. rsync You can also delete : amount to rm command .

CentOS5.X The built-in rsync Version are 2.6.x Version of , The main problem with this version is performance , such as , When a large number of small files are synchronized, it is easy to cause memory overflow or synchronization interruption , This is mainly due to 2.6 The version uses the list of pre listed files , It's about the mechanism of synchronization , When dealing with large file synchronization, incomplete synchronization files also occur in the market . But in the CentOS6.x series rsync All versions have been upgraded , This article uses linux System is CentOS6.7, Self contained rsync The version is 3.0.6.

2. RSYNC Characteristics of

The following features are from rsync Official website It's copied on the Internet :

  1. support for copying links, devices, owners, groups, and permissions

    Support copy link , equipment , user , User groups and attribute permissions .
  2. exclude and exclude-from options similar to GNU tar

    Supporting image tar The same rule out ( Or exclude from the file ) Features of backup files
  3. a CVS exclude mode for ignoring the same files that CVS would ignore

    Support CVS Exclude file mode
  4. can use any transparent remote shell, including ssh or rsh

    Supports remote shell(ssh,rsh etc. ) The data transfer
  5. does not require super-user privileges

    You don't need super user rights
  6. pipelining of file transfers to minimize latency costs

    rsync3.x After the version, the files are compared and transferred at the same time , Reduces file transfer latency
  7. support for anonymous or authenticated rsync daemons (ideal for mirroring)

    Support anonymous or need to verify rsync Daemons mode

3. Working scenarios used

  1. Synchronize all client data to the backup server ( It can cooperate with timing tasks , Achieve daily scheduled backup )

  2. Real time synchronization ( Solve the single point of failure problem of storage server ),rsync combination inotify Do real-time data synchronization .

4. RSYNC How it works

Rsync There are three main ways to transmit data :

  • Data transmission between single host computers ( This is similar to cp Command functionality )[ Local transport mode ]
  • With the help of rcp,ssh And so on ( This is similar to scp Command functionality )[ long-range shell transmission mode ]
  • To daemons (socket) The way to transmit data ( This is rsync Its own important function )[ Daemons mode ]

The official introduction :

1. Local:  rsync [OPTION...] SRC... [DEST]
2. Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST
3. Access via rsync daemon:
Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

That's all about rsync Some basic theories of . It's boring , Let's start the actual combat journey :

5. Introduction to the actual combat environment

as everyone knows ,linux There are different publishers , There are also different versions of the kernel , Here is an experimental version of this article :

[root@backup ~]# cat /etc/redhat-release
CentOS release 6.7 (Final)
[root@backup ~]# uname -a
Linux backup 2.6.32-573.26.1.el6.x86_64 #1 SMP Wed May 4 00:57:44 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@backup ~]#

Rsync Version of :

[root@backup ~]# rsync --version
rsync version 3.0.6 protocol version 30

First of all confirm Rsync Has it been installed? :

[root@backup ~]# rpm -qa rsync
rsync-3.0.6-12.el6.x86_64
[root@backup ~]#
This message shows that it has been installed , If it's not installed on the system rsync You can use the following command to install :
yum install rsync -y

6. The actual local transmission mode

The local transmission mode realizes the local synchronization function , Equivalent to cp,rm Wait for the order , The simple point is to copy data from one place to another or delete data on the same machine ,rsync Than cp The high level of command is ,rsync Incremental copy is used when copying , That is, it will only copy different contents , This greatly improves performance .

  1. Grammar format

     rsync [OPTION...] SRC... [DEST]
  2. Syntax description :

     1. rsync For synchronous commands 
    2. [option] Parameters needed for synchronization
    3. SRC As source , The partition to be copied , File or directory
    4. [DEST] Partition for the purpose of , File or directory
  3. Example :

      Local copy command 
    [root@backup ~]# cp /etc/hosts /tmp # Use cp Command copy file
    [root@backup ~]# rsync /etc/hosts /opt # Use rsync Command copy file
    [root@backup ~]# ls /opt/hosts
    /opt/hosts
    [root@backup ~]# Local delete command :( This function is dangerous , Generally, it is not recommended to use )
    [root@backup ~]# mkdir /null -p # Create an empty directory without any files , Name at will
    [root@backup ~]# ls /tmp/
    a.txt hosts today_backup_file.txt
    [root@backup ~]# rsync -avz --delete /null/ /tmp # Then copy the empty directory to the directory to be deleted , Add --delete Parameters are implemented rsync Delete function of , Here we need to pay attention to /null/ This directory must be followed by /, Can not write /null such , In this way , It will /null This directory ( Contains the directory name ) copy to /tmp Under the table of contents , You can't delete it .
    sending incremental file list
    ./
    deleting .ICE-unix/
    deleting today_backup_file.txt
    deleting hosts
    deleting a.txt
    deleting .pwd.lock sent 29 bytes received 15 bytes 88.00 bytes/sec
    total size is 0 speedup is 0.00
    [root@backup ~]#
  4. Introduction of important parameters

     -r --recursive  Recursive mode , All directories under subdirectories are transmitted the same way 
    -t --times Keep file time information
    -o --owner Keep file owner information
    -p –perms Keep file permissions
    -g --group Keep file properties
    -P --progress Display the synchronization process and the progress of transmission and other information
    -D --devices Keep device file information
    -l --links Keep the soft connection
    -e --rsh=command The channel protocol used , Specify an alternative rsh Of shell Program , for example ssh etc.
    --exclude=PATTERN Specifies that file modes that do not need to be transferred... Are excluded
    --exclude-from=file ( The directory file where the file name is located )
    --bwlimit=RATE Limit the speed of transmission
    More parameters , Please refer to :
    http://www.samba.org/ftp/rsync/rsync.html

Three common parameters :

    -v –verbose  Detailed output mode , The progress of transmission and other information 
-z –compress Compression during transmission to improve transmission efficiency , --compress-level=NUM Can be compressed by level
-a --archive Archiving mode , Indicates that files are transferred recursively , And keep all file properties , be equal to -rtopgDl

7. By remote shell Data transfer (remote shell mode)

For data transmission between two machines , For example, back up important data to the server . This model is equivalent to scp The function of command implementation , however rsync You can still achieve incremental copy , And it supports directory copying , This is the ratio rsync A good place to use .

By remote shell(rcp,ssh etc. ) Transmission can be divided into two situations , The grammar is :

 PULL : rsync [OPTION...] [USER@]HOST:SRC... [DEST]
PUSH : rsync [OPTION...] SRC... [USER@]HOST:DEST

Syntax description :

1.	rsync  For synchronous commands 
2. [OPTION...] Parameter options for synchronization
3. [USER@]HOST:SRC... For synchronous remote link user and host address
4. SRC As source , That's it copy The partition , Files or directories, etc , and HOST Connect them with a colon
5. [DEST] Partition for the purpose of , Files or directories, etc

Pull means to synchronize the data from the remote host to the corresponding directory of the local host executing the command ;

Push means to execute the command from the local host to synchronize the local data to the specified directory of the remote host .


Examples of pull Syntax :

rsync -avz -e 'ssh -p 22' root@172.16.1.41:/opt/ /tmp/

Push syntax example :

rsync -avz -e 'ssh -p 22' /etc/ root@172.16.1.41:/tmp/

hot tip :

Notice the difference between the two commands :

rsync –avz /opt/ /tmp/
rsync -avz /opt /tmp/

/opt/ Push or pull /opt Contents under the directory ,/opt Push or pull /opt The contents of the catalog and its contents .

8. The actual use of RSYNC Daemons mode ( a key )

so to speak , What I've talked about before is all foreshadowing , This is the top priority .

working principle :

RSYNC Daemons mode , Including servers and clients , Start on the server rsync Daemons of , The client configures the corresponding user name and password , Realize the backup function from client to server .RSYNC The machine started by the daemons is the backup server mentioned in the title of our article . Other machines as clients , Push data to this machine , Or pull data from this machine .

In the actual combat of this article, the server side host So backup, So we call this server for short backup, Client side hostname yes nsf01, Empathy , We call this client for short nfs01.

8.1 Server side (bakup machine ) To configure

a. First, make sure the software is installed :

[root@backup ~]# rpm -aq rsync
rsync-3.0.6-12.el6.x86_64
[root@backup ~]#

b. Create user

[root@backup ~]# useradd rsync -s /sbin/nologin -M  # This is rsync Users used by the process 
[root@backup ~]# id rsync
uid=501(rsync) gid=501(rsync) Group =501(rsync)
[root@backup ~]#
The client connects to the server rsync User access rights access data

c. Create a server-side configuration file (rsyncd.conf)

#rsync_config_______________start
#created by grewan 07::23 2016-05-28
#Email:wangqj541@163.com blog: http://www.cnblogs.com/greta/
##rsyncd.conf start##
uid = rsync
gid = rsync
use chroot = no
max connections = 200
timeout = 300
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log # This is rsync Of log file , More useful
[backup]
path = /backup/
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
#rsync_config_______________end

Profile description :

Parameters explain
uid=rsync rsync Users used ID. default uid by -2, Usually it is nobody
gid=rsync rsync Groups used ( User's group ). default gid by -2, Usually it is nobody
use chroot=no If true,daemon Before transferring files to the client “chroot to the path”. This is a rsync A configuration of security , Because most of us use it on the intranet rsync, So you don't have to configure it
max connections=200 Set the maximum number of connections , The default is 0, It means unlimited , A negative value turns off the module
timeout=300 The default is 0, You mean for no timeout, The suggestion is 300-500(s)
pid file=/var/run/rsyncd.pid rsync daemon Start the process after it starts PID Write to this file . If this file already exists ,rsync The process does not override the file , It will end
lock file=/var/run/rysncd.lock Appoint loc Files are used to support max connections Parameters of , So that the total number of connections will not exceed the limit , The default is /var/run/rsyncd.lock
log file = /var/log/rsyncd.log rsync Log file
ignore errors Ignore IO error
read only = false Specifies whether the client can upload files , By default, it is... For all modules true
list = false Whether to allow the client to view the list of available modules ( similar ls), The default is true
hosts allow = 172.16.1.0/24 Specify the host name of the client that can be contacted or ip Address or address segment , This parameter is not available by default , That is, they can be connected
hosts deny = 0.0.0.0/32 Specify a non contactable client host name or IP Address or address segment , Forbid them to connect . This parameter is not available by default , That is, they can be connected
auth users = rsync_backup Specify which modules can be used by users separated by spaces or commas , The user does not need to exist in the local system , By default, all users have access without password
secrets file = /etc/rsync.password Specify the file where the user name and password are stored . Format : user name : password , The password does not exceed 8 position
[backup] Module name , It needs to be enclosed in brackets , There is no special requirement for naming , But it's better to have a meaningful name , Convenient for future maintenance
path = /backup/ In this module ,daemon The file system or directory used , The permissions of the directory should be consistent with those in the configuration file , Otherwise, there will be reading and writing problems .

d. Create backup directory

[root@backup ~]# mkdir /backup
[root@backup ~]# chown -R rsync.rsync /backup/ # Be sure to remember the user and user group who modified the file
[root@backup ~]# ls -ld /backup/
drwxr-xr-x 2 rsync rsync 4096 5 month 22 15:05 /backup/
[root@backup ~]#

e. Create password file ( Generate according to the configuration file )

[root@backup ~]# echo "rsync_backup:grewan" > /etc/rsync.password
[root@backup ~]# chmod 600 /etc/rsync.password

f. Start the service

[root@backup ~]# rsync --daemon
[root@backup ~]# lsof -i :873 #rsync The port of is 873, Use this command to check rsync Whether the service started successfully
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsync 3894 root 4u IPv4 15807 0t0 TCP *:rsync (LISTEN)
rsync 3894 root 5u IPv6 15808 0t0 TCP *:rsync (LISTEN)
For more information, please refer to :[root@backup ~]# man rsyncd.conf
8.2 client (nsf01 machine ) To configure

a. Configure password file

[root@nfs01 ~]# echo "grewan" > /etc/rsync.password

b. Modify password file permissions

[root@nfs01 ~]# chmod 600 /etc/rsync.password
8.3 Sample test

Be careful : The examples here are all on the client side (nsf01) Operating on , The general use scenario is to backup data from the client to the server .

Grammar format :

Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST]    # Push data to server , This way is more commonly used 
rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST] Push: rsync [OPTION...] SRC... [USER@]HOST::DEST # Pull data from the server , This way is more commonly used
rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST

1. On the client side (nfs01) To create a /backup Catalog , And test files

[root@grewan backup]# mkdir /backup -p   # Create directory 
[root@grewan backup]# touch {1..10} # establish 10 Test files
[root@grewan backup]# ls
1 10 2 3 4 5 6 7 8 9

2. Put the client /backup Contents of the catalog , Back up to the backup server :

[root@nfs01 backup]# rsync -avz /backup/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
sending incremental file list
./
1
10
2
3
4
5
6
7
8
9 sent 450 bytes received 201 bytes 1302.00 bytes/sec
total size is 0 speedup is 0.00
[root@nfs01 backup]#
Command specification :
rsync -avz This is the command and the corresponding parameters
/backup/: hold backup The contents of the directory are pushed to the server
rsync_backup@172.16.1.41::backup : This is the server information ,rsync_backup It's a configuration file /etc/rsyncd.conf The user name configured in ,
172.16.1.41 It's server's ip Address ,backup yes /etc/rsyncd.conf Module name configured in , There must be no mistake in this place ,
You can go back to the configuration file , Deepen the impression
--password-file: Specify password file , Without this parameter , To enter the password manually .

3. On the backup server side (backup machine ) View the results of the backup

[root@backup ~]# cd /backup/
[root@backup backup]# ls
1 10 2 3 4 5 6 7 8 9
[root@backup backup]#

4. The second way to push

Use rsync Push by protocol

[root@nfs01 backup]# rsync -avz /backup/ rsync://rsync_backup@172.16.1.41/backup --password-file=/etc/rsync.password
sending incremental file list
./
1
2
3
4
5 sent 239 bytes received 106 bytes 690.00 bytes/sec
total size is 0 speedup is 0.00
[root@nfs01 backup]#

5. Pull content from backup server

[root@nfs01 backup]# rsync -avz rsync_backup@172.16.1.41::backup /backup/ --password-file=/etc/rsync.password
receiving incremental file list
./
1
2
3
4
5 sent 162 bytes received 326 bytes 325.33 bytes/sec
total size is 0 speedup is 0.00
[root@nfs01 backup]# ls
1 2 3 4 5
[root@nfs01 backup]#

6. The second way to pull

[root@nfs01 backup]# rm -f *
[root@nfs01 backup]# rsync -avz rsync://rsync_backup@172.16.1.41/backup /backup/ --password-file=/etc/rsync.password
receiving incremental file list
./
1
2
3
4
5 sent 162 bytes received 326 bytes 976.00 bytes/sec
total size is 0 speedup is 0.00
[root@nfs01 backup]#

7. Delete rsync Synchronization process

kill `cat /var/run/rsyncd.pid`
kill `/var/run/rsyncd.pid`

9. Multi directory sharing

Multi directory sharing means that the client can push or pull multiple directories to the server . It's easy to implement , It is to configure multiple modules in the configuration file , Each module can specify a different user name , Passwords and so on . If the basic information of all pushed modules is the same , You can put configuration information on multiple modules , The module can only configure a corresponding path , Like the following example :

[root@backup backup]# cat /etc/rsyncd.conf
#rsync_config_______________start
#created by grewan 07::23 2016-05-28
#Email:wangqj541@163.com blog: http://www.cnblogs.com/greta/
##rsyncd.conf start##
uid = rsync
gid = rsync
use chroot = no
max connections = 200
timeout = 300
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
[backup] # Multiple modules are configured here , Common configuration items , Move to the top of the module
path = /backup/
[test]
path = /test/
#rsync_config_______________end

How to use RSYNC More articles on building backup servers

  1. Nine 、 Build a backup server Use rsync service

    brief introduction Rsync Is open source fast . multi-function , Can achieve full and incremental local or remote data synchronous backup excellent tool . Incremental backup is more efficient , You can synchronize content or properties [root@backup-41 ~]# rpm -qa rsy ...

  2. adopt rsync Build a remote backup system ( Two )

    Rsync+inotify Back up data in real time rsync When synchronizing data , All the files need to be scanned and compared , And then differential transmission , If the file reaches a million or more levels, it is , It's also a long time to scan files , And if only a few files have changed ...

  3. adopt rsync Build a remote backup system ( One )

    Preface Our company is an e-commerce company , All is linux System , The number of websites is increasing every day , In order to ensure safety , We need to build a remote disaster recovery system , Put website data every morning 1 Point backup to remote server , Because of the amount of data , Incremental backup every day , Just back up when ...

  4. [ original ]Rsync Build and use

    rsync The construction and use of services *** Download and install : #wget https://download.samba.org/pub/rsync/src/rsync-3.1.2.tar.gz #tar -zxvf ...

  5. rsync build

    The server : See if it is installed :rpm -qa rsync If not installed :yum install -y rsync add to rsync user useradd -s /sbin/nologin -M rsync edit /e ...

  6. Use windos Simulation building web colony ( Two )

    One . adopt rsync Build a backup server We need to make real-time hot standby for these three directories , They are   Script directory of the system   System configuration file directory   The timing task directory of the system [root@mage-monitor- ~]# cat /se ...

  7. Rsync+inotify Build and use

    ## Rsync build ### 1.1 Environmental preparation ``` Rsync-Server 192.168.1.174 Client-Rsync 192.168.1.173 Service startup users are root, The use of the client ...

  8. rsync+inotify Achieve real-time file synchronization between servers -- turn

    I did before “ssh Trust and scp Auto transfer scripts ” Technical documentation for , This solution is used as a backup method in the company , But in practice , Because the primary server is transmitting to the backup server , Our main server needs to back up the files in real time . It's constantly coming into being , I don't know ...

  9. inotify+rsync Real time file synchronization

    The original is translated from http://dl528888.blog.51cto.com/2382721/771533/ I did before “ssh Trust and scp Auto transfer scripts ” Technical documentation for , This solution is used as a backup method in the company , But in actual operation ...

Random recommendation

  1. hadoop note (zookeeper)

    1. install It needs to be installed in advance java Environmental Science , Download this article zookeeper-3.3.6.tar.gz package . 1.1 tar -zxvf zookeeper-3.3.6.tar.gz 1.2 modify conf Medium zoo_ ...

  2. js Some knowledge of Excerpt from http://img0.pconline.com.cn/Pc_intranet/1105/13/313647_7.pdf

    Js Problem analysis --js Analysis of the current situation of page performance : The problem statement analyzes the problem : Abstract the root of the problem , Prove the seriousness of the problem through examples or reasoning : Start to spread by focusing on existing problems , What other problems will this lead to , Or a summary of the same type of problems : Start with dispersion ...

  3. 003_ About IntellJ IDE 2016 1. 4 Use

    IDEA Full name IntelliJ IDEA, yes java Integrated environment for language development ,IntelliJ Recognized as the best in the industry java One of the development tools , Especially in smart code assistants . Code autoprompt . restructure .J2EE Support . Various versions of tools ( ...

  4. OpenCV Way of learning (1)

    OpenCV The first code of : #include <opencv2/core/core.hpp> #include <opencv2/highgui/highgui.hpp> #i ...

  5. [Asp.net MVC]Asp.net MVC5 series —— Add view

    Catalog Series articles summary Add view summary Series articles [Asp.net MVC]Asp.net MVC5 series —— First project summary In this section we add a new controller HelloWorldController class , ...

  6. HDU 2676 Network Wars 01 Fractional programming , Minimum cut difficulty :4

    http://acm.zju.edu.cn/onlinejudge/showProblem.do?problemId=1676 To the summit i,j, The starting point s=1, End t=n, It can be said that the meaning of the question requires a group of 01 matrix use ...

  7. 《mysql Database backup script 》

    vim mysql.sh #!/bin/bashDAY=`date +%Y-%m-%d` // The date is shown in mm / DD / yyyy and assigned to DAY Variable SIZE=`du -sh /var/lib/mysql // see mysql ...

  8. 6、 ... and 、vue How to cache pages

    vue How and ionic It's the same caching mechanism as , You can cache pages , stay A Page skip to B Page back to A When the page is A The data on the page is still ? stay app.vue Lieutenant general router-view Use keep-alive wrap up , Use v-if To judge that ...

  9. 27 python Beginners ( Semaphore 、 Condition variables, 、 Synchronization conditions 、 queue )

    Reference blog : www.cnblogs.com/yuanchenqi/articles/5733873.html  semaphore Semaphore : condition Condition variables, : event Synchronization conditions : Conditions ...

  10. css Regression testing tools :backstopjs

    I've been watching an open class recently , A teacher talked about an automated tool ,backstopjs, Can automatically compare UI The picture is the same as the picture written at the front end , Inconsistencies will be marked with , Good use of , But there are also some problems in the process of writing , Now write it out , Make a note of First , want ...