Bacula Heartbeat Interval

Источник

При использовании резервного копирования Bacula вылезает ньюанс, если между Director-сервером и FD-сервером (клиентом) находится маршрутизатор (используется NAT) - выполняемая задача (job) может вдруг ни с того, ни с сего остановиться с ошибкой "Fatal error: Network error with FD during Backup: ERR=Connection reset by peer".
Дело всё в том, что Director, инициируя задачу на клиенте, в процессе выполнения просто ждёт - весь трафик идёт между FD-сервером (клиентом) и SD-сервером (storage). Маршрутизатор спустя установленное время (timeout, у cisco - 7200 секунд) решив, что соединение между Director-сервером и FD-сервером (клиентом) умерло, просто удаляет информацию о соединении из своей таблицы соединений, обрывая таким образом выполняемую задачу.
Решается проблема тривиально - на FD-сервере (клиенте) в конфигурационном файле в секцию FileDaemon добавить:

Heartbeat Interval = 60

При действующем Heartbeat-интервале FD-сервер в процессе выполнения работы пересылает heartbeat-пакеты Director-серверу, создавая таким образом активность в соединении и маршрутизатор, видя, что соединение "живо", не вмешивается.

Ссылки:

1. Client/File daemon Configuration
2. Timeout (?) problems with some Full backups

Из официального FAQ Bacula project:

My backup starts, but dies after a while with "Connection reset by peer" error

This is usually due to some router/firewall having a connection timeout, and it kills connections that are “idle” for too long. To fix, you should do both of:
(1) activate keepalive by adding

 Heartbeat Interval = 60

in all of SD, FD and DIR configurations. Note that by itself this still won't be enough if you have some slow operations like SQL queries, accurate backups enabled, etc. You need the second point too.
(2) lower the system time before keepalives are sent
Setting the system SO_KEEPALIVE timeouts is needed as the defaults might be quite long (like 2 hours of inactivity or even longer before system starts to send keepalives). See http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/ for instructions for GNU/Linux systems, or some other systems at http://www.gnugk.org/keepalive.html or http://www.starquest.com/Supportdocs/techStarLicense/SL002_TCPKeepAlive.shtml
You can check if SO_KEEPALIVE timeout is setup correctly by restarting bacula and starting a new job, and then check the current state of TCP connections with “netstat -to”. Here is how it looks when it is WRONG and needs fixing (this example shows it will not start sending keepalives for about 7200 seconds, that is 2 hours!):

# netstat -to
tcp        0      0 client:9102       server:54043      ESTABLISHED  keepalive (7196.36/0/0)
tcp        0      0 client:43628      server:9103       ESTABLISHED  keepalive (7197.26/0/0)

In that case you can try setting the system defaults to lower value, for example on GNU/Linux systems (or check URLs above for other systems) with:

sysctl -w net.ipv4.tcp_keepalive_time=60

Put that in /etc/sysctl.conf or /etc/sysctl.d/* to keep it across reboots.
Alternatively, one could try to increase router/firewall timeouts and/or number of simultaneous connections, and/or reduce time needed for backup (turning off accurate backups, reducing filesets, etc) or reducing the time network connection is idle during backup (for example, running Full backup instead of Incremental will take longer time, but the network connection will be idle for much less time, as bacula won't have to check if files have changed, which can take some time).
Some routers/firewalls (those having connection tracking / stateful firewall / NAT capabilities) will also reset all running connections if they reboot. Not much helping here, apart from avoiding condition which may make them reboot, or turning off their connection tracking / firewall / NAT (which might make them useless, of course)