I logged into our firewalls today to find the clock of our primary firewall was 45 minutes ahead and our backup firewall was 35 minutes behind. Embarrassing. It’s important to keep the clocks of all the machines in our cluster synchronized in case we ever need to investigate an intrusion or correlate activity between multiple machines.
The thing is, I have ntpd installed and running on both of these machines but it wasn’t updating the clock. Investigating /var/log/messages, I found ntpd was generating error messages like:
Sep 25 17:29:42 fw2 ntpd[2090]: sendto(xx.xx.xx.34) (fd=19): Operation not permitted
Sep 25 17:30:48 fw2 ntpd[2090]: sendto(xx.xx.xx.34) (fd=19): Operation not permitted
Sep 25 17:31:54 fw2 ntpd[2090]: sendto(xx.xx.xx.34) (fd=19): Operation not permitted
Sep 25 17:33:00 fw2 ntpd[2090]: sendto(xx.xx.xx.34) (fd=19): Operation not permitted
Sep 25 17:34:03 fw2 ntpd[2090]: sendto(xx.xx.xx.34) (fd=19): Operation not permitted
I received these errors with ntpdate as well but if I used the -u argument for an unprivileged port, it worked. Smells like firewall. This is where my sixty seconds of Google didn’t give me an immediate answer so I started by adding some logging rules to the firewall to give me more information about what packets were failing:
iptables -A OUTPUT -o eth0 -p udp -d <IP address of time server> --dport 123 \
-j LOG --log-prefix "NTP output: "
iptables -A INPUT -i eth0 -p udp -d <IP address of client> --dport 123 \
-j LOG --log-prefix "NTP input: "
Running ntpdate again, I trapped the following error:
Sep 25 17:57:18 fw2 kernel: NTP output: IN= OUT=eth0 SRC=xx.xx.xx.3 DST=xx.xx.xx.34
LEN=76 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=123 DPT=123 LEN=56
Ah ha! The SPT=123 DPT=123 shows that both the source port and destination port are 123! This is unusual for a daemon; usually the source port will be a unprivileged port between 1024 and 65535. Knowing this, it was easy to correct the firewall rule:
# allow client to query and receive timeserver
iptables -A OUTPUT -o eth0 -p udp \
-s <IP address of client> --sport 123 \
-d <IP address of time server> --dport 123 \
-m state --state NEW -j ACCEPT
the -m state is for stateful filtering; you could just as well leave it off and use -j ACCEPT to blanket accept traffic for this rule.
Comments are closed.