NTP: Time is of the Essence

Time synchronization is not something many people may consider to be a critical component of a properly functioning enterprise; however, it is vital for managing, securing, debugging, and investigating security incidents on a network. Desynchronized timekeeping across distributed servers in a corporate network can cause serious headaches for IT staff trying to diagnose network issues or investigators trying to get to the bottom of a data breach.

Without accurate timestamps on log files, log aggregation and security information and event management (SIEM) tools are unable to accurately correlate log files for proactive alerting and post-incident forensic analysis. Proper time correlation is necessary for authentication systems, determining network usage, file system updates, diagnosing problems affecting multiple devices and components, billing and other transactional services, and networks holding personal health information (PHI) – the Health Insurance Portability and Accountability Act (HIPAA) security rules require accurate timestamping.

This post covers the basics of time synchronization, some considerations for what can go wrong, and some recommendations to avoid pitfalls or external threats of poorly configured timekeeping.

NTP

Typically, network administrators implement the Network Time Protocol (NTP) to synchronize timekeeping across a set of distributed time servers and clients. NTP runs over the User Datagram Protocol (UDP) using port 123. The NTP network of servers typically obtains its time from highly accurate atomic clocks or GPS clocks linked to time servers. These specialized servers directly communicate with NTP servers using Coordinated Universal Time (UTC)—adjusted by adding leap seconds to match the rotation of the earth—and relies on local hosts to account for time zone changes. NTP is used to distribute this time across internal networks. The NTP client requests the time from the NTP server and the client then calculates the link delay and local offset to adjust its time to match the server’s clock. Several of these exchanges occurs within the first five to ten minutes after configuration to initially synchronize its clock and, typically, the client then updates its clock every few minutes thereafter.

Considerations

The distributed nature and use of multiple time servers provides NTP with some security; however, it is not secure by default and administrators should use an access list-based restriction scheme and an encryption authentication mechanism. NTP is susceptible to both man-in-the-middle (MitM) and distributed denial-of-service (DDoS) attacks. Previous versions of NTP have also been vulnerable to stack-based buffer overflow exploits. Additionally, NTP message spoofing could be used to adjust the time on client computers.

As with all software, administrators must continue to update to the latest version of NTP in order to limit a potential attackers’ ability to exploit known vulnerabilities. Earlier this month, the Network Time Foundation’s NTP Project released NTP version 4.2.8p10 to address multiple vulnerabilities in NTP Daemon (ntpd) that could allow a remote attacker to cause a denial-of-service (DoS) condition. A successful DoS attack on an NTP server could desynchronize timekeeping and result in secondary and tertiary effects that impact network performance.

On February 11, 2016, a security researcher posted a YouTube video that demonstrated how time synchronization issues can affect the everyday user; he found that manually setting the date of an iPhone or iPad back to January 1, 1970 would “brick” the device, rendering it useless. This occurred because Apple’s iPads and other devices consistently check various NTP servers around the globe to sync their internal date and time clocks. Brian Krebs later reported on his blog that two other researchers, Patrick Kelley and Matt Harrigan, discovered that if an attacker builds a hostile WiFi network to compel the device to download date and time updates from a malicious NTP time server, they could force the device to reset their clocks to January 1, 1970.

Time Synchronization Options

Organizations have the option to procure their own dedicated network time server, which protects against inherent security risks from obtaining time from internet time servers. If a time server is installed within the network firewall, risks from the outside are greatly minimized and the timing accuracy increases. Various products allow for time synchronization with non-network external clocks like GPS satellites, CDMA cell phone networks, and government radio signals, such as WWVB. At a minimum, organizations should review their network’s time synchronization configuration and determine if utilizing NTP or synchronizing with a non-network external clock is best for their network in terms of cost, reliability, and security.

Close Public NTP and Secure Port 123

Ideally, NTP is configured in a client-server model where the server contacts the NTP source to sync time.

Administrators should configure their system to ensure the NTP server acts as a client only. Additionally, to prevent NTP server floods, it is recommended to shut down the NTP daemon and block inbound traffic from port 123, the NTP default. Technical details on the commands used to configure these recommendations can be found here.

NTP DDoS Mitigation

The following are some measures to put in place to ensure that your NTP servers are not used to attack others, and to minimize the impact of NTP DDoS attacks against your network:

• Test if your NTP server is vulnerable to being used in an attack, follow the instructions provided by the OpenNTP Project at openntpproject.org.
• Upgrade to the newest version of NTP as quickly as possible once an update is available.
  • All versions of ntpd server prior to 4.2.7 are vulnerable to NTP amplification attacks by default, due to the monlist command. The latest versions of NTP remove the monlist command entirely, and therefore mitigate the risk from that tactic.
  • Implement a version of NTP that does not utilize the monlist command, such as OpenNTPD.
  • If you are unable to upgrade your server to the latest version, disable the monlist query feature by adding “disable monitor” to your ntp.conf file and restarting the NTP process.
• Implement firewall rules that restrict unauthorized traffic to the NTP server.
• Attackers usually spoof the source IP address of DDoS attacks – if possible, ensure firewalls are configured to filter and restrict forged inbound traffic from the internet.
• Implement tighter Access Control Lists (ACLs) in their public facing transit edges across Layer 3 devices like switches and hardware-based routers. This may help to mitigate the malicious traffic from reaching targeted resources; however, if the attack amplification factor is large, transit edge devices will still be disrupted.

If you are the target of an NTP amplification DDoS attack:

• Investigate your network logs and look for inbound traffic with a source port of 123/UDP and a specific packet size.
• Once identified, contact your upstream network service provider and provide them with the attacking IP addresses and the packet sizes used in the attack. Upstream providers may be able to filter and drop the inbound NTP traffic using the specific packet size that you provide.

More resources on properly configuring and maintaining of NTP servers:

• Microsoft: How to configure an authoritative time server in Windows Server
• InfoSec Institute: Network Time Protocol (NTP): Threats and Countermeasures
• Wikipedia: Network Time Protocol