Network Monitoring - Information for ITSS

Overview

Welcome to the ITSS support page for OxCERT's network monitoring services. This page provides information for IT Support Staff on our intrusion detection system and related alert procedures, as well as guidance on data collection requirements and logging of network access.

Intrusion detection system (IDS)
Data collection requirements for Network Managers
Tracing network abuse

Intrusion detection system (IDS)

In order to monitor network traffic between the University network and the internet for malicious activity and policy violations, OxCERT is implementing a new intrusion detection system, based on the Open Source product Suricata. The IDS tool will initially be using Open Source signatures (from Emerging Threat) and publicly available indicators of compromise (IOCs), and will be integrated with OxCERT's Request Tracker and security information and event management system, SAVANT (more information on this integration).

High-level IDS architecture

The schematic drawing below illustrates the high-level architecture of the IDS system. Beginning with Network Traffic, leading to IDS Signatures, then Data Enrichment, SAVANT, Custom Script and finally Alerts.

Alert procedure

OxCERT will send alerts flagged by this IDS system directly to responsible teams across the colleges and departments of the University. Alerts will be generated when an indicator of a specific potential cyber threat activity is flagged. A ticket with OxCERT's Request Tracker (RT) will be created and an email will be sent to the team responsible for the IPs upon which the potential threat has been flagged. The ticket, depending on the specific alert, will generally contain the following information:

A timestamp for when the suspicious connection was made
Transport
Source IP
Destination IP
URL
Information regarding the network traffic and what indicator has been flagged
Links to further information on the indicator that has been flagged

If similar traffic from new IP addresses is seen on your network, OxCERT will alert you straight away on the same ticket. OxCERT will also update this ticket with daily summary emails if the suspicious traffic is still being detected. These daily summary emails will be sent at 06:00(GMT).

Example alert from our intrusion detection system

If you receive an email alert from OxCERT's IDS system, it will appear in the format shown in the example below. The content will vary depending on the indicator of compromise campaign that is currently running. In this example, details are given of the potential cyber threat activity including the timestamp, source, destination and URL, as well as the action that should be taken and who to contact for further support. (If you need to receive this example in a different format, please contact oxcert@infosec.ox.ac.uk).

From: OxCERT <oxcert-NNNNNN@infosec.ox.ac.uk>
To: Department of Important Studies <security-contact@unit.ox.ac.uk>
Subject: [CRITICAL] Possible Shlayer trojan traffic from your network (unit)

Hello,

This is an automated alert from OxCERTs Intrusion Detection System (IDS). The IDS has flagged that the following is an indicator of potential cyber threat activity:

Suspicious connection(s) made from your network consistent with the Shlayer trojan:

Timestamp                 | Source                           | Destination               | URL
--------------------------+----------------------------------+---------------------------+------------------------
2021-01-01T00:00:00.010Z  | SOURCE ADDRESS (source.ox.ac.uk) | DESTINATION ADDRESS (tcp) | http://example-url.com

(For brevity, we've only printed the first 5 detections for any given Oxford IP in the table above. You can find all detections in the attached .json file.)

Please can you identify the machine(s) used, isolate them from the network, and have them checked for malware. Please let us know the results of your investigation. We recommend that you also install the standard University endpoint-protection (Sophos) on the device before allowing them to reconnect to the network.

This network traffic is consistent with variants of the Shlayer Trojan, some Mac-specific malware. You can get more information about Shlayer Trojan from:
* https://blog.malwarebytes.com/detections/adware-shlayer/
* https://macsecurity.net/view/327-shlayer-trojan-mac

If we see similar traffic from new IP addresses on your network, we will alert you straight away. We will also update this ticket with daily summary emails if detections continue.

As always, please do contact us if you have any questions about this alert or suggestions for improvement to this system.

Best wishes,

Your friendly network alert bot

OxCERT has no access to your unit's infrastructure, upon receiving an alert please take action to investigate and remediate the flagged issue. OxCERT is happy to answer any questions you have and offer help, please do just get in contact via the ticket raised or telephone.

SAVANT dashboard

OxCERT will be creating a dashboard in SAVANT which provides a detailed view of all IDS alerts in your unit's sub-estate. More information on this dashboard will become available in due course.

Data collection requirements for Network Managers

It is vital within the university environment that the source of any abusive or malicious network traffic can easily be traced and isolated. Depending on the nature of the problem, it may be necessary to determine either the user or the computer responsible for particular traffic at a particular time. OxCERT will expect colleges and departments to be able to trace either upon request.

Reasons for requiring traceability include, but are not limited to, the following:

malicious network traffic (scanning, sending spam, communicating with known Command and Control system, etc.)
determining users affected by a security incident (e.g. users whose passwords have been exposed to a keylogger)
suspected access of illegal materials
alleged copyright infringement
violations of University IT regulations
troubleshooting

Note in particular that the prevalence of keylogger infections means that it is increasingly necessary to be able to determine who has used a known compromised system.

Tracing network abuse

The information below explains the procedures generally used in order to trace network abuse. It is the responsibility of each college and department to ensure that the necessary information is logged for users of their networks.

Traceability

In the vast majority of cases, a problem is seen to involve a particular IP address at a particular time, and will involve an address on a standard college or department backbone connection. Frequently it will be possible for OxCERT to identify the MAC address using the IP address at the time, for instance from ARP data or from DHCP logs.

Tracing single-user systems

In many cases, a particular computer is used by just one person. Without NAT, the IP address and timestamp are sufficient to identify both the computer and the person. Where the backbone routers can see the computer's true MAC address, the system can be isolated by a MAC address block. OxCERT will generally use MAC address blocks at the backbone router where possible, as this minimises the danger of collateral damage in environments using dynamic IP address pools.

If OxCERT can only supply an IP address, units may need to be able to map this to a MAC address themselves, for instance from their DHCP server logs. One of IP or MAC address should be sufficient to identify the system and its user.

Communal systems

Some abuse comes from communal systems, for instance those in a computer room or library. To determine who was using the system at a particular time, the system should require authentication before use, ideally with logs stored on a separate machine. In the case of "kiosk-mode" terminals where the functionality is severely restricted, it may be appropriate to allow usage without authentication. However if general access is permitted (for instance, access to arbitrary websites) then we would strongly recommend that adequate authentication and logging measures are in place.

Users should be reminded to log out when they are finished and to screen-lock the system should they leave it briefly unattended, in order to avoid others abusing the system under their account.

Multi-user systems

The trickiest to handle are multi-user systems, for instance general-purpose servers such as the GNU/Linux service. Such systems permit concurrent usage by multiple users. Utilities such as process accounting allow for some degree of per-user accountability.

Some servers are more specific in purpose, for instance mail relays or web proxies. These may authenticate users, and if not should at the very least log the IP address from which a connection originated.

Network Address Translation

NAT (strictly NAT/PAT) setups with multiple systems sharing a single public IP address, make tracing problems far harder, owing to the loss of the usual one-to-one mapping between host and public IP address. It is still essential that network traffic be traceable to a single host on the internal network.

Many NAT setups provide a certain amount of logging, for instance a log of all HTTP connections. While useful, these data are not sufficient, and will fail to log anything for certain types of malware.

In order to reliably trace the source of an outgoing network connection from a system behind NAT, it must be possible to map from the public data (timestamp, protocol, source and destination IP addresses, source and destination ports where appropriate) to the IP addresses and port numbers used on the internal network.

A tool such as Argus running on a mirror port may be useful for recording details of every network flow on the inside of a NAT gateway. For almost all purposes the data gathered will be sufficient. However, Argus will not log the NAT gateway's port number mappings as packets traverse the gateway. This may cause problems under certain circumstances. Ideally the gateway itself should log the mappings as used within its own state table. Please bear in mind that every outbound connection attempt which reaches the backbone network needs to be logged. It is not sufficient merely to log successful connections, as such logs will be missing information that is often vital to OxCERT.

Additionally, please see the network team's documentation on Network Address Translation for more general details on NAT.

DNS server logs

OxCERT may under certain circumstances need to make use of DNS resolver logs when handling incidents. Where clients are configured to use a local DNS resolver as opposed to the central University servers, OxCERT will identify potentially malicious DNS lookups originating from the IP address of the local resolver. To aid investigations, OxCERT strongly recommend that local DNS resolvers are clearly identifiable as such (via hostname or DNS comment), and, subject to local privacy policies allowing it, retain logs of queries so that the source of malicious DNS requests can be identified.

Timestamps

It is essential that all logs bear precise, accurate timestamps. OxCERT recommend that logs bear timestamps to a precision of at least the nearest second, and the logging systems' clocks are synchronised with the central NTP service. Please ensure that you are familiar with the timezone used in your logs (local time? always GMT?), and beware of confusion that may result from transitions to or from daylight savings time.

Incident handling

Where centrally-collected logs are insufficient to provide OxCERT the information necessary for an investigation, OxCERT may request a college or department to provide information from their own logs. For example, this may be system logs from a compromised server, or network and port translations from a NAT device.

Depending on the nature of the incident, OxCERT may request that the unit send their full logs for a stated time period. OxCERT will aim to specify as short a time period as is reasonably possible. Nevertheless if malicious traffic is observed on several occasions during a time period (for instance overnight), then OxCERT may request logs for the full time period as there may be more than one infected host active during that time.

Units may be concerned about providing personally-identifiable information (PII) in logs to OxCERT. Data provided to OxCERT will be handled in confidence, accessible only to members of OxCERT and for no longer than is necessary for the purposes of the investigation. Incident summary information will be stored in accordance with OxCERT's data retention policy.

Note that NAT logs will provide OxCERT with information regarding traffic from a unit's internal IP addresses, but in general will have no means of linking those to individuals; further data would be required in order to do so. Consequently most NAT logs should not contain PII; in cases where they do then units should feel free to remove or anonymise PII before sending data to OxCERT.

For further information regarding incident handling, please see the Incident Management section of our website.

Retention policy

OxCERT recommends that you have a policy in place as to how long logs should be retained. Too long and there may be significant storage overheads and issues with data protection legislation, but too short and logs may expire before an incident can be investigated.

The Data Protection Act 2018 is the UK's implementation of the General Data Protection Regulation (GDPR). It requires that personal data must not be kept for longer than necessary for the purposes for which it was collected. Retention for a reasonable period to prevent/investigate abuse is fine. Industry best current practice suggests that routine logs be kept for 3-6 months; longer if they are still required for a specific investigation. While the government is encouraging providers to keep logs for much longer for anti-terrorism reasons, such moves currently remain voluntary and affect only public networks; the University network and JANET are private. Whilst in most cases it will not be necessary to go back through logs more than a few days OxCERT's recommendation is that you store a minimum of 90 days' data.

Potential problems

Where logging is purely in terms of IP and MAC addresses, there are dangers of placing absolute trust in the data gathered. Users assigned one particular IP address can trivially switch to another. This can (usually) easily be spotted through use of tools such as arpwatch. Spoofing MAC addresses is only marginally more difficult, but is harder to track. Logging the MAC addresses associated with each switchport, and any changes, is a possible solution for some setups, for instance if each switchport connects to a different student's room. Ports in public areas are more difficult, and if occurrences of MAC-address spoofing become more frequent, then the long-term solution would be to go for a technology such as 802.1x, requiring the user to authenticate in order to gain network access.

For certain attacks, in which the attacking host is not interested in receiving a response, the source address of a packet may not even be an IP address actually used by the host. Egress filters at the boundary of your network and the University backbone prevent packets with source addresses other than those within your subnet allocation(s) from reaching other networks, but this is no defence against them spoofing another address within your allocation.