NSIT: Network Security Center - flow-extract man page

Network Security Center

flow-extract man page

NAME

flow-extract --- flow-extract - Displays flows from a flow file.

SYNOPSIS

flow-extract [-aDRBbnh] [ -e program ] [ -E program ] [ -f file ] [ -F file ] [ -d dumpfile ] [-o outfile ] [ -z complevel ]

DESCRIPTION

The flow-extract is used for selecting flows from a binary log file created by Flow-tools(1). The program uses the same syntax as the Netlogger extract program to select flows.

OPTIONS

-a

Output matching flows in ASCII format (the default).

-B

Specify the endianness of the input ('big' or 'little').

-b

Output matching flows in binary format. This is useful for selecting data which can then be processed by flow-tools such as flow-stat(1). This option implies -n.

-n

Do not resolve IP address or port information when displaying the flows in ASCII format.

-D

Do not port information, but do resolve IP addresses when displaying the flows in ASCII format.

-R

Do not resolve IP address, but do resolve port information when displaying the flows in ASCII format.

-n

Do not resolve port information when displaying the flows in ASCII format. IP addresses will be resolved, however.

-h

Print an informative help message describing the options and then exit.

-e program

Specify an extract program on the command line.

-E program

Specify an extract program on the command line.

-f file

Specify a script containing an extract program.

-F file

Specify a script containing an extract program.

Multiple -e, -f, -E, and -F options can be specified. They are all combined together and treated as a single program. Programs specified by -e and -f are appended to the program, while programs specified with -E and -F are prepended. -E and -F can be used with extract scripts to insert addition conditions at the top of the script without having to edit it each time.

-d dumpfile

Specify the datafile to read as input. Defaults to standard input.

-o outfile

Specify the output file. Defaults to standard output.

-z complevel

Specify the compression level. Flow-extract compresses files by default using zlib. This option specifies how much the files should be compressed. Defaults to 6, valid numbers are 0 (no compression) to 9.

USAGE

Programs

program consists of selector-action statements. Statements have the form

: selector { action }

A missing selector means to always select the flow. Flow-extract processes the entire program for each flow in the log file unless a matching statement contains a next action.

Actions

There are three actions currently define, print, printall and next.

Print outputs the flow in the format selected using the -a and -b switches.

Printall outputs the flow with all of the information in it. This format displays the data that is specific to various flow versions. Its format is consistent between all versions. Those fields which do not exist in the flow version that is being processed contain a dash. If -b is used for binary output there is no difference between .BI print and printall as the binary version contains all of the flow information regardless.

Next skips to the next flow and restarts scanning of program.

Selectors

The following selectors are available.

net =, != netname|ipaddr[/netmask]

Select flows if the source or destination network is [not] in the network specified. The optional netmask allows subnets to be specified.

srcnet =, != netname|ipaddr[/netmask]

Select flows if the source network is [not] in the network specified. The optional netmask allows subnets to be specified.

dstnet =, != netname|ipaddr[/netmask]

Select flows if the destination network is [not] in the network specified. The optional netmask allows subnets to be specified.

host =, !=, <, >, ><, <=, >= hostname|ipaddr

Select flows based on the source or destination host.

srchost =, !=, <, >, ><, <=, >= hostname|ipaddr

Select flows based on the source host.

dsthost =, !=, <, >, ><, <=, >= hostname|ipaddr

Select flows based on the destination host.

iface =, !=, <, >, ><, <=, >= interface

Select flows based on the router interface.

srciface =, !=, <, >, ><, <=, >= interface

Select flows based on the source interface.

dstiface =, !=, <, >, ><, <=, >= interface

Select flows based on the destination interface.

hp =, !=, <, >, ><, <=, >= hostname|ipaddr service-name|port

Select flows based on the source or destination host-port combination. For example, hp = mail.uchicago.edu SMTP will match any flows where the source host is mail.uchicago.edu and the source port is SMTP or any flows where the destination host is mail.uchicago.edu and the destination port is SMTP.

srchp =, !=, <, >, ><, <=, >= hostname|ipaddr service-name|port

Select flows based on the source host-port combination.

dsthp =, !=, <, >, ><, <=, >= hostname|ipaddr service-name|port

Select flows based on the destination host-port combination. srcport =, !=, <, >, ><, <=, >= service-name|port

Select flows based on the UDP or TCP source port. Service-name is an entry from the /etc/services file. Port is an integer number indicating the port.

dstport =, !=, <, >, ><, <=, >= service-name|port

Select flows based on the UDP or TCP destination port. Service-name is an entry from the /etc/services file. Port is an integer number indicating the port.

proto =, !=, <, >, ><, <=, >= number

Selects flows based on the protocol number.

pkts =, !=, <, >, ><, <=, >= number

Selects flows based on the number of packets in the flow.

octets =, !=, <, >, ><, <=, >= number

Selects flows based on the number of octets (bytes) in the flow.

flag FIN|SYN|RST|PUSH|ACK|URG

Selects flows based on the TCP flags set on the flow.

flags safrpu/safrpu

Selects flows based on the TCP flags set on the flow. The flags on the right of the slash are the mask for the flags on the left. That is, if the right hand side has saf set and the left hand side has s set, any flows which have the SYN packet set and the ACK and FIN packets not set will be selected.

Select flows based on the date of the flow.

time =, !=, <, >, ><, <=, >= HH:MM:SS | HH:MM

Select flows based on the time of the flow.

before YYYY-MM-DD [HH:MM] | YY-MM-DD [HH:MM] | MM/DD/YY [HH:MM] | today [HH:MM] | yesterday [HH:MM]

Select flows based on the time of the flow.

since YYYY-MM-DD [HH:MM] | YY-MM-DD [HH:MM] | MM/DD/YY [HH:MM] | today [HH:MM] | yesterday [HH:MM]

Select flows based on the time of the flow.

Combining Selectors

Selectors can be combined using the following operators.

and,: && Short circuited boolean AND.

or,: || Short circuited boolean OR.

not,: ! Unary negation operators.

Parenthesis can be used to group expressions.

EXAMPLES

Print in ASCII any records showing attempts to ssh to host myhost.

flow-extract -e 'dsthost = myhost && dstport = ssh {print}' -d ft-v06.2002-02-15.200000

Show all traffic on 2001-02-15 to the 128.135.119.0/255.255.255.0 subnet and sent it to flow-stat to find the top bandwidth user by port.

flow-cat ft-v06.2002-02-15* | flow-extract -be 'net = 128.135.119.0/255.255.255.0 {print}' | flow-stat -f7 | sort -nr +2 -3

Show the details of all traffic crossing router interface 43 that is a successful TCP connection.

flow-extract -e 'iface = 43 && flag ack && ! flags ar/arf {printall}'

Sample script called unusual to print records for unexpected TCP traffic. This would have to be tailored to a site.

#!/opt/bin/flow-extract -f
# Script 'unusual' for printing unusual traffic
port = ssh || port = SMTP || port = http || port = 113 {next}
hp = news.uchicago.edu NNTP {next}
# If we made it to here, then print the record
{print}

Run the previous script, but only show records for today.

unusual -E 'before today {next}'

BUGS

This man page needs more examples.

AUTHORS

flow-extract was ported from the TAMU extract program to use flow data by E. Larry Lidz. Large portions of the code (and this man page) come directly from there. Many flow specific features have been added to the program by E. Larry Lidz as well. If you're interested in the changes, see the ChangeLog with the source, or compare this man page to extract(8).