CLI Commands for Troubleshooting FortiGate Firewalls
This blog post is a list of common troubleshooting commands I am using on the FortiGate CLI. It is not complete nor very detailled, but provides the basic commands for troubleshooting network related issues that are not resolvable via the GUI. I am not focused on too many memory, process, kernel, etc. details. These must only be used if there are really specific problems. I am more focused on the general troubleshooting stuff. I am using it personally as a cheat sheet / quick reference and will update it from time to time.
Coming from Cisco, everything is “show”. With Fortinet you have the choice confusion between show | get | diagnose | execute. Not that easy to remember. It is “get router info6 routing-table” to show the routing table but “diagnose firewall proute6 list” for the PBF rules. Likewise the sys | system keyword. It is always “diagnose sys” but “execute system”.
#show system interface port1
#get system global
#diagnose fmnetwork netstat list
#get system status
#execute lvm info
#execute add-vm-license "-----BEGIN FAZ VM LICENSE---- ---"
https://docs.fortinet.com/document/fortianalyzer/7.2.0/cli-reference/486545/add-vm-license
https://help.fortinet.com/fdb/5-0-0/html/source/tasks/t_network_configuration_cli.html
Configuring Network Settings using the CLI
This topic describes the steps to configure your network settings using the CLI. For details about each command, refer to the Command Line Interface section.
Entering the correct vdom/gobal config
Remember to enter the correct vdom or global configuration tree before configuring anything:
1
2
3
|
config global
config vdom
edit <vdom–name>
|
To execute any “show” command from any context use the sudo keyword with the global/vdom-name context followed by the normal commands (except “config”) such as:
1
2
3
4
|
sudo {global | <vdom–name>} {diagnose | execute | show | get} ...
sudo global show system admin
sudo root get system interface physical
|
Show running-config & grep & scp
To show the running configuration (such as “show run” on Cisco) simply type:
1
|
show
|
To show the entire running configuration with default values use:
1
|
show full-configuration
|
When you are in a config submenu you can list the subsequent configuration options with all further submenus with:
1
|
tree
|
For example:
To omit the “–More–” stops when displaying many lines, you can set the terminal output to the following, which will display all lines at once. This is similar to “terminal length 0” from Cisco. Be careful with it, because this command is persistent. Set it to default after usage!
1
2
3
|
config system console
set output standard
end
|
To find a CLI command within the configuration, you can use the pipe sign “|” with “grep” (similar to “include” on Cisco devices). Note the “-f” flag to show the whole config tree in which the keywords was found, e.g.:
1
2
|
show | grep –f ipv6
show full–configuration | grep –f ipv6
|
Example with grep but WITHOUT the -f option (which makes no sense at all):
1
2
3
4
5
6
|
FGT90D # show | grep ipv6
set gui–ipv6 enable
config ipv6
config ipv6
config ipv6
set ipv6 2001:db8::cafe
|
Now with the -f option. Note the “<—” at the end of every line that has the “ipv6” keyword in it, while the full configuration part around it is listed.
You can even extend your grepping by using multiple expressions to grep, wrapped into single quotes and \|, such as: (Thanks to Ulrich’s comment!)
1
2
|
show | grep –f ‘internal\|wan’
diag vpn tunnel list | grep ‘name\|esp\|ah’
|
In order to copy the configuration via SCP from a backup server you must first enable the SCP protocol for the admin:
1
2
3
|
config system global
set admin–scp enable
end
|
before you can grab it from the backup server, e.g. Linux with:
1
2
|
scp <username>@<FortiGate–ip/name>:sys_config <destination>
scp admin@192.168.7.106:sys_config ~/fortigate–config–2017–11–20.txt
|
To save your config through the CLI in order to have it in the GUI under <username> -> Configuration -> Revisions, use:
1
|
execute backup config flash
|
Even better, you should enable the following feature which saves a backup of your configuration after each logout automatically:
1
2
3
|
config system global
set revision–backup–on–logout enable
end
|
General Information
The very basics:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
get system interface physical #overview of hardware interfaces
get hardware nic <nic–name> #details of a single network interface, same as: diagnose hardware deviceinfo nic <nic-name>
fnsysctl ifconfig <nic–name> #kind of hidden command to see more interface stats such as errors
get system status #==show version
get system performance status #CPU and network usage
execute sensor list #power supply, temperature, fans
execute sensor detail
diagnose sys top #top with all forked processed
diagnose sys top–summary #top easier, incl. CPU and mem bars. Forks are displayed by [x13] or whatever
execute dhcp lease–list
get system arp
diagnose ip arp list
diagnose ipv6 address list
diagnose ipv6 neighbor–cache list
diagnose sys ntp status
diagnose autoupdate versions #lists the attack definition versions, last update, etc.
diagnose log test #generated all possibe log entries
diagnose test application dnsproxy 6 #shows the IP addresses of FQDN objects
diagnose debug crashlog read #shows crashlog, a status of 0 indicates a normal close of a process!
|
After rebooting a fresh device which is already licensed, it takes some time until it is “green” at the dashboard. The following commands can troubleshoot and start the “get license” process. Use the first three to enable debugging and start the process, while the last one disables the debugging again:
1
2
3
4
|
diag debug app update –1
diag debug enable
exec update–now
diag debug disable
|
To reboot your device, use:
1
|
execute reboot
|
To shutdown your device, use:
1
|
execute shutdown
|
General Network Troubleshooting
Which is basically ping and traceroute. Unluckily it is shitty difficult to use those commands since you need a couple of subcommands to source pings from a different interface, and so on. Furthermore, the traceroute for IPv6 uses its options on the CLI directly such as “-i <interface>”, while traceroute for IPv4 uses the “traceroute-options …” subcommands:
1
2
3
4
5
6
7
8
9
10
11
12
|
execute ping6–options ?
execute ping6–options source <source–interface–IP>
execute ping6 <hostname|ip>
execute ping–options ?
execute ping–options source <source–interface–IP>
execute ping <hostname|ip>
execute tracert6 <hostname|ip>
execute traceroute <hostname|ip>
execute traceroute–options ?
|
To view the current …-options, use this:
1
2
3
|
execute ping–options view–settings
execute ping6–options view–settings
execute traceroute–options view–settings
|
Routing
Routing table, RIB, FIB, policy routes, routing protocols, route cache, and much more. 😉 Note the differences between IPv6 and legacy IP.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
get router info6 routing–table #routing table = active routes
get router info routing–table all #IPv4 needs an “all” at the end
get router info6 routing–table database #Routing Information Base WITH inactive routes
get router info routing–table database
get router info6 kernel #Forwarding Information Base
get router info kernel
diagnose firewall proute6 list #Policy Routes + WAN Load Balancing
diagnose firewall proute list
get router <routing–protocol> #basic information about the enabled routing protocol
diagnose ip rtcache list #route cache = current sessions w/ routing information
|
High Availability
Diagnose and managing: (Just another **** example on how “get | diagnose | execute” is mixed along with “sys | system”.)
1
2
3
4
5
|
get system ha status
diagnose sys ha status
execute ha manage ? #switch to the CLI of a secondary unit
execute ha manage <device–index>
diagnose sys ha checksum show #verify the checksum of all synchronized peers
|
Manually test a failover by decreasing the priority of the current master (since highest priority wins):
1
|
execute ha set–priority <serial–number> <new–priority>
|
Don’t forget to restore the priority value to your original one!
Start a sync at a secondary device to (from?) the master: (Honestly, I am not sure what “synchronize” means in this command. I would like to decide which config to push to the other device. The Fortinet documentation reads: “Use this command from a subordinate unit in an HA cluster to manually synchronize its configuration with the
primary unit or to stop a synchronization process that is in progress.”)
1
|
execute ha synchronize {start | stop}
|
Session Table
Display the current active sessions:
1
2
3
4
5
6
7
|
get system session list #rough view with NAT, only IPv4
diagnose sys session filter clear
diagnose sys session filter ?
diagnose sys session filter dst 8.8.8.8
diagnose sys session filter dport 53
diagnose sys session list #show the session table with the filter just set
|
Remote Server Authentication Test
In order to test user credentials against some (remote) authentication servers such as LDAP or RADIUS or even local:
1
2
3
|
diagnose test authserver ldap <server_name> <username> <password>
diagnose test authserver radius <server_name> <chap | pap | mschap | mschap2> <username> <password>
diagnose test authserver local <group_name> <username> <password>
|
FSSO User Authentication
When you’re using some kind of Fortinet single sign-on (FSSO) features such as the agentless/agent polling mode to a Windows AD you can use the following commands to get some information about the recognized users and agent servers:
1
2
3
|
diagnose debug authd fsso list
diagnose debug authd fsso server–status
diagnose firewall auth list
|
The first one shows all monitored users with details concerning their LDAP groups:
while the last one shows the users with their corresponding FortiGate user groups and traffic counters:
If you need further debugging messages you can enable it for the Fortigate non-blocking auth daemon and the FSSO daemon:
1
2
3
|
diagnose debug enable
diagnose debug application fnbamd 255
diagnose debug application fssod 255
|
Sniffer / Packet Capture
Sniff packets like tcpdump does. (Only if the built-in packet capture feature in the GUI does not meet your requirements.) This can be used for investigating connection problems between two hosts. There are no details of the firewall policy decisions. Use the debug flow (next paragraph) for analysis about firewall policies, etc.
1
|
diagnose sniffer packet <interface|any> ‘<tcpdump-filter>’ <verbose> <count> <time–format>
|
with:
verbose:
1: print header of packets
2: print header and data from ip of packets
3: print header and data from ethernet of packets (if available)
4: print header of packets with interface name <<<<<< good default choice
5: print header and data from ip of packets with interface name
6: print header and data from ethernet of packets (if available) with intf name
count: number of packets
time-format:
a: UTC time
l: local time
Examples: (Thanks to the comment from Ulrich for the IPv6 example)
1
2
3
4
|
diagnose sniffer packet any ‘host 8.8.8.8’ 4 4 l
diagnose sniffer packet any ‘host 8.8.8.8 and dst port 53’ 4 10 a
diagnose sniffer packet wan1 ‘dst port (80 or 443)’ 2 50 l
diagnose sniffer packet any ‘net 2001:db8::/32’ 6 1000 l
|
Here are two more examples on how to show LLDP or CDP packets in order to reveal the connected layer 2 ports from switches. Kudos to Joachim Schwierzeck.
1
2
3
4
5
|
LLDP:
diagnose sniffer packet port1 ‘ether proto 0x88cc’ 4 1 a
CDP:
diagnose sniffer packet port1 ‘ether[20:2] == 0x2000’ 6 1 a
|
Flow
If you want to see the FortiGate details about a connection, use this kind of debug. E.g., it shows the routing decision and the policy, which allowed the connection.
1
2
3
4
5
6
7
8
9
|
diagnose debug reset
diagnose debug flow filter ?
diagnose debug flow filter saddr 172.16.23.11
diagnose debug flow filter daddr 8.8.8.8
diagnose debug flow show function–name enable
diagnose debug enable
#display the next 10 packets:
diagnose debug flow trace start 10
diagnose debug disable
|
Example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
fg2 # diagnose debug reset
fg2 # diagnose debug flow filter daddr 1.1.1.1
fg2 # diagnose debug flow show function-name enable
show function name
fg2 # diagnose debug enable
fg2 # diagnose debug flow trace start 10
fg2 #
id=20085 trace_id=169 func=print_pkt_detail line=5455 msg=“vd-root received a packet(proto=17, 194.247.4.10:1492->1.1.1.1:53) from local. “
id=20085 trace_id=169 func=resolve_ip_tuple_fast line=5535 msg=“Find an existing session, id-06868db4, original direction”
id=20085 trace_id=169 func=__ip_session_run_tuple line=3332 msg=“run helper-dns-udp(dir=original)”
id=20085 trace_id=170 func=print_pkt_detail line=5455 msg=“vd-root received a packet(proto=17, 194.247.5.6:37400->1.1.1.1:53) from internal. “
id=20085 trace_id=170 func=init_ip_session_common line=5619 msg=“allocate a new session-0686a887”
id=20085 trace_id=170 func=vf_ip4_route_input line=1604 msg=“find a route: flags=00000000 gw-194.247.4.1 via wan1”
id=20085 trace_id=170 func=fw_forward_handler line=748 msg=“Allowed by Policy-1: AV”
id=20085 trace_id=170 func=ids_receive line=277 msg=“send to ips”
id=20085 trace_id=170 func=av_receive line=293 msg=“send to application layer”
id=20085 trace_id=171 func=print_pkt_detail line=5455 msg=“vd-root received a packet(proto=17, 194.247.5.6:37400->1.1.1.1:53) from local. “
id=20085 trace_id=171 func=resolve_ip_tuple_fast line=5535 msg=“Find an existing session, id-0686a887, original direction”
id=20085 trace_id=171 func=__ip_session_run_tuple line=3332 msg=“run helper-dns-udp(dir=original)”
id=20085 trace_id=172 func=print_pkt_detail line=5455 msg=“vd-root received a packet(proto=17, 194.247.4.10:1492->1.1.1.1:53) from local. “
id=20085 trace_id=172 func=resolve_ip_tuple_fast line=5535 msg=“Find an existing session, id-06868db4, original direction”
id=20085 trace_id=172 func=__ip_session_run_tuple line=3332 msg=“run helper-dns-udp(dir=original)”
fg2 # diagnose debug disable
|
VPN
To show details about IKE/IPsec connections, use these commands:
1
2
3
4
5
6
|
get vpn ike gateway <name>
get vpn ipsec tunnel name <name>
get vpn ipsec tunnel details
diagnose vpn tunnel list
diagnose vpn ipsec status #shows all crypto devices with counters that are used by the VPN
get router info routing–table all
|
To debug IKE/IPsec sessions, use the VPN debug:
1
2
3
4
5
6
7
|
diagnose debug reset
diagnose vpn ike log–filter clear
diagnose vpn ike log–filter ?
diagnose vpn ike log–filter dst–addr4 1.2.3.4
diagnose debug app ike 255 #shows phase 1 and phase 2 output
diagnose debug enable #after enough output, disable the debug:
diagnose debug disable
|
To reset a certain VPN connection, use this (Credit):
1
|
diag vpn tunnel reset <phase1 name>
|
Log
For investigating the log entries (similar to the GUI), use the following filters, etc.:
1
2
3
4
5
6
7
|
execute log filter reset
execute log filter category event
execute log filter field #press enter for options
execute log filter field dstport 8001
execute log filter view–lines 1000
execute log filter start–line 1
execute log display
|
Defaults
Just a reminder for myself:
- IP: 192.168.1.99
- Login: admin
- Password: <blank>
To change the IP address of the mgmt interface (or any other) via the CLI, these commands can be used:
1
2
3
4
5
6
|
config system interface
edit mgmt
set ip 192.168.1.1 255.255.255.0
set allowaccess ping http https ssh snmp
next
end
|
Password Recovery & Factory Reset
Just the links here: Resetting a lost Admin password and How to reset a FortiGate with the default factory settings.
Links
- Fortinet: FortiOS Admin Guides
- itsecworks: Fortigate troubleshooting commands
- https://weberblog.net/cli-commands-for-troubleshooting-fortigate-firewalls/