Tuesday, April 8, 2014

Exporting firewall rules to a CSV

Sometimes it can be useful to export and analyze rules in a CSV type format. This comes in especially handy when working with long and complex firewall policies.

I came across the perl script below that takes firewall policies from a text file and performs the CSV conversion for you.

Syntax: csvparse.pl rules.txt

<rules.txt> should be in the following format:
config firewall policy
     edit 1
         set srcintf "internal"
         set dstintf "wan1"
             set srcaddr "all"
             set dstaddr "all"
         set action accept
         set schedule "always"
             set service "ANY"
         set logtraffic-app disable
         set webcache enable
         set nat enable
     next
end
And here's the Perl script.
#!/usr/bin/perl
#

my $output = "policies-out.csv";

my $policyid = 0;
my $setting = "";
my %policies;
my %seen;
my $in_policy_block = 0;
my @order_keys;
my $order_key = 0;

open(OUTFILE,">$output") || die "Can't open file $output: $!\n";

while (<>) {
	if ($in_policy_block) {
		if (/^\s*edit\s+(\d+)/i) {
			# start of new policy
			$policyid = $1;
		} elsif (/^\s*set\s+(\S+)\s+(.*)$/i) {
			# it's a setting
			my ($key,$value) = ($1,$2);
			$value =~ tr/\"\015\012\n\r//d;
			$order_keys[$order_key++] = $key unless $seen{$key}++;
			$policies{$policyid}{$key} = $value;
		} elsif (/^\s*end/i) {
			$in_policy_block = 0;
		}
	} elsif (/^\s*config firewall policy/i) {
		$in_policy_block = 1;
	}
}

# print out our header
print OUTFILE "id";
foreach my $key (@order_keys) {
	print OUTFILE ",$key";
}
print OUTFILE "\n";

# now print out each record
foreach my $policy (sort keys %policies) {
	print OUTFILE "$policy";
	foreach my $key (@order_keys) {
		if (defined($policies{$policy}{$key})) {
			print OUTFILE ",$policies{$policy}{$key}";
		} else {
			print OUTFILE ",";
		}
	}
	print OUTFILE "\n";
}


close(OUTFILE);

24 comments:

Unknown said...

Great script Sebastian. Tks for sharing!

Francois Ropert said...

oh my! perl devs still exists in 2014??? :)

Alexander N said...

Thanks for sharing!

This brings up another question I never managed to figure out...

How do you list the rule order in cli (or from a configuration backup file) ?

Thank you in advance!

Simon said...

That's really cool...but for me we have a lot of clients with identity based profiles...and this cause the script to break, I guess because of the extra "end" statements.

I may try and take a look sometime as I love the concept, but I'm no Perl expert.

Thanks for sharing.

Anonymous said...

Nice script Sebastian.

I ported it to Python and made it public on github : https://github.com/maaaaz/fgpoliciestocsv

I also took the liberty of putting your version in the same repo.

I tested both scripts against several production configuration files. No bug so far. Feel me to report some.

Unknown said...

I know this can help me soon, so I already bookmarked it :)
firewall appliance is very important specially for the sake of my kids.

Unknown said...

I've already bookmarked this post, thanks! I am also using utm firewall appliance to help me protect My pc.

Unknown said...

Hi, thank you for great script, but i need little help with making the sort of rules in original sequence and i'm not strong in perl - if i delete the 'sort' in 'foreach my $policy ..' it will output rules in random order, so i need to put some index (something like $linenum) instead of policy id in hash and i cannot figure out, how to do it. Thank you ...

Unknown said...

Sorry, being unknown, i'm Eric ;)

Unknown said...

Well, i finally figure out, how to sort it in sequential order.

It's pobably not so elegant, but it works for me.

here is diff ...

12a13
> my $policyseq = 1;
15c16,17
<
---
> $key = "id";
> $order_keys[$order_key++] = $key;
20a23
> $policies{$policyseq}{$key} = $policyid;
26c29
< $policies{$policyid}{$key} = $value;
---
> $policies{$policyseq}{$key} = $value;
28a32,33
> } elsif(/^\s*next/i) {
> $policyseq++;
36c41
< print OUTFILE "id";
---
> print OUTFILE "seq";
43c48
< foreach my $policy (sort keys %policies) {
---
> foreach my $policy (sort {$a <=> $b} keys %policies) {


It shows 'seq' along with 'id' at the beginning.

PR said...

Has anyone tried mapping firewall objects to IP addresses?

Pierrot said...

Bonjour, est il possible de prendre en compte l'affichage des VDOM avec ce script ?

Merci

Pierre

Unknown said...

I updated it a bit to make it work for identity based policy (and keep the order)
seems to work for me, hope for someone else too

this is my diff to the original file

6a7
> my $id_policyid = 0;
11a13
> my @pol_list;
20a23
> push @pol_list, $policyid;
26a30,45
> } elsif (/^\s*config identity-based-policy/i) {
> ID_POL: while (<>) {
> if (/^\s*edit\s+(\d+)/i) {
> # start of new policy
> $id_policyid = $1;
> push @pol_list, "$policyid-$id_policyid";
> } elsif (/^\s*set\s+(\S+)\s+(.*)$/i) {
> # it's a setting
> my ($key,$value) = ($1,$2);
> $value =~ tr/\"\015\012\n\r//d;
> $order_keys[$order_key++] = $key unless$seen{$key}++;
> $policies{"$policyid-$id_policyid"}{$key} = $value;
> } elsif (/^\s*end/i) {
> last ID_POL;
> }
> }
43c62
< foreach my $policy (sort keys %policies) {
---
> foreach my $policy (@pol_list) {
54d72
<

Unknown said...

At line 56, I replaced:
^\s*end
by:
^end

then AMAZING!!!

Thx a lot for this script :)

Unknown said...

Could you please upload a final script? I'm not familiar with diff files.
Thanks.

Unknown said...

Hey guys
Does it possible for any one to share video of how to do it ??


Tnx

Unknown said...

worked like charm.
tnx man!

Unknown said...

Perfect !

SAIASP_Admin said...

Great stuff. Thanks for sharing. Saving me loads of work.

LVAC said...


# "show firewall policy" on router then save results to configfirewallpolicy.txt python version 3
import os
import sys
import csv

def procLines(lines):
AllRules=[]
d={}
current=''
sts={}

for l in lines:
l=l.replace('\n','')
if 'config' in l:
continue
if 'end' in l:
continue
if 'uuid' in l:
continue

if 'edit ' in l:
rule=l.split('edit ')[1]
current=rule
sts['rule']=current
d[current]={}

if 'set' in l:
sl=l.split(' ')
sts[sl[9]]= l.split(sl[9]+' ')[1].replace('" "',', ').replace('"','')

if 'next' in l:
d[current]=sts
sts={}
#print(d[current])
return d

def ExportCsv(dic):
fieldnames=[]
# print( dic['49'].keys())
for r in dic.keys():
for c in dic[r].keys():
if c not in fieldnames:
fieldnames.append(c)
#print(fieldnames)
with open('firewall.csv','w+') as f:
w=csv.DictWriter(f,fieldnames=fieldnames,dialect='excel')
w.writeheader()
for r in dic.keys():
w.writerow(dic[r])








if __name__ == '__main__':
f=open('configfirewallpolicy.txt','r')
rlines = f.readlines()
firewall=procLines(rlines)
f.close()
ExportCsv(firewall)

Anonymous said...

Hi, have you tried it for version 5.6 ?? thank you, I appreciate your response

Anonymous said...

I have configured multiple schedule in my firewall and i want to export it to csv file , Please anyone can share the script for the same

n1c00 said...

5.6.9 here. Seems to be working like a charm!

Abhishek Singh said...

Thanks man. Saved a lot of time and effort. Had 100+ policies. Thanks again for a this solution.