Tuesday, April 8, 2014

Exporting firewall rules to a CSV

Sometimes it can be useful to export and analyze rules in a CSV type format. This comes in especially handy when working with long and complex firewall policies.

I came across the perl script below that takes firewall policies from a text file and performs the CSV conversion for you.

Syntax: csvparse.pl rules.txt

<rules.txt> should be in the following format:
config firewall policy
     edit 1
         set srcintf "internal"
         set dstintf "wan1"
             set srcaddr "all"
             set dstaddr "all"
         set action accept
         set schedule "always"
             set service "ANY"
         set logtraffic-app disable
         set webcache enable
         set nat enable
     next
end
And here's the Perl script.
#!/usr/bin/perl
#

my $output = "policies-out.csv";

my $policyid = 0;
my $setting = "";
my %policies;
my %seen;
my $in_policy_block = 0;
my @order_keys;
my $order_key = 0;

open(OUTFILE,">$output") || die "Can't open file $output: $!\n";

while (<>) {
	if ($in_policy_block) {
		if (/^\s*edit\s+(\d+)/i) {
			# start of new policy
			$policyid = $1;
		} elsif (/^\s*set\s+(\S+)\s+(.*)$/i) {
			# it's a setting
			my ($key,$value) = ($1,$2);
			$value =~ tr/\"\015\012\n\r//d;
			$order_keys[$order_key++] = $key unless $seen{$key}++;
			$policies{$policyid}{$key} = $value;
		} elsif (/^\s*end/i) {
			$in_policy_block = 0;
		}
	} elsif (/^\s*config firewall policy/i) {
		$in_policy_block = 1;
	}
}

# print out our header
print OUTFILE "id";
foreach my $key (@order_keys) {
	print OUTFILE ",$key";
}
print OUTFILE "\n";

# now print out each record
foreach my $policy (sort keys %policies) {
	print OUTFILE "$policy";
	foreach my $key (@order_keys) {
		if (defined($policies{$policy}{$key})) {
			print OUTFILE ",$policies{$policy}{$key}";
		} else {
			print OUTFILE ",";
		}
	}
	print OUTFILE "\n";
}


close(OUTFILE);

24 comments:

  1. Great script Sebastian. Tks for sharing!

    ReplyDelete
  2. oh my! perl devs still exists in 2014??? :)

    ReplyDelete
  3. Thanks for sharing!

    This brings up another question I never managed to figure out...

    How do you list the rule order in cli (or from a configuration backup file) ?

    Thank you in advance!

    ReplyDelete
  4. That's really cool...but for me we have a lot of clients with identity based profiles...and this cause the script to break, I guess because of the extra "end" statements.

    I may try and take a look sometime as I love the concept, but I'm no Perl expert.

    Thanks for sharing.

    ReplyDelete
  5. Nice script Sebastian.

    I ported it to Python and made it public on github : https://github.com/maaaaz/fgpoliciestocsv

    I also took the liberty of putting your version in the same repo.

    I tested both scripts against several production configuration files. No bug so far. Feel me to report some.

    ReplyDelete
  6. I know this can help me soon, so I already bookmarked it :)
    firewall appliance is very important specially for the sake of my kids.

    ReplyDelete
  7. I've already bookmarked this post, thanks! I am also using utm firewall appliance to help me protect My pc.

    ReplyDelete
  8. Hi, thank you for great script, but i need little help with making the sort of rules in original sequence and i'm not strong in perl - if i delete the 'sort' in 'foreach my $policy ..' it will output rules in random order, so i need to put some index (something like $linenum) instead of policy id in hash and i cannot figure out, how to do it. Thank you ...

    ReplyDelete
  9. Sorry, being unknown, i'm Eric ;)

    ReplyDelete
  10. Well, i finally figure out, how to sort it in sequential order.

    It's pobably not so elegant, but it works for me.

    here is diff ...

    12a13
    > my $policyseq = 1;
    15c16,17
    <
    ---
    > $key = "id";
    > $order_keys[$order_key++] = $key;
    20a23
    > $policies{$policyseq}{$key} = $policyid;
    26c29
    < $policies{$policyid}{$key} = $value;
    ---
    > $policies{$policyseq}{$key} = $value;
    28a32,33
    > } elsif(/^\s*next/i) {
    > $policyseq++;
    36c41
    < print OUTFILE "id";
    ---
    > print OUTFILE "seq";
    43c48
    < foreach my $policy (sort keys %policies) {
    ---
    > foreach my $policy (sort {$a <=> $b} keys %policies) {


    It shows 'seq' along with 'id' at the beginning.

    ReplyDelete
  11. Has anyone tried mapping firewall objects to IP addresses?

    ReplyDelete
  12. Bonjour, est il possible de prendre en compte l'affichage des VDOM avec ce script ?

    Merci

    Pierre

    ReplyDelete
  13. I updated it a bit to make it work for identity based policy (and keep the order)
    seems to work for me, hope for someone else too

    this is my diff to the original file

    6a7
    > my $id_policyid = 0;
    11a13
    > my @pol_list;
    20a23
    > push @pol_list, $policyid;
    26a30,45
    > } elsif (/^\s*config identity-based-policy/i) {
    > ID_POL: while (<>) {
    > if (/^\s*edit\s+(\d+)/i) {
    > # start of new policy
    > $id_policyid = $1;
    > push @pol_list, "$policyid-$id_policyid";
    > } elsif (/^\s*set\s+(\S+)\s+(.*)$/i) {
    > # it's a setting
    > my ($key,$value) = ($1,$2);
    > $value =~ tr/\"\015\012\n\r//d;
    > $order_keys[$order_key++] = $key unless$seen{$key}++;
    > $policies{"$policyid-$id_policyid"}{$key} = $value;
    > } elsif (/^\s*end/i) {
    > last ID_POL;
    > }
    > }
    43c62
    < foreach my $policy (sort keys %policies) {
    ---
    > foreach my $policy (@pol_list) {
    54d72
    <

    ReplyDelete
  14. At line 56, I replaced:
    ^\s*end
    by:
    ^end

    then AMAZING!!!

    Thx a lot for this script :)

    ReplyDelete
  15. Could you please upload a final script? I'm not familiar with diff files.
    Thanks.

    ReplyDelete
  16. Hey guys
    Does it possible for any one to share video of how to do it ??


    Tnx

    ReplyDelete
  17. Great stuff. Thanks for sharing. Saving me loads of work.

    ReplyDelete

  18. # "show firewall policy" on router then save results to configfirewallpolicy.txt python version 3
    import os
    import sys
    import csv

    def procLines(lines):
    AllRules=[]
    d={}
    current=''
    sts={}

    for l in lines:
    l=l.replace('\n','')
    if 'config' in l:
    continue
    if 'end' in l:
    continue
    if 'uuid' in l:
    continue

    if 'edit ' in l:
    rule=l.split('edit ')[1]
    current=rule
    sts['rule']=current
    d[current]={}

    if 'set' in l:
    sl=l.split(' ')
    sts[sl[9]]= l.split(sl[9]+' ')[1].replace('" "',', ').replace('"','')

    if 'next' in l:
    d[current]=sts
    sts={}
    #print(d[current])
    return d

    def ExportCsv(dic):
    fieldnames=[]
    # print( dic['49'].keys())
    for r in dic.keys():
    for c in dic[r].keys():
    if c not in fieldnames:
    fieldnames.append(c)
    #print(fieldnames)
    with open('firewall.csv','w+') as f:
    w=csv.DictWriter(f,fieldnames=fieldnames,dialect='excel')
    w.writeheader()
    for r in dic.keys():
    w.writerow(dic[r])








    if __name__ == '__main__':
    f=open('configfirewallpolicy.txt','r')
    rlines = f.readlines()
    firewall=procLines(rlines)
    f.close()
    ExportCsv(firewall)

    ReplyDelete
  19. Hi, have you tried it for version 5.6 ?? thank you, I appreciate your response

    ReplyDelete
    Replies
    1. 5.6.9 here. Seems to be working like a charm!

      Delete
  20. I have configured multiple schedule in my firewall and i want to export it to csv file , Please anyone can share the script for the same

    ReplyDelete
  21. Thanks man. Saved a lot of time and effort. Had 100+ policies. Thanks again for a this solution.

    ReplyDelete