program to read a file contents

Question

I have a snort log file named "logs" and want to extract IP addresses from it and store them to another file named "blacklist". it can extract unique IP Addresses but if I run the program again, it adds the previous IPs as well. I want the program to first check whether IP is already in blacklist file? if so, just ignore it otherwise add unique IPs from logs file to blacklist. code:

#!/usr/bin/python
import re
mylist1 = []
mylist2 = []
mylist3 = []
mylist4 = []
logfile = open('/var/log/snort/logs', 'r')
blklist = open('blacklist', 'ab+')

for line in open ('blacklist', 'r').readlines():
  mylist4.append(line)

for l in logfile.readlines():
  l = l.rstrip()
  ip = re.findall(r'[0-9]+(?:\.[0-9]+){3}',l)
  if ip is not None and ip not in mylist1:
    mylist1.append(ip)
for ip in mylist1:
  addr = ",".join(ip)
  if ',' in addr:
    a = addr.split(',')
    for ip in a:
        addr = "".join(ip)
        if addr is not '':
            mylist2.append(addr)
        else:
            mylist3.append(addr)
for x in blklist:
  mylist2.append(x.strip())
for x in mylist2:
  if x not in mylist3 and x not in mylist4:
    blklist.write(x+'\n')
    mylist3.append(x)

Logs file is:

12/16-10:34:27.070967 [**] [1:10000001:1] snort alert [1:0000001] [**][classification ID: 0] [Priority ID: 0] {ICMP} 192.168.40.19 -> 192.168.50.29

12/16-10:34:27.070967 [**] [1:10000001:1] snort alert [1:0000001] [**][classification ID: 0] [Priority ID: 0] {ICMP} 192.168.50.29 -> 192.168.30.20

Output of blacklist file after first program run:

192.168.30.20
192.168.50.29
192.168.40.19

Output of blacklist file after second program run:

192.168.30.20
192.168.50.29
192.168.40.19
192.168.30.20
192.168.50.29
192.168.40.19

any help please?

Answer 1

You can read everything in from your blacklist file and log into lists. Join those list and then ouput a set back to the blacklist file (sets are unique values) since the read empties the file your will have a unique list of all new and old IPs. If the order matters (doubt it does) then a set will cause issues. Let me know and I can revamp the below.

if __name__ == '__main__':
    import re
    blacklist = list(open("blacklist", 'r').read().split('\n'))
    logfile = list(open("/var/log/snort/logs", 'r').read().split('\n'))

    newentry = []
    for entry in logfile:
        ips = re.findall( r'[0-9]+(?:\.[0-9]+){3}', entry)
        for ip in ips:
            newentry.append(ip)

    newblacklist = blacklist + newentry

    with open("blacklist", 'w+') as f:
        f.write('\n' .join(set(newblacklist)))
        f.close()

Answer 2

You could utilize the Python container type set which stores only unique elements. The procedure below should work for you:

create a 'current' blacklist set
read the blacklist file IP's into the current set

create a 'delta' blacklist set

for each IP address in the log file
  if not already in current blacklist
    add the IP into the delta set

append (by writing) the delta set into the black list file