python Script to parse server log between two different timestamps

Question

I want to parse server accesslogs to find 4xx and 5xx errors count . But failed to get data between two timestamps . Somewhere getting mistakes in logic.

Timestamps formate date / month /year / Hr /min /sec : 27/Aug/2013:12:45:54

if time >= start_timestamp and  time <= end_timestamp : 
      if check_point_flag == 0  :
      start_time = request['request'].split(" ")[0].split('[')[1]
      check_point_flag=1
      total_patient_count, total_encounter_count = self.get_count_information( access_token, request , resource_name, total_patient_count, total_encounter_count)
      count_patient_5xx ,count_encounter_5xx = self.get_count_information(access_token,request,resource_name,count_patient_5xx,count_encounter_5xx, '500', '600')
      count_patient_4xx ,count_encounter_4xx = self.get_count_information(access_token,request, resource_name,count_patient_4xx,count_encounter_4xx, '400', '500')             
      count_patient_2xx ,count_encounter_2xx = self.get_count_information(access_token,request, resource_name,count_patient_2xx,count_encounter_2xx, '200', '300')             
      end_time = request['request'].split(" ")[0].split('[')[1]

Here is some logs after changing to groupdict()

{'status': '404', 'protocol': 'HTTP/1.0', 'request_time': '0.000', 'mongo_exec_time': '-', 'url': '/ready', 'request': '[19/Aug/2013:16:31:20 -0400]', 'hostname': 'xxxxxx', 'bytes_sent': '168', 'audit_response_time': '-', 'application': '-', 'user_agent': '-', 'upstream_response_time': '-', 'queries_count': '-', 'clientip': '12.255.25.13', 'hma_exec_time': '-', 'method': 'GET', 'user': '-'}
{'status': '404', 'protocol': 'HTTP/1.0', 'request_time': '0.000', 'mongo_exec_time': '-', 'url': '/ready', 'request': '[19/Aug/2013:16:31:22 -0400]', 'hostname': ''xxxxxx', 'bytes_sent': '168', 'audit_response_time': '-', 'application': '-', 'user_agent': '-', 'upstream_response_time': '-', 'queries_count': '-', 'clientip': '13.255.25.13', 'hma_exec_time': '-', 'method': 'GET', 'user': '-'}
{'status': '404', 'protocol': 'HTTP/1.0', 'request_time': '0.000', 'mongo_exec_time': '-', 'url': '/ready', 'request': '[19/Aug/2013:16:31:52 -0400]', 'hostname': ''xxxxxx', 'bytes_sent': '168', 'audit_response_time': '-', 'application': '-', 'user_agent': '-', 'upstream_response_time': '-', 'queries_count': '-', 'clientip': '13.255.25.13', 'hma_exec_time': '-', 'method': 'GET', 'user': '-'}
{'status': '404', 'protocol': 'HTTP/1.0', 'request_time': '0.000', 'mongo_exec_time': '-', 'url': '/ready', 'request': '[19/Aug/2013:16:31:54 -0400]', 'hostname': ''xxxxxx', 'bytes_sent': '168', 'audit_response_time': '-', 'application': '-', 'user_agent': '-', 'upstream_response_time': '-', 'queries_count': '-', 'clientip': '13.255.25.13', 'hma_exec_time': '-', 'method': 'GET', 'user': '-'}
{'status': '200', 'protocol': 'HTTP/1.1', 'request_time': '0.000', 'mongo_exec_time': '-', 'url': '//nginx_stub_status', 'request': '[19/Aug/2013:16:31:55 -0400]', 'hostname': ''xxxxxx', 'bytes_sent': '109', 'audit_response_time': '-', 'application': '-', 'user_agent': 'python-requests/1.2.3 CPython/2.6.6 Linux/2.6.32-358.14.1.el6.x86_64', 'upstream_response_time': '-', 'queries_count': '-', 'clientip': '127.0.0.1', 'hma_exec_time': '-', 'method': 'GET', 'user': '-'}

Please help to find logic to parse files between two timestamps

Answer 1

If I understand your question correctly, you are trying to find a way to filter log messages based on their timestamps. You could use the datetime module:

import datetime
time = datetime.datetime.strptime('27/Aug/2013:12:45:54', '%d/%b/%Y:%H:%M:%S')
start_timestamp = datetime.datetime.strptime('20/Aug/2013:12:45:54', '%d/%b/%Y:%H:%M:%S')
end_timestamp = datetime.datetime.strptime('30/Aug/2013:12:45:54', '%d/%b/%Y:%H:%M:%S')

if time >= start_timestamp and  time <= end_timestamp: print 'it worked'