how to unpack stxetx data in python

Question

I'm receiving a data packet sent from a RS485 level serial bus to a ESeye

Hammerkop transmitter which then URL encodes the data and sends it to my server. The request looks like the following:

at=info method=GET path="/write/?type=stxetx&packet=ArcYX%01%00%00%00%00%00%00%03%00%F0%00%06%00%F0%008%0E%B0%27%00%00%E85&localpackettime=2016-12-20+16%3A00%3A27&serial=868324023356343&packettime=2016-12-20+16%3A00%3A27&receivetime=2016-12-20+16%3A00%3A28&timezone=UTC" host=dry-hollows-46605.herokuapp.com request_id=4d5bec23-0676-4e32-ae66-af26e26f0394 fwd="81.94.198.91" dyno=web.1 connect=1ms service=232ms status=500 bytes=63266

I'm using Django 1.9 as my framework and to get the packet data I use:

packet_data = request.GET.getlist('packet')   

The data i receive is the following:

packet_data = [u"A\x90cYX\x01\x00\x00\x00\x00\x00\x00\x03\x00\xf0\x00\x06\x00\xf0\x008\x0e\xb0'\x00\x00\xe85"]

I then must split this into bytes, the first letter is an ascii letter which is straight forward and can be ignored, the rest are bytes. I'm trying to split out this unicode string into the values i need for example the first 4 bytes after the 'A' are a byte int timestamp.

I've tried encoding the values, for example:

a = packet_data[1].encode('utf-8')
b = packet_data[2].encode('utf-8')
c = packet_data[3].encode('utf-8')
d = packet_data[4].encode('utf-8')

which converts them to str rather than unicode and then I unpack them:

timestamp = unpack('i',a+b+c+d)

however this only works some of the time and sometimes I get the error:

error: unpack requires a string argument of length 4

I got the error for these timestamp values for example:

before encoding u'\x90', u'c', u'Y', u'X'

after encoding '\xc2\x90', 'c', 'Y', 'X'

and suggestions would be so helpful I've been going round for days on this.

My only thought is that I'm not packing/unpacking the data correctly, maybe I'm missing out the correct padding but I've tried different [padding types and they don't seem to help either. What would someone try next?

Answer 1

Django has unhelpfully decoded the GET parameter as text instead of binary data. You will need to parse the contents of request.META['QUERY_STRING'] yourself.

>>> import urlparse
>>> urlparse.parse_qs('...&packet=ArcYX%01%00%00%00%00%00%00%03%00%F0%00%06%00%F0%008%0E%B0%27%00%00%E85&...')['packet'][0]
"ArcYX\x01\x00\x00\x00\x00\x00\x00\x03\x00\xf0\x00\x06\x00\xf0\x008\x0e\xb0'\x00\x00\xe85"