JSON passed through html form input is not decoding

485 views Asked by At

I'd rather not reprocess all of the data twice for my users, so I am trying to send the data as a hidden form input, encoded as JSON, to generate a CSV export in next function:

<h3> <form action="/c/flex_csv/" method="post">
<input type="hidden" name="data" value='%s'>
<input type="hidden" name="questions" value='%s'> 
<input type="submit" name="submit" value="Download a CSV" />
</form></h3>

And the data in the html source looks okay. I had a lot of trouble with quotes and Unicode characters breaking it, so I am now manually recoding these problem characters:

[the 'data' is a list of dictionaries]

def safe_listdict(data):
    # makes a list of dict that will pass through HTML and CSV without character encoding problems    or quotes breaking form input fields
    import unicodedata
    safe_data = []
    for x in data: 
        line = {}
        for k,v in x.items():
            if type(v) == str:
                line[k] = v.replace("'",'&#39;').replace('"','&quot;').encode('ascii','xmlcharrefreplace')
            elif type(v) == unicode:
                # first normallizes the unicode, then converts to ascii, replacing xml as needed
                line[k] = unicodedata.normalize('NFKD',v.replace("'",'&#39;').replace('"','&quot;')).encode('ascii','xmlcharrefreplace')
            else:
                line[k] = str(v)
            safe_data.append(line)
    return safe_data
import json
safe_data = json.dumps(safe_listdict(data))

This is what the html-source looks like:

<input type="hidden" name="data" value='[{"q0": "65604", "q3": "HOW VAP HAD HELP ME", "q2": "Before three months back i had nothing like knowledge in my mind about HIV/AID prevention, but now i had something in my mind. I had teach my three friends about it and when the had been reped where to visit before 72 hours.     And after that i had also learned how to use ordinary towels which i didn&#39;t know how to use before. And how to maintain proper Hygine.    Through VAP program i had know how to relate with other people like my parents, friends and boys in school.    The last one is that i had now high parepresure, how to relate with my teachers and other pupils in school.    For now i had teach over twenty girls in our community.", "q5": "HIV/AIDS AND PROPER HYGINE", "q4": "VIJANA AMANI PAMOJA", "q7": "NAIROBI", "q6": "KENYA", "q9": "Less than 1 month ago", "q8": "MATHARE 4B", "q15": "", "q14": "0720374037", "q16": "0735775820  [email protected]", "q11": "14", "q10": "Female", "q13": "The right people", "q12": "Heard about it happening", "q19": "Knowledge"},  ...more dicts of same format...]

But when I decode the JSON in my next function, within python, I always get errors parsing:

ERROR:cherrypy.error.18520784:[07/Jan/2014:19:18:18] HTTP Traceback (most recent call last):
  File "/home/djotjog/webapps/cp2/power_drill_down_flex.py", line 620, in flex_csv
    data = json.loads(data)
  File "/usr/local/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python2.7/json/decoder.py", line 381, in raw_decode
    obj, end = self.scan_once(s, idx)

**ValueError: Expecting ',' delimiter: line 1 column 3108 (char 3107)**

The data looks like this in the error log BEFORE I run json.loads(data) to convert it from json back into a python list of dictionaries. I cannot see anything obviously wrong...

DEBUG:root:[{"q0": "65784", "q3": "Getting students to contribute to the environment", "q2": "We went to the schools to do education serious on the environment so they are more aware.   We had evaluation and the student said they are more aware of what to do with their waste and on environmental issues.   They said they are more willing to contribute to segregating their waste and stopping environmental issues like illegal logging.  We suggested things like informing + warning people about the implication of what they are doing. They said they would do it", "q5": "how students were encouraged to take part in envir", "q4": "High school students", "q7": "Bohol, Pilar", "q6": "Philippines", "q9": "1-2 months ago", "q8": "Classrooms in Virgen del Pilar Academy", "q15": "Yes", "q14": "09156646213", "q17": "None", "q16": "09175442613", "q23": "70", "q10": "Male", "q13": "The right people", "q12": "Helped make it happen", "q11": "20", "q19": "Food and shelter, Knowledge, Self-esteem", "q18": "Inspired"}, ...more dictionaries...]

Note: The goal is to enable user to save data via a CSV export button. I have to send it to another page, so python functions can organize it first.

0

There are 0 answers