requests cookiejar - send multiple cookies from list of dictionaries

Question

I'm hoping someone can help me with creating a cookie jar with multiple cookies from a list of dictionaries. This is in python3.

I've tried a few different methods, but they either overwrite the previous cookie or don't accept non-standard cookie attribs (such as httpOnly)

The cookies are returned from Flaresolverr (which I believe is the same output as selenium) as multiple dictionaries in a list, e.g.:

cookies = [
      {
        "name": "ASP.NET_SessionId",
        "value": "SOMEVALUE",
        "domain": "sub.mydomain.co.uk",
        "path": "/",
        "expires": -1,
        "size": 41,
        "httpOnly": true,
        "secure": false,
        "session": true
      },
      {
        "name": "__cfduid",
        "value": "SOMEVALUE",
        "domain": ".mydomain.co.uk",
        "path": "/",
        "expires": 1622898293.967355,
        "size": 51,
        "httpOnly": true,
        "secure": true,
        "session": false,
        "sameSite": "Lax"
      }
   ]

So if I then try to add them all in a cookie jar, you can see "__cfduid" is the only cookie in the jar:

r = requests.session()
for c in cookies:
      r.cookies.update(c)

print(r.cookies)
<RequestsCookieJar[<Cookie domain=.mydomain.co.uk for />, <Cookie expires=1622898293.967355 for />, <Cookie httpOnly=True for />, <Cookie name=__cfduid for />, <Cookie path=/ for />, <Cookie sameSite=Lax for />, <Cookie secure=True for />, <Cookie session=False for />, <Cookie size=51 for />, <Cookie value=SOMEVALUE for />]>

I've also tried jar = requests.cookies.RequestsCookieJar() and jar.set(xxxx) but it doesn't like non-standard cookie attributes (e.g. httpOnly)

I can't do a r= requests.get and take the cookies from r.cookies because the cookies are returned as a json inside the requests.text (or requests.json), not as part of the session itself.

I've done plenty of searching but I can't find anything that works - any help would be very much appreciated, thank you!

Answer 1

The method you are looking for is set(), not update(). It can take all its arguments as keyword arguments, but it must match as documented by http.cookiejar. Also, the session parameter is called discard in a Cookie object. With some list comprehensions, we can convert it to the form it needs:

cookie_props = ['name', 'value', 'port', 'domain', 'path', 'expires', 'secure', 'discard']
cookiejar = [dict([(k, item['session' if k == 'discard' else k])
             for k in cookie_props if k in item]) for item in cookies]

r = requests.session()
for c in cookiejar:
    r.cookies.set(**c)
print(r.cookies)
print(len(r.cookies))

The list comprehension renames session to discard and only copies over the name/value pairs that match properties accepted by the set() method on http.cookiejar.