allow Response class to be pickled

[git2samus]

Python 2.7.4 (default, Apr 19 2013, 18:32:33) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle, requests
>>> pickle.dumps(requests.get('http://example.org'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 725, in save_inst
    save(stuff)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 615, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 725, in save_inst
    save(stuff)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/usr/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

[Lukasa]

Out of interest, what's the use case for this?

[git2samus]

I want to use requests with the multiprocessing module and it uses pickle as the serialization method.

[git2samus]

I may be able to produce a patch if you give me some directions.

[sigmavirus24]

If you want to take a crack at a patch @git2samus, take a look at the Session class. That's picklable and has all the info there. If you would rather one of us do it, I can start to tackle it in half an hour.

[djsutho]

You can't pickle the Response as it contains a link to the connection pool. If you aren't reusing connections you might consider the following:

r = requests.get('http://example.org')
r.connection.close()
pickle.dumps(r)

[tanelikaivola]

tanelikaivola@229cca8 is how I solved it.. Just remove the raw-attribute from the getstate.

tanelikaivola@a5360de is an alternate way of doing it (which mimics the code style of kennethreitz@590ce29).

[heston]

Any update on this? I just ran into the same issue trying to cache Responses for testing purposes.

I ended up with something like this:

r = requests.get('http://example.org')
r.raw = None
pickle.dumps(r)

[ionrock]

This would be a big help for caching via Cache Control. At the moment, it is very difficult to cache a response outside of memory because it is difficult to rebuild the object hierarchy.

I did look into caching the raw response and using it to rebuild the response from scratch when a cache is hit, but that would have taken some pretty serious monkey patching of httplib and urllib3.

The patches by @tanelikaivola seem like they would world without too much trouble.

[sigmavirus24]

@ionrock why not do something like Betamax instead of trying to use pickle?

[ionrock]

@sigmavirus24 Thanks for pointing out Betamax! I'd argue the serialize/deserialize_response functions would be a great addition to the Response object. If the goal is to avoid pickle, this seems like a great option. Sometimes pickle is a good option though, so I still believe it is worthwhile to add the functionality. I've tested the patch @tanelikaivola and they work well. What else would need to be done to potentially get them merged? Obviously some tests would be helpful. I'd also be happy to see about adding the serialize/deserialize code from Betamax if that would be alright with @sigmavirus24.

Let me know if there is a good way to proceed. I'd be happy to put the code together.

@sigmavirus24 thanks again for the Betamax suggestion. I will be switching Cache Control to use that methodology.

[sigmavirus24]

I'm 👎 for adding the serialize/deserialize code from Betamax to requests. For one, I don't believe it takes care of all of the information that could be serialized. As it is Betamax only serializes what it needs to so that it can maintain compatibility with VCR's cassette format. If other people want it in requests proper, we could do that, but I'm not sure it works very well without the rest of betamax.

[ionrock]

@sigmavirus24 Fair enough. I'm most definitely not the expert on everything that a Response object, so I'm happy to take your word for it. When I looked at what a response object contained it was semi-complex as it was a wrapper around a urllib3 HTTPResponse, which is in turn a wrapper around a httplib response, which assumes the content comes directly from a socket (not a file like object).

With that being the case, it seems to make sense to support pickling of Response object. Again, I'm happy to write some tests for @tanelikaivola's patches if there is a consensus. Otherwise, I'd like to understand where it falls short. At the very least I'd like to try and fix it for myself.

Thanks for the discussion!

[sigmavirus24]

Personally I prefer tanelikaivola@a5360de so if you want to add tests around that, that would be a great start.

[ionrock]

I added a test and made sure there is some assurance that response and the request are in tact before and after pickling. Feel free to take a look 512beb8.

[sigmavirus24]

@ionrock