Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make GLPickle more extensible. #180

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

srikris
Copy link
Contributor

@srikris srikris commented Feb 18, 2016

Summary of changes

  • Made GLPickle more extensible
  • Used the extensibility mechanism to implement things for Model, SGraph,
    SFrame, and SArray.
  • Added some additional tests for the same.

Details

Previously, GLPickle could not be extended without making changes directly to
it. Now, we added a mechanism to extend it by implementing 2 simple
methods:

  1. gl_pickle_save: Save your object to a filename (not handle) given to you.
  2. gl_pickle_load: Load your object from a filename (not handle) given to you.

As an example, here is a simple class that was extended to be pickled via
GLPickle.

class SampleClass(object):
  def __init__(self, member):
     self.member = member

  def __gl_pickle_save__(self, filename):
      with open(filename, 'w') as f:
          f.write(self.member)

  @staticmethod
  def __gl_pickle_load__(filename):
      with open(filename, 'r') as f:
          member = f.read().split()
      return SampleClass(member)

@@ -62,6 +62,8 @@
from .version_info import version
from .version_info import __VERSION__

from _gl_pickle import GLPickler
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For PY3 support, you'll also need to always use relative imports:

from ._gl_pickle import GLPickler
from ._gl_pickle import GLUnpickler

Notice the periods.

@ylow
Copy link

ylow commented Feb 18, 2016

@srikris Can you do a merge with master again?

@srikris
Copy link
Contributor Author

srikris commented Feb 18, 2016

@ylow Done.

@ylow
Copy link

ylow commented Feb 19, 2016

  • Lets not bump the version number. This can be avoided simply by type checking of the type_tag.
  • Also, the mechanism of serializing the load method will cause problems. This means that if any point, there is a bug in the load method, any previously saved object will no longer be loadable since it is depending on the implementation of the load method which is fully serialized into the pickle.
  • This may also make Python 2 saved objects not loadable in Python 3 and vice versa.

Instead you should just save the module import path to the load method and find it on deserialization.

@ylow
Copy link

ylow commented Apr 12, 2016

@srikris progress on this?

@rbkreisberg
Copy link

@srikris @ylow is this going into the 1.9 release?

@srikris
Copy link
Contributor Author

srikris commented Apr 14, 2016

This will go in right after 1.9. I plan to fix it soon.

srikris added 5 commits April 28, 2016 07:56
Summary of changes
------------------
- Made GLPickle more extensible
- Used the extensibility mechanism to implement things for Model, SGraph,
  SFrame, and SArray.
- Added some additional tests for the same.

Details
-------
Previously, GLPickle could not be extended without making changes directly to
it. Now, we added a mechanism to extend it by implementing 2 simple
methods:

1. __gl_pickle_save__: Save your object to a filename (not handle) given to you.
2. __gl_pickle_load__: Load your object from a filename (not handle) given to you.

As an example, here is a simple class that was extended to be pickled via
GLPickle.

  class SampleClass(object):
      def __init__(self, member):
         self.member = member

      def __gl_pickle_save__(self, filename):
          with open(filename, 'w') as f:
              f.write(self.member)

      @staticmethod
      def __gl_pickle_load__(filename):
          with open(filename, 'r') as f:
              member = f.read().split()
          return SampleClass(member)
- No bump of version number.
- Loading module and then class name.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants