Using Snappy compression on Gob data in Go?

Question

I need to save a structure to disk and read it in again later, I'm trying to keep IO down to a minimum, but also not spend ages compressing and uncompressing the file, so I intend to use Snappy for compression as it's very fast and relatively efficient.

Normally I would gzip compress a gob when saving it to file, like this:

func (t *Object) Save(filename string) error {
    // Open file for writing
    fi, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer fi.Close()
    // Attach gzip writer
    fz := gzip.NewWriter(fi)
    defer fz.Close()
    // Push from the gob encoder
    encoder := gob.NewEncoder(fz)
    err = encoder.Encode(t.Classifier)
    if err != nil {
        return err
    }
    return nil
}

But Snappy does not attach to these Reader/Writer interfaces that everything else seems to use. Instead it just provides the basic functionality: https://godoc.org/code.google.com/p/snappy-go/snappy

func Encode(dst, src []byte) ([]byte, error)

func Decode(dst, src []byte) ([]byte, error)

What would be the most efficient way to use this Snappy package for compressing the Gob data as its being saved to file (and also reading it back)? Ideally I don't want to use ioutil.ReadAll to just read from the gob reader into a slice of bytes and then compress that all over again as that seems to be a very heavy way of doing it which would create a lot of wasted memory.

I admit that I don't fully understand how the reader and writer interfaces work.

Answer 1

package snappystream

import "github.com/mreiferson/go-snappystream"

snappystream wraps snappy-go and supplies a Reader and Writer for the snappy framed stream format.

Have you considered package snappystream? For example,

package main

import (
    "encoding/gob"
    "fmt"
    "os"

    "github.com/mreiferson/go-snappystream"
)

type Object struct {
    Classifier struct{}
}

func (t *Object) Save(filename string) error {
    // Open file for writing
    fi, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer fi.Close()
    // Attach snappy writer
    fs := snappystream.NewBufferedWriter(fi)
    // Push from the gob encoder
    encoder := gob.NewEncoder(fs)
    err = encoder.Encode(t.Classifier)
    if err != nil {
        return err
    }
    err = fs.Close()
    if err != nil {
        return err
    }
    err = fi.Close()
    if err != nil {
        return err
    }
    return nil
}

func main() {
    obj := Object{}
    err := obj.Save("obj.sz")
    if err != nil {
        fmt.Println(err)
    }
}