Store intermediate results as pickle

[betatim]

I was wondering what the thinking was behind choosing np.save/np.load instead of pickle. I was preparing data for a fairly small region and ended up with 156250 tiles in total. Of which about 50000 are in one of my classes. Running label-maker images took a long time so I started investigating and switching from .npz to a pickle means it starts downloading tiles more or less immediately. Before I had to wait a very long time (minutes or more?) to get to the bit when it starts downloading.

If this sounds like something you'd be open to I'd contribute a PR.

[drewbo]

@betatim thanks for the idea. We originally chose np.save/np.load since it's likely that folks will already have numpy imported. We'd be open to a PR on this if you can make a small demonstration/benchmark to show it's faster.

For your specific task, you could try using the --sparse flag (although note there is one pending bug: #46) to limit the saved tiles to only the ones containing your classes.

[drewbo]

@betatim something like this might provide a good benchmarking framework https://github.com/mverleg/array_storage_benchmark; label data tends to be pretty sparse so we should optimize with that in mind or expose options to the user in the case of dense label masks