Safe HDF5 parallel access

[toobaz]

Is something like this is worth including in pandas (although it will not work i.e. in Windows)?

If yes, I will put together a PR (also supporting Python2).

(I could also implement a platform independent alternative... but locking would not be atomic, so race conditions would be possible - although improbable)

[jreback]

You can add this as a link to the SO question in the cookbook. Doing this is quite dangerous, and the user should know exactly what they are doing.

[toobaz]

OK (by "dangerous" you mean if you mix the "safe" and the "standard" approach, or something else I'm missing?)

[jreback]

I mean 'advanced' / 'magical', meaning that if the user doesn't understand what they are doing they get the impression that you CAN write in parallel to HDF5, which is a very-very bad idea (your serialization strategy is fine though), but this is not something pandas should enforce. In theory I agree with you, but I don't think pandas can/should really support this kind of enforcement.

[jreback]

all that said am happy with a cookbook link. Something like this might be supportable in blaze