I'm writing (or more precisely, appending) a real-time data stream to an instance of python's gzip module. If the program that's writing the stream crashes, and relaunches, I would like the stream to automatically be appended to the original file. Unfortunately this fails in practice, as I get an "unexpected end of file" error that corresponds to the exact point at which the original program crashed.
What's the underlying approach for handing this situation, as I can't imagine this should be a hard problem. My approach is outlined below:
f = gzip.GzipFile( 'filename_json.txt.gz' ), mode='at', compresslevel=9) while(something_is_true): f.write(stream['message'] + '\n') f.close()
This runs continuously, but if the program crashes (or gets killed), the end-of-file operator never gets appended, and the gzip file becomes corrupt. In which case, any data appended after that point becomes unreadable.