python gzip "unexpected end of file" when writing a data stream

Python gzip”遇到意外的文件结尾”写的一个数据流时

问题 (Question)

I'm writing (or more precisely, appending) a real-time data stream to an instance of python's gzip module. If the program that's writing the stream crashes, and relaunches, I would like the stream to automatically be appended to the original file. Unfortunately this fails in practice, as I get an "unexpected end of file" error that corresponds to the exact point at which the original program crashed.

What's the underlying approach for handing this situation, as I can't imagine this should be a hard problem. My approach is outlined below:

f = gzip.GzipFile( 'filename_json.txt.gz' ), mode='at', compresslevel=9)
while(something_is_true):
    f.write(stream['message'] + '\n')
f.close()

This runs continuously, but if the program crashes (or gets killed), the end-of-file operator never gets appended, and the gzip file becomes corrupt. In which case, any data appended after that point becomes unreadable.

Thanks!

我写(或者更准确的说,添加)的实时数据流到Python的gzip模块实例。如果程序是写作流崩溃,和球迷,我想流自动附加到原始文件。不幸的是,在实践上是失败的,因为我得到一个“意外的文件结尾”错误,对应于精确点的原程序崩溃。

把这种情况的基本方法是什么,我不能想象,这应该是一个很难的问题。我的方法如下:

f = gzip.GzipFile( 'filename_json.txt.gz' ), mode='at', compresslevel=9)
while(something_is_true):
    f.write(stream['message'] + '\n')
f.close()

这个连续运行,但如果程序崩溃(或被杀),文件操作结束,从不附加,以及gzip文件损坏。在这种情况下,任何数据追加后,变为不可读。

谢谢!

最佳答案 (Best Answer)

with gzip.open('filename_json.txt.gz', mode='at', compresslevel=9) as f:
    while something_is_true:
        f.write(stream['message'] + '\n')

(This works for me on python 2.7.6)

But if that for some reason isn't working, you can do it the old fashioned way:

try:
    f = gzip.open('filename_json.txt.gz', mode='at', compresslevel=9)
    while something_is_true:
        f.write(stream['message'] + '\n')
finally:
    f.close()

Note that the error will still propagate with this code unless you catch the error. But the file will be closed.

with gzip.open('filename_json.txt.gz', mode='at', compresslevel=9) as f:
    while something_is_true:
        f.write(stream['message'] + '\n')

(这对我的作品在Python 2.7.6)

但如果因为某些原因没有工作,你能做的最老式的方法:

try:
    f = gzip.open('filename_json.txt.gz', mode='at', compresslevel=9)
    while something_is_true:
        f.write(stream['message'] + '\n')
finally:
    f.close()

请注意,错误还是会传播此代码,除非你捕获错误。但是,该文件将被关闭。

本文翻译自StackoverFlow,英语好的童鞋可直接参考原文:http://stackoverflow.com/questions/23451101