[_gdbm.error: Database needs recovery] while calling .collect_observations()

[sk-drop]

Hi! I found myself in a pitiful situation..
I don't know if bug fits the description, but it is an error I can't solve..
It's a problem with some dbf file.
I tried reinstalling wetterdienst + dogpile.cache and PyCharm, but nothing seems to work, as I still get the same error message.

code:

ob_daily = DWDObservationData(
    station_ids=station_ids_d,
    parameter=params_daily,
    time_resolution=TimeResolution.DAILY,
    start_date="2015-01-01",
    end_date="2020-10-10",
    tidy_data=True,
    humanize_column_names=True,
)

for df in ob_daily.collect_data():
    name = str(df.STATION_ID.iloc[0]).strip(".0")
    df.to_csv('./data/daily/{}.csv'.format(name))
    print('{} done'.format(name))

and the error he throws:

Traceback (most recent call last):
  File "/Users/sashakaun/PycharmProjects/wttr/main.py", line 84, in <module>
    for df in ob_daily.collect_data():
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/api.py", line 178, in collect_data
    df_parameter = self._collect_parameter_from_station(
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/api.py", line 243, in _collect_parameter_from_station
    df_period = collect_climate_observations_data(
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 82, in collect_climate_observations_data
    filenames_and_files = download_climate_observations_data_parallel(remote_files)
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 106, in download_climate_observations_data_parallel
    return list(zip(remote_files, files_in_bytes))
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 611, in result_iterator
    yield fs.pop().result()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/wetterdienst/dwd/observations/access.py", line 124, in _download_climate_observations_data
    return BytesIO(__download_climate_observations_data(remote_file=remote_file))
  File "<decorator-gen-2>", line 2, in __download_climate_observations_data
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 1356, in get_or_create_for_user_func
    return self.get_or_create(
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 954, in get_or_create
    with Lock(
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/dogpile/lock.py", line 185, in __enter__
    return self._enter()
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/dogpile/lock.py", line 94, in _enter
    generated = self._enter_create(value, createdtime)
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/dogpile/lock.py", line 178, in _enter_create
    return self.creator()
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/dogpile/cache/region.py", line 920, in gen_value
    self.backend.set(key, value)
  File "/Users/sashakaun/PycharmProjects/wttr/venv/lib/python3.8/site-packages/dogpile/cache/backends/file.py", line 239, in set
    dbm[key] = pickle.dumps(value, pickle.HIGHEST_PROTOCOL)
_gdbm.error: Database needs recovery

Thank you for your help!

[amotl]

Dear Sasha,

sorry for that. That might be related to the cache files. You might want to just drop them completely. Recognizing you are on macOS, they are probably located at /Users/sashakaun/Library/Caches/wetterdienst.

With kind regards,
Andreas.

[sk-drop]

Dear Andreas,
It works after clearing the cache!! Thank you for the kind help!!

[amotl]

Dear Sasha,

you are welcome, good that this worked for you.

Seeing what you are doing over at weather, I would like to humbly note that dwdweather2 is about to be phased out. Wetterdienst is clearly the future for doing FOSS weather data acquisition using Python.

We will be happy to see you joining us here. If you see any features not fulfilled by the library yet, we would like to encourage you creating issues, contributing to the core library or just adding reasonable examples about what you are aiming at within the example/ folder of this repository. Anything will help!

With kind regards,
Andreas.

[sk-drop]

Dear Andreas,

thank you for the heads up!
I will continue to use your service for a research project!
I can't help but notice the amount of thought that has gone and is going into the process of making this service accessible and easy to use, so I hereby salute you and everyone else, who is working on this project. Your making the world a better place!!
Stay safe!

Regards,
Sasha