Web crawler for hydrological data

Globally, new sources of raw data are being made available via the web all the time. However, ECMWF often isn’t aware of these. Manually identifying and gathering information on these new sources is both time consuming and error prone. 

This challenge was aimed at developing a tool to search the web systematically, identifying data sources for observed environmental data. The software automates the discovery, analysis and assessment of the candidate web pages in order to find new datasets. The resulting data can be used to improve global predictive weather forecasting models. 

Follow the developments on GitHub