Blog CS A Repository of Things

Scraping Weather Data

We recently had a requirement for UK weather data. A little digging around revealed that hourly weather observation data is available, free of charge from the UK met office, via the uk gov open data site

However, this link above only gives one hour or results at a time, which is a little restrictive...

undefined

Then I took a look at the met office's own API (Datapoint), which is supposed to provide data for a given datetime for it's weather stations.Unfortunately, although I could get xml/json data feeds from the api, every time I polled for data before today, it returned 'no matching records'.

In the end I decided to write the following python script to loop through every day and hour I was interested in on the data gov site, and pull back the csvs into one folder so I could get the full dataset. 

Click here to see the code in gist or else click permalink below, where I have included the code.

Open Data in the UK

There are literally thousands of free open data sets for the UK, with which you can do some pretty awesome things. Or if you're interested in just London, then there's a another load of localised sets

A quick search on here, plus an hour or two of munging and crunching, means you might never have to take a newspaper at its word again. Looking to open a shop, check where the best footfall, least crime is. Fast food chain? Check out obesity rates across the UK.

Home