Two new APIs and a couple of chunky zip files…

A crack team of data ferrets working for the Task Force have returned from their first forays into the bowels of Government. And lo, they come bearing gigabytes of juicy data, most of which is ready to share in handy API format. Check out the Office of National Statistics Neighbourhood Statistics API for an avalanche of local data. Or the NHS Choices API, for those of you who fancy re-using much of the advice available on the NHS website. Or the 47MB of zipped XML making up the last year of official notices Governement and others are required by law to publish in the London Gazette. Or even the contents of EduBase listing the address of every school in England and Wales with details of the headteacher, age range of pupils and school status (e.g. foundation school).

What’s more, the ferrets assure me that there’s more to come…

Edit: Clarified that Edubase is only data from England and Wales

About these ads

15 Comments

Filed under Sources

15 responses to “Two new APIs and a couple of chunky zip files…

  1. The Edbubase data is interesting, but it would be better available in .sql format (which could then be fed direct into a database).

    Excel is.. nice, but it’s not a rigorous way to work with data. And having to re-export what one assumes already exists as an SQL file is a bit of a “why are we doing this?” step.

  2. How does this “crack team” feel about being called “data ferrets”?? :-)

    Good start though!

  3. Rob

    @ Charles: Working in Excel perhaps isn’t perhaps as rigorous, but far more accessible; also, I’d have thought the format of the sql file would be dependent on the underlying DBMS.

    Good Work, keep feeding the ferrets.

    Rob

  4. Pingback: What would you create with Public Information?

  5. @Rob: a text file is accessible, but if you’re going to put it into a database you still need to import it. An SQL file can have a header line that tells you what the database structure is.

    So CREATE TABLE event (name VARCHAR(20), date DATE, type VARCHAR(15), remark VARCHAR(255)); gives you a table called “event” with fields name, data, type and remark. Simple. Powerful. Better than Excel.

  6. trog

    Totally agree about getting this data in SQL input file rather than Excel. Even csv would be better. To re-use this data most easily it needs to go into a database! Surely different SQL files for different db vendors could be produced? (mysql,ms-sql,oracle?). Excel is a licensed and windows based product…. not the best for sharing with everyone who may want it.

  7. When you say “listing the address of every school in the country” turns out this data is just for England and Wales – to save us all going to data sources that we have no interest in can you be explicit in indicating where exactly the data sets relates to – UK wide, England, England and Wales, Scotland etc. I take it that this is a UK government website and also discusses data available for Scotland?

  8. Agreed it’d be better as a .csv, but frankly I’m just delighted we’ve got it at all at all for now…

  9. Pingback: Free Our Data: the blog » Blog Archive

  10. OK, I’ve exported the schools database into SQL format (including CREATE TABLE etc), which is now available for download from the Free Our Data site: http://www.freeourdata.org.uk/blog/?p=213

    Also included MD5 checksums (which I think would be a good idea for other one-off files offered here).

    Government is welcome, obviously, to grab the file back from us and host it at SUABW.com. With the checksum…

  11. Jack Doe

    @Charles\Rob: Valid point about SQL, the syntax only works for certain databases, good look creating a column of type DATE in SQL Server 2005. It requires DATETIME, so keep it simple, CSV.

    - but yes, Excel works for me!

  12. On this side of the world we’re very impressed by the steps you’re taking in the UK to improve government service offerings and increase transparency through effective use of online.

    I’m mentioning what you’re doing in my eGov AU blog to help stimulate interest in these types of initiatives here in Australia.

    A quick question – I’m preparing to begin releasing information from my agency via XML feeds, so I am very interested in your choice of data format and the rationale behind it.

  13. An API is not really an API, in the sense that most people would use the term today, if using it to bring data into a web-page requires a change to the browser configuration, and a SOAP request.

    (At least that’s the case with the Neighbourhood data…for the NHS data they’re going to ‘get back to me within 10 days’, so I don’t know yet whether that is the same.)

    Nowadays, people would expect to make RESTful calls, rather than have the overhead of using SOAP. And in particular, they would expect to have the option of retrieving the data in a JSON format, so that it can be easily used in web-pages.

    Of course, this could all be done if we were to make our own copies of the database, but seems to defeat the purpose of the whole thing! And certainly takes us a long way away from being ‘an API’.

    I know, I know…journeys of thousands of miles begin with single steps, and all that kind of thing. But so far this only shows me how much there is to do, before we get data that is *really* usable by lots of people.

    Regards,

    Mark

  14. This is all very welcome.
    The “new” API exposed by NHS Choices is very nice, but unfortunately less useful (to me) than their old API.
    The new API is RESTful (good) but concentrates on editorial content such as news stories and the like. It does expose some NHS service data, but only via calls to a local search method. It lacks any way to access the patient feedback and responses on the site.
    The previous API (pre-NHS Choices) was a SOAP API which did allow one to navigate the “tree” of NHS services from top to bottom (regions > trusts > sites > services). It was most useful for hospital services. In other areas the data didn’t seem to be complete.
    So, here’s what is needed now:
    1: Add NHS service tree navigation back into the new API.
    2: Populate the backend with data so there really is a tree to navigate.
    3: Add some way to get at patient feedback and NHS responses, by trust.
    This would be really useful for those of us working on using public feedback to help improve local services.
    Incidentally, on the assumption that NHS Choices had meant to include all this anyway, we have mashed up patient feedback from NHS Choices with that coming directly via Patient Opinion.
    This means that patients and NHS comissioners and providers now only have to look in one place to find all their feedback.
    All the best
    James

  15. Pingback: Journalismus & Recherche » Blog Archiv » Futter für die Data-Mining-Nerds

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s