back to article Oi, drag this creaking, 217-year-old UK census into the data-driven age

The UK's Office for National Statistics is under pressure. Every decade since 1801, it has carried out one of the world's most comprehensive statistical undertakings, the census. Now, it has until 2021 to prove it can do so without the massive surveys it still relies upon. The population survey, which was sent out to 25 …

  1. jake Silver badge

    I'm sure that, for a price,

    google will be more than happy to provide HMG with any info they ask for.

    Sorted.

    1. Anonymous Coward
      Anonymous Coward

      Re: I'm sure that, for a price,

      Quite honestly they could do that or just make it up based on a quick survey in the office because the government never acts on the data its got.

      The don't provide enough schools even when they could see a growth in the number of school age children coming. They act all surprised about an ageing population even though that's been obvious for decades.

      The just look at the short term and what will make them popular and if it goes wrong the usual cookbook is something like this.

      1. Leak some stories showing that s small group of people are at fault, immigrants are always a popular target.

      2. If it's not immigrants then it's some kind of "cheat" who's depriving the rest of us.

      3. Publish some distorted statistics to prove they're spending more than ever before.

      4. Publish another made up number to show the problem sin't as big as it is.

      5. Publish another story showing that it's our fault as we're not using the service correctly.

      6. Blame it on the previous government.

      7. Pretend that the big boys (EU) mad them do it.

      8. If this doesn't work then stick your fingers in your ears and repeat some of the above in a loud voice.

      Never spend long defending any of the above, just move on to the next one in the hope that no-one will notice their lies.

      1. Tom 7

        Re: I'm sure that, for a price,

        Can I just say for government here read party in charge of parliament, The civil service part of the government (and many councils) make efforts to use as much real data as they can. Bless em.

  2. Pascal Monett Silver badge
    Stop

    It's a fundamentally bad idea to use administrative data

    One of the primary uses of a census is as a reality check : this is the actual state of the country.

    You cannot have that reality check if you are just scraping databases for info. In that case, you're just compiling existing data and you'll miss the possibility of including pointed questions to uncover some specific behavioural or societal change.

    Do it online, by all means, supplement the census with administrative data if you wish, but do not reduce the census to just perusing databases.

    1. MyffyW Silver badge

      Re: It's a fundamentally bad idea to use administrative data

      Agree that re-using administrative data is likely to become a turd-polishing exercise.

      As a secondary consideration, I do relish the chance - once every 10 years - to declare my faith as "Jedi" and fear this may be lost in my on-grid footprint (unless they also trawl the comments section of El Reg.

      1. Anonymous Coward
        Anonymous Coward

        Re: It's a fundamentally bad idea to use administrative data

        I do relish the chance - once every 10 years - to declare my faith as "Jedi" and fear this may be lost in my on-grid footprint

        That's a bit last century, don't you think? How about a social media campaign to encourage people to assert Dawkinism as their religion? That should leave almost nobody unoffended.

        1. handleoclast
          Coat

          Re: It's a fundamentally bad idea to use administrative data

          Dawkinsism.

          Me, I claim pedanticism as my religion.

          1. Anonymous Coward
            Anonymous Coward

            Re: It's a fundamentally bad idea to use administrative data

            Dawkinsism. Me, I claim pedanticism as my religion.

            Blasphemer!

            As an adherent of a different abominable religion, you are not entitled to make this claim! Curse you, infidel! It is and always will be Dawkinism, until Saint Richard both dies and is then reborn in the perfect form - that is, without the "s".

            And then it still will be Dawkinism

        2. tiggity Silver badge

          Re: It's a fundamentally bad idea to use administrative data

          Couple it with Dworkinism and you will offend a few more

    2. I ain't Spartacus Gold badge

      Re: It's a fundamentally bad idea to use administrative data

      The article's initial premise is wrong. I'm sure that government would like to save money - and that's a driver of what's going on.

      But one of the initial reasons that it was thought 2011 would be the last census is that people just weren't filling out the forms. Worse it was the poorest people most unrepresented in the data - and they're the ones who need government services most. There's a particular problem with people in temporary or multi-occupancy accommodation - again likely to be the poorest.

      A lot of money was spent on paying people to knock on doors and go round checking this stuff - and we're talking tens of thousands of people here, so actual serious money (even by government standards). Plus the forms then come back hand-written to had to be scanned electronically and lots of it input by hand. Again that costs serious money.

      So online makes sense, but again, it's the poorest who're least likely to get spotted.

      Another huge problem is people with poor english. Again, likely to be poorer, and more in need of government services. Much less likely to fill out a form, and often not likely to answer the door to government surveyors - or let them in even if they do open up. This is a hugely significant part of the population in some places. Especially ones with high levels of immigration.

      This then ties in with my final point. Time. The data gets increasingly out-of-date - so doesn't pick up changes in patterns of population movement.

      Take the example of Slough (has all the above problems). They were begging the Labour government for more cash because they had high levels of immigration from both EU and non-EU sources during the last decade. Plus got a lot of the asylum seekers who come through Heathrow (and need much more government support than most immigrants). But their local government grant from the Treasury was based on census data from 2001 - when immigration levels were much lower.

    3. BobChip
      Stop

      Re: It's a fundamentally bad idea to use administrative data

      Absolutely! Scraping existing data = "garbage in, garbage out". A reality check requires real data.

  3. Adelio

    Data protection.

    I would have thought that information collected by (say councils) can only be used for the original purposes they were collected for. they would have to get permission from the invidivial users to re-use that data for other purposes. Unless government (local and national) can just ignore these rules!

    1. Kane
      Thumb Up

      Re: Data protection.

      "I would have thought that information collected by (say councils) can only be used for the original purposes they were collected for. they would have to get permission from the invidivial users to re-use that data for other purposes. Unless government (local and national) can just ignore these rules!"

      They would simply post on their local gov.uk portion of the councils website that they are making use of the data opt-in by default, and then bury that info as deeply as they can - if you don't want your info to be collated, simply send in your details to this P.O. box, on the back of a smutty 1960's postcard from Blackpool, with the simple message "opt-out" written in lemon juice, using a penny-black stamp! We'll make sure it's not used! Promise!!

      1. IamStillIan

        Re: Data protection.

        Won't get away with that under the GDPR changes, which will be full in force by 2021, unless there;s a Brexit based get out put in place.

        1. Anonymous Coward
          Anonymous Coward

          Re: Data protection.

          Why do everyone say Ooo GDPR won't allow that....

          Exceptions.

          -other important public interests, in particular economic or financial interests, including budgetary and taxation matters, public health and security;

          Quite easily would a census stat come under that, as the census is used to predict and validate government budgets, health provision and education as well as a thousand other thing.

          1. I ain't Spartacus Gold badge

            Re: Data protection.

            Also the raw census data doesn't get published for 100 years - at least that's how the law always used to be. I believe the last census to go public was 1911. Great if you're a historian, economist or into family history.

            In the meantime anonymised data sets will be released in varying levels of detail.

            1. DavCrav

              Re: Data protection.

              "Also the raw census data doesn't get published for 100 years - at least that's how the law always used to be. I believe the last census to go public was 1911. Great if you're a historian, economist or into family history."

              the next one will go live in 2021 (1921 census) but then we'll have to wait a while: the 1931 census (at least for England and Wales) was destroyed in a fire, and the 1941 census didn't happen because the UK had more important things to be getting on with at the time.

            2. To Mars in Man Bras!
              Thumb Down

              Re: Data protection.

              >>Great if you're a historian, economist or into family history...

              Except when, for some reason, a huge American commercial enterprise [ie. Ancestry. com] seems to have somehow managed to be appointed gatekeepers of this info and are allowed to charge us for access to it —profiting from selling us back the data which our ancestors were legally obliged to provide.

              [One of the reasons I refuse to fill in any of these census or census-like forms]

              1. DavCrav

                Re: Data protection.

                "Except when, for some reason, a huge American commercial enterprise [ie. Ancestry. com] seems to have somehow managed to be appointed gatekeepers of this info and are allowed to charge us for access to it —profiting from selling us back the data which our ancestors were legally obliged to provide."

                Anyone can go to Kew and look at it. It appears, although I'm not sure, that Ancestry.com digitized the lot, and so they have access to it. It's not clear to me whether they paid for the privilege of doing this, I suspect so. If another company wants to do this, I would guess that the ONS not only would be happy to allow it, but would legally be required to do so.

          2. IamStillIan

            Re: Data protection.

            That exemption is a Member State Derogation. ie. Member states can execute discretion to legislate over these areas. The UK won't be a member state, so we won't be entitled to derogations of our own making unless we obtain an agreement to the contrary, dispite the fact we're pretty likely to stiill have to comply with thme.

  4. Pen-y-gors

    Sharing data

    "The Digital Economy Act, which received royal assent last year, should make it easier for data sharing within government"

    Oh. Goody. What could possibly...?

  5. Pen-y-gors

    A historian writes...

    As usual, government only interested in saving money, and creating Big Brother.

    The paper census has a lot going for it. I can still look at the schedules from 1841, without having to wonder how I can get a copy of Windows -37 or Office 1848 to read it. Censuses are a primary source for studying and understanding our history, albeit imperfect - not that our governments are famous for wanting to learn from the past. Live in it? Yes, but not learn from it. And given their habit of totally ignoring evidence that disagrees with their knee-jerk prejudices, why do they want any data at all?

    We need to keep the paper census. Yes it costs, but it's worth it. Add a few more questions each time, and keep the data completely separate from other government databases. And in a hundred years our great-grand-children can look at them, see how bad great-grandma's handwriting was, and marvel at great-grandpa being a Jedi Knight!

    1. LucreLout

      Re: A historian writes...

      And in a hundred years our great-grand-children can look at them, see how bad great-grandma's handwriting was, and marvel at great-grandpa being a Jedi Knight!

      Way before that ever happens, some civil servant type will have left the document on a train/pub/hookers nightstand.

      The main advantage of digital that I see is that it removes one leg of incomoeptence from our overssers. On the other hand, remind me again when lessons will actually be learned about keeping our digital data secure too?

      1. TRT Silver badge

        Re: A historian writes...

        Talking of hookers... visits a location during working hours more than twice a week... Apparently my occupation is "rent boy". Serves me right for having a client who lives in a seedy room in Soho.

      2. JimC

        Re: it removes one leg of incomoeptence from our overssers

        You jest, surely. This proposal adds any numbers of layers to the process that are vulnerable to cockup, stupidity, malignity, carelessness and management incompetence.

        1. LucreLout

          Re: it removes one leg of incomoeptence from our overssers

          You jest, surely. This proposal adds any numbers of layers to the process that are vulnerable to cockup, stupidity, malignity, carelessness and management incompetence.

          Yes, but the data is awfully hard to hack (until digitised).

    2. smudge

      Re: A historian writes...

      We need to keep the paper census. Yes it costs, but it's worth it. ... And in a hundred years our great-grand-children can look at them, see how bad great-grandma's handwriting was, and marvel at great-grandpa being a Jedi Knight!

      But the paper census is already damaged. The 2011 Census was the first at which you could fill in the questionnaire online instead of on paper. The percentage of online submissions was in the mid-teens.

      Yes, the paper forms were scanned. and stored on micro-film, so if anyone is still around in 2111, they will be available to all.* But the information submitted online will never be seen.

      * Pro tip - one way of sending a message to your descendants is to scrawl it on your Census form. If it's not obscene or defamatory, it will appear in 100 years' time!

    3. I ain't Spartacus Gold badge

      Re: A historian writes...

      Pen-y-gors,

      See my (over-long) post above for some reasons why this got changed, other than money. Obviously saving money may have also been a major factor.

      But to cut a long post story short, the census was becoming increasingly unreliable. Far fewer people were filling the form in, paritcularly amongst poorer people in temporary/shared occupancy accommodation. And most particularly amongst immigrants, becoming a much larger group in recent years and often concentrated in specific areas.

      That was making the census much less useful for its most important job, planning local services, and that use trumps the historical one.

      1. John Brown (no body) Silver badge
        Coat

        Re: A historian writes...

        "and that use trumps the historical one."

        Oh FFS, do we have bring that man into....oh...err...sorry, as you were.

  6. IamStillIan

    Missing the point

    One of the really important points of the census is to validate all the other data excercises going on day to day. ie. is the admin data any good, or does government miss big sections of the population (always shows that it does..).

    This move threatens to render missed groups off the radar indefinately, which has all kinds of knock on in service provision, democracy etc.

    Doing bits of it online to save postage is seems fine form that this point of view, but not doing it at all is a huge failing.

    1. Crambo

      Re: Missing the point

      Strongly agree - the increased use of digitally enabled services and data collection on government funded service provision, and infrastructure can make use of the administratively collected records to reduce the reliance on another centrally managed detailed survey (census).

      The reality is that there are data quality issues within each and every database and data collection. The art in this digitally enabled world is to leverage off available data sources and importantly make use of Data Science with consistent and considered quality reviews and processes, in order to obtain an integrated data asset for the community that is fit for purpose.

  7. Anonymous Coward
    Anonymous Coward

    Anonymised linking

    So they make the data anonymous then link it at an individual level, how on earth do you do that?

    Also, I'm sure if they use the tax records there will be a lot of well to do areas where people are earning nothing.

    1. Whitter
      Thumb Down

      Re: Anonymised linking == Deanonymisation

      "Once it goes through the anonymisation process... our researchers ... link the data together to produce the aggregate insights"

      Otherwise known as deanonymisation.

      1. Tridac

        Re: Anonymised linking == Deanonymisation

        Link the data from the various data sets *after* anonymisation ?...

      2. Spanners Silver badge
        Meh

        Re: Anonymised linking == Deanonymisation

        "Once it goes through the anonymisation process... our researchers ... link the data together to produce the aggregate insights"

        Otherwise known as deanonymisation.

        If you aggregate information, it becomes even more anonymous. Instead of knowing who's a Jedi, you just know how many there are in whatever container you aggregated down to,

        Unfortunately, the usefulness of a survey in the future is greater if the actual data can be used. The 2021 release of the 1921 survey will be interesting to me as I will be able to find out about my grandparents. Aggregated data would tell me that one area had more people but the other was still down following WWI and the Spanish flu.

  8. rh587

    As someone who has recently been putting together a bit of family history... the census is an absolute boon. Sure, you can go through the Parish and Civil records for Birth/Death/Marriage/Baptism but particularly in the 1800s some of them are vague, hard to find or can be ambiguous where you have two John Smiths both getting married in the same year and you're not sure which one is yours. A household census that shows the whole family (with servants, lodgers, occupations and the rest) paints a broader social picture.

    The government could produce reports based on benefits or tax data for internal usage, but unless they use that to publish a derived dataset (household, dependents, etc) in a Census-like format, then that data is unlikely going to be searchable in the future.

    It would be a great loss to sociology and social history (both today and for future researchers) to abandon any sort of survey. The 2011 census cost £485m which (at £48m/yr) is chump change and - IMO - worth every penny.

    1. John Brown (no body) Silver badge

      "The 2011 census cost £485m which (at £48m/yr) is chump change and - IMO - worth every penny."

      To put that into perspective, it's only about 3 Persimmon Homes CEO bonuses.

      1. MrZoolook

        Or just under 1/4 of an "NHS Post-Brexit budget increase", but that doesn't exactly roll of the tongue.

  9. John Sager

    Bitrot

    Paper forms will last hundreds of years under the right storage conditions. CDs won't. I know ONS are probably acutely aware of this, but digital data will need to be re-stored (and perhaps restored) every so often, and we all know what happens to any funds that are supposed to be dedicated to this.

    Apparently Elon Musk's roadster is carrying a quartz disk with Asimov's Foundation Trilogy on it. Something like that may possibly be the future of long-term storage, as long as the design of the reader is stored in some more basic long-term medium!

    1. smudge

      Re: Bitrot

      Paper forms will last hundreds of years under the right storage conditions.

      2011 paper forms were shredded, pulped and recycled. After being archived onto microfilm.

      1. hplasm
        Devil

        Re: Bitrot

        "2011 paper forms were shredded, pulped and recycled chucked on a tip en masse..."

        FTFY

  10. Spacedinvader
    Happy

    2011 eh?

    It was that long ago I didn't fill that in and send back? How time flies (when you throw your watch...).

    So, 2021 for me to not do it again? Looking forward to it!

  11. Anonymous Coward
    Anonymous Coward

    "[...] so it might represent religion even if it doesn't capture religion.""

    CofE and RCC parish rolls are notorious for listing as members everyone who was baptised as a baby. They are also now making it very difficult for people to remove their names from those registers.

    That will inflate their perceived numbers for those still alive who were born before the 1970s when baptism was effectively mandatory. Official forms until the end of the 20th century still insisted on a tick box for an organised religion. It was a long time before a tick box for "none" caught up with the actuality in society.

    1. I ain't Spartacus Gold badge

      Are you sure about that? CofE rules are that parish rolls are updated every year and people have to add themselves each time.

      The church attendance is done from a separate survey, where churches are given a form to fill out for each service for a month - and then it gets averaged out.

      1. Anonymous Coward
        Anonymous Coward

        "Are you sure about that? "

        Not totally. I was going on reports that both Churches - but particularly the RCC - make it difficult to achieve. It is possible that there are two parallel counts. One from the baptismal archives - and the other as estimates of attending congregations every so often.

        I know of one church where it is reckoned that non-religious visitors regularly outnumber the sparse congregation - on account of the quality of the choir.

  12. Anonymous Coward
    Anonymous Coward

    The beauty of a census was that it was a frozen point in time.

    Analyses of different sets of dynamic data will not necessarily give the same correlation between sets. Whether that is better or worse than a single point in each decade is an interesting question.

    The single point census gave a snapshot. The new proposals seem likely to facilitate regular tracking of people and their activities. Even if the ONS themselves use anonymised data - other fingers in the pie may not be so scrupulous. Theresa will just love that.

    1. I ain't Spartacus Gold badge

      It's been really hard for local councils, in particular, to plan in an era when we've had 300,000 - 500,000 net immigration a year for over a decade. Obviously data from 5 years ago is bugger-all good if you get a disproportionate amount of that immigration to your area. Not so much of a problem if you have an ID card system of course, but we don't. So I think they've decided the needs of government planning are higher than those of historians.

      Getting people to fill out the forms is increasingly difficult too.

      Of course it doesn't help that other government data might be pants. It was discovered about 5 years ago that HMRC had created one or two million extra National Insurance numbers - basically they had duplicates for some people and were randomly assigning some of their contributions between them. I think it emerged when calculating pension contributions, and people complaining they'd paid in more than the computer said.

  13. B83
    Holmes

    Data Dark Ages

    This will be the census of the Data Dark Ages. There are too many people who wont trust an online survey. Im just thinking of my parents, their mates, my mates parents and their mates. "Oh online survey, wont be trusting that", and I cannot blame them.

    Most of my mates have real fancy smart phones but are generally inept at using them. They have traded their laptops/desktops for smart phones and wont be spending any time filling in an online form like the census.

    Yes it costs money but after the Car Tax** someone needs to the give the Mandarins a good shake and tell them its too early for this type of survey.

    Im no luddite but there are quite a few folk out there that are when it comes to this type of thing.

    ** http://www.bbc.co.uk/news/business-42009111

    1. I ain't Spartacus Gold badge

      Re: Data Dark Ages

      The problem is, people already weren't filling in the paper forms.

      1. B83

        Re: Data Dark Ages

        Hi

        I had to go and have a look and for the 2001 census there was a 98% return rate but did not find anything for the last census. I would be surprised if there was a massive drop in return rates though.

        https://www.ons.gov.uk/census/2001censusandearlier/dataandproducts/qualityofthecensusdata/responserates

  14. x 7

    I'm aged 60, and so far I've managed to avoid or ignore every census since I moved away from my parents. I really don't want those nosy governmental beggers to know anything about me..........I can see this will make things harder

    1. Fogcat

      I can't speak for other government departments, but in the past I've worked with people from the ONS and their "paranoid" approach to any one being able to individually identify any person or household from data they release is impressive.

      If other data holders were like them our data would be a lot safer.

    2. MrZoolook

      Remember this comment when you wish to draw a pension, and you suddenly about-face and give them every snippet of info they want so you can start drawing it.

  15. Kay_terra

    I can't help but wonder if they've got past the old storage issue that put this one on the back burner in 2011 after Francis Maude realized that the proposed aggregation of data would require IL6 storage (aka the same used by the spooks) to prevent national embarrassment, outrage and so on.

    Anonymising data after it has been received is well and good, but you have to store the original data somewhere ... and last I checked, the cost of adequately securing the data was higher than the cost of the paper census.

    1. ButlerInstitute

      IL6 ?

      "IL6 / Interleukin-6 Protein"

      That's from searching for "IL6 storage". Nothing in the first few pages of results that looks like it might be relevant to this issue.

      1. EnviableOne

        Re: IL6 ?

        Information level 6 its an old security level marking under the HMG IS5

        Would now be TOP SECRET or FIVE EYES ONLY

        ANyway the more Data Points they collect, the easier it is to De-Annomise the data, which is now a criminal offense under the DPA2017

  16. Gene Cash Silver badge

    So basically they've decided to not do the census and just guess

    On this side of the pond, the census decides a lot of government resources, such as health "care", housing/city planning, emergency police/fire funding, welfare, roads, and other important stuff.

    And yes, we have morons that don't fill it out. I threatened to evict a roommate over that. You wanna be that kind of idiot, don't do it in my house.

  17. Potemkine! Silver badge

    "At the other end of the spectrum are population characteristics like ethnicity and religion"

    Why the fuck would a government need to know about one's ethnicity or religion??

    Does the government also have to know about political preference or sexual habits?

    1. John Robson Silver badge

      Of course it needs to know your political leanings, how else can it draw new constituency boundaries?

    2. Weeble

      "Does the government also have to know about political preference or sexual habits?"

      Well, I don't know about political preferences, but when it comes to disease control and other health matters I can see value in knowing what "the people in general" get up to in their bedrooms.

  18. StuntMisanthrope

    We're crap, but we've gained access.

    Never a more frightening combination, when applied to the fashionable payment opportunity du jour. #platofirst

  19. Kernel

    Meanwhile

    Out here in a remote colony, on the far side of the world, the missus and I completed our entry in NZ's latest census last night - online, for the second time a census has been done this way.

    Get with the program guys, it's the 21st century already!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like