Posts by ibmalone

Re: CSV?

CSV in general is schema-less. CSV following a schema you've agreed for data interchange is by definition not schema-less and import libraries will let you specify your expected data types and fields (often trivially). Anyway, the concept only applies here in watered-down form, since we're talking about multiple records of the same form, it's only the field types and order to be specified, not the relation between different types of observation.

There are definitely other choices which may be better suited to a task, but what csv has over excel is that it's easily generated and read, doesn't have the complexity of multiple sheets and non-data objects, isn't primarily intended to be loaded into software where a field can be altered with an accidental keypress, and doesn't have hard-coded limits on record or field counts (but yes, there are better formats for really big data). And to top it off, xls / xlsx are also schema-less.

Re: CSV?

Yes, that kind of mess does exist, but here they should have enough control of the whole process to avoid it. (And it's more of an issue when free text is included, numbers are unaffected by those minor encoding mixups, only if you're off into the wilds of things like ebcdic is it a problem, but I'm not sure you can even run excel on systems that generate that kind of encoding.)

By the time it gets to tens of thousands of results the project is big enough that it requires a person in charge who recognises that their desire to see the raw data does not justify using excel as the primary data storage.

Re: Excel isn't that bad.....

The fundamental problem is spreadsheets mix data, code and presentation, which should be separate, particularly if data is going to be passing through many hands. It certainly has its uses, the ease of setting up some simple calculations or graphs with little training is handy. The problem here is not the use of excel for a report, it's using excel to manage the data storage and transfer.

Re: Familiar kind of scenario.

Very much disagree, there is database and stats software out there that eats this stuff for breakfast. The problem is putting people on the job who are unaware of this and have been given only a hammer.

Re: CSV?

While I sympathize with the sentiment, it only really applies in the locked room scenario where somebody throws a file over the wall to you. If you're setting up a data transfer you are allowed to talk to the other side in advance, and even if not you can provide a data dictionary to specify fields and how they're stored (date format is the most serious problem, numerical data generally isn't unless there are very large integers, string quoting can be a bit of a pain, but field count checks will catch anything outrageous). In the scenario where you're the person both controlling the generating and reading ends of the process you have the luxury of making it fit your every need. Yes, there are more robust formats, but excel certainly isn't one of them.

Re: you know

csv code golf, R:

mydata <- read.csv("mydata.csv")

Re: CSV?

But the excel excuse relies on it being used as an interchange format, "oh we were just using it to move data between systems, we aren't so completely incompetent as to maintain all the test records in excel". If it's being combined into a single excel file then that is worse.

As for csv lacking integrity checks, well, so does excel. Send it with a checksum if you're worried. Yes, excel might produce a warning if a file has been corrupted, but it can't tell you data in a row hasn't been accidentally modified. There is a pretty reliable convention for csv now, and if you control both the sending and receiving end then you can make sure they're in agreement about field data types. If you don't want to do that, R, Stata, SPSS all have their own data formats which are designed for this stuff. Using a data format that is designed to be opened up and monkeyed around with rather than machine processed for data at this scale is a mistake.

Re: Not the same thing

Everyone pushing this angle is of course neglecting that vaccines can be given as boosters, that we could re-vaccinate vulnerable people once the thing is under control (as we do with flu every year), that some vaccines in testing have produced stronger antibody responses than wild type infection, that part of the reason cold and flu viruses are hard to immunise against is their mutation rate, which is lower for coronaviruses (though there are other human coronaviruses, there aren't many and we haven't had a serious need for a vaccine before), that most of the vaccines target the spike protein and a virus that has mutated to escape that response could well be less virulent and, lastly, that even attenuated immunity may lead to less serious disease.

For anecdotal re-infection, it's early days, but as over 34 million people worldwide have had it if re-infection within six months was at all common we should have seen more than anecdotal cases by now.

Sure, even given all that a vaccine might not work. But let's give it a chance, hey?

Re: I like Electron but...

Funny, I was expecting this one, or maybe this one, or it might have been this one.

Re: data use policy

Well, fine, but if you want to be that paranoid why are you connecting to the internet and not in a bunker in the desert? This is the same risk every other device you use that isn't completely open from the ground up carries, so why specifically raise it as a privacy concern about the contact tracing app? It does seem to be rather muddying the waters, we're talking about people who have smartphones anyway.

Re: data use policy

"Obviously", except not. You could read about how it works if you wanted, people have explained it here before.

*Sigh* okay.

The only identifiable bit is the test results token https://www.gov.uk/government/publications/nhs-covid-19-app-privacy-information/the-nhs-test-and-trace-app-early-adopter-trial-august-2020-data-protection-impact-assessment#test-results once the phone receives a token for a positive test it uploads the diagnosis keys, which are not connected to the test result token and cannot be traced to the user (in any case, you've had a test, so the people who conducted the test know this about you anyway). The list of diagnosis keys that have been sent in after positive tests is made available, people's individual devices retrieve this and use them to check against the exposure events they have collected and whether your exposure to a matching positive testing device was above the defined threshold. No central server knows which exposure events you were in contact with.

Re: data use policy

The short version is the data is kept on your phone until you report a positive test, at which point your randomly generated tokens for the preceding 14 days are anonymously uploaded to allow others to check against them. There isn't scope for a data breach as identifiable data isn't centrally held, unless the app is collecting data not documented in the policy, and the source code is available so that can be audited.

Re: Compulsion or not

My understanding (although the law appears to change by the hour, so would advise people to check the legislation for themselves rather than rely on a comment on the register...) was that fines for breaching isolation were based on being instructed by a public health official to isolate. The app is not a public health official.

Re: Is the QR code check part of the app in a legal requirement for venues?

My understanding is venues must have the qr code, but it's not required that visitors specifically use the qr code to check in, however venues now have to collect contact tracing details on visitors. So either you use the qr code or they have to get your contact details. I suppose it's then up to them whether they want to do both or say qr only.

(Not sure about under 16s in either case as their data may be more protected, do they even have to contact trace?)

Re: United? Kingdom

So far as Northern Ireland goes, it made sense to have an app that interoperated with the already running Ireland one, and the English effort had vanished up its own fundament anyway.

Curious difference between NHS Covid-19 and StopCovid NI; StopCovid NI reports number of installations (currently ~370k), NHS Covid-19 does not report anything about number of users.

My phone is on android 9, with bluetooth and location on for the stop covid ni app battery drain is noticeably up, I usually could get 3 days light use out, with those features on for covid notification I get about 1.5 days. No, I don't have twitter installed and facebook while pre-installed is never used. Turn off bluetooth or location and battery life goes back up, there's only one system that needs both on to operate. The app itself doesn't actually register as using any battery power, but I've never found those stats to be at all accurate.

So it will be interesting to see whether NHS Covid 19 does the same thing. I don't know why your experience is different. Older android have the notification feature through a google play services update, and this may mean it operates differently in regard of power saving. I also have a relatively compact phone which is normally in a battery saving mode, maybe covid notifications screw that up while those with slabs don't notice any difference.

As someone who has been posting the same thing not as AC I tend to agree having skimmed the data use policy. This looks okay on first inspection, the main differences to the anonymous bluetooth-only apps like StopCovid NI being the recording on your phone of venues via QR codes (not uploaded, unlike the bluetooth tokens not even uploaded if you have a positive test, has to have been reported by a manual contact tracer) and the area/start-of postcode risk status (it is not entirely clear to me that this is not shared on reporting a positive test, but you'd think they have it from the test records). Haven't gone through it with a fine-toothed comb, but it does look much better.

While the duration of data retention link is broken (first thing I noticed), it is set out in https://www.gov.uk/government/publications/nhs-covid-19-app-privacy-information/nhs-test-and-trace-app-early-adopter-trial-august-2020-privacy-notice which, despite the link title, is the privacy notice accessible from https://www.gov.uk/government/publications/nhs-covid-19-app-privacy-information

I wouldn't say the functionality is that much more extensive than, say, the StopCovid NI app, only setting up the QR code infrastructure (letting venues get unique codes) is significant. Months wasted here.

Re: Sounds familiar

The thing is, if they have such a poor understanding then they shouldn't be selling locks based on it. You don't even need to understand the actual bluetooth stack to come up with a secure scheme, just use it as your transport protocol and put something that's actually secure on top of it (which does not re-use tokens, and no I wouldn't trust these people to achieve that securely either, but a replay attack is the digital equivalent of a shimmable lock).

Re: Sounds familiar

There are traditional padlocks that are very hard to defeat, and there is such a thing as strong cryptography (this being elReg I'm sure there will be people who insist that they only do banking transactions face to face, but most of us rely on computer security to look after our bank accounts). Combining the two at a given price point though... it means needing multiple engineering disciplines and extra components to marry the two together, which means more points for vulnerabilities. The best illustration of this (but not the only problem) is that many have a mechanical bypass for when the electronics don't work, you now have two opening mechanisms to attack before even contemplating the physical security blunders many of them have.

I'm sure a secure electronic lock can be built (comparably secure to a mechanical one), it just doesn't seem that anyone has any interest in making one commercially viable.

Oh, and blockchain. Reminds me of an argument last year about blockchain being the solution to every problem relating to electronic voting. "You can't be sure the software hasn't been compromised" "Yes you can because blockchain". Right have fun with that.

The hasp is part of the bike lock I think, so still a fail if that's the case. (Have looked at the video in slow motion and don't think the shackle breaks. There's some kind of collar on one side which means you see a dark line when it's spinning round.)

The biggest physical crime though (if making a lock from zinc wasn't bad enough) is being able to get into the electronics and locking mechanism with a screwdriver. Seems to be the case with a lot of 'electronic' locks that they're constructed the same as other consumer electronics, just attached to a locking mechanism.

As for cutting yes, you will be able to cut any lock with the right tools, but you're going to have a much harder time doing it with well hardened steel. LPL aficionados will be familiar with his hydraulic cutter series, they have failed (just not on padlock shackles) https://www.youtube.com/watch?v=lvn3_CNVSFs

Re: OA publishing

"the journals tend to only select sufficient-quality papers"

That's the point where it falls down most often, there's no guarantee a "top" journal will accept good work (and they do accept work that's not so good) the prestige is often self sustaining, both on the submissions and readership sides, but decisions to send for review are not transparent and good work can, and does, get sent back almost unread. Then it gets submitted elsewhere, but over a wider spread of journals. Conversely, the biggest name journals have themselves been hit by fraudulent research before and it will continue to happen.

This shouldn't be a problem these days; it's possible to track article impact independently, it's possible to search for and find new articles without looking through a journal's contents page, journal impact isn't necessary anymore but still used. There are better (still flawed) metrics, yet you will still see "publication in high impact journals" as a career development criteria. The goal of publication should be effective communication of your research, not which banner is on the front page.

Re: I hate techies!

Zoom is also a weird niche solution in that it requires custom software installed. Both Jitsi and gotomeet will work from a browser (typically Chrome) and work well. Heck, I know of one library service that is now doing its streaming story-time (for pre-school children) via Zoom because that's been mandated from on-high, rather than facebook which is, you know, facebook, but does have the advantage of viewers not needing to install special software and is actually more appropriate to what they're doing. Our entire organisation has access to teams, which works absolutely fine (including giving you access to meeting notes and slides), just doesn't have the 'fun' party view for panopticon enthusiasts. Zoom seems to be one of those services that is popular because you can set up an account easily if you have access to a credit card, thus it's easy to look proactive and cutting edge, especially with a cool name, Zoom. Zoom. Zoom Zoom. Pity they didn't call it Zoomer.

That said we now have an institutional subscription to zoom too, because apparently it's the most reliable way for students in China to connect. Make of that what you will.

Re: Elon-gated kit

You can talk about "human bias", but people value what they value and those trillions are only hypothetical a long way down the road (another difficulty with applying the trolley problem; it assumes perfect knowledge of outcomes). Additionally, if colonisation offers an escape to only a tiny fraction of a percent of people, it will save an even tiny percentage of Earth's biodiversity. The reality of space travel is that it's slow, those future people may never find another Earth and never know a life beyond four metal walls. I'm not against it as an insurance strategy, but we have to recognise it solves no problems for anyone alive today.

Space-based hardware isn't reliant on such an effort, we already have weather satellites and communication gear. If we want better asteroid detection and diversion strategies then we'd better get working on those.

Re: Elon-gated kit

Can't ignore maths. Colonising space may help guarantee humanity's future survival, but it will never solve population or resource problems on Earth due to the lifting requirements, an ark is only an escape mechanism for a few (remember what happened to the rest of mankind in that story, and indeed everything else, except fish, the almighty is apparently happy with the fish).

Yep, the goals stated aren't compatible with one single chip at a particular location, that's just not how the brain works, it's not a data bus you can tap into. Creating a control surface is probably do-able, the primary motor area is quite well defined, on the surface, and has a kind of topographic mapping to the body. Monitoring vision? That's at the back of the brain, and lots of it is in the calcarine sulcus, possibly the deepest fold in the brain. At this stage we should mention the brain in its normal state has a consistency only a little firmer than jelly.

Memories? Well, that's an entirely different ball game, we first have to ask what type of memory. If it's how to read or wield a hammer then in the associative regions at the sides, but also tying in other parts of the brain (like the visual and auditory areas for reading and motor areas---secondary I think---for hammers). Ah, you want episodic memory? Hippocampus, folded deep within the brain, good luck getting in there without touching anything else. Try not to damage it or no new memories for you my friend.

Hang on, there's a bit called the thalamus that a lot of connections go through (but not all), let's try that! Oh, it's a dense 3D junction box located right in the middle of the brain.

All this is of course after the risks involved in opening the skull up at all.

Potentially useful for people with spinal injuries and other such life-changing problems (like optic nerve damage)? Yes. Wire you in to your xbox? No.

Re: WD40?

I had an acquaintance, many years ago who ran a small locksmiths shop, he was the go to guy whenever the plod needed to get past a lock or into a safe.

Sounds like a useful person to know. My experience so far includes two different types of locksmith.

I encountered the first when I got locked out of a first floor flat with a street door. The neighbours were kind enough to lend me a phone, but as it was a weekend and the only other person with keys was away, I had to call a locksmith. The people I ended up using (something like all service 3 u) had a generic branded websites of the type you'll recognise if you've ever looked for cleaning services. As most of the ones I could find were of this type, no call-out charge and seemingly quicker to get hold of than the more traditional looking ones, I made the mistake of calling them out. The guy turns up, takes a look at the pretty standard (5 pin, I've still got one of the spares) rim-lock for the latch (the lever deadlock hadn't been used) and announces, without even touching it, that he can do nothing with it and the only option is drilling it (incidentally, tries to convince me to do a weird thing about agreeing the lock is faulty when I've actually lost my keys, and despite me saying no to this still invoices me as such). So he drills it (takes a couple of minutes) and I get charged for two hours labour and a new lock at about 200% mark up on retail price. Overall cost about £250, should have argued more, but I'd been wearing wet running gear for a couple of hours at this point and it was 8pm with work the next day.

The second one I never actually met. Got home one day (to a different first floor flat, with a somewhat more sophisticated latch lock and euro cylinder deadlocks), to discover a letter from SGN (the gas distribution company) on our kitchen table. A leak had been discovered at the external meter (while testing some installation for another flat), so they had shut off our gas at the meter and apparently had to enter the flat for some reason, leaving a letter to tell us what had happened. Our landlord did not know about this, there was no forced entry, and no other key-holders. The most likely explanation is they got a competent locksmith (rather than a guy with a black and decker and a van full of yale locks) to let them in.

There's a moral to this story that shouldn't be too hard to spot... If you look at the master locksmiths association guidance, a weekend evening call-out rate for one of their members might be up to £120 hourly, so maybe when I got locked out I could have ended up paying one of them about the same amount of money instead. But I wouldn't have been stuck with someone I wasn't certain I could trust drilling out my front door lock, and I'd have had a better idea beforehand what I was actually going to pay. (Having not thought about the cowboys for a while, I had a quick look on google, and if the one-star reviews are to be believed it seems their prices have doubled since then in any case.)

Re: So we need to upgrade the physical locks....

A few already do, avocet abs keys for one, https://www.youtube.com/watch?v=zq5rGjt-9rQ you don't really need to go that far though, this will probably only work easily with straightforward in-line pin tumbler locks, there are a host of other types once you get to slightly more secure models, including locks with multiple locking systems (say, combined dimples and cuts for one), which this would have great difficulty disentangling.

Re: Like to have an expert check the privacy statement & app

Having used the Stop Covid NI app while I was visiting for two weeks shortly after launch I was pretty impressed by the data use policy, very clear about purposes of use, what data was collected, what data stripped and at what point, conservative on retention period and with an opt out available at any time. This is in very stark contrast to the policy for the previous NHSX app (and the contact tracing programme) which said approximately, we'll hold all the data you give us for 20 years, we'll be using it for whatever we can think of, no you can't withdraw consent, yes, we'll make it available to Serco.