* Posts by Frumious Bandersnatch

2662 publicly visible posts • joined 8 Nov 2007

This storage startup dedupes what to do what? How?

Frumious Bandersnatch

Re: I'm a little confused

The analogy did put me in mind of dictionary-based compression (including LZW), too, as well as Merkle trees (for the sync problem).

The idea of re-using dictionaries sounds good, but you need good algorithms and data structures to make it tractable. You can't just blindly try compressing all blocks against all dictionaries to find the best compression ratio. That would take forever. I think that some clever tree-based/recursive construction (probably along with Bloom filters or maybe Rice-Golomb encoding) and some dynamic programming could make this work.

Frumious Bandersnatch

OK, this is my final post on the question, Vic. I promise.

I understand better now the point you're making about "entropy". You're saying that, eg, if I hashed all the natural numbers, represented in ascii, in a range (let's say they're zero-padded up to some minimum block length to make a fair comparison) that there will be fewer collisions than if the blocks all contained random 8-bit data. I think that a well-designed hash won't have significantly different collision rates over the two kinds of data for a given capacity range (ie, the number of messages to be stored).

And I'm, saying that, in the general case, [that the risk is "vanishingly small enough"] is simply not true

Mathematically speaking, you're absolutely right. Any hash can and will have collisions as proved by the counting argument/pigeonhole principle (or birthday paradox). All I'm saying is that in practice assuming that the same hash implies the same contents can be a reasonable engineering assumption. As evidenced by how git does it.

The assumptions that git makes on the actual hash size (160 bits) and expected number of commits (and, if I were to accept it, a factor to account for "entropy") aren't going to hold for a massive block-based de-dupe system, but you can plug the numbers into some formulas to find the expected collision rate and choose your digest size to make the risk "low enough" that it's not worth worrying about (eg, a mean time to collision of 50 years, if that's what you want).

Frumious Bandersnatch

I don't know what you're trying to say, there, Vic. By your logic, git having less entropy in the source should imply less entropy in the output hashes, which would in turn imply more collisions.

Anyway, entropy is irrelevant since the point of a good (cryptographic) message digest is to decorrelate patterns in the input from those in the output hash. Entropy measures over the set of output hashes should be the same regardless of input (again, for a "good" hash algorithm).

I'm just making the point that while you can't get away from the pigeonhole principle, if you have vastly more holes than pigeons, you can put some sort of bound on what's an acceptable probability of collision and design your application with the assumption that this risk is "vanishingly small enough".

It's all a trade-off, like Trevor explained in the article and in his post above.

Frumious Bandersnatch

That is not reasonable.

The git tool uses large digests (sha-1) to identify all commits. This is a 160-bit hash and apparently you'd need 2^80 random messages to get a 50% chance of a collision.

I recall reading that git hasn't any safeguards or double-checks to see if two commits hash to the same value. So Linus was just trusting the maths and making a calculated risk. For his purposes, at least, the chance of collision was so vanishingly small that's it's a risk that he was willing to take (for the sake of simplifying the code, one assumes).

I get what you're saying about the pigeonhole principle, it's not always unreasonable to assume that collisions won't happen.

Hillary Clinton: Stop helping terrorists, Silicon Valley – weaken your encryption

Frumious Bandersnatch

What?

From what I've read, these guys used no encryption at all.

Is "bandwagonesque" something other than a word to describe the minds of politicians (or a Teenage Fanclub album, obviously)?

It's come to this for IBM: Watson is now a gimmick app on the iPhone

Frumious Bandersnatch

"kids like legos"

It's "Lego", not "Legos".

http://english.stackexchange.com/questions/10839/legos-not-lego

Reg reader achieves bronze badge, goes directly to jail

Frumious Bandersnatch
Paris Hilton

Re: Actually I think there should be some form of quality control

To avoid random spamming by the desperate.

I've often thought of exactly the opposite. I should get off my arse(*) one of these days and set up "bewildering-upvotes.com" (**) where each week members vote on the most boring/anodyne post on their favourite site and then everyone goes and gives it a thumbs up. Perfect for sad gits everywhere who fancy themselves as anarchists, but who are too polite to do anything naughty.

* and probably sit back down again if I want to do web coding

** motto: "Dada's not dead; it just smells a little funny" perhaps?

Thanks for playing: New Linux ransomware decrypted, pwns itself

Frumious Bandersnatch

Re: Keep your secret secret

you'll have a million different possibly keys

A 20-bit key is eminently crackable using brute force, especially if the file is of a known type. (<insert shell one-liner calling 'file' program on each one here>)

What you need to do is use "cryptographically secure" RNG like reading /dev/random (which may give the game away) or gather entropy from the running system in much the same way that the kernel and other RNG seeding functions do. Definitely don't use something like the time as the sole value for seeds.

Irish roll out obligatory drone register

Frumious Bandersnatch

It seems to make sense to have some kind of weight limit, but I'm wondering whether 1kg is a good cut-off? It seems a little on the low side if someone wanted to try their hand at building their own (with Pi and Arduino, say) rather than buying a kit or pre-built model. It seems that an extra 250 to 500 grams would be more practical in these cases.

Riding on the memory bus: Micron brings out 8GB flash DIMM

Frumious Bandersnatch

Re: So this uses an already scarce RAM slot on the MoBo?

HOW DOES THIS WORK DAMMIT?

If they fit in a RAM slot then presumably they look just like regular RAM to the OS.

There are a few ways it could be used:

* reserve it the same way as you would a block of physical memory and have something like /proc/kcore so that userland processes can mmap parts of it (use it like regular memory)

* make it look like a block device that you can put partitions on and mount and so on (treat it like a disk drive)

* keep it all for the kernel and use it transparently in some subsystems like software raid, filesystem journal, disk cache, etc.)

Untamed pledge() aims to improve OpenBSD security

Frumious Bandersnatch
Terminator

"I promise not to kill anyone" (was Re: Stupid idea)

Killing it if it tries to do one is a terrible idea

Not necessarily. Look at Erlang. There, a major plank in the philosophy for handling errors is to just let the process die, along with any linked processes. Mind you, it also has the concept of supervisors that can restart processes after a shutdown. You have to actively work with this paradigm and code your application to take advantage of this, though. You can't just bolt it onto existing programs.

As for assert(), they do you no good if the code has a buffer overflow or something that wasn't explicitly trapped and the bad guys get to run arbitrary code. The pledge() idea gets around this nicely as injected code is sandboxed and probably won't be able to do much damage.

As for something managing to deliberately kill your anti-virus, the AV vendor can just use the idea from Erlang: let the affected daemon/process die, have a supervisor watching for any termination, log it and restart the process. If it consistently crashes on a given input file, mark that file as possibly containing malware and skip it on subsequent runs. Though really, I don't expect AV software to be crashing on arbitrary input files...

The main downside to this, as I see it, is that it does no good against viruses that are aware of how it works. Simply trace the call to pledge(), replace it with a jump to the viral code and there's no barrier to doing whatever "unsafe" operation it wants. Ditto running any untrusted program, which definitely won't call pledge().

I can see the reason for wanting this to be a system call rather than an external "rights" database. It's a lot simpler, easier to support and very lightweight. It isn't a complete solution, but for certain things (code injection) it's rather a neat defence.

Drones are dropping drugs into prisons and the US govt just doesn't know what to do

Frumious Bandersnatch

Bolas

A post with words

AMD sued: Number of Bulldozer cores in its chips is a lie, allegedly

Frumious Bandersnatch

It's hard to see how this can succeed

1. I'm sure that technical documentation was available, both from AMD and review sites

2. Marketing bumf (not direct from Intel) often counts hyperthreaded "cores" as full cores

3. I'm sure that he had the opportunity to return the part as not matching what was offered, but declined to do so

4. As the article points out, what constitutes a core isn't well defined (and perhaps main ALU and supporting stuff does count)

5. Probably hard to prove AMD intended to mislead (though perhaps plaintiff doesn't have to go that far)

6. Balancing actual loss versus what he's claiming, did the loss of half an FPU core per "real" core really affect him that much (who saturates their FPU units in desktop/laptop PCs anyway?)

It's not nice when you buy something that doesn't live up to expectations, but seriously, I think this guy doth protest too much.

OmniRAT malware scurrying into Android, PC, Mac, Linux systems

Frumious Bandersnatch

"presumably only until the police move in"

As I've read elsewhere, there's nothing illegal about remote access tools in most places. I think it needs to be pointed out that the version of the software being sold is distinct from the viral form, which uses the toolkit.

It's trivial to find the site selling "omnirat" and browsing the site (with lynx/w3m/in a VM) lists the features as basically only providing a remote administration tool. No mention of exploits or nefarious installation routes. For comparison, I can still download back orifice from sourceforge (for free).

Implying that the plods should/will take down the vendor's site just demonstrates a fundamental misunderstanding of the above, IMO.

In-a-spin Home Sec: 'We won't be rifling through people's web history'

Frumious Bandersnatch

"world leading oversight arrangements"

For if this negligee is fit for the UK queen, then other leaders would scarce deign but to try it on.

Kevin Spacey becomes pitch-hound for Swiss security bods

Frumious Bandersnatch

For relaxing times

Make it Suntory time.

Post-pub nosh neckfiller special: The WHO bacon sarnie of death

Frumious Bandersnatch

Walking in the forest when ferns are releasing spores

I think you're talking about bracken? A lot of sources confirm that the spores are indeed carcinogenic, but I also came across this page that “Regarding bracken mentioned in your newsletter, if you are worried about breathing spores, we understand now that bracken doesn't spore in this country [the UK] and its expansion is only by vegetative growth."

Unfortunately, I couldn't find any other corroborating evidence to support that, but check out the article on UK's Cancer Research site, which does suggest wearing masks on dry, hot days, but that it's "unlikely [to be] a major cause of cancer".

Cops use terror powers to lift BBC man's laptop after ISIS interview

Frumious Bandersnatch

Re: So What?

To combat extremism you must first know the mentality of an extremist and in order to know that you must talk to extremists

Exactly this. It reminds me of this rather excellent article about what ISIS is, what it wants and how it intends to get there.

The thing is, much of what's in that article is stuff that I'd never seen discussed on TV. Like how central the idea of a caliphate (and ISIS's self-identification as primarily a religious movement) is, or how ISIS justifies beheadings and other acts of terror (bizarrely, they're seen as a way to avoid bloodshed by hastening everyone's conversion to their brand of Islam), and why direct assault on their territory would actually play into their hands (they feel that it would legitimise their claim of being a valid caliphate).

There are several controversial people interviewed in that article and together they provide a compelling picture of who ISIS are and why they are doing what they're doing. Imagine a situation (like the one reported on here) where even talking with these people could cause journalists themselves to be subjected to police attention and have their livelihoods threatened (what else was on that laptop? would any other person be willing to talk to them again, even on unrelated issues?). In fact, this is the classic example of the "chilling effect" you sometimes read about. How can a free press and proper democratic processes survive when talking to the wrong person results in state-sponsored searches, confiscation of your property and the threat of imprisonment (via not handing over encryption keys, assuming they exist)?

Bacon can kill: Official

Frumious Bandersnatch

Re: To which I say....

You foisted margarine

Margarine wasn't created for health benefits. It was created "for the armed forces and lower classes" as a substitute for butter [Wikipedia]. I don't think that anyone ever claimed it was healthy.

TalkTalk attack: 'No legal obligation to encrypt customer bank details', says chief

Frumious Bandersnatch

weasel words

As Steve Wright said, "Eagles may soar, but weasels don't get sucked into jet engines"

I can only assume that Harding would fail to see the satire in the above.

El Reg revisits Battle of Agincourt on 600th anniversary

Frumious Bandersnatch

Re: @Elmer Phud ....now we see ex-squaddies with PTSD trying to get shelter in doorways....."

I knew that Mr. Kipling made exceedingly good cakes, but I never knew he was such a good spinner of yarns.

Caption this: WIN a 6TB Western Digital Black hard drive with El Reg

Frumious Bandersnatch

Re: Calm Your Mind

I know... hence "last week".

Frumious Bandersnatch

A quick shufti in the Perliscope and a foolproof pattern for winning entries stood revealed: [-a-zA-Z0-9,!.:;()'"?/ ]+

Frumious Bandersnatch

"See? There is no brace", repeated Doolittle one more time. Teaching the laser cannon phenomenology was a lot harder than he had imagined.

Frumious Bandersnatch
Happy

Re: Calm Your Mind

Sirsasana? That sounds sooo last week.

Frumious Bandersnatch

Nobody could accuse Charles Atlas of being a Luddite after seeing how graciously he yielded to Google Maps.

Frumious Bandersnatch

"I don't know what the heck a "hard disk" is, but if that's what baby wants, then I'm the guy to get it for ya!"

Frumious Bandersnatch

Sadly, all that most VW engines had to look forward to in the aftermath of the emissions scandal was a life of servitude as robot butlers.

Frumious Bandersnatch

Wow. Proctology sure has advanced in the last 100 years.

Frumious Bandersnatch

Fritz Lang's Metropolis, where men were men and women had integrated toast racks.

Frumious Bandersnatch

Disintegrating eye, eh? Wait til you get a load of my smashing arm!

Frumious Bandersnatch

I gave you one thumbs up. I think maybe it deserves another

Frumious Bandersnatch

"Dutch" Schulz stood admiring his creation. "Finally," he thought, "I'll never have to stick this thumb in a dyke ever again"

Frumious Bandersnatch

Pierce Brosnan gets a few last-minute acting tips prior to starring in Treehouse of Horror XII.

Frumious Bandersnatch

This "Dolly" he'd ordered from russianbrides.ru sure wasn't like all the other broads.

Frumious Bandersnatch

You think I've got a big bicep, eh? Well whey don't you try playing Counterstrike with a thumb stick this big?

Frumious Bandersnatch

My mate Anatoli Bugorski assured me this is perfectly safe.

Frumious Bandersnatch

Let's get one thing straight, Glados: you do the working, and I do the ordering!

Frumious Bandersnatch

You got it wrong, bud, I'm gonna probe you later.

Frumious Bandersnatch

"An bhfuil tusa ag labhairt liomsa?"

(http://filmbase.ie/short-film-yu-ming-is-ainm-dom/)

Frumious Bandersnatch

Young Davros was born into a privileged life with flat floors everywhere. No wonder that he never considered that casters might not be the most practical means of locomotion in a battle droid.

Frumious Bandersnatch

Slick could hardly believe it. The shimmering face in the picture tube was really speaking to him. From Nigeria! And a prince, no less!

Frumious Bandersnatch

The "tea cannon alarm clock" prototype. Not a good idea then, and it's still not, despite what IoT vendors tell you.

Frumious Bandersnatch

Boy, those schmoes at Sony, Microsoft and Nintendo are gonna be mincemeat when my lawyers get through with them. Nobody gets to use my joystick patent unless I say they can!

Bacon as deadly as cigarettes and asbestos

Frumious Bandersnatch

Ne vous inquiétez pas

Maintenant ils disent que la glace est vraiment bonne pour la santé.