GPU-flingers' bash: Forget the Matrix, Neo needs his tensors Last week, Nvidia held its biggest ever GPU Technology Conference (GTC). The big walk-away is that GPUs are rapidly becoming an expected and standard component of computing, table stakes in many cases, across the computing platform. Big deal right there and hence the frothiness of much of the coverage. Before the conference, I …
A 'scalar' is a single number. Think of a single point in space.
A 'vector' is a column (or row, depending on your notation) of numbers. Stretch the dot out into a line, and the values of the numbers gives the direction of the line in however many dimensions you are using. e.g. A two number vector is like coordinates on graph paper, adding a third gives height as well, and so on into harder to visualise directions.
A 'matrix' is a grid of numbers, and can be thought of as a collection of stacked vectors wrapped up in an easy to process format. It can also represent a transformation - a way of converting one vector into another. The representation is the same, just the use has changed.
All of these are also tensors, of zero, one and two dimensionality. This can be extended, so a three dimensional tensor could be displayed as a 3D grid of numbers - a stack of matrices. Then stack those into 4D, etc, until your head turns inside-out.
However, the term 'dimension' starts to get confusing, as a vector with three numbers in it can be said to represent a vector in three dimensions, but it is also a one dimensional tensor.
A tensor is a linear function from a bunch of vectors (and dual vectors, but don't worry about those) to numbers. Vectors are pretty much what you think -- little brass arrows of various lengths -- and 'linear' means that T(u + v) = T(u) + T(v) and things like that. This is an oversimplified description, of course, but the whole 'tensors are matrices with certain transformation rules' thing is what physicists still learn (I was one) and it's just not helpful.
@ Frank ly
I can barely remember anything from that book
One thing I remember is that on my first reading of the book (many years ago) I thought the spelling of the characters' names was weird, bizarre, arty-farty bollocks and that nobody would ever do things like that in reality.
And then SMS came along to prove me completely wrong.
Weighing in at a robust 815mm2, oodles of flops of various kinds, and 300W, it’s big, it’s fast, and it’s hot
Hooray more half-filled racks :)
Seriously, it's getting to the point that datacentres are going to have to up their power and cooling if they have any real computational geeks as the servers/accelerators become so power hungry. My work's "high density" racks were optimised for blades in a corporate DC (think SAP, Oracle). We've just put in 100ish P100s and have had to spread them out in two racks.
"Hooray more half-filled racks"
5 years ago I was asked to name worst-case scenario heat generation for our server room and the facilities people visibly gulped when I said "Well, I can fairly trivially put 45kW in a single rack and it's only going to get worse"
The limiting factor isn't cooling capacity but how much power we can get onsite. They wrote a report saying I was asking for impossibly large numbers when I stuck my finger in the air and said "I need 50kW _now_ and the capacity for up to 250kW in future would be nice, but 100kW would probably do(*)" - sure enough about 6 months later someone filed a request for systems which would have maxxed out the 100kW in the first pass and the 250kW fairly easily. Thankfully he didn't get budget for it (I told him that doing this would require major infrastructure in addition to the rack-mounted kit). You have to love academics shooting for the moon.
(*) I have about 25kW in the room and maxxed-out cooling due to manglement rejecting my initial estimate of 75kW as "too high", with another 20kW in old computer rooms that was supposed to be migrated in, but can't be. Pointing out that even a very low 3kW/rack over 18 racks adds up to 52kW didn't go down well.
Somehow, while I wasn't looking, the word "matrix", referring to N-dimensional arrays of scalars which are the subject of linear algebra, got renamed "tensor".
I find that a bit odd, since a tensor is a very specific use of the matrix by physics, a use which bears little or no relation to the use of matrices in the programming of neural networks.
I'll give you this, though: the word "tensor" is a lot cooler and edgier than the far more appropriate, but old and stodgy term, "matrix".
Source: someone who started studying linear algebra and physics around 1972.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
