What does TensorFlow mean?
We explore tensors and the dataflow paradigm to ground the term TensorFlow.
A tensor is a mathematical object represented as a multidimensional array of numbers. Based on the context it can mean many things. For example, a tensor can be a linear transformation between vectors, scalars, and even other tensors. More on the mathematical aspects of tensors, here and here.
For the purposes of TensorFlow, think of a tensor as storage unit capable of storing objects in a structured manner. The following diagram illustrates the structure and indexing of tensors as they grow in dimension.
In the above diagram, a cell is denoted with the indexing scheme used to access the stored value. When the dimensions grow, index for the higher dimension is tacked onto the left side. For example, an element in a 2-D tensor (matrix) is indexed as
(row, column), whereas in a 3-D tensor, it is
(depth, row, column).
It’s sometimes hard to visualize the structure of tensors larger than three dimensions. One trick is to think of higher dimensions as a container to store an array of lower dimensional tensors in an structured (indexed) manner. For example, a 4-D tensor is an array of 3-D tensors.
Similarly a 5-D tensor is an array of 4-D tensors.
Now, try imagining a 6-D tensor.
In Imperative Programming, the code is a sequence of commands. The control flows through the commands in an ordered manner to generate the desired output. The primary focus of an imperative program is concerning what statement needs to be executed next, rather that what data is required for said execution. More succinctly, imperative programming is control flow rather than data flow.
In DataFlow programming, the code is a collection of operation definitions. An operation can be something as simple as addition of two numbers, to some complicated optimization operation. Operations are dependent on some input data, or on the output of other operations (they can be stand-alone as well, but it is uninteresting). The primary requirement for evaluation of an operations is the availability of all its inputs. This requirement makes the code reactive to data, hence making it data flow.
The code of a dataflow program can be visualized as a directed graph whose nodes are the operations and the incoming edges signify the data that flows between the operations. Thus, ‘execution’ in the dataflow programming paradigm can be seen as the evaluation of a directed graph when data flows through it.
A numerical computation library built using the dataflow paradigm, where the data that flows between the operations are tensors.
The above image (credit) is an animation that depicts the flow of input tensors through the dataflow graph. It also showcases how parallelism comes as an inbuilt feature with the dataflow paradigm.
The TensorFlow Workflow
From a design perspective, a TensorFlow program has two separate concerns: Graph Building, and Evaluation of the graph.
The two processes of building the graph and its evaluation, separates the concepts of computation (higher-level) and execution (lower-level). This separation allow the execution of the same graph on multiple devices with little to no change.
Building a dataflow graph involves defining all the inputs and operations in your desired learning network. Usually the graph has three identifiable components: input, inference, and training.
Evaluation of the Graph
The graph is loaded on a TensorFlow runtime session by specifying an operation to be evaluated. The session looks at the graph and runs the dependent operations (the incoming edges) that are needed for the evaluation of the requested operation. The runtime converts the necessary graph components into parallelizable kernels, and is then sent to the chosen device(s) (CPU, GPU, Android, etc.) for execution.
The name TensorFlow comes from two concepts: the usage of tensors to hold input data, and the dataflow programming paradigm. In the TensorFlow library, a learning algorithm is represented as a dataflow graph, and the input data as tensors that flow through this graph. Hence the name TensorFlow.