PyTorch: Artificial Intelligence Explained
Contents
PyTorch is an open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing, primarily developed by Facebook's AI Research lab (FAIR). It is free and open-source software released under the Modified BSD license. Although the Python interface is more polished and the primary focus of development, PyTorch also has a C++ interface.
PyTorch provides two high-level features: Tensor computing (like NumPy) with strong acceleration via graphics processing units (GPU), and Deep Neural Networks built on a tape-based autograd system. You can reuse your favorite Python packages such as NumPy, SciPy, and Cython to extend PyTorch when needed.
Understanding PyTorch
At its core, PyTorch provides a few key features. A multidimensional array, called a tensor, is a key data structure in PyTorch. Tensors are similar to NumPy’s ndarrays, with the addition being that Tensors can also be used on a GPU to accelerate computing. PyTorch supports a variety of tensor types, such as FloatTensor, DoubleTensor, and LongTensor.
PyTorch implements a version of dynamic computational graph called a tape-based computational graph. This means that as you execute operations, a graph is built, and the operations are recorded on this graph. This is particularly useful for backpropagation, as the graph can be processed in the reverse order for calculating gradients.
Dynamic vs Static Graphs
There are two types of computational graphs in the world of deep learning libraries. One is the static computational graph, used by libraries like Theano, TensorFlow, and CNTK. The other is the dynamic computational graph, used by libraries like PyTorch, Chainer, and DyNet.
Static computational graphs are nice because you can optimize the graph up front. For instance, you can see that you have multiple GEMM operations in a row, so you decide to merge them into a single, larger GEMM operation for efficiency. However, the downside is that the graph is static, so you can’t change it on the fly. This is a problem for certain models that require dynamic graphs, such as recursive neural networks and neural Turing machines.
Autograd
Autograd is the automatic differentiation system of PyTorch. With autograd, you can automatically compute the gradient or derivative of any function. This is especially useful for backpropagation in neural networks, where you need to update the parameters of your model based on the computed gradients.
In PyTorch, you can mark a tensor to be tracked by autograd by setting its attribute .requires_grad to True. From then on, any operation on this tensor will be tracked and included in the computation graph. To compute the gradients, you call the .backward() function on the output tensor, and the gradients are computed and stored in the .grad attribute of each input tensor.
Building Neural Networks with PyTorch
PyTorch provides a module nn that makes building networks much simpler. Here’s how to build a simple neural network with PyTorch.
First, you define a class that extends nn.Module. This is the base class for all neural network modules, which includes layers. Your models should also subclass this class. Then, you define the layers as class attributes in the constructor. PyTorch will automatically create the necessary parameters, which are tensors. Parameters are a special kind of tensor that are automatically added to the list of your model’s parameters when they’re assigned as an attribute of a Module.
Forward Propagation
Next, you define the forward function. This is where you specify how your model is going to be run, from input to output. This is where the actual computation happens. Note that you don’t have to specify the backward function because it’s automatically defined for you using autograd.
Finally, you can create an instance of your model and call it like a function. Here’s an example of a simple feedforward neural network that classifies digit images. It’s a simple, linearly stacked network of layers, commonly known as a fully connected network or a multilayer perceptron.
Loss Functions
A loss function takes the (output, target) pair of inputs, and computes a value that estimates how far away the output is from the target. There are several different loss functions under the nn package. A simple loss is nn.MSELoss which computes the mean-squared error between the input and the target.
For instance, if you’re doing multi-class classification, you might want to use something like Cross Entropy Loss (nn.CrossEntropyLoss), which computes the cross entropy between the output and the target. If you’re doing binary classification, you might want to use something like Binary Cross Entropy Loss (nn.BCELoss).
Training Neural Networks with PyTorch
Once you have your model built, you need to train it. There are many ways to do so in PyTorch, and the library provides a lot of flexibility in this regard. Here’s a basic outline of what you might want to do.
First, you need to define a loss function. This is what you want to minimize during training. PyTorch provides many standard loss functions in the torch.nn module. For instance, if you’re doing binary classification, you might want to use Binary Cross Entropy (nn.BCELoss).
Optimizers
Next, you need to define an optimizer. This is what will update the parameters of your model based on the gradients computed during backpropagation. PyTorch provides many standard optimizers in the torch.optim module, such as Stochastic Gradient Descent (SGD), RMSProp, and Adam.
Then, you need to define a training loop. This is where the actual training happens. In each iteration of the loop, you do the following: Forward Propagation, Loss Computation, Backward Propagation, and Parameter Update.
Model Evaluation
After training your model, you need to evaluate it. This is where you see how well your model performs on a separate test dataset. You do this by running your model on the test data and computing the loss and other metrics (such as accuracy, precision, recall, F1 score, etc.). Then, you can compare these metrics to those of your training set to see if your model is overfitting, underfitting, or just right.
Finally, you might want to save your trained model for future use. PyTorch provides a simple function to save your model, torch.save(). You can also load a saved model with torch.load(). This is useful for inference, when you want to use a pre-trained model without having to retrain it.
Advanced Features of PyTorch
PyTorch provides a lot of advanced features and utilities for deep learning research and development. Some of these are: TorchScript, a way to serialize your models for production; a multiprocessor package torch.multiprocessing to leverage multiple CPUs and GPUs; and torch.distributed, a package that supports distributed computing.
Another advanced feature of PyTorch is its support for custom and complex architectures. Unlike some other libraries, PyTorch doesn’t force you to conform to a specific template or structure for your models. You can build your models however you want, using any Python code. This is particularly useful for research, where you might be experimenting with new and different architectures.
Custom Layers
PyTorch provides a lot of pre-defined layers that you can use in your models. However, sometimes you might need to define your own custom layers. This is easy to do in PyTorch. You just need to define a class that extends nn.Module and implement the forward function.
The forward function is where you define how your layer transforms its input into output. This could be as simple as applying a mathematical operation, or as complex as running a whole sub-network. You can use any Python code in the forward function.
Onnx Support
ONNX is an open format to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools and choose the combination that is best for them. PyTorch provides support for ONNX out of the box.
Once you have your model trained in PyTorch, you can export it in ONNX format. This allows you to run your model on a variety of platforms and devices. You can also import ONNX models into PyTorch, which allows you to use models trained in other libraries in PyTorch.
Conclusion
PyTorch is a powerful, flexible, and intuitive deep learning library. It provides a wide range of functionalities, from basic tensor operations and automatic differentiation, to high-level layers and training utilities. Its dynamic computational graph and Pythonic design make it a great tool for both research and production.
Whether you're a researcher pushing the boundaries of machine learning, a data scientist building a production model, or a hobbyist experimenting with deep learning, PyTorch has something to offer you. With its active community and strong support from its parent company, Facebook, PyTorch is a library worth considering for any deep learning task.
Looking for software development services?
-
Web development services. We design and build industry-leading web-based products that bring value to your customers, delivered with compelling UX.
-
Mobile App Development Services. We develop cutting-edge mobile applications across all platforms.
-
Artificial Intelligence. Reshape your business horizon with AI solutions