What is Transfer Learning?
Transfer learning is a research problem in machine learning that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem. For example, knowledge gained while learning to recognize cars could apply when trying to recognize trucks. This area of research bears some relation to the long history of psychological literature on transfer of learning, although formal ties between the two fields are limited.
It is a deep learning technique that enables developers to harness a neural network used for one task and apply it to another domain. A blog post on NVIDIA attempts to explain the process in layman terms with convolutional horse prediction:
- First, you delete what’s known as the “loss output” layer, which is the final layer used to make predictions, and replace it with a new loss output layer for horse prediction. This loss output layer is a fine-tuning node for determining how training penalizes deviations from the labeled data and the predicted output.
- Next, you would take your smaller dataset for horses and train it on the entire 50-layer neural network or the last few layers or just the loss layer alone. By applying these transfer learning techniques, your output on the new CNN will be horse identification.
This can be used for image recognition, but it can also be used for sound. However, you’ll need a lot of data, sometimes as much as a million hours of speech.
Transfer learning is useful when you have insufficient data for a new domain you want handled by a neural network and there is a big pre-existing data pool that can be transferred to your problem.
NVIDIA Transfer Learning Toolkit offers GPU-accelerated pre-trained models and functions to fine-tune your model for various domains such as intelligent video analytics and medical imaging.
TensorFlow has a tutorial on how to do this for beginners in Keras and those that have a machine learning background.
In an energy efficiency perspectivea blog post from IBM argues: ”Transfer learning is an opportunistic way of reducing machine learning model training to be a better steward of our resources.”
I stil have not made up my mind, but transfer learning is a topic that I will have to pursue further.