Introduction

Over the past few years, the predominant focus of artificial intelligence (AI) and machine learning (ML) research has been on technical and theoretical aspects. With the explosion in the volume of data, especially unstructured data, coupled with exponential increase in processing power, businesses are now aggressively looking for ways to gain competitive advantage by deploying these technologies (Davenport & Ronanki, 2018) to develop new products, serve their customers, market research, process automation, etc. (Duan et al., 2019). Practitioners and researchers alike suggest that Artificial Intelligence and Machine Learning (AIML) will provide the next frontier for both productivity and competition (Dwivedi et al., 2021). Some have even gone to claim that this is a revolution that will fundamentally transform how the industry itself operates (A?gerfalk, 2020). Firms are now beginning to figure out how investment in AIML can generate business value. There is a surge in companies venturing in the AIML world (Ransbotham et al., 2018) and investing in integrating AIML into the various aspects of the business.

Large volumes of unstructured data in the form of text, audio and images are an important by-product of the digital age. Posts on Facebook and Twitter and blogs provide deep insights into consumer behaviour and presents unprecedented potential opportunities to companies as well as academic researchers.

Existing models for deep learning, such as Convolutional Neural Networks (CNN) and Recurring Neural Networks (RNN), demonstrate issues with modelling of long-term contexts as they learn within a specific context which leads to a locality bias (Qiu et al., 2020). Also, since RNNs process inputs sequentially, that is, one word after the next, it makes limited use of parallel processing. A deep learning model, referred to as transformers, proposed by Vaswani et al. (2017), was designed to overcome these challenges. The model utilises ‘self-attention’ in learning, leading to self-supervised learning by the models. In essence, the model architecture design in self-attention utilises more parallel learning as compared to RNNs. The self-attention model can also take into account the long-term contexts. This is because every permutation and combination of all meaningful words, referred to as tokens, from the input sequence are used for learning (Vaswani et al., 2017). The architecture of transformers allows for it to learn even from complex language information. At the same time, it is an expensive as well as time-consuming process for generating humongous volumes of labelled data in the natural language processing (NLP) domain from the easily available unlabelled data.

Recent students suggest that transformer-based pre-trained language models have the potential to overcome the learning challenges of existing models and can likely successfully deliver in a variety of NLP tasks. Generative pre-trained transformer (GPT) and bidierctional encoder representations from transformers (BERT) are frontrunners in the evolution of such self-supervised learning models which have utilised the transformer architecture. As a result, the transformer-based pre-trained language models can potentially utilise self-supervised learning to learn any universal representations from humongous volumes of data. Subsequently, such knowledge can be transferred to downstream tasks. Since the self-supervised learning models provide good background knowledge for downstream tasks, the need to train downstream models from scratch is eliminated. In fact, the downstream models can then be trained with minimal data which provides the right context for the model.

Deep Learning Models

The earlier NLP systems were mainly rule-based and were subsequently replaced by machine learning models. For machine learning models to deliver successfully, domain expertise of the modeller is critical and is, thus, a time- and resource-consuming process. Prominent learning models in recent years are unsupervised (USL), supervised (SL) and self-supervised learning models (SSL).

Supervised learning enables models to learn from data labelled by humans. This supervised learning has been a critical part in the progress that AI has achieved in recent years. Such models work well on specific tasks as they are trained using data labelled for that specific task. A drawback of such supervised training is that to attain good model performance, a large number of labelled data is required. Collecting and labelling such data is an expensive and time-consuming task. Additionally, for many domains, such as medical and legal, it is difficult to access labelled data. Furthermore, since these models only use the supplied labelled data for training, they are likely to suffer from a bias as there may be many other relationships that exist in reality which have not been captured by the labelled data used for training, thus, leading to generalisation error, and the models may probably even accept spurious relationships.

In summary, the drawbacks of supervised learning are that it has significant dependence on data labelled by humans, which is usually time-intensive and cost-ineffective, and that it lacks the ability to generalise the models that emerge from such learning. This can, at times, also lead to spurious relationships. In many areas, such as legal or medical, availability of labelled data could be a challenge, thereby creating a limit on applications for such models in such domains. These models are unable to capitalise on learning for easily available unlabelled data.

As the computer hardware accelerators such as Graphic Processing Units (GPU) and Tensor Processing Units (TPU) along with word association algorithms such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) evolved, the use of deep learning models such as CNN (Kalchbrenner et al., 2014) and RNN (Zhou et al., 2016) for NLP applications increased significantly (Zhou et al., 2016). As stated earlier, the major challenges in training these models were that they had to be trained from scratch (with the exception of word embeddings) and required large volumes of the labelled data for training, and such labelled data is expensive to generate.

Self-Supervised Learning and Transformers: A Game Changer?

Self-Supervised Learning

Self-supervised learning (SSL), a newer learning paradigm, has attracted significant attention from the AI research community. Its biggest advantage is its ability to learn from unlabelled data for gaining universal knowledge about language, speech or images when fed into pre-trained models.

For SSL to work effectively, first the system is pre-trained with a humongous volume of data. Such pre-training can involve one or more pre-training tasks. Since the pretraining is achieved with extremely large volumes of data, the model learns the universal language relationships for both information: syntactic and semantic. When performing downstream tasks with these generalised models, these representations enable improved performance as they now require only a few instances of label data. In simpler terms, with the pre-training with large volumes of unlabelled data, the model learns fundamental common sense and basic knowledge. Now for specific tasks, these models require minimal labelled data to train.

While supervised learning has been a revolutionary step in the development of deep learning models over the last couple of decades, its drawbacks have pushed the research agenda towards developing alternate paradigms for learning. SSL is one such new learning paradigm. The biggest advantage of SSL is that it requires any human labelled data, and rather it can learn from humongous volumes of unlabelled data without any human supervision. The architecture that enables such learning was proposed by Vaswani et al. (2017) and has gained significant traction since then. Vaswani et al. (2017) named this architecture ‘Transformers’. Since SSL is both data efficient and has the ability to generalise, it finds a variety of applications in the deep learning fields of NLP (Liu et al., 2020; Qiu et al., 2020), robotics (Liu et al., 2020), computer vision (Han et al., 2020; Khan et al., 2021) and speech (Baevski et al., 2020; Sivaraman & Kim, 2020).

SSL can be viewed as a hybrid of both supervised and unsupervised learning and some points of similarity and dissimilarity with both. As in unsupervised learning, SSL does not require labelled data. The primary objective of USL is to identify hidden patterns in the data, in contrast, the value of SSL is to identify and model meaningful relationships. In comparison to SL, SSL is similar to SL as they both require supervision and are different as SSL is able to generate labels without human involvement and while the goal of SL is to model a specific objective, SSL’s aim is to train the model with a very large set of general knowledge. Thus, SSL uses the universal knowledge as background because of which downstream learning is possible with minimal training. Since the SSL models are trained on humongous amounts of freely available unlabelled data, such learning is more generalizable as compared to USL and SL which are useful for the specific objectives they were built for. As a result of these features, models such as GPT-1, BERT (Devlin et al., 2018), XLNet (Yang et al., 2020), Roberta, T5, ELECTRA, BART, ALBERT and PEGAUSUS have demonstrated encouraging success with learning using NLP and transferring this knowledge (also referred to as transfer knowledge) to downstream learning and models. Transformer-based architecture, a more recent development for deep learning, has made such pre-training of models with these humongous amounts of data possible.

Transformers

For many business applications, we can utilise deep-learning models that are pre-trained. This eliminates the intensive task of developing and training complex deep learning models. Rather we can use a model that is proven to be optimised and work well. This also eliminates the task of collecting and labelling large amounts of data. This can conserve resources that can be deferred to other core aspects of the business application.

Typically, when we want to utilise deep learning models that are pre-trained, we need to find a model that is trained for our specific task. Besides this, a pre-trained model can be fine-tuned for our specific task. This still requires us to collect and label considerable data. In few-shot learning, we only need to provide a few example cases as input along with our query to the model. With this, the same model can be used for a variety of tasks without requiring any extra engineering. Extremely large transformer models have shown great success in few-shot learning. Transformers are able to perform few-shot learning by having extremely large number of parameters and training data. Since transformers can utilise parallel computing or hardware acceleration, they can achieve a scale large enough for few-shot learning that previous sequential models like long short-term memory (LSTM) could never achieve.

GPT-3 was one of the first popular and mainstream success for its breakthrough application with few-shot learning. GPT-3 can achieve few-shot learning due to its 175 billion trainable parameters being trained on 45TB of unlabelled text data collected partly from a data set of eight years of web crawling.

GPT-3 can be used for many breakthrough applications and produce astonishing results. There have been many applications that were built using GPT-3. For example, one application was a Figma plugin. In this, by just giving the text description of your desired application, a prototype can be generated within a matter of seconds. If one gave the description of an Instagram-like application, then the plugin will generate an appropriate prototype that is comparable to Instagram. Another application was a ReactJS application generator. In this, by giving the text description of a web app, the application would generate ReactJS code for it. When the text description of a to-do app was given to it, a code for the functioning of the to-do app was generated.

Transformers can enable businesses to use pre-trained models like GPT-3 to build many such breakthrough applications to complement their existing or create new products/services.

Transformer Network Architecture

Transformer networks contain a sequence encoder which takes the input sequence and encodes it to a sequence of vectors. The encoder of the transformer network uses a self-attention module which helps it decide which parts of the input are important. It is used to ignore the unrelated tokens for the given token in the sequence and focus on the related tokens. For example, in the sentence ‘The car is red’, the model might assign high attention to the token ‘red’ for the token ‘car’. For every token in the input sequence, an attention score is computed with respect to every other token in the sequence. This creates an attention matrix of size N × N, where N is the sequence length. This attention matrix could be a memory bottleneck if the sequence length is large. The encoder also uses positional embeddings which helps the network (Figure 1).

Figure 1. Architecture of the Transformer Network

Source: Vaswani et al. (2017).

Real-World Applications

Deep learning is widely used for several applications in various domains such as NLP, computer vision, video processing, audio processing, etc. There are several supervised and unsupervised deep learning techniques which are used in a wide variety of real-world applications. The supervised techniques require several labelled data points which are used to train a model. For example, to build an application to recognise objects present in an image, we would need several images with ground truth labels of the objects present. It is said that adding more training data usually gives us a higher prediction accuracy. That is not necessarily true, because the accuracy might start to saturate after a point. That is why it is beneficial to use models which are more scalable, such as the transformer network (Vaswani et al., 2017).

Transformer networks (Vaswani et al., 2017) were proposed for sequence-based tasks such as text processing. The authors proposed transformers for sequence-to-sequence applications and showed the efficacy of transformer networks on machine translation. The authors showed that transformer networks outperform other sequence processing techniques such as LSTM networks (Sundermeyer et al., 2012).

Transformer networks have been shown to be more scalable compared to LSTM networks, that is, they can generalise better on larger data sets. LSTM networks process each token of the sequence sequentially, whereas transformer networks process them in parallel; hence, they can better exploit hardware accelerators such as GPU or TPU as it allows parallel processing. Recent models like GPT-3 (Brown et al., 2020), which use transformer networks, are trained on several machines connected through a network. For NLP applications, transformer networks can be first pre-trained for the task of next token prediction, where the network has to predict the next word in the sentence. This unsupervised pre-training is very useful because getting large amounts of labelled data is very expensive.

Unlike LSTM networks or CNNs, transformers do not make any prior assumptions about the input data, which helps them generalise better with large amounts of training data. Although models which do make assumptions about the structure of the data might perform better with smaller data sets.

Transformer Networks for Text-Based Applications

Transformer networks have been shown to be effective in several applications involving text data such as machine translation, question answering, named entity recognition, text summarisation, etc. In the application of machine translation, the model takes a sentence in language A and translates it to language B. This is usually done by training a sequence-to-sequence model on a parallel corpus having sentences of both language A and B. The application of the named entity recognition is to find named entities such as names of people, places, etc., in a sentence. This could be useful for analysing relationships between several entities from text data. Automatic question answering is very useful for chatbots, search engines and customer support. Here, the models take a query as input and select an appropriate response from a database of documents. The response is then re-worded to match the grammar according to the question. This uses a ranking model which ranks the responses by assigning similarity scores to all the responses from the database. Several models such as BERT (Devlin et al., 2018), ET (So et al., 2019) and transformer extra long (Transformer-XL) (Dai et al., 2019) have shown good performance for text-based applications.

Transformers can enable businesses to build highly accurate chatbots that are practically indiscernible from real human beings. They can also be used to summarise large texts accurately or make documents with legal jargon more comprehensible.

Transformer Networks for Computer Vision Applications

Transformer networks have shown good efficacy for image classification, object detection, semantic segmentation and image generation. Transformer networks can be used to replace the convolution operations in a convolutional neural network, or they can be used in conjunction with convolutional operations. The self-attention in transformer networks allows it to learn high-level information in the image. VIT (Dosovitskiy et al., 2020) splits the image into multiple patches and applies linear projection on each patch before passing it to the transformer module (Figure 2). The authors of VIT show that their method is superior to several CNN-based image classification models.

Figure 2. Architecture of the VIT Network

Source: Dosovitskiy et al. (2020).

DETR (Zhu et al., 2020) is a transformer-based end-to-end network to detect the classes and location of multiple objects in an image. VideoBERT (Sun et al., 2019) is a transformer-based model which learns a joint representation of text and videos, which can be used to generate captions from videos. The authors trained VideoBERT on data from YouTube. DALL-E (Ramesh et al., 2021) and CogView (Ding et al., 2021) are transformer-based models which are used to generate images by using a text description as input.

Using the idea of transfer learning, computer vision researchers trained large CNN models (He et al., 2016; Simonyan & Zisserman, 2014; Szegedy et al., 2016; Tan & Le, 2019) with the use of large data sets (e.g., ImageNet [Krizhevsky et al., 2012; Russakovsky et al., 2015]). Such models learn the common relationships from these images. Subsequently, these models pre-trained with large volumes of data are fine-tuned to downstream tasks with a very small data set specific to the task at hand (Kaur & Gandhi, 2019). Such models have demonstrated significant success in computer vision tasks (He et al., 2016; Ren et al., 2015).

Researchers from the area of NLP are optimistic with such pre-trained models and are increasingly combining the power of SSL and transformers in models such as GPT-3 (Brown et al., 2020), PANGU-(200B) (Zeng et al., 2021) and GShard (600B) (Lepikhin et al., 2020) which are trained using billions of parameters and use trillions of parameters for their switch-transformers (Fedus et al., 2021).

Transformer Networks for Audio Applications

Transformer networks have been used to encode audio signals for solving audio-based applications. Audio speech transformer (Dong et al., 2018) splits the audio signals into chunks and applies a linear projection on each chunk to get a sequence of embeddings. The sequence is passed to a transformer network, which is then used to perform classification of audio clips. Streaming transformer (Moritz et al., 2020) is another transformer-based model which takes a chunk-wise stream of audio and performs real-time speech recognition. Music transformer (Huang et al., 2018) is a generative network which is used to generate music with long-term structure.

Audio chatbots can be built using transformers, which will be able to have normal human such as interactions and conversations. These bots can be used in place of human customer service agents and continue to provide comparable service.

Discussion and Conclusion

Supervised- and unsupervised-based learning models have found increasing use in the real-world applications. Such models have been found to be prone to and amplifying the biases that exist in the data sets used for training. Therefore, the decisions based on these models may end up being unintentionally biased. By using transformer-based pre-trained learning models, many of these biases are expected to reduce. A limitation of such transformer-based pre-trained models is that they are expensive to train. But with increasing processing power of computers, these costs are likely to come down.

Subsequent to the success with the transformer-based pre-trained learning models in the general domain of English, such architecture and learning is also being used in various other domains such as legal (Articles 32 and 33), programming (Ahmad et al., 2021; Feng et al., 2020; Guo et al., 2020; Lu et al., 2021; Phan et al., 2021), finance (Yang et al., 2020), news (Gururangan et al., 2020), networking (Louis, 2020), biomedical (Alsentzer et al., 2019; Gu et al., 2021; Lee et al., 2020; Peng et al., 2019, 2021), dialogue (Wu et al., 2020) and academics (Beltagy et al., 2019; Liu et al., 2021; Peng et al., 2021).

This article discusses several learning methods using transformer networks and provides their real-world business applications. Transformer networks show great potential due to their efficacy and scalability. While this article presents several transformer-based applications in the domains, such as text processing, image processing and audio processing, it has shown promising results in multiple domains such as legal, academics, finance and news. Further research in business applications of these models will help the business community adopt these for meaningful decision-making.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The authors received no financial support for the research, authorship and/or publication of this article.