Google’s Bidirectional Encoder Representations from Transformers (BERT) is a large-scale pre-trained autoencoding language model developed in 2018. Its development has been described as the NLP community’s "ImageNet moment", largely because of how adept BERT is at performing downstream NLP language understanding tasks with very little backpropagation and fine-tuning needed (usually only 2–4 epochs).
For context, traditional word embeddings (e.g. word2vec and GloVe) are _non-contextual. T_hey __ represent each word token with a single static vector, and learn by word co-occurence instead of the sequential context of words. This may be problematic when words are polysemous (i.e. where the same word has multiple different meanings), which is very common in law.
For instance, a single static context-free vector will be used to represent the word "lemon" in the sentences "the lemon is sour" and "that car is a lemon" (i.e. an interesting reference to lemon laws, which protects consumers when new vehicles turn out to be defective).
In contrast, BERT is a contextual model which generates context-specific representations based on the words surrounding each token. Interestingly, Ethayarajh (2019) shows that instead of creating one representation of "lemon" per word sense (left option), BERT will create infinitely many representations of "lemon", each highly specific to its context (right option). This means that instead of generating a single dense vector to represent the token "lemon", the token is dynamically represented by "the __ is sour" and "that car is a _____".
BERT’s ability to capture linguistic polysemy is particularly relevant in the legal domain, where polysemy abounds. For instance, the word "consideration" in law represents the idea of reciprocity in contractual relationships, and has a different meaning from the word’s usual connotation of being considerate. Furthermore, the same term can have multiple definitions in statute and case law. For instance, the term "worker" has four different definitions in EU law, and can be used to connote different kinds of workers, even in the same document. The development of contextual language models is hence significant for the legal AI landscape.
The technicalities of the BERT architecture are beyond the scope of this article (and has been extensively written about). Nevertheless, it is worth noting that one of BERT’s key innovations is its bidirectionality, i.e. it overcomes the traditional problems of sequential text parsing. Notably, bidirectionality in this context does not mean bidirectional in a sequential sense (i.e. parsing from both left-to-right and right-to left at the same time) but bidirectional in a simultaneous sense (i.e. it learns information from both the left and right contexts of a token in all layers at the same time). This is done by using methods like Masked LMs (MLMs) and Next Sentence Prediction (NSP) to achieve better contextual word embedding results.
Advantages of Domain-Specific BERTs
While BERT has been very effective in performing general language representation tasks, a problem is that – being trained on unlabelled generic Wikipedia and open-source articles – it lacks domain-specific knowledge. This is an important caveat, as a language model can only be as good as its corpus. As an analogy, applying a vanilla BERT model to legal domain-specific issues might be equivalent to asking a liberal arts student to approach a legal problem, rather than a law student trained on years of legal material. This is problematic since there is a lot of disconnect between the language as found in general open-source corpora (e.g. Wikipedia and news articles) and legal language, which may be esoteric and Latin-based.
While there has not yet been a legal domain-specific BERT model developed at the time of writing, examples of other domain-specific BERTs that have been developed include BioBERT (biomedical sciences), SciBERT (scientific publications), FinBERT (financial communciations), and ClinicalBERT (clinical notes). Although they all share the similarity of being domain-specific BERTs, their architecture exhibits many key differences.
Training Legal Domain-Specific BERTs
I will explore some of the approaches in training domain-specific BERTs:
Completely Pre-training BERT
This involves completely re-doing BERT’s pre-training process with large-scale unlabelled legal corpora (e.g. statutes, precedents). In this sense, the domain-specific knowledge would be injected during the pre-training process. This was the approach taken for the SCIVOCAB SciBERT.
While this does significantly enhance BERT’s performance on domain-specific tasks, the key problem is that the time, costs, and quantity of data required for complete re-training might simply be too massive to be feasible. BERT-Base itself is a neural network with 12 stacked layers and approximately 110 million weights/parameters, and its pre-training corpora required 3.3 billion tokens. Re-training a domain-specific BERT would require a similar amount of training data (e.g. SciBERT used 3.17 billion tokens). It would require a week or more to train the model (e.g.the SCIVOCAB SciBERTs took 1 week to train on a v3 TPU) and be extremely costly, which is simply not feasible for the average enterprise.
Further Pre-Training BERT
As such, a compromise might be to not completely re-train BERT, but instead initialise weights from BERT and pre-train it further with legal domain-specific data. This has been shown to enhance BERT’s performance, and Sun (2019) shows that models that were further pre-trained performed better across all seven datasets than the original BERT-Base model.
This seems to be the most popular method, and was used for BioBERT, the BASEVOCAB SciBERT, and FinBERT. Instead of completely re-training from scratch, researchers initialised the new BERT model with learned weights from BERT-Base, then trained it on domain-specific texts (e.g. PubMed abstracts and PMC full-text articles for BioBERT). Lee et al. (2020) reported that BioBERT achieved higher F1, precision, and recall scores than BERT on all datasets. Similarly, Beltagy et al. (2019) reported that SciBert outperformed on biomedical, computer science, and multi-domain tasks.
Fine-Tuning BERT
Another, even simpler, option is to use the pre-trained BERT but fine-tune it with legal domain-specific corpora in downstream NLP tasks. The concept of fine-tuning comes from the domain of transfer learning, which – in simple terms – means taking a model that was aimed to solve task x, and repurposing it to solve task y. In practice, it usually means taking the pre-trained BERT model and adding an additional output layer of untrained neurons at the end, which is then trained using labelled legal corpora (fine-tuning is usually a supervised task). After being fine-tuned on domain-specific data, the resulting model will have updated weights that more closely resembles the characteristics and vocabulary of the target domain
The advantage of this approach is that it requires much less data and is significantly faster and cheaper to train. Given that BERT-Base’s pre-training has already encoded knowledge of most words in the general domain, it only requires a few tweaks to adapt it to function in a legal context. Instead of a timespan of weeks, fine-tuning usually requires just 2–4 epochs (a matter of minutes). This approach also requires less data (although it usually requires labelled data), which makes it more feasible.
Furthermore, if desired, fine-tuning can be done in conjunction with domain-specific pre-training, i.e. they are not mutually exclusive methods. For instance, BioBERT was first pre-trained with biomedical domain-specific corpora and then fine-tuned on biomedical text mining tasks like named entity recognition, relation extraction, and QA.
Conclusion
With the recent hype around mega-scale models like OpenAI’s GPT-3, the initial excitement surrounding BERT seems to have somewhat waned. Nevertheless, it bears remembering that BERT is still an incredibly powerful and agile model that arguably sees more practical application. As seen from the experience of domain-specific BERTs like SciBERT and BioBERT, fine-tuning and/or pre-training BERT on domain-specific corpora is likely to improve performance with regard to legal NLP tasks. To this end, depending on the researcher’s time and monetary resources, there are various ways to improve and tailor BERT’s performance.
