AI in innovation research: an overview of transformers

Abstract

Patent text documents are a valuable and increasingly important data source for researchers in the field of innovation. Recent advances in natural language processing—particularly those centred on Transformers—have opened radically new opportunities for extracting information from patent text. Transformers, large language models (LLMs) built on deep learning architectures, rely on key underlying components such as attention mechanisms and word embeddings that enable a semantic understanding that is unparalleled compared to traditional approaches to text analysis. As such, Transformers represent a fundamental leap in how innovation researchers can extract meaning from patent documents. In this paper, we bridge a technical and applied perspective by unpacking the core components of Transformers, drawing on essential concepts from machine learning and linguistic theory. We then illustrate how Transformers can be leveraged in patent research, highlighting several potential applications, with a focus on measuring technological novelty. To ground our discussion, we present some exploratory analyses to demonstrate how Transformers can be used in practice.