1. What is Generative Artificial Intelligence?
The human language system contains complex and subtle patterns that were historically very difficult to discover. This is because, in human language, the number of possible combinations of words and phrases is vast, and there is a lack of obvious discernible patterns. Natural Language Processing (NLP), a key branch of artificial intelligence, aims to enable machines to understand, generate, and respond to human language. However, for a long time, its development faced numerous difficulties and progressed slowly.
The limitations in NLP development were mainly due to the following reasons: First, insufficient data. Although we have massive amounts of text data, the quantity of annotated datasets is relatively limited, and creating annotated datasets requires significant human effort and time, becoming a major bottleneck for NLP development. Second, model complexity. NLP models usually have complex structures, requiring substantial computational resources and time for training and optimization. For example, the GPT-3 model has 175 billion parameters (note: original text incorrectly states 17.5 million, corrected based on common knowledge) and requires extensive GPU computing resources for training. Third, task diversity. NLP tasks are rich and varied, such as text classification, sentiment analysis, machine translation, and question answering. Each task requires specific model structures and training methods, and designing and optimizing models for specific tasks demands considerable time and experience. Fourth, the complexity of semantic understanding. Semantics in human language are rich and ambiguous; the same word can have different meanings in different contexts. Enabling machines to understand these semantics is a daunting task. Fifth, lack of standardization and openness. The NLP field lacks unified standards and specifications, and interoperability and compatibility between different libraries and frameworks are poor. This makes migrating models and code between different frameworks difficult, and there is also a lack of unified evaluation standards, making it hard to compare the performance of different methods. Sixth, technical bottlenecks. Although deep learning has made significant progress in the NLP field, it still faces challenges such as improving model generalization, handling long-range dependencies, and generating high-quality text.
Taking machine translation as an example, when we use various translation tools to translate English into Chinese, we often get rather stiff and inaccurate results. Many times, the translation is done word by word, failing to accurately convey the semantics and context of the original text. This fully reflects the difficulties NLP faced in the past.
However, 2017 was a landmark year for the field of Natural Language Processing. Eight engineers from Google, in their paper "Attention Is All You Need," proposed a model called Transformer, the appearance of which completely changed the landscape of NLP. The core idea of the Transformer model is the "Self-Attention Mechanism," also known as "self-attention" or simply "attention." This mechanism allows the model, when processing a word or phrase, to simultaneously consider information from other related words or phrases, enabling it to find global information in the input sequence and adaptively adjust the representation of each position. Specifically, the self-attention mechanism calculates the attention weights for each position in three steps: first, it calculates the relevance score between the current position and all other positions in the sequence, usually using dot product, scaled dot product, etc.; next, it uses the relevance scores to calculate attention weights, employing the Softmax function to normalize the relevance scores into a probability distribution; finally, it performs a weighted sum of the representations of all positions in the sequence based on the attention weights to obtain the output representation for the current position. This mechanism enables the model to better understand the context of the language, thereby translating or generating text more accurately.
The Transformer model has numerous significant advantages. Compared to traditional sequence-to-sequence (Seq2Seq) models that typically use Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs), RNNs capture dependencies in a sequence by processing each element sequentially. However, due to their sequential nature, they are difficult to parallelize and are prone to gradient vanishing or explosion problems when processing long sequences. CNNs process multiple positions in a sequence in parallel through convolutional operations, but their computational complexity is still related to the sequence length, and multiple convolutional layers are needed to capture long-range dependencies. The Transformer model, by utilizing the self-attention mechanism instead of traditional recurrent networks, allows the model to disregard the distance of dependencies in the input or output sequences, overcoming the limitations of traditional recurrent models. This achieves higher efficiency and better performance, which is particularly important for tasks requiring the processing of large amounts of data and complex pattern recognition, such as text generation and machine translation in NLP.
After the Transformer model was proposed, models gradually gained the ability to predict the next word. During the training process, substantial human intervention is required to ensure better "prediction" performance. Human intervention primarily manifests in several key areas: First, data cleaning and preprocessing. By cleaning and preprocessing raw data, noise can be removed, missing values handled, and data converted into a format suitable for model training. For example, for text data, operations like tokenization, stop-word removal, and stemming can be performed to reduce data dimensionality and noise. Second, feature engineering. Selecting and extracting appropriate features can improve model performance. Third, annotation and error correction. Annotating data provides supervised information for the model, while correcting the model's output helps it learn better. Fourth, the application of active learning and semi-supervised learning. By actively selecting valuable data for annotation and utilizing unlabeled data for learning, model efficiency and performance can be improved.
Companies like OpenAI, based on the Transformer model, collected vast amounts of text data to train this model. When trained to a certain extent, this model, known as the Generative Pre-trained Transformer (GPT), showed significant changes. Its response quality improved dramatically, providing replies very close to human ones, and even exhibiting the ability to reason about complex problems—a phenomenon referred to as "emergent abilities."
In 2018, OpenAI launched its first GPT model, named GPT-1. Since then, the GPT series has continued to evolve, with the latest GPT model being GPT-4, released in early 2023. In May 2024, OpenAI announced the launch of the multilingual and multimodal GPT-4o, capable of processing audio, visual, and text input in real-time. Based on the Transformer architecture, GPT models possess powerful text generation and language understanding capabilities. Through pre-training on massive amounts of text data, they learn the patterns and rules of language, enabling them to generate fluent, coherent, and diverse text. Its core components include Multi-head Self-Attention and Position Encoding. The self-attention mechanism allows the model to understand information from different positions in a sequence, enhancing generation capabilities; position encoding addresses the issue of long text output by providing contextual information to the model.
On November 30, 2022, OpenAI provided ordinary users with a chat interface, allowing them to experience GPT's capabilities, marking the birth of ChatGPT. Once launched, ChatGPT quickly became popular worldwide, gaining over 1 million users within just one week of its release. By the end of January 2023, just two months after its launch, its monthly active users had exceeded 100 million, making it the fastest-growing consumer application in history. It not only excels at analytical or mechanical cognitive computations but is also adept at creating or generating entirely new, meaningful, and even aesthetically pleasing content, such as writing poetry, designing products, creating games, and writing program code. The emergence of ChatGPT signifies that artificial intelligence technology has entered a new era, and it will profoundly impact the future transformation of the entire economy and society. It has driven disruptive changes in a series of models, including content generation, knowledge creation, and information distribution and acquisition. With its extremely simple natural language interaction method, it addresses numerous user pain points, further opening up new spaces and imaginative possibilities for the implementation of AI business models and the digital and intelligent transformation of the knowledge service industry.