Dialogue丨Bowen Zhou: How does the general large-scale model break through vertical industry scenarios?

Author: Wang Yongang Editor: Dong Zibo

**Source:**AI Technology Review

Image source: Generated by Unbounded AI tool

After leaving JD.com, Zhou Bowen hasn't been this excited for a long time.

ChatGPT turned out to stir up the world, like a spring thunder awakened practitioners from all walks of life, so that they all heard the footsteps of AGI entering reality.

Under the upsurge, people saw Wang Huiwen and Wang Xiaochuan start their own businesses, and also saw Baidu and Alibaba dominating the market. Zhou Bowen, as the former dean of the AI Research Institute of IBM and JD.com, has studied the basic theory of artificial intelligence and core cutting-edge technologies, applications and industrialization for more than 25 years. He founded Lianyuan Technology as early as the end of 2021. Model, with generative AI, multi-round dialogue and human-computer collaboration technology, helps enterprises and consumers complete product innovation and digital intelligence transformation in the new era of artificial intelligence. "It's not so much that I decided to start a business in this field, it's better to say that this business found me." Zhou Bowen described this as something that must be done, as if a sense of mission was urging him to act.

Zhou Bowen graduated from the University of Science and Technology of China, and then went to the University of Colorado at Boulder to obtain a doctorate. As the former president of IBM Research's US headquarters artificial intelligence basic research institute, he returned to China after presiding over AI-related work for many years, and successively served as senior vice president of JD Group, chairman of the group technical committee, president of JD Cloud and AI, and founding president of JD Artificial Intelligence Research Institute and other positions. As the founder of JD AI, he is responsible for the technical research and business development of JD AI. He established the JD AI Division, AI Research Institute, and JD AI Accelerator from 0 to create a national open platform for intelligent supply chain artificial intelligence, realizing the daily call volume from 0. To tens of billions, led the technical reconstruction of Jingdong artificial intelligence customer service and launched external productization, managed billions of technical service business and thousands of integrated technology, product, marketing and sales teams.

In 2021, Zhou Bowen predicted that generative AI would explode in the near future, so he decided to leave JD.com to found Lianyuan Technology, which is committed to helping vertical enterprises carry out product innovation and digital intelligence transformation with general large-scale model capabilities, and reshape with AI Commodity value; in 2022, he will serve as the Huiyan Chair Professor of Tsinghua University and the tenured professor of the Department of Electronic Engineering, and in May of the same year, he will establish the Collaborative Interactive Intelligence Research Center of Tsinghua University. Coincidentally. **

ChatGPT is about to come, and Zhou Bowen also posted in Moments: "I believe that China's OpenAI needs to explore a new path!" Under the pride, he is eager to seek talents. But different from other entrepreneurs, Zhou Bowen and Lianyuan Technology chose to rely on tens of billions of parameters and unique training methods to make the large model better at understanding the relationship between people and commodities on the basis of general capabilities. Intelligent technology helps companies reconstruct the full-link innovation system from product insight, positioning, design, R&D to marketing.

Zhou Bowen once stated in public that his entrepreneurial direction is to take the lead in integrating artificial intelligence with traditional industries to bring higher value to enterprise digital intelligence innovation, that is, to achieve a breakthrough in the ability of general large models in vertical scenarios.

Recently, a reporter from AI Technology Review had a conversation with Zhou Bowen. The following is the transcript of the conversation. AI Technology Review has edited the content without changing its original meaning:

Let AI learn human wisdom, a new paradigm of interaction and collaboration

**AI Technology Review: ChatGPT has brought this interaction method, what do you think is different from the previous interaction method? **

Zhou Bowen: One of my research directions is the interaction between AI and people, and learning in the interaction. Human-computer interaction is different from human-computer dialogue. Through human-computer interaction, AI can learn things in the process, so this is not a simple task to perform, but a means to achieve learning.

As recorded in "The Analects of Confucius", it is the story of Confucius and his seventy-two disciples learning through interaction. In the West, similar to the Athens Academy of Plato and Aristotle, the inheritance of the oldest knowledge and wisdom is accomplished through dialogue between people, and teachers help students better complete their studies through interaction with students .

For example, if the teacher asks the students to pour a glass of water, it is difficult for such simple "command-execution" actions to increase wisdom; How to overcome difficulties, this is the interaction that can increase wisdom, and it also reflects my core point of view on the collaborative interaction between humans and AI.

The essence of AI is collaboration and interaction with humans. It learns continuously from interactions, and then cooperates with humans to better solve problems. This point of view will become more and more important in the near future, and at the same time, it will face more technical and ethical challenges. In the end, it will not be easy to keep the bottom line. Like the AI Alignment that everyone says, humans can pass their will to AI, and then break down tasks with AI, allowing AI to learn and realize human will in the process. This is a new way of collaboration, that is, collaborative interactive intelligence.

**AI Technology Review: Do you think achieving value alignment through interaction is an effective way for the human brain and GPT to collaborate? How should humans and AI work better together? **

**Zhou Bowen: **After the explosion of generative AI, AI that learns through collaborative interaction with humans will become stronger and stronger.

Daniel Kahneman, winner of the Nobel Prize in Economics in 2002, proposed in his best-selling book Thinking Fast And Slow that there are two modes of human thinking—system 1 and system 2, and system 1 is fast. Thinking, intuitive judgment; System 2 is slow thinking, which requires a lot of reasoning and calculation.

Initially, people thought that AI was more suitable for the work of "system 1", such as face recognition and quality inspection, which were based on the pattern recognition of "system 1". But I insist that the real value of AI lies in 2, which is to help humans better complete complex logical reasoning tasks. The emergence of ChatGPT has verified the feasibility of AI as System 2, which means that AI can discover new knowledge, and the discovery of new knowledge will help humans design better AI, such as brain science and computing optimization Discovery, and a flywheel to create new knowledge emerges. The flywheel effect means that AI can enable the entire system to better discover new knowledge, and this new knowledge can help design better AI systems, thus forming a virtuous circle. Therefore, a mutually reinforcing relationship has been formed between AI, knowledge and innovation, which requires that the way AI and humans collaborate must be transformed.

I have proposed a "3+1" research direction before, that is, to use reliable AI as the research base and long-term goal, to focus on multi-modal representation interaction, human-computer collaborative interpretation, and environmental collaborative evolution. The core is to be human The collaboration and co-creation of machines can realize the goal of human beings helping AI to innovate and AI helping human beings to innovate.

One of them is multimodal representation interaction, where there may be a grand unified theory. In 2022, people are still skeptical about this, but with the advent of GPT-4, this multi-modal unified representation interaction has become more convincing; Another point is human-computer collaborative interaction. People were also skeptical about this in 2022, but now this interaction method has become more credible, and people began to believe that it is likely to happen; The third point is the co-evolution of AI and the environment, This means that AI not only needs to cooperate with humans, but also must adapt to the surrounding environment. We first proposed this concept in early 2022, and so far we have not seen any successful cases in this direction, not even OpenAI.

If you can't learn OpenAI, but you can't do Microsoft, you need to subtract for large-scale domestic business startups

**AI Technology Review: The special feature of the Transformer model is that it uses an attention mechanism (Attention) to model text. We noticed that you have carried out research related to AI attention mechanism very early. **

**Zhou Bowen:**The core highlights of Transformer are self-attention mechanism and multi-head mechanism. In June 2017, "Attention is All You Need" published by Google Brain introduced the concept of self-attention (self-attention) mechanism and Transformer. Later, OpenAI's GPT was also deeply influenced by this paper.

Prior to this, I published the first paper as a corresponding author to introduce a multi-hop self-attention mechanism to improve the encoder - "A Structured Self-Attentive Sentence Embedding". This paper was completed and uploaded to arXiv in 2016, and was officially published in ICLR in early 2017. We are also the first team to propose this mechanism, and more importantly, this is the first natural language representation model that does not consider downstream tasks at all. Everyone has used attention or self-attention in some cases before, but they are all task-dependent.

**AI Technology Review: In this paper, what did you find? How did these discoveries affect the subsequent Transformer technology changes? **

Zhou Bowen: We proposed in the paper that the best representation method is to use structured self-attention to represent natural language (NLP). This paper has been cited more than 2,300 times since its publication.

Prior to this, Ilya Sutskever, chief scientist of OpenAI, believed that the best representation method was "sequence-to-sequence (Seq2Seq)", that is, to train the model to convert the sequence of one domain into the sequence of another domain, such as the corresponding source language in machine translation and the target language; or in question answering, where the question is a sequence and the answer is a sequence. On this basis, the mapping relationship between the two represented by the deep neural network is learned.

But later, the team of deep learning expert and Turing Award winner Yoshua Bengio proposed an "attention mechanism", the core of which is that not all words are equally important when answering questions; By identifying the more critical parts, and then paying more attention to this part, you can give a better answer. This attention model quickly gained very wide acceptance. In 2015, I led the IBM team to start research based on the "Seq2Seq+Attention Mechanism" architecture and ideas at the same time, and successively launched several earliest generative models for AI writing in natural language. Related papers have also been cited more than 3000 times.

But I was not satisfied with the content of the paper at the time, because there was a problem in it, that is, attention was constructed based on the answer. The AI trained in this way is like a student who asks the teacher to mark the key points before the final exam of the university, and then reviews the key points with targeted attention. In this way, although the performance of AI on specific problems can be improved, it is not universal. Therefore, we proposed that it does not depend on the given task and output at all, and only based on the internal structure of the input natural language, through AI multiple readings to learn which parts are more important and the relationship between them, this is self-attention plus Representation Learning for Multi-Head Mechanisms. This kind of learning mechanism only looks at the input, more like students study and understand the course multiple times and systematically before the exam, instead of learning in a targeted and fragmented way based on the key points of the exam, which is closer to the purpose of general artificial intelligence, and greatly Enhanced AI's ability to learn.

**AI Technology Review: What is the relationship between the paper "Attention is All You Need" and you? **

Zhou Bowen: We know that all the large models of this wave come from Transformer, so when you see a T in the model, then the T most likely represents Transformer. I am very honored to have done some forward-looking work in this area. At the end of 2017, researchers from Google published "Attention is All you need", a milestone paper that brought the Transformer model to the world. And our paper "A Structured Self-attentive Sentence Embedding", which first proposed a "multi-hop self-attention mechanism" published in early 2017, was cited. And the first author of this paper, Ashish Vaswani, was a student I had mentored at IBM. The title of the paper "Attention is All You Need" also expresses the meaning of "self-attention is very important, multi-head is very important, but RNN may not be as important as we thought before" proposed by us.

**AI Technology Review: What consistent technical judgments do you and OpenAI have? **

**Zhou Bowen: This paper and the Transformer architecture completely changed everything, and it solved the problem of long-distance memory of the model. Ilya Sutskever recalled in a recent interview that OpenAI completely switched to the Transformer architecture the day after the paper appeared. **

We know that GPT is very different from Bert's model, and the reason why Bert was very successful at first, but not as good as GPT later is that it uses both left-to-right and right-to-left information. . In other words, Bert uses future information to help AI learn how to represent, while GPT insists on predicting what the next word will be based on past information only. **OpenAI's approach on this point is in line with our team's thinking, that is: try not to use answers to learn. **From attention to self-attention, from BERT to GPT-3, the core idea is when you no longer rely on future information such as the output or the context of the word to be predicted, or when more data can be used to fully When training AI models, we start to see the possibility of AGI.

In addition, OpenAI believes that large models learn world knowledge through natural language, thereby compressing world knowledge into large models. The GPT series large models and ChatGPT are also promoted according to this concept. The same is true of the concept and vision of my team and I, that is, to build a general-purpose large-scale model, and to enable it to exert higher value and capabilities in the vertical field through professional training, and integrate consumers' complex emotions, needs and experiences, as well as product innovation, Design, product parameters, materials, functions, etc., are compressed into a large model to reconstruct the binary relationship between people and products, and use AI to reshape the value of products.

**AI Technology Review: In addition to technical strength, what other aspects of OpenAI make you think there are merits? **

Zhou Bowen: Not only in terms of technical judgments, OpenAI’s entire business approach is representative, including: the establishment of the ecosystem, the announcement of the new Moore’s Law, the reduction of API prices by 90%, etc. Expand the imagination space of capital and users for the commercial application of large models, and derive almost unlimited application scenarios. In addition, OpenAI's plans for ethical governance, business development, ecological technology, and future development are also very clear.

**AI Technology Review: Will the next OpenAI appear in China? **

** Zhou Bowen: ** The technical difficulty of building a large model is actually beyond the imagination of many Chinese entrepreneurs. Therefore, I do not recommend that domestic companies blindly follow and copy the "OpenAI+Microsoft" model, because most Chinese technology companies in China are not as good at business decision-making as Microsoft, and their technology judgment is not as good as OpenAI. **

The success of OpenAI is the result of many factors. For example, Ilya Sutskever made technical judgments, Greg Brockman did functions, and Sam Altman integrated resources, including research on the impact of AI on ethics and society. If domestic companies simply imitate OpenAI, the distance between each other will only get further and further away.

OpenAI's technical judgment can be seen from the latitude of data, because not all data in the world are equally important. Why did OpenAI choose to use Github's programming language to train the thinking chain? Because the semantics and syntax of the programming language are extremely simple, the logic of the execution process is rigorous. This also represents a characteristic and advantage of OpenAI: it will not attack blindly. Therefore, I think that China's AI development needs to find another path, that is, relying on the ability of general large models to start from the application of vertical scenarios, which is more likely to succeed.

Generative AI will disrupt the existing consumer experience

**AI Technology Review: Why are you targeting the consumer sector? **

Zhou Bowen: When I was in JD.com, I saw a huge business opportunity of "dynamically matching consumer demand and product design with artificial intelligence". In 2021, I decided to leave my job and start a business to develop a general large language model for vertical industries (the large language model had not exploded at the time), hoping to cover all consumer behaviors from non-specific scenarios. We know that the time and space scenes are different from Monday to Friday, and the focus of white-collar workers or other occupations is also different. Behind these cultural symbols that affect shopping behavior are consumers' complex emotions, experiences and product choices. Logically, this is exactly the valuable information that businesses need. When making products on the supply side, it includes creativity, design, product parameters, functions, materials, and brand positioning, slogan, marketing, advertising, marketing, promotional images, etc. There is actually a strong corresponding relationship behind all these factors .

This kind of corresponding relationship has never been passed by human beings before. Practitioners in planning, marketing, and sales only understand the links they are responsible for. And we are going to make the world's first general-purpose large-scale model of the commodity supply chain, that is, to compress all these information into a general-purpose model with high fidelity, and based on this large-scale model to empower the entire life cycle of enterprise products, including: opportunities Insight (Discover), product definition (Define), program design (Design), drive R&D (Develop), marketing transformation (Distribute). In this way, enterprises can discover innovation opportunities more efficiently, design and produce more creatively, carry out marketing promotion, reach users and complete transformation more effectively.

**AI Technology Review: In terms of business model, this seems to be more advanced. **

Zhou Bowen: For any entrepreneurial team, it is very important to be able to cultivate more professional capabilities after having the general technical capabilities of large models. At present, the breakthrough of GPT is mainly in its general ability, but its value for specific industries and vertical fields has yet to be developed. For example: GPT can draw very realistic artistic paintings, but it cannot draw circuit diagrams, because it does not have enough knowledge of physical knowledge. In-depth, and relevant judgments are not professional enough.

Therefore, I think there is a need for such a tool (a general-purpose large model with professional capabilities) to make it easier for consumers to find and be more willing to buy the products they need, which may completely change people's existing shopping paths. Generative AI can compress massive amounts of business information into such large models, so as to learn all aspects of the commodity supply chain, and improve the efficiency of key links centered on consumers. This is the idea and creativity that has already been generated in 2021.

**Lianyuan Technology is developing a large model with general capabilities. This large model has expertise in linking products and consumers. **We have 37 large-scale model evaluation indicators, 2/3 of which are general-purpose capabilities such as reasoning ability and computing ability, and more than a dozen items are specially applied to the connection between products and consumers, so as to realize "let every product should be Born out of need, let every consumer get what they want" goal.

**AI Technology Review: How can generative AI better integrate with consumption scenarios such as e-commerce? **

**Zhou Bowen:**Humans can either only understand the business logic of planning, or the logic of marketing, but AI can open up all the business chains.

Consumers need a lot of professional vocabulary to find the products they want in scenarios such as e-commerce platforms; but on the other side, merchants do not understand the real needs of consumers and can only reach consumers through e-commerce transactions , Through consulting research institutions to further understand consumers. After introducing a multi-round dialogue function like ProductGPT, the dynamic matching efficiency of merchants and consumers on products will be more efficient than market research, so that e-commerce platforms can participate more deeply in product innovation, design, research and development, and marketing. promotion etc.

In the actual commercial society, there is actually a strong correspondence between the demand side and the supply side. Our self-developed leading Collaborative Innovation Platform SaaS is based on the multi-modal understanding, reasoning and generation capabilities of large models, and helps companies discover business opportunities and product innovation through deep insights into consumers, scenarios, products, product references, and R&D. At the same time, the ProductGPT multi-round dialogue platform of Lianyuan Technology provides each employee of the enterprise with a personal assistant deeply customized according to different professional roles, and meets their specific work needs by providing role-specific skills and knowledge. For example, Lianyuan Technology's consumer research personal assistant will provide professional skills and relevant knowledge such as researching market trends, understanding consumer needs, and market research.

**AI Technology Review: You have used generative AI to make money in JD.com, how did you do it? **

**Zhou Bowen:**In 2019, I led JD.com’s AI team to apply generative artificial intelligence to create product copywriting and select pictures. That is also JD.com’s first generative large-scale model. At that time, our AI model mainly accomplished three things:

First, you can read the content on the product details page by yourself, and directly generate 8-9 selling points of this product through analysis;

Second, when a consumer browses a certain product, the big model will quickly find out which selling points can impress the user more by analyzing the behavior data of different consumers;

Third, AIGC will generate exclusive slogans around the selling points that consumers are most concerned about based on user portraits.

After a period of implementation, the conversion rate of product recommendations has increased by 30% compared to before. Consumers may not realize that when they search and shop on JD.com, the product categories and descriptions they see are actually automatically generated by AIGC word by word according to the user's preferences and the selling points of the products at the moment he browses the products.

**AI Technology Review: What do you think of OpenAI's open API, and what does it mean for the industry? **

Zhou Bowen: Speaking from personal experience, I used to be the chief scientist of IBM Watson Group. At that time, the data of some industries in the United States was regulated, and such enterprises were generally unable to cooperate, and could only deploy private clouds. For this reason, in 2015-2016, I was determined to be a public cloud. In order to achieve this, it is necessary to APIize Watson's AI capabilities. At that time, I led the launch of dozens of APIs including dialogue and natural language understanding. Put these APIs on the cloud platform, and now IBM's AI business mainly makes money from it.

I returned to China at the end of 2017, and in April 2018, I released JD.com's artificial intelligence open platform. At that time, there was basically no AI platform in China, which also brought considerable income to JD.com. In 2019, the JD.com AI team led by me generated 170 million yuan in revenue, which is not bad for a team of 200 people.

**AI Technology Review: There is a perception in the industry that the risk of making a vertical large model is very high. What do you think? **

Zhou Bowen: I think that in the future, those well-defined, high-value workflows will be completed by professional AI models rather than general AI models. It is easy to further improve the basic capabilities of a general-purpose large model after a certain vertical scene is successfully completed. In addition, if we start from a vertical scenario, our past accumulation in terms of computing power, data, and algorithms can be more fully utilized. Therefore, in Lianyuan Technology, the large model must have the basic capabilities of general large model technology in the underlying framework of the technology, and it must be evaluated with scientific methods, but it also requires professional training.

In 2023, because of the sudden popularity of ChatGPT, the market began to use AI 2.0 to describe its huge potential. In addition, almost all technology giants have joined the battle, the venture capital market is trying to seize new opportunities, and the market environment is also changing rapidly. GPT is a systematic entrepreneurial opportunity, but just copying, following, and catching up is risky and difficult.

After founding Lianyuan Technology, we have communicated with more than 100 customers, saw the real needs, and improved the technology realization path by continuously optimizing the large model: "In 2022, we demonstrated the commercial value and technical feasibility of this scenario, This means that even if we are making a large model, we are on a different track from OpenAI, and the profit model is also different.

What I want to do is a better world knowledge compressor than the current GPT, which requires very interactive data, and the data is obviously closely related to the scene. As for what kind of data has the meaning of higher human intelligence, there is actually a lot of theoretical work to be done in it, and it is a direction worth exploring in the future.