Abstract
At present, the research on the theoretical system of multimedia image linguistics in my country is very limited. In order to further improve and develop the theoretical system of multimedia picture linguistics, this article extracts effective knowledge from the research results of educational technology, linguistics, psychology, and semiotics and establishes a teaching system based on image linguistics. Multimedia picture linguistics consists of three parts: structure, semantics, and pragmatics, corresponding to the artistic laws of multimedia pictures, the cognitive laws of multimedia pictures, and the humanization and naturalization of picture design. Therefore, the research in this article is an important part of establishing multimedia image linguistics. Starting from the research methods of multimedia technology and picture linguistics, this article introduces the theoretical framework and the relationship between the multimedia learning cognitive theory and Mayer’s multimedia learning cognitive theory and conducts in-depth discussion and demonstration. The thesis constructs a teaching system that combines picture linguistics and multimedia technology and makes it a language tool for developing and designing multimedia learning materials. Experiments have proved that the teaching system combining picture linguistics and multimedia technology proposed in this paper can provide a framework, ideas, and methods for the related research of multimedia picture linguistics and multimedia technology and make multimedia picture language a multimedia learning material design and development The language tools that can be followed can effectively improve the teaching effect under the background of information technology.
1. Introduction
“Multimedia Picture Linguistics” is a new subject category that has emerged recently. The direct reason for its emergence is the need to carry out the construction of multimedia teaching materials widely at home and abroad at present, and the deeper reason is that in the information age. It is urgent to recognize and standardize the popular language of this era, the multimedia picture language [1]. Multimedia picture linguistics is an innovative theory born and grown up in China. It is a new design category formed in the information age. It has important application value and is an important research field of educational technology [2]. At present, the research of the theoretical system is still in the primary stage, and it is necessary to further explore and improve the theoretical framework, research content, research methods, and theoretical basis and so on. It can be used to provide a reference framework, ideas, and methods for the follow-up related research.
“Symbol” is a meaningful medium used to convey information in the process of communication. Multimedia pictures are considered to be a language because it conforms to the definition of language; that is, it can be regarded as a symbol system and can be used to transmit knowledge information to view aesthetic hearing [3]. This definition shows that language has two basic properties: language is a “symbol” system, and language is used for communication and thinking. This paper is divided into six chapters. The first section is the introduction part, which mainly leads to the design method of this paper. The related work is given in the second section. The third section introduces the theory and core concepts of the teaching system which combines picture linguistics and multimedia technology, and it can give readers a more intuitive feeling. Section 4 describes in detail how to construct the teaching system of the combination of picture linguistics and multimedia technology [4]. The fifth section mainly from the experimental point of view proves that the combination of the picture linguistics and multimedia technology teaching system can better improve the teaching efficiency. Finally, Section 6 gives a summary of the article.
The research contributions of the thesis include the following: (1)The thesis establishes a teaching system based on image linguistics(2)This article starts with the research methods of multimedia technology and picture linguistics, introduces the theoretical framework and relationship between multimedia learning cognitive theory and Mayer multimedia learning cognitive theory, and conducts in-depth discussion and demonstration(3)This paper constructs a teaching system that combines picture linguistics and multimedia technology and makes it a language tool for developing and designing multimedia learning materials
2. Related Work
In 2002, Professor you Sewing, a famous educational technology expert in China, put forward the concept of “multimedia picture language”. In recent years, under the guidance of professor yet, the research team of our country has been guided by professor yet. On the basis of “multimedia picture language” as the starting point, this paper makes a reasonable creation of the multimedia picture art theory and designs some rules to be followed in view of the multimedia learning material interface. In recent years, the Chinese research team has put forward the theory system of “multimedia picture linguistics”, which has further expanded its research to the level of linguistics [5]. The main purpose of the research on the theory of “multimedia picture linguistics” is to improve the teaching effect in the context of information technology and to make the application, development, and design of various multimedia language materials follow certain rules. It is an innovative theory growing up in China, and the value of its application is very high [6]. We need to continuously explore and improve it in order to provide better methods, frameworks, and ideas for subsequent research [7]. It plays an important role in the field of Chinese educational technology research. However, in view of the current research status, the research on the theoretical system of multimedia picture linguistics is still in its infancy.
From the perspective of general linguistics, this paper reveals the essential attributes and linguistic characteristics of the multimedia picture language. This paper discusses the theoretical basis for the division of multimedia picture linguistics into three parts: construction, semantics, and pragmatics, in order to verify the reasonableness of multimedia picture linguistics [8]. The ideas and conclusions of studying multimedia picture linguistics can be found in general linguistics. General linguistics defines language as the symbolic system of human communication and thinking. Multimedia infrastructure is shown in Figure 1.
Computer network platform can provide a good environment for students, so that students can receive information both visually and audibly, thus improving their learning efficiency. We made statistics on a large number of data: visual can get 82% of the information, and auditory can get 11% of the information [9]. Therefore, the human sense of hearing and vision is the most accepting information. We compared the same material, using hearing alone and hearing at the same time to get 70% of the information. Two days later, the amount of information held down to 10%, and visual can get 83% of the information. Then, two days later, the information kept down to 20%. If both vision and hearing receive information at the same time, the amount of information held can still reach 65% two days later. Therefore, it is very important to build a computer network platform, which can provide students with a good learning environment and improve their English ability [10].
3. The Theory and Core Concepts of the Teaching System Based on the Combination of Picture Linguistics and Multimedia Technology
3.1. Multimedia Graphics
Multimedia picture is a new kind of comprehensive information picture that appears with the appearance of multimedia. First of all, the multimedia learning materials and the knowledge information exchange between people, in the multimedia picture. It also has the following several important characteristics: (1) based on the basis of plane display, multimedia images can only be based on electronic plane display, such as mobile phones and computers. In order to complete the input and output of various media information in the multimedia interactive environment, the information it needs to present and the related interactive operations also need to be carried out with the help of the display screen. (2) The picture is interactive and dynamic. With the change of time, multimedia picture will also change, and multimedia picture is not a single picture but a “picture group.” In this way, the information interaction between human and machine can be realized on the basis of controlling picture group connection and change. (3) The audio and visual media can be combined effectively to form a multimedia picture, in which the sound of the screen is synchronized with the change of the picture. Although its performance has some differences, it is an essential part of multimedia pictures. (4) There are many types of multimedia pictures. Multimedia learning materials include network course, teaching course, electronic teaching plan, teaching animation, and multimedia courseware. From the point of view of presentation form, multimedia images can be divided into tablet screen, mobile phone screen, and computer screen. Two languages meet the conditions of “language” that is shown in Table 1.
The multimedia picture language is a symbol system. The basic elements of traditional art, such as painting, photography, animation, film, and television, have already formed the symbol system in each art category. The basic elements of multimedia picture are composed of the basic elements of the above traditional art, which are divided into four media types: picture, text, sound, and image [2]. And the use of multimedia picture art rules (that is, the grammatical rules of picture language) is to regulate the relationship between the four media symbols, so that the symbol system formed a set of rigorous picture language structure as shown in the following expression.
The multimedia picture language can be used to convey knowledge or visual sense of hearing. The multimedia teaching material is written in the multimedia picture language, because it contains all kinds of media such as picture, text and audio, and video, which can make full use of the visual, auditory, and visual and conceptual channels of learners. So, the effect of transmitting knowledge and aesthetic sense is better than that of written language and traditional art. It must be pointed out that the use of multimedia screen language symbols for communication is necessary to pay attention which is different from the two characteristics of the character symbol, namely, “in the form of meaning” and “time-sharing presentation” as shown in the following expression.
The multimedia picture language can also be used for thinking. Multimedia picture language introduces information content into the human brain through visual and auditory visual channels and conceptual channels. These information content can be processed in the form of representation and concept in the human mind [6]. Therefore, it can be considered that this is the form of thinking using the multimedia picture language. Experience of the towers is given in Figure 2.
3.2. Multimedia Pictures Have Three Features of General Language
In linguistics, language should generally have three characteristics: symbolism, systematizes, and venerability [2]. Multimedia pictures also have these characteristics of linguistic description language. Symbolic: linguistics holds that language symbols have hierarchical structures: phrases are composed of morpheme and words, and sentences are composed of phrases [11]. Multimedia picture language symbols also have a similar hierarchy [3]. For example, the hierarchical structure of graphic language symbols is composed of points, lines and surfaces, color, texture, and shadow tone, by form, attribute, sound, motion, and so on constitute the moving picture (namely the television picture, the computer picture, the multimedia picture).
Systematic: linguistics holds that language is a system of linguistic symbols maintained by grammatical rules. The multimedia picture language is a language symbol system, its language symbols are maintained by multimedia picture art rules (grammar rules), and this maintenance should pay attention to two main points: (1) the rules of multimedia picture art regulate the symbol of visual (auditory) perception, that is, the evolution of the symbol of the basic element of the norm in the picture [6]. This point is emphasized because it clarifies the thinking of the basic theory of the language (multimedia picture art theory), and grammar rules need to be applied according to this idea [12]. (2) On the scope, the multimedia picture art rules are standardized from eight aspects. The basic rules of “prominent theme (or subject),” “Media matching,” and “orderly change” are used to cover the static presentation and dynamic presentation of “Graph.” The text and interpretation of the two aspects of the art rules on the form of the text language sound in the presentation of the screen to regulate [13]. The emphasis on this point is to illustrate that the rules are inextricably linked to the basic theory of the language; that is, the grammatical rules are extracted from the theory as shown in the following expression [9].
Generative: linguistics holds that although the system of linguistic symbols is made up of limited linguistic symbols and grammatical rules, it can produce an infinite variety of sentences [14]. The multimedia picture language is a language symbol system composed of limited language symbols (basic elements, visual elements) and grammar rules (multimedia picture art rules) [10]. It is precisely because of this limitation that the multimedia picture language can be made to be learnable and can be used to develop infinite kinds of multimedia teaching materials [15]. Therefore, the multimedia picture can be regarded as the same as the text, the two belong to two different types of artificial language [16].
3.3. Theoretical Framework and Research Content of Multimedia Picture Linguistics
3.3.1. Determination of Theoretical Framework Based on Semiotics Theory
Charles William Morris, a famous American pragmatism philosopher, divides semiotics into three parts based on logic and law, empiricism semantics, and pragmatism semiotics that are linguistics, semantics, and pragmatics [4]. Morphology mainly studies the relationship between charms, namely, the structure of symbols, which is also called grammar, construction, or syntax. Semantics mainly studies the interrelation between the symbol form and the object represented by the symbol, that is, the relationship between the information content of the symbol object expressed and transmitted by the symbol form and the symbol [17, 18]. Pragmatics mainly studies the relationship among the patterns of symbols, objects, and the specific situations in which symbols are applied [5]. From the perspective of the three components of semiotics, semiotics studies the three relationships between symbols and their related factors. As a symbolic tool for interpersonal communication, language also belongs to the category of semiotics [6].
If the multimedia picture language is compared to the symbolic language, the following viewpoints can be obtained: (1) the basic symbols used in the multimedia picture are all kinds of media; (2) the content of the information expressed and transmitted through the various media symbols is the teaching content; (3) the concrete situation of all kinds of media symbols in multimedia picture is the real information teaching environment. Based on the theoretical framework of semiotics and on the basis of the above viewpoints, the theoretical framework of multimedia picture linguistics can be preliminarily formed. Multimedia picture linguistics consists of three parts: picture construction, picture semantics, and picture pragmatics. Picture construction studies the structure and relationship between various media, while screen semantics studies the relationship between various media and the information they express or disseminate. Based on the theoretical framework of semiotics, and based on the above viewpoints, the theoretical framework of multimedia picture linguistics can be initially formed [19, 20]. Picture pragmatics mainly studies the relationship between various media and information teaching environment. The theoretical framework is shown in Figure 3 on the page below. The theoretical framework of multimedia picture linguistics is given in Figure 3
4. The Basic Theory and Application of Constructing Picture Linguistics and Multimedia Technology
4.1. The Concepts of “Language” and “Speech” Exist in Linguistics
After entering the 20th century, the Swiss linguist Saussure first puts forward the concepts of “language” and “speech” in linguistics, which he called “the first bifurcation of the theory of speech activity.” That is to say, for linguistics, it is necessary to distinguish language from speech [4]. Saussure’s “language” actually refers to the language structure regulated by grammatical rules, while “speech” refers to the process and result of expressing with grammar [21, 22]. If “language” is regarded as a tool, then “speech” can be regarded as the use of tools. In 1957, Chomsky (American linguist) based on the theory of cognitive psychology created “transformational generative linguistics”, which proposed the concepts of “language competence” and “language use.” It corresponds to Saussure’s concepts of “language” and “speech.” Functional linguistics, which emerged in the late 1970s, is characterized by its emphasis on the function of language, which holds that grammar alone is not enough, and needs to be explained by function and cognition to explain syntax and other linguistic phenomena [23].
In the study of multimedia picture textualism, the research path of “constitution” can be adopted, and the constituent elements are analyzed in terms of the design rules of the elements and the relationship between the elements, which generalize the design object into a visual and auditory object with certain characteristics. This particular object is analyzed as a constituent element, and then the design rules of the constituent elements and the relationship between the elements are summarized as the design rules [7]. Therefore, it can be determined that there are three aspects in the study of screen construction, namely, the analysis of the constituent elements of multimedia pictures, the study of the design rules of each element, and the study of the relationship between the elements [24, 25]. The analysis of the constituent elements of multimedia picture is the basis of the other two parts of the research content in the construction of picture, and the study of the design rules of each element and the relationship between the elements are actually the grammatical rules of the multimedia picture language. The basic attributes of the elements in the multimedia picture are given in Table 2.
It can be seen from the development of linguistics that although the study of linguistics is deepening, Saussure’s original idea of distinguishing language from speech has continued to this day. Based on the existing research results, the idea can be described as follows. Language is a relatively stable lexical and grammatical system which has formed a consensus. It can help people communicate with the outside world and can help people to think internally. One refers to speech (including writing), which is an act of communication, and the other is that the words spoken (including the words written) need to take into account the results of the communication [26]. The expression is given as follows.
4.2. Analysis of the Elements of Multimedia Picture
The constituent elements are the basic unit of the picture composition. The constituent elements of multimedia picture can be summarized as four kinds of media symbols, such as picture, text, sound and image, and the interactive function of the picture, that is, graph picture, including drawing figure and still image. The intersection-interactive function is essentially a multimedia picture of the group function. The presentation of the interactive output result and the trigger of the interactive input operation are realized by the multimedia screen which contains the elements of the interactive function [8]. The interactive form of multimedia picture not only depends on the input and output of these two kinds of picture, but also depends on the characteristics of the different multimedia picture presentation terminal interactive information input technology. Teaching content classification diagram is given in Figure 4.
Second, this paper analyses the new visual and auditory effects derived from the coordination of various attributes in the picture elements, studies its rules, and then comes up with the corresponding picture grammar rules in turn. Although the visual and auditory effects are also related to the basic attribute components of the elements of the picture, they are not equal to the sum of these elements, but they are a new whole completely independent of these elements [4]. This whole has an independent “new quality” that transcends all parts, which is called the “Gestalt quality”, a process in which implicit stimuli are generated from explicit stimuli. Taking text as an example, new visual effects can be derived from the changes, layout, combination, and collocation of the basic properties of the text, such as font, font size, color, spacing, location, and layout. At the same time, it presents a comprehensive style of text, which generates “new quality” in the process of learners’ perception. If this comprehensive style meets the aesthetic needs of learners, it will produce harmonious visual effects; otherwise, it will become a failure in multimedia pictures, which will have a negative impact on learners’ learning. Investigation on the situation creation of mental health education in primary and secondary school is given in Table 3. The first 2 columns show the results, and the back shows the number of votes.
From the perspective of the social function, language is nothing more than a tool. The relationship between language and speech is like the relationship between tools and the use of tools. In general linguistics, sentences are the dividing line between “language” and “speech.” The static units below sentences, including words, sounds, and meanings, belong to the study category of language, which is the internal structural model of language as a relatively stable communication tool [5]. The dynamic units above sentences, including the flow of language, intonation, sentence group, paragraph, paragraph group, and text, are completed in a certain context by people using the language system, which belongs to the category of speech research. Moreover, the model of sentence belongs to the category of language, and the use of specific sentence belongs to the category of speech.
4.3. The Distinction between Language and Speech in Multimedia Picture Linguistics
In multimedia picture linguistics, according to Saussure’s idea of distinguishing language from speech, the language of multimedia picture is divided into two aspects: basic theory and theoretical application. In the view of multimedia picture linguistics, the teaching content in multimedia teaching material is expressed through multimedia picture, or in a broad sense, it is expressed by moving picture (including TV picture, animation picture, and multimedia picture). This shows that the moving picture should be equivalent to the “sentence” in general linguistics, which is the dividing line between language and speech in multimedia picture linguistics. The expressions are given as follows.
The art of multimedia picture is divided into the art of media presentation and the art of picture group, which embodies the idea of dividing the picture according to the multimedia picture. The former regards the basic elements of all kinds of media on the screen as “words” in the multimedia picture language (symbol) system. The rules of multimedia picture art are regarded as “grammar” to regulate the evolution of these words (the basic elements after evolution are called visual elements). Therefore, the multimedia picture is through these “grammar” standard “vocabulary” and has formed each kind of performance teaching content structure pattern, namely, “sentence.” Obviously, these belong to the theory of multimedia picture art, corresponding to the category of “language” in linguistics. Obviously, these contents belong to the application of basic theories and correspond to the study of “phonetics” in linguistics. The latter uses editing functions or interactive functions to group each picture according to the requirements of “disciplines”, according to the requirements of different disciplines and environments. The purpose is to accurately express the content of each discipline and achieve good teaching results.
4.4. Interpretation of Three Parts: Construction, Semantics, and Pragmatics
Modern linguistics holds that linguistics is a discipline that studies language tools (i.e., symbolic systems), tool internal structure systems, and the processes and results of people using language system tools in different contexts [6]. Linguistics is divided into “linguistic linguistics” and “linguistic linguistics.”
Language linguistics is a kind of linguistics which takes “language” as its only object. “language” refers to the lexical and grammatical system of the combination of sound and meaning. Linguistic study of speech: the process and results of using the linguistic system tools in different contexts [7]. American philosopher Miles further divides semiotics into three parts: morphology semantics and pragmatics. The study of the relationship between symbolism morphology is also called grammar or syntax, and semantics studies the relationship between symbols and symbolic objects, that is, the information about symbolic objects transmitted by symbols through symbols. Pragmatics studies the relationships among charms, objects, and symbolic situations. Symbolic context refers to the specific environment in which symbols are used between users to convey their thoughts and feelings [8].
Multimedia picture linguistics draws lessons from the views of modern linguistics and semiotics, and it is divided into three parts: construction, semantics, and pragmatics [9]. Four elements of the teaching system are given in Figure 5.
In this teaching system, the relationship between teaching content and all kinds of media belongs to the above multimedia picture semantic research content. The medium in the teaching system refers to the physical carrier of storing and transmitting information, such as information equipment and network. The relationship and law between the media and various kinds of media as the form of information presentation in multimedia pictures belong to the research content of picture pragmatics. It focuses on the different characteristics of different media carriers such as computers and mobile devices and the matching rules of various media symbols in multimedia images [6].
Among them, the study on the structure of multimedia picture belongs to the category of “linguistic linguistics”, which consists of linguistic symbols and grammatical rules. In addition, according to modern linguistics, the context in linguistics of speech can be divided into small context and large context. Multimedia picture pragmatics uses the “big background” as the classroom teaching environment to study different teaching forms of picture language to meet the needs of classroom teaching strategies. According to the needs of compiling multimedia teaching materials, multimedia picture semantics extends the concept of “small situation” to the teaching content of different subjects and studies different design formats of screen languages to meet the needs of these subjects. The small context refers to the context before and after oral or written language, while the large context refers to the identity, situation, emotion, and social and cultural environment of both parties. Therefore, multimedia picture semantics and pragmatics can be regarded as the study category of “linguistics of speech.”
5. Experimental Verification Analysis
Picture semantics studies the relationship between all kinds of media and the teaching content it expresses or conveys, from which to sum up some understanding of the regularity of different types of teaching content and form the “picture semantic rules”. According to pedagogy, the teaching content refers to all kinds of materials and information dynamically generated by teachers and students in teaching communication activities to serve the purpose of teaching. The teaching content includes not only the contents of teaching materials [9]. It also includes the role of motivation, guidance, methodology guidance, normative concepts, and value judgment. It is the sum total of the knowledge and skills, the indoctrination of ideas and views, and the behaviors and habits cultivated by schools and teachers. Although the teaching material content is not the only component of the teaching content, it is the most important component. As a form of teaching material, multimedia learning material is responsible for presenting or carrying all kinds of knowledge information in the teaching content. Therefore, more accurately, picture semantics is actually to study the relationship between all kinds of media and the expressed or transmitted knowledge information and to excavate the matching rule between various media and different types of knowledge information—“semantic rules of picture.” Evaluation in the PE learning content is given in Table 4.
As a result of the above analysis, it can be confirmed that the research content of picture semantics is to study the picture, text, sound, image, interactive form, and the factual knowledge, conceptual knowledge expressed, or transmitted in the multimedia picture. The relationship and matching rules between procedural knowledge and metacognitive knowledge and the semantic rules suitable for different knowledge types are summarized. Through the specification of semantic rules, the design of multimedia picture can present and express all kinds of knowledge more accurately as follow. Student achievement statistics is shown in Table 5.
It can be seen from Table 5 that there is a direct proportional relationship between the student’s performance and the student’s cognitive level. Different theories of teaching elements reflect different perspectives and emphases of scholars, but in essence, there is no fundamental opposition and conflict between different theories of teaching elements. Teaching is a communication activity between teachers and students as a way of cultural inheritance. Therefore, teaching is essentially a cognitive activity, a form of existence, and a social phenomenon, which includes both the subject and the object of action [7]. Teaching is by teachers and students as the main body, culture as the content, the media as a means of cultural exchanges, and inheritance of an activity. Therefore, the basic elements of the teaching system should include teachers, students, teaching content, and media. The four elements are not independent of each other, but an organic whole that interacts and interrelates, thus forming a relatively stable teaching system, as shown in Figure 6.
As can be seen from the Figure 6, with the continuous development of picture linguistics and multimedia technology, the system performance of the teaching system is constantly improving. This is because the teaching system based on the combination of picture linguistics and multimedia technology can better combine the advantages of picture linguistics and multimedia technology. Fully applying the advantages of the two to the teaching system can better adapt to the development of the teaching model.
It can be seen from the Figure 7, compared with the traditional teaching system, that this paper fully combines picture linguistics with multimedia technology, so that it can better cope with the teaching changes in multimedia technology. As a result, the teaching system has been in a steady growth of efficiency situation. Figure 8 shows the comparison between model prediction accuracy and other methods.
We compare the model established in this article with the methods in the latest international references, such as support vector regression machine (SVR) and extreme learning machine (ELM). The comparison uses model-based accuracy as an indicator. Figure 8 shows the comparison between model prediction accuracy and other methods. As shown in Figure 8, in the prediction comparison, the model established in this paper has better accuracy and higher prediction accuracy under the same test condition.
6. Conclusion
Multimedia picture linguistics is composed of three parts: construction, semantics, and pragmatics, which correspond to the art law of multimedia picture, the cognitive law of multimedia picture, and the humanization and naturalization of picture design. Therefore, the study of this paper is an important part of creating multimedia picture linguistics. At present, the research on the theoretical system of multimedia picture linguistics in China is very limited. In order to further improve and develop the theoretical system of multimedia picture linguistics, this paper draws lessons from the research results of educational technology, linguistics, psychology, and semiotics. Then, starting from the research methods and theoretical framework of multimedia technology and picture linguistics, the relationship between multimedia learning cognitive theory and Mayer's multimedia learning cognitive theory is deeply explored and demonstrated. Experiments have proved that the teaching system that combines picture linguistics and multimedia technology proposed in this paper can provide a framework, ideas, and methods for the research of multimedia picture linguistics and multimedia technology. It is an information teaching application. The method proposed in this paper can provide more frameworks, methods, and ideas to construct a teaching system that combines picture linguistics and multimedia technology and make it a language tool for developing and designing multimedia learning materials. In the future, we will further devote ourselves to related research to provide reference and reference for the development of the education industry.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The author(s) declare(s) that they have no conflicts of interest.