International Journal of Knowledge Content Development & Technology
[ Article ]
International Journal of Knowledge Content Development & Technology - Vol. 11, No. 3, pp.63-79
ISSN: 2234-0068 (Print) 2287-187X (Online)
Print publication date 30 Sep 2021
Received 25 May 2021 Revised 11 Jun 2021 Accepted 18 Jun 2021
DOI: https://doi.org/10.5865/IJKCT.2021.11.3.063

A Study on the Improvement of Korean Journal Citation Index Service

Suntae Kim*
*Assistant Professor, Department of Library and Information Science, Jeonbuk National University, Jeonju, Korea kim.suntae@jbnu.ac.kr

Abstract

The KJC and KCI services operated by the National Research Foundation of Korea (NRF) need to refer to the global citation index database to set the service direction that researchers want and develop new functions. Therefore, in this study, the usability of Korea's KJC and KCI web databases was evaluated. In addition, FGI verification was performed for a group of experts on KJC and KCI improvement measures derived by academic information experts. The evaluation committee consisted of the researcher group, the society related group, the publisher (academic information distribution organization) group, and the policy-making organization group. A total of 16 experts performed usability evaluation and service improvement proposals for KJC and KCI services. As a result of usability evaluation of KJC and KCI web database using SUS measurement tool, KJC service was evaluated as 73.44, and KCI service was evaluated as good as 64.38. KJC improvement proposals were evaluated positively with 4 points or more for all improvement items, and KCI improvement proposals were positively evaluated with 3.5 points or more for all improvement items. Overall, KJC and KCI error improvements were evaluated with a score of 4 or more, showing strong affirmations. In the future, it is hoped that the improvement proposals and improvements proposed in this study will be applied to the development of KJC and KCI services, so that more quality services will be used by domestic researchers.

Keywords:

Korean Journal Citation Index Service, KCI, KJC, Korea Journal Central, System Usability Scale, SUS

1. Introduction

The Web of Science and Scopus citation index databases are databases that are systematically constructed with information on citing and cited documents. The citation index proposed as a tool for academic research provides researchers with a variety of access points to relevant research resources. Provides access to papers in the same subject area, other papers written by the same author, and references to relevant papers. It can also be useful for analyzing research trends in a specific subject area (Kyunghee University Engineering Library, 2017). The specific utility of the citation index database can be examined in three main aspects. First, as the effectiveness of search and discovery, it provides a variety of access points for researchers to find papers or researchers on research topics of interest. Second, it is possible to grasp the latest research trends as the effectiveness of setting the research direction, and to track the research activities of other researchers. It can also be used to decide which journal to submit research results. Third, as the effectiveness of policy making, it can be used as a tool for policy making and support by selecting nationally excellent researchers, excellent academic journals, and excellent research fields.

On the other hand, among the journal articles published with public funding, 71% of the articles have been published in international journals, and most of them are published in subscription-based journals (Kim, 2016). Free access is available to 1,264 papers in Korea, 67% of which are listed on the Korea Citation Index (KCI) (Kim & Chung, 2017). The KCI citation index database is also very similar in character to the above databases. KCI is a specialized information service that analyzes citation relations between papers by converting journal information, published paper information, and references into a database for academic journals published by domestic academic organizations. It is used as a tool to evaluate the quantitative level of journals published in a specific subject field because it can calculate the influence of journals according to various statistical data and citation frequency required for research resource management. In addition, by grasping and researching the relationship between the cited documents and the cited documents by papers in domestic academic journals, it can be used as an objective tool for grasping the academic journal issuing institution and the level of each academic field and for evaluating research results by researchers (Pukyong National University School library webzine, 2013).

However, unlike Web Of Science or Scopus citation index database, KCI builds and services citation data limited to domestic journals. Therefore, there is a big difference in the range of the provided data and the services to be provided. Meanwhile, the National Research Foundation of Korea (hereinafter referred to as NRF) is operating Korea Journal Central (hereinafter referred to as KJC) as a service designed to promote domestic journals to foreign researchers and to introduce submissions to journals. KJC is partially linked with KCI. KJC only provides services at the level of linking detailed information of thesis to KCI and does not provide a tightly coupled service linking function.

In this study, the current KJC and KCI service levels were analyzed, Web of Science and Scopus services were analyzed, and new developments or functional improvements were required for KJC and KCI, and a method of seamless linkage between KJC and KCI services was suggested.


2. Literature Review

Research cases using the citation index database can be easily found until recently. Overseas, Deshmukh et. al. (2020) collected articles on COVID-19 over the last three months from PubMed, Medline (EBSCO & Ovid), Google Scholar, Science Direct, Scopus, and Bio-Medical databases, A study was conducted to review and summarize histopathological changes.

Kosman et. al. (2021) is the development of an overseas study program hosted by CINHAHL, EBSCO, ERIC, Google Scholar, MEDLINE, PsycINFO, Scopus, and Web of Science databases by searching literature for students participating in the study abroad program as higher education medical students. The impact on the community of developing countries was studied. Chung (2020) analyzed 14 data journals indexed in Web of Science, 6,086 data papers, and 84,908 cited references to investigate the intellectual structure of data journals and data papers in the academic community.

The case of domestic research is as follows. Back et. al. (2020) searched Web of Science, Scopus, PubMed, Korea Education Research Information Service, and Korea Citation Index to identify 64 domestic and foreign research trends. Park and Seo (2020) conducted a study to analyze abstracts by collecting 1,043 papers on the subject of technological innovation from 2015 to 2019 in the Scopus database. Nguyen et. al. (2020) analyzed the bibliographic characteristics and contents of educational papers written by Vietnamese authors published in Scopus registered journals. Shin et. al. (2020) evaluated bibliographic indicators in the field of environmental science with 3,212 papers written by domestic authors out of 74,089 papers published in 57 environmental science journals in Scopus. Kim et. al. (2020) conducted a study to verify the herbal and herbal medicines frequently used for the treatment of obesity in Korea by collecting papers from 2015 to 2019 for 12 databases including Scopus. Choi et. al. (2020) analyzed distributed academic papers DB schema and DB metadata to derive an integrated DB schema. In addition, the final academic information data items were determined through comparison and analysis using Web of Science and Scopus schema.

On the other hand, studies related to general usability of web services have been conducted variously in recent years. Kang and Hwang (2020) conducted a study to improve usability through content utilization analysis of KISTI science and technology information service. Fabisiak (2018) conducted a study to analyze the usability of web services from the user's point of view. Kwak et. al. (2020) conducted a study on the improvement method of map-based web platform service.

Looking at the preceding studies above, the use of the citation index database service has been conducted until recently in various ways, including bibliographic studies targeting papers registered in the database, and studies for designing an integrated database. However, it is difficult to find a study to improve the quality of the citation index database. Therefore, this study is judged to be a timely study.


3. Research Objectives and Methods

In this study, we evaluate the usability of KJC and KCI web databases. The purpose of this study is to verify and suggest KJC and KCI improvement proposals derived through this study through an expert group. As a research method, interest groups related to KJC and KCI services were grouped into four. The four groups consisted of the researcher group, the society related group, the publisher/academic information distribution group, and the other/policy-making agency group. Each group consisted of four people. A total of 16 experts were formed, and the KJC system and the KCI system were evaluated by dividing them into two areas, respectively. The first evaluation was to evaluate the usability of the KJC and KCI web databases using the System Usability Scale (hereinafter referred to as SUS) measurement tool. Second, through Focus Group Interview (hereinafter referred to as FGI), suggestions for improvement were derived for KJC and KCI services. The content itself provided by KJC and KCI is not analyzed. Therefore, the accuracy, completeness, up-to-date, coverage, and expertise of the content are not evaluated. After deriving improvement proposals focusing on the service interface and function development, opinions of experts were collected through FGI.

Evaluation target and evaluation method

System Usability Scale (here in after, SUS) was developed by John Brook in 1986 and is often used as a software satisfaction evaluation tool. The SUS evaluation tool consists of 10 questions that evaluate the level of consent. It consists of mixed positive and negative questions about the service. It generally provides a high level of subjective view of usefulness. Therefore, it is often used to compare the usefulness of services. As an advantage, since detailed element evaluation is excluded, test fatigue is relatively less than that of other evaluation tools. In addition, it is possible to increase the concentration of the respondents' evaluation by mixing positive and negative questions. However, the disadvantage is that it is only used to classify the convenience of use, and the service itself cannot be diagnosed. Also, scoring can be complicated. A cautionary note when applying the SUS methodology is that the best results can be obtained only when the scores are standardized to calculate the percentile ranking when interpreting the results. In addition, the evaluator should be guided to respond promptly to all questions, and the respondent should not think for a long time. If the evaluator thinks for a long time or is unable to answer a specific question, 3 points are marked.

3.1. Professional Composition

In order to evaluate the usability of the KJC and KCI services, four experts were recruited into each group, divided into researchers, conference officials, publishers (academic information distribution institutions), and others (policy-making institutions). FGI (Focus Group Interview) was conducted by recruiting experts who have used academic information services such as Scopus, SciVal, Web of Science and InCites as well as KJC and KCI services in the research process. The composition of experts is shown in Table 2 below. In the case of academic experts, there were two researchers who actually run the academic society and two experts who promote the journal member project. Researchers were composed of four researchers in the fields of forestry, rural areas, geology, and information to include researchers in various subject fields. In the case of publishing, it consisted of one expert from a non-profit organization that collects and distributes academic information, one expert from a commercial publishing company, and one expert from a specialized library and a medical library. In the case of other policy-making support fields, it consisted of four researchers who were involved in the decision-making process for research support.

Field of evaluation committee and how to organize expert

3.2. Evaluation Tools SUS Tools and Evaluation Methods

Fig. 1 shows the SUS measurement items and score input form that the evaluators respond to. Table 3 shows the range of KJC and KCI response scores and the conversion method of SUS evaluation scores. If the final SUS score is 68, it is interpreted as an average score. Therefore, a score of 68 or higher is interpreted as an average score or higher.

Fig. 1

KJC SUS (System Usability Scale) measurement questions and response tools

Range of KJC and KCI response score and conversion method of SUS score

The SUS evaluation was carried out with a zoom non-face-to-face evaluation method in consideration of the COVID-19 situation. After explaining the evaluation outline to 16 evaluation committee members, explaining the KJC and KCI services, the SUS evaluation method was explained. Since the SUS evaluation itself aims to proceed with the evaluation without using the service, a detailed description of each service has been omitted. However, only the purpose of each service and background information for evaluation were explained before evaluation.


4. KJC & KCI Service Usability Evaluation

4.1. Expert Evaluation on KJC & KCI Service Improvement Proposals

The evaluation was performed on a group of experts who participated in the KJC & KCI service usability evaluation. The evaluation method was based on a proposal for improvement previously derived by one academic information expert for the KJC service and KCI service. The evaluation steps are as follows.

  • Step 1: KJC, KCI service analysis
  • Step 2: Investigation and analysis of similar services
  • Step 3: Identifying improvements in KJC and KCI services compared to similar services
  • Step 4: Derivation of KJC, KCI service errors
  • Step 5: Expert evaluation
  • Step 6: Comprehensive opinions and suggestions from experts

Prior to the evaluation, KJC improvement proposals were derived into 2 cases for interface improvement and 10 cases for function addition. As an interface improvement proposal, a change to an interface that lists all the volume and issue list and unified color of clickable text was derived. The proposal for adding a function was derived with the following contents. 1) It is necessary to provide boiler templates to conferences that do not provide copyright information. 2) A function of providing an advanced search interface is required. 3) When clicking the conference name, the function to move to conference service is required. 4) It is necessary to provide an interface for selecting the subject area. 5) When clicking the author's name in the author details, it is necessary to provide a list of authors' articles. 6) When clicking on a volume and an issue in the author details, it is necessary to provide a list of papers on the volume and issue. 7) When you click ‘Number of Cites’, you need to provide a list of articles citing the article. 8) A function of collecting identifier information such as DOI and providing a link is necessary so that thesis other than KCI can be accessed. 9) It is necessary to provide a list of articles citing the article and a list of authors citing the article. 10) The thesis navigation provision function is required on the thesis detail screen.

KJC errors and improvements derived before the evaluation were identified as 4 functional improvements and 2 interface improvements. The function improvement was derived with the following contents. 1) The Best Practice menu should be deleted. 2) Key metadata must be provided in the journal list. 3) The page should be switched to an independent tab. 4) The original text is needed in cooperation with KISTI/KERIS. The interface improvement was derived from the following contents. 1) The tip that guides you to enter the search word needs to be modified. 2) All field names must be provided on the journal search result screen.

As for KCI improvement proposals derived before the evaluation, 4 interface improvement proposals and 35 function addition proposals were derived. Interface improvement proposals are: 1) Need to modify the expression 'KCI Cited' among the names of field items provided on the thesis detail screen, 2) Provide a list of references built up to the point of service, 3) Change the search for limited publication date by year, 4) Consist of improving the interface of constraints on the classification of overseas listings.

The feature addition proposal includes the following contents. 1) A function that can select multiple analysis units and compare them is required. 2) Dynamic filtering function of the analysis field is required. 3) The user's preferred research subject field selection and mapping function are required. 4) Detailed performance preview function is required. 5) Topic confirmation function with high research competitiveness is required. 6) It is necessary to check the funding status of the institution. 7) It is necessary to check the status of research cooperation. 8) A function of quantitative/qualitative performance analysis by subject area is required. 9) Excellent researcher search function is required. 10) It is necessary to present research results in the form of Pie Chart, Tree map, and Donut Chart. 11) Journal quartile (Q1~Q4) presentation function is required. 12) It is necessary to check the paper list of the quartile. 13) Depending on the research field, research topics and topic cluster alignment functions are required. 14) Topic presentation function is required as a heat map. 15) It is necessary to provide a list of institutions, conferences, and researchers related to specific topics. 16) It is necessary to create a dynamic graph by changing the evaluation index. 17) It is necessary to present the status of research cooperation by filtering by the number of co-authors by the relevant institution and researcher. 18) It is necessary to select and analyze the region where the institution is located and the research funding support institution. 19) It is necessary to provide information to joint research institutes. 20) It is necessary to check the researchers of the two institutions that have studied jointly. 21) It is necessary to provide a list of researchers who have not cooperated so far among the 100 researchers with the best research results of the two institutions. 22) When clicking the number provided on the screen, the function of providing a list of related information is required. 23) When entering a search word, it is necessary to provide a help interface. 24) When there is information on the affiliated organization and research field on the page, the function developme nt function is needed as the research trend by issuing institution and the research trend by research field. 25) It is necessary to edit search word and provide storage interface. 26) It is necessary to register search word and provide notification service. 27) A search support function using various author identifiers (ResearcherID, Author ID, scientific engineer registration number, researcher registrat ion number, etc.) is required. 28) A function of providing search tips is required. 29) Funding information related search field provision function is required. 30) A search function using an agency (Affiliation) identifier (ISNI, ORCID, etc.) is required. 31) It is necessary to provide a detailed search interface for entering a search expression through the Advanced search menu. 32) After searching for an institution, it is necessary to provide information on organizations belonging to the institution and cooperative research institutions. 33) On the thesis detail screen, it is necessary to provide the thesis navigation function. 34) A function of providing previous search words is required. 35) A complete match search function is required.

KCI errors and improvements derived before the evaluation were identified as 2 errors, 3 interfaces, and 6 functional improvements. The case of double errors was derived as the following contents. 1) The error in which the downloaded file name is stored abnormally must be corrected. 2) When selecting a university-affiliated institution in the type of institution, the error in which the society is selected must be corrected. In the case of the interface, the following contents were derived. 1) It is necessary to provide information on the basis of calculation for the number of citations. 2) The error that the cursor does not appear in the search word input window needs to be corrected. 3) Cited, Citing, Number, and Times notation needs to be modified. In the case of functional improvement, the following contents were derived. 1) Analysis information service Research trend keyword analysis service needs to be tuned. 2) It is necessary to improve the speed of the thesis citation information search screen. 3) It is necessary to add a description of what exactly the UCI information means to users. 4) It is necessary to consider the method of integrating KCI and interface. 5) In the case of overseas listing classification, SCI and SCIE are integrated in WoS and provided as SCIE. Therefore, it is necessary to modify the constraints. 6) It is necessary to switch to an independent tab.

4.2. SUS evaluation and KJC·KCI Improvement Proposal Evaluation Results

As shown in the following Table 6, the KJC service was evaluated at 73.44, and the KCI service was evaluated at 64.38. If the SUS conversion score was 63 or higher, the usability of the service to be evaluated was judged to be excellent, and the usability of the KJC and KCI services was evaluated as good. The following Table 4 shows the SUS score for each KJC evaluation participant and the average for each item. Table 5 shows the SUS score for each KCI evaluation participant and the average for each item. In each table, R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the research paper support agency (policy agency). Among the collected SUS evaluation tables, 6 points were marked on item 3 of KJC of the S1 evaluator, but it was confirmed that the highest point 5 was incorrectly marked, and the code was corrected to 5 points. Table 6 shows the SUS conversion scores of KJC and KCI.

SUS evaluation score for each KJC evaluation participant and average for each question. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

SUS evaluation score for each KJC evaluation participant and average for each question. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

SUS conversion score of KJC and KCI. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

The following Table 7 summarizes other opinions of evaluators on KJC and KCI services. The opinions of the evaluators are divided into interfaces, new functions, and others.

Other opinions of evaluators on KJC and KCI services

For the KJC improvement proposal, all improvement items were evaluated with 4 points or more, showing a strong positive overall. Regarding the improvement of KJC error, overall evaluation was 4.4 or higher, showing strong affirmation. For the KCI improvement proposal, all improvement items were evaluated with a score of 3.5 or higher, showing positive overall. Regarding the improvement of the KCI error, the overall evaluation was 4 points or more, showing strong affirmation. Table 8 shows the score and average of each evaluation committee member for the KCI improvement proposal. Table 9 shows the score and average of each evaluation committee member for the improvement of KCI error. The 'No.' field presented in each table is the identification number of the improvement proposal and error improvement.

SUS evaluation score for each KJC evaluation participant and average for each question. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

Score and average of each evaluator on the improvement of KCI error. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

4.3. Expansion and Connection of KJC & KCI Services

Fig. 2 below shows a part of the analysis result of KJC service function development. It shows that it is possible to connect to the “Research Trend” service of KCI through the Article menu of KJC service. Through this, it is possible to link with services such as thesis with the highest number of citations and the researcher's h-index Top 100.

Fig. 2

KJC functional development diagram

Fig. 3 below shows a part of the analysis result of KCI service function development. It shows that it is possible to expand the service from the search results of institution information to research trends by research field, and to expand the service from the search results of academic conferences to detailed institution information services. This shows that it is possible to expand the service to the institution's website and academic journal details (quotation details).

Fig. 3

KCI's functional development (thesis search, academic journal search, institutional information search)

The following Table 10 shows KJC and KCI service linkage and extension proposals. Source Service represents the location of a service that needs to be linked and extended. Target Sercie represents the location of the target service to be linked and extended. In the case of the KJC service, a total of six links and extensions were proposed, including the extension of the journal citation index, and in the case of the KCI service, eight links and extensions were proposed, including the extension functions commonly required in the detailed page of the journal.

Proposal for linkage and expansion of KJC and KCI services


5. Discussion and Conclusion

In this study, the usability of KJC and KCI web databases was evaluated. In addition, KJC and KCI improvement proposals derived by academic information experts were verified and suggested through expert groups. The group of interests related to KJC and KCI services is composed of the researcher group, the society related group, the publisher/academic information distribution organization group, and the other/policy-making agency group. The evaluation was performed by dividing into two areas.

As a result of usability evaluation of KJC and KCI web database using SUS measurement tool, KJC service was evaluated as 73.44 and KCI service was evaluated as 64.38. Therefore, the usability of KJC and KCI services was evaluated as good. As for other opinions on KJC, 7 opinions for interface improvement including 'mobile-oriented reorganization', 5 new functions including the 'function to check SCI registration in domestic journals', and 2 other opinions were proposed. As for other opinions on KCI, 4 interface improvement opinions, including 'mobile-oriented reorganization', and 6 new functions, including 'DOI information provision' were proposed.

KJC improvement proposals were evaluated positively with 4 points or more for all improvement items, and KCI improvement proposals were positively evaluated with 3.5 points or more for all improvement items. Overall, KJC and KCI error improvements were evaluated with a score of 4 or more, showing strong affirmations.

In this study, the importance of the KCI citation index database, which conducted usability evaluation and suggested improvement proposals, is constantly increasing. The method of using the citation index database can be said to vary depending on the subject using the database. Kim et al. (2019) argued that the problem of identifying similar journal names is the largest in identifying references, and argued that the use of journal title authority files, etc. He also argued the possibility of using a domestic integrated database as a service-based database and citation analysis reflecting the domestic situation in order to improve the reference literature mapping rate. As the KCI citation index database is an integrated database of academic journals published in Korea, it can be said that data reinforcement and service reinforcement are necessary so that it can function as a pseudonym database and an authority database. In addition, Cho (2020) is interested in the immediate and complete open access of publicly funded research achievements, internationally demanding journals and storage infrastructur e, and the globally accelerating transition contracts. He said that he should seek an alternative. McKiernan et al. (2016) reported that when researchers share data used in research, citations of papers vary from field to field, but increase by 9% to 35% compared to those not. From this point of view, it is clear that current KCI service must also establish citation data for data papers to be submitted to data journals and have a legitimate need to develop related services. In the future, it is hoped that the improvement proposals and improvements proposed in this study will be applied to the development of KJC and KCI services, so that more quality services will be used by domestic researchers.

Acknowledgments

The research was supported by “Research Base Construction Fund Support Program” funded by Jeonbuk National University in 2021.

References

  • Back, H., Yi, K., Lee, J., Kim, J., & Moon, J. (2020). Analysis of Domestic and International Biomechanics Research Trends in Shoes: Focusing on Research Published in 2015-2019. Korean Journal of Sport Biomechanics, 30(2), 185–195. [https://doi.org/10.5103/KJSB.2020.30.2.185]
  • Cho, J. (2020). Analysis of Open Access Status of Domestic Author's Papers Published in International Journals: Based on Highly Cited Papers. Journal of the Korean Society for Library and Information Science, 54(1), 325–341. [https://doi.org/10.4275/KSLIS.2020.54.1.325]
  • Choi, W., Hwang, H., Kim, J., Lee, K., & Lim, S. (2020). Comparison and Analysis of Metadata Schema for Academic Paper Integrated DB. The Journal of the Korea Contents Association, 20(2), 689–699. [https://doi.org/10.5392/JKCA.2020.20.02.689]
  • Chung, E. (2020). An Investigation of Intellectual Structure on Data Papers Published in Data Journals in Web of Science. Journal of the Korean society for information management, 37(1), 153–177. [https://doi.org/10.3743/KOSIM.2020.37.1.153]
  • Deshmukh, V., Motwani, R., Kumar, A., Kumari, C., & Raza, K. (2020). Histopathological Observations in COVID-19: a systematic Review. Journal of Clinical Pathology, 74(2), 76–83. [https://doi.org/10.1136/jclinpath-2020-206995]
  • Fabisiak, L. (2018). Web Service Usability Analysis Based on User Preferences. Journal of Organizational and End User Computing, 30(4), 1–13. [https://doi.org/10.4018/joeuc.2018100101]
  • Kang, N.-G., & Hwang, M.-N. (2020). A Study on the Content Utilization of KISTI Science and Technology Information Service. Journal of Internet Computing and Services, 21(4), 87–95. [https://doi.org/10.7472/JKSII.2020.21.4.87]
  • Kim, G. (2016). An OA Policy Study on Research Outcomes Funded by Public Institutions. Korean Library and Information Science Society Winter Conference, November 11, 2016: 337-353.
  • Kim, G. & Chung, K. (2017). “Characteristics of Open Access Journals in Korea: Focusedon KCI Journals.” Korean Society for Information Society, 34(3): 251-267.
  • Kim, J., Kim, S. Y., Lim, S. J., & Hwang, H. (2019). Case Study of Journal Article and Reference Mapping. The Journal of the Korea Contents Association, 19(11), 262–269. [https://doi.org/10.5392/JKCA.2019.19.11.262]
  • Kim, S.-J., Seo, Y.-H., Lee, H.-S., Chang, H.-K., Cho, J.-H., Kim, K.-W., & Song, M.-Y. (2020). Research Trends of Herbal Medicines for Obesity: Mainly since 2015 to 2019. Journal of Korean Medicine Rehabilitation, 30(4), 89–103. [https://doi.org/10.18325/jkmr.2020.30.4.89]
  • Kosman, B. A., Etxebarria, N., & Chipchase, L. S. (2021). The Impact of Learning Abroad Programs in Developing Countries: A Scoping Review. Nurse Education Today, 97, 104716. [https://doi.org/10.1016/j.nedt.2020.104716]
  • Kwak, S., Kwon, G., & Kwon, J. (2020). Usability Design Improvement Method of Map-based Web Platform Service. Journal of Next-Generation Convergence Information Services Technology, 9(3), 211–221. [https://doi.org/10.29056/jncist.2020.09.03]
  • Kyunghee University Engineering Library. (2017). Citation Index Database Comparison. https://khuelibrary.tistory.com/83
  • McKiernan, E. C., Bourne, P. E., Brown, C. T., Buck, S., Kenall, A., Lin, J., ... & Spies, J. R. (2016). Point of view: How open Science helps Researchers succeed. Elife, 5, e16800. [https://doi.org/10.7554/eLife.16800]
  • Nguyen, C. H., Nguyen, L. T. M., Tran, T., & Nguyen, T.-T. (2020). Bibliographic and Content Analysis of Articles on Education from Vietnam indexed in Scopus from 2009 to 2018. Science Editing, 7(1), 45–49. [https://doi.org/10.6087/kcse.188]
  • Park, E.-M., & Seo, J.-H. (2020). Analysis of Research Trends in Technology Innovation: Focus on SCOPUS DB. Journal of Convergence for Information Technology, 10(8), 120–126. [https://doi.org/10.22156/CS4SMB.2020.10.08.120]
  • Pukyong National University Library Webzine. (2013). Do you know the Citation Index of KCI's Domestic Academic Journals? https://bit.ly/2OxlCXQ
  • Shin, W.-G., Park, M.-K., Kim, D.-H., Jang, H.-J., & Min, T.-S. (2020). Analysis of Research Performance and Trends in Environmental Science. Journal of Environmental Science International, 29(3), 283–297. [https://doi.org/10.5322/jesi.2020.29.3.283]
[ About the author ]

Suntae Kim is a assistant professor in library and information science at Jeonbuk University, Jeonju, Korea. The author mainly conducts research related to research data, which is becoming more important in the era of the 4th industrial revolution and data-intensive science. The author is actively conducting research related to metadata schema design and data repository for systematic management and sharing of research data.

Fig. 1

Fig. 1
KJC SUS (System Usability Scale) measurement questions and response tools

Fig. 2

Fig. 2
KJC functional development diagram

Fig. 3

Fig. 3
KCI's functional development (thesis search, academic journal search, institutional information search)

Table 1.

Evaluation target and evaluation method

Evaluation Target Assessment Methods
KJC service and KCI service SUS measurement by 16 experts
Improvement proposals derived by expert FGI performed by 16 experts

Table 2.

Field of evaluation committee and how to organize expert

Field Professional composition
4 researchers 4 researchers in the fields of forestry, rural areas, geology, and information
4 academic experts Two researchers who actually run the conference
Two experts who promote the academic journal project
4 Publisher (information distribution agency) One non-profit organization expert
One expert in a commercial publishing company
One expert at a specialized library and a medical library
4 policy-making support experts 4 researchers involved in the decision-making process for research support

Table 3.

Range of KJC and KCI response score and conversion method of SUS score

Range of KJC, KCI response scores SUS score conversion method
Step 1: For odd items, subtract 1 from the score.
Step 2: For even items, subtract the response score from 5 points. After the above procedure, the score of all items is converted into a score between 0 and 4 points. The higher the score, the more positive the response to the service. Step 3: After adding up all the converted scores, multiply by 2.5 to calculate the final SUS score. Through this method, a score of 0 to 40 points is converted into a score between 0 and 100 points.

Table 4.

SUS evaluation score for each KJC evaluation participant and average for each question. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

No. R1 R2 R3 R4 S1 S2 S3 S4 P1 P2 P3 P4 F1 F2 F3 F4 Avg.
1 5 3 3 4 1 5 4 4 3 3 4 3 4 3 5 5 3.69
2 1 1 1 4 5 3 2 2 2 4 3 2 1 2 1 1 2.19
3 5 4 5 4 5 3 4 4 2 2 3 3 4 3 2 5 3.63
4 1 1 1 1 1 3 2 3 3 4 2 2 1 1 5 1 2.00
5 3 4 5 4 3 5 3 3 2 3 4 3 2 3 5 3 3.44
6 1 1 1 2 1 1 4 3 3 4 3 2 2 3 1 1 2.06
7 5 4 5 3 5 3 5 5 3 3 4 5 5 4 3 5 4.19
8 1 1 1 3 1 1 2 2 2 3 2 2 1 2 1 1 1.63
9 5 5 5 3 5 3 4 4 3 3 4 4 5 4 1 5 3.94
10 1 1 1 2 1 3 2 2 1 3 3 1 2 2 1 1 1.69

Table 5.

SUS evaluation score for each KJC evaluation participant and average for each question. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

No. R1 R2 R3 R4 S1 S2 S3 S4 P1 P2 P3 P4 F1 F2 F3 F4 Avg.
cy1 5 5 5 5 5 5 3 4 4 3 5 3 4 4 5 4 4.31
2 3 2 1 4 1 3 4 2 2 4 4 4 3 4 1 1 2.69
3 5 2 5 4 4 3 3 4 2 2 4 3 3 2 3 4 3.31
4 1 3 1 1 1 3 3 1 2 4 3 2 3 3 4 1 2.25
5 5 4 5 2 2 5 3 3 3 3 5 3 5 3 5 5 3.81
6 1 4 1 2 2 1 4 3 3 3 2 1 1 4 2 1 2.19
7 5 4 5 3 4 1 2 4 2 2 2 3 2 3 3 5 3.13
8 1 4 1 3 1 5 4 2 2 4 3 3 3 4 3 1 2.75
9 5 4 5 3 5 3 2 4 2 2 4 2 4 2 3 5 3.44
10 1 1 1 3 1 5 4 1 2 4 2 3 2 3 4 1 2.38

Table 6.

SUS conversion score of KJC and KCI. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

Div. R1 R2 R3 R4 S1 S2 S3 S4 P1 P2 P3 P4 F1 F2 F3 F4 Avg.
KJC 95 88 95 65 78 70 70 70 55 40 65 73 83 68 68 95 73.44
KCI 95 63 100 60 85 50 35 75 55 33 65 53 65 40 63 95 64.38

Table 7.

Other opinions of evaluators on KJC and KCI services

KJC KCI
<Interface> <Interface>
- Basically, it should be reorganized to focus on mobile. - Improvements are needed so that the journal cover can be seen.
- Many parts such as data, interfaces, and functions require maintenance. - Basically, reorganization is needed to focus on mobile.
- After defining the main user group, it is necessary to design functions and interfaces. - In order to ensure consistency, a separate website for thesis search and IF (journal evaluation) is required.
- It is necessary to secure web accessibility or explain the recommended browser. - It is necessary to clearly and concisely explain the basis for calculation of statistics and indicators.
- It is necessary to provide major functions and information on the main screen.
- It is necessary to provide an interface suitable for the purpose.
- There is no need to bring Aims & Scope to the fore.
<New Functions> <New Functions>
- Improvements are needed so that the journal cover can be seen. - DOI information needs to be provided.
- Advanced search function is required in institutional search.
- It is necessary to check whether or not SCI is registered in domestic journals. - A function of saving and retrieving search expressions is required.
- The English culture of the web page of the academic journal linked to the KJC service is required. - It is necessary to accept the management direction for data journals.
- Before and after signature change information needs to be provided as well. - It is necessary to solve the problem of accuracy and recall of the complete search result.
- Abstract There is a need for a device that allows you to directly copy and write the reference form above. - It is necessary to link with universities, national institutions, and donation associations.
<Etc.> <Etc.>
- NAVER needs to take action to be searched. n/a
- Use guidance in various ways is necessary.

Table 8.

SUS evaluation score for each KJC evaluation participant and average for each question. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

No. R1 R2 R3 R4 S1 S2 S3 S4 P1 P2 P3 P4 F1 F2 F3 F4 Avg.
1 4 4 5 5 5 5 4 4 4 5 4 5 5 3 4 5 4.44
2 4 3 3 4 5 3 4 3 4 5 4 4 2 3 4 5 3.75
3 5 5 5 5 5 5 4 3 4 5 5 4 4 5 5 5 4.63
4 4 4 5 4 5 5 4 4 3 5 4 4 3 3 4 5 4.13
5 5 4 5 5 5 5 4 5 3 5 4 4 5 3 5 5 4.50
6 4 4 5 4 4 3 4 4 4 5 4 5 2 3 5 5 4.06
7 4 4 5 5 4 3 4 4 4 5 4 4 2 3 5 5 4.06
8 5 5 5 5 5 5 4 3 3 5 4 3 4 3 5 5 4.31
9 4 5 5 5 4 5 4 4 3 5 4 3 4 3 5 5 4.25
10 4 3 5 4 4 5 4 3 3 5 5 3 3 4 3 4 3.88
11 4 3 1 4 5 5 3 5 4 5 4 4 4 1 4 5 3.81
12 4 4 5 4 5 5 3 5 4 5 3 4 4 1 4 5 4.06
13 4 4 5 5 4 5 4 4 5 5 4 4 3 5 4 5 4.38
14 5 3 5 5 5 5 4 4 5 5 4 3 3 5 5 5 4.44
15 5 3 5 5 5 5 4 4 5 5 4 3 3 5 5 5 4.44
16 5 3 1 4 4 5 4 3 4 5 5 3 3 3 3 5 3.75
17 5 4 5 4 4 5 4 2 4 5 5 4 2 1 5 5 4.00
18 5 4 1 4 4 3 4 2 4 5 5 5 2 1 5 5 3.69
19 5 5 5 4 5 5 4 3 4 5 4 4 2 1 5 5 4.13
20 5 5 5 4 4 5 4 3 4 5 4 4 1 1 5 5 4.00
21 3 3 5 3 4 5 3 3 5 5 3 3 1 1 5 5 3.56
22 4 4 5 4 5 5 4 5 4 5 5 3 4 1 3 5 4.13
23 4 4 5 5 5 5 4 5 4 5 4 2 3 5 4 5 4.31
24 5 5 5 5 5 5 4 5 4 5 4 3 4 5 4 4 4.50
25 5 3 1 5 5 5 4 5 3 5 4 4 3 5 4 5 4.13
26 4 3 5 4 5 5 4 4 4 5 3 3 4 4 5 5 4.19
27 5 3 1 5 5 5 4 5 3 5 5 3 4 3 3 5 4.00
28 5 3 5 4 5 5 4 4 3 5 4 3 2 5 3 5 4.06
29 4 5 5 5 4 5 4 5 5 5 5 4 5 4 5 5 4.69
30 4 4 5 5 4 5 4 5 3 5 5 4 3 5 4 5 4.38
31 4 4 5 4 5 5 3 5 4 5 5 3 3 5 4 5 4.31
32 4 5 1 4 5 5 4 3 4 5 5 4 2 5 5 5 4.13
33 4 5 5 5 5 5 4 5 5 5 4 4 2 5 5 5 4.56
34 5 2 5 5 5 5 4 3 5 5 5 5 4 4 5 5 4.50
35 4 4 5 4 4 5 4 3 4 5 4 4 2 3 5 4 4.00
36 4 3 5 5 5 5 4 5 5 5 5 3 3 5 5 4 4.44
37 5 4 5 5 5 3 4 4 4 5 5 4 4 4 5 5 4.44
38 5 4 5 4 5 5 4 4 3 5 5 3 3 5 3 4 4.19
39 5 4 5 5 5 5 4 3 3 5 5 5 5 3 3 5 4.38

Table 9.

Score and average of each evaluator on the improvement of KCI error. (R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the field of research support agency (policy agency))

No. R1 R2 R3 R4 S1 S2 S3 S4 P1 P2 P3 P4 F1 F2 F3 F4 Avg.
1 5 3 5 5 5 5 4 5 5 5 5 3 5 3 4 5 4.50
2 5 4 5 5 5 5 4 5 4 5 5 4 5 5 5 5 4.75
3 5 3 5 5 5 5 4 5 5 5 4 4 5 5 4 5 4.63
4 4 4 5 5 4 5 4 3 3 5 4 2 3 5 5 5 4.13
5 5 4 5 5 4 5 4 5 5 5 4 4 5 4 5 5 4.63
6 4 5 5 5 5 5 4 5 5 5 5 5 5 5 5 5 4.88
7 5 1 5 4 5 5 4 4 3 5 5 4 3 3 3 5 4.00
8 5 4 5 5 5 5 4 5 4 5 5 4 3 5 5 5 4.63
9 5 4 3 4 4 5 4 4 4 5 4 4 2 4 4 5 4.06
10 4 4 5 5 5 5 4 3 3 5 5 3 4 5 5 5 4.38
11 5 4 5 5 5 5 4 5 5 5 5 3 5 4 4 5 4.63

Table 10.

Proposal for linkage and expansion of KJC and KCI services

Source Service Target Service
KJC > About > Citation Index KCI > Journal Citation Index
KJC > About > Journal Information KCI > Journal Detail Information > Research Trends by Research Field, etc.
KJC > About > Publisher KCI > Institutional Information Details> Citation Index Details, etc.
KJC > Article > All Issues KCI > Search by Journal Volume
KJC > Article KCI > Research Trends
KCI > Institutional Information Search results KCI > Research trends by research field, etc.
KCI > Academic Conference Search Institution Information Details > Citation Index Details, etc.
KCI > Citation Information Search > Researcher Citation Info. > Author performance details KCI > Research Trends by Research Field, etc.
KCI > Analyzed Info. Service > Research Trends by Journal > KCI > Journal Details (Citation Index Details)
KCI > List of all KCI papers KCI > Article Details, Author Details, Citation Index Details etc.
KCI > Article Details KCI > Journal Citation Index, Author Details etc.
KCI > Citation Index Details KCI > Research Trends by Research Field, etc.
KCI > Journal Details KCI > Research Trends by Research Field, etc.