Online First

In order to evaluate the usability of the KJC and KCI services, four experts were recruited into each group, divided into researchers, conference officials, publishers (academic information distribution institutions), and others (policy-making institutions). FGI (Focus Group Interview) was conducted by recruiting experts who have used academic information services such as Scopus, SciVal, Web of Science and InCites as well as KJC and KCI services in the research process. The composition of experts is shown in Table 2 below. In the case of academic experts, there were two researchers who actually run the academic society and two experts who promote the journal member project. Researchers were composed of four researchers in the fields of forestry, rural areas, geology, and information to include researchers in various subject fields. In the case of publishing, it consisted of one expert from a non-profit organization that collects and distributes academic information, one expert from a commercial publishing company, and one expert from a specialized library and a medical library. In the case of other policy-making support fields, it consisted of four researchers who were involved in the decision-making process for research support.

Fig. 1 shows the SUS measurement items and score input form that the evaluators respond to. Table 3 shows the range of KJC and KCI response scores and the conversion method of SUS evaluation scores. If the final SUS score is 68, it is interpreted as an average score. Therefore, a score of 68 or higher is interpreted as an average score or higher.

Fig. 1 
KJC SUS (System Usability Scale) measurement questions and response tools

Range of KJC, KCI response scores

SUS score conversion method

Step 1: For odd items, subtract 1 from the score. Step 2: For even items, subtract the response score from 5 points. After the above procedure, the score of all items is converted into a score between 0 and 4 points. The higher the score, the more positive the response to the service. Step 3: After adding up all the converted scores, multiply by 2.5 to calculate the final SUS score. Through this method, a score of 0 to 40 points is converted into a score between 0 and 100 points.

The SUS evaluation was carried out with a zoom non-face-to-face evaluation method in consideration of the COVID-19 situation. After explaining the evaluation outline to 16 evaluation committee members, explaining the KJC and KCI services, the SUS evaluation method was explained. Since the SUS evaluation itself aims to proceed with the evaluation without using the service, a detailed description of each service has been omitted. However, only the purpose of each service and background information for evaluation were explained before evaluation.

The evaluation was performed on a group of experts who participated in the KJC & KCI service usability evaluation. The evaluation method was based on a proposal for improvement previously derived by one academic information expert for the KJC service and KCI service. The evaluation steps are as follows.

• Step 1: KJC, KCI service analysis
• Step 2: Investigation and analysis of similar services
• Step 3: Identifying improvements in KJC and KCI services compared to similar services
• Step 4: Derivation of KJC, KCI service errors
• Step 5: Expert evaluation
• Step 6: Comprehensive opinions and suggestions from experts

Prior to the evaluation, KJC improvement proposals were derived into 2 cases for interface improvement and 10 cases for function addition. As an interface improvement proposal, a change to an interface that lists all the volume and issue list and unified color of clickable text was derived. The proposal for adding a function was derived with the following contents. 1) It is necessary to provide boiler templates to conferences that do not provide copyright information. 2) A function of providing an advanced search interface is required. 3) When clicking the conference name, the function to move to conference service is required. 4) It is necessary to provide an interface for selecting the subject area. 5) When clicking the author's name in the author details, it is necessary to provide a list of authors' articles. 6) When clicking on a volume and an issue in the author details, it is necessary to provide a list of papers on the volume and issue. 7) When you click ‘Number of Cites’, you need to provide a list of articles citing the article. 8) A function of collecting identifier information such as DOI and providing a link is necessary so that thesis other than KCI can be accessed. 9) It is necessary to provide a list of articles citing the article and a list of authors citing the article. 10) The thesis navigation provision function is required on the thesis detail screen.

KJC errors and improvements derived before the evaluation were identified as 4 functional improvements and 2 interface improvements. The function improvement was derived with the following contents. 1) The Best Practice menu should be deleted. 2) Key metadata must be provided in the journal list. 3) The page should be switched to an independent tab. 4) The original text is needed in cooperation with KISTI/KERIS. The interface improvement was derived from the following contents. 1) The tip that guides you to enter the search word needs to be modified. 2) All field names must be provided on the journal search result screen.

As for KCI improvement proposals derived before the evaluation, 4 interface improvement proposals and 35 function addition proposals were derived. Interface improvement proposals are: 1) Need to modify the expression 'KCI Cited' among the names of field items provided on the thesis detail screen, 2) Provide a list of references built up to the point of service, 3) Change the search for limited publication date by year, 4) Consist of improving the interface of constraints on the classification of overseas listings.

The feature addition proposal includes the following contents. 1) A function that can select multiple analysis units and compare them is required. 2) Dynamic filtering function of the analysis field is required. 3) The user's preferred research subject field selection and mapping function are required. 4) Detailed performance preview function is required. 5) Topic confirmation function with high research competitiveness is required. 6) It is necessary to check the funding status of the institution. 7) It is necessary to check the status of research cooperation. 8) A function of quantitative/qualitative performance analysis by subject area is required. 9) Excellent researcher search function is required. 10) It is necessary to present research results in the form of Pie Chart, Tree map, and Donut Chart. 11) Journal quartile (Q1~Q4) presentation function is required. 12) It is necessary to check the paper list of the quartile. 13) Depending on the research field, research topics and topic cluster alignment functions are required. 14) Topic presentation function is required as a heat map. 15) It is necessary to provide a list of institutions, conferences, and researchers related to specific topics. 16) It is necessary to create a dynamic graph by changing the evaluation index. 17) It is necessary to present the status of research cooperation by filtering by the number of co-authors by the relevant institution and researcher. 18) It is necessary to select and analyze the region where the institution is located and the research funding support institution. 19) It is necessary to provide information to joint research institutes. 20) It is necessary to check the researchers of the two institutions that have studied jointly. 21) It is necessary to provide a list of researchers who have not cooperated so far among the 100 researchers with the best research results of the two institutions. 22) When clicking the number provided on the screen, the function of providing a list of related information is required. 23) When entering a search word, it is necessary to provide a help interface. 24) When there is information on the affiliated organization and research field on the page, the function developme nt function is needed as the research trend by issuing institution and the research trend by research field. 25) It is necessary to edit search word and provide storage interface. 26) It is necessary to register search word and provide notification service. 27) A search support function using various author identifiers (ResearcherID, Author ID, scientific engineer registration number, researcher registrat ion number, etc.) is required. 28) A function of providing search tips is required. 29) Funding information related search field provision function is required. 30) A search function using an agency (Affiliation) identifier (ISNI, ORCID, etc.) is required. 31) It is necessary to provide a detailed search interface for entering a search expression through the Advanced search menu. 32) After searching for an institution, it is necessary to provide information on organizations belonging to the institution and cooperative research institutions. 33) On the thesis detail screen, it is necessary to provide the thesis navigation function. 34) A function of providing previous search words is required. 35) A complete match search function is required.

KCI errors and improvements derived before the evaluation were identified as 2 errors, 3 interfaces, and 6 functional improvements. The case of double errors was derived as the following contents. 1) The error in which the downloaded file name is stored abnormally must be corrected. 2) When selecting a university-affiliated institution in the type of institution, the error in which the society is selected must be corrected. In the case of the interface, the following contents were derived. 1) It is necessary to provide information on the basis of calculation for the number of citations. 2) The error that the cursor does not appear in the search word input window needs to be corrected. 3) Cited, Citing, Number, and Times notation needs to be modified. In the case of functional improvement, the following contents were derived. 1) Analysis information service Research trend keyword analysis service needs to be tuned. 2) It is necessary to improve the speed of the thesis citation information search screen. 3) It is necessary to add a description of what exactly the UCI information means to users. 4) It is necessary to consider the method of integrating KCI and interface. 5) In the case of overseas listing classification, SCI and SCIE are integrated in WoS and provided as SCIE. Therefore, it is necessary to modify the constraints. 6) It is necessary to switch to an independent tab.

As shown in the following Table 6, the KJC service was evaluated at 73.44, and the KCI service was evaluated at 64.38. If the SUS conversion score was 63 or higher, the usability of the service to be evaluated was judged to be excellent, and the usability of the KJC and KCI services was evaluated as good. The following Table 4 shows the SUS score for each KJC evaluation participant and the average for each item. Table 5 shows the SUS score for each KCI evaluation participant and the average for each item. In each table, R is a researcher, S is an academic society, P is a publisher (information distribution agency), and F is an evaluator in the research paper support agency (policy agency). Among the collected SUS evaluation tables, 6 points were marked on item 3 of KJC of the S1 evaluator, but it was confirmed that the highest point 5 was incorrectly marked, and the code was corrected to 5 points. Table 6 shows the SUS conversion scores of KJC and KCI.

The following Table 7 summarizes other opinions of evaluators on KJC and KCI services. The opinions of the evaluators are divided into interfaces, new functions, and others.

For the KJC improvement proposal, all improvement items were evaluated with 4 points or more, showing a strong positive overall. Regarding the improvement of KJC error, overall evaluation was 4.4 or higher, showing strong affirmation. For the KCI improvement proposal, all improvement items were evaluated with a score of 3.5 or higher, showing positive overall. Regarding the improvement of the KCI error, the overall evaluation was 4 points or more, showing strong affirmation. Table 8 shows the score and average of each evaluation committee member for the KCI improvement proposal. Table 9 shows the score and average of each evaluation committee member for the improvement of KCI error. The 'No.' field presented in each table is the identification number of the improvement proposal and error improvement.

Fig. 2 below shows a part of the analysis result of KJC service function development. It shows that it is possible to connect to the “Research Trend” service of KCI through the Article menu of KJC service. Through this, it is possible to link with services such as thesis with the highest number of citations and the researcher's h-index Top 100.

Fig. 2 
KJC functional development diagram

Fig. 3 below shows a part of the analysis result of KCI service function development. It shows that it is possible to expand the service from the search results of institution information to research trends by research field, and to expand the service from the search results of academic conferences to detailed institution information services. This shows that it is possible to expand the service to the institution's website and academic journal details (quotation details).

Fig. 3 
KCI's functional development (thesis search, academic journal search, institutional information search)

The following Table 10 shows KJC and KCI service linkage and extension proposals. Source Service represents the location of a service that needs to be linked and extended. Target Sercie represents the location of the target service to be linked and extended. In the case of the KJC service, a total of six links and extensions were proposed, including the extension of the journal citation index, and in the case of the KCI service, eight links and extensions were proposed, including the extension functions commonly required in the detailed page of the journal.