Online First

When compared with general information, healthcare information needs to be more reliable and readable as the information is actually used for disease management and health promotion. Frequently asked questions (FAQs) in healthcare provide such information as they are often written and compiled by healthcare professionals in response to specific questions of healthcare consumers. The healthcare consumers thus tend to read those web pages that have healthcare FAQs (Sillence et al., 2004), and many healthcare information providers have collected and maintained a large number of healthcare FAQs for healthcare consumers.

Therefore, given a healthcare question as a query, the retrieval of relevant FAQs is essential for the utility of the reliable and readable healthcare information in the FAQs. Given a database of FAQs and a query, a FAQ retriever ranks the FAQs based on the relevancy of the FAQs to the query. The FAQ retrieval task is challenging as both the query and the FAQs are often quite short, making it difficult to collect helpful evidences to identify relevant FAQs.

A FAQ consists of a question part and an answer part. Several previous techniques considered both parts for FAQ retrieval (e.g., Wu, Yeh, & Chen, 2005; Wu, Yeh, & Lai, 2006; Xue, Jeon, & Croft, 2008), and in some cases other information concerning the FAQ was employed as well (e.g., the content of the web page containing the FAQ, Jijkoun & de Rijke, 2005). In this paper, ECA focuses on the question part, which is the most important part in FAQ retrieval (Jeon, Croft, & Lee, 2005). It provides the conceptual similarity scores between the question parts of queries and FAQs. ECA can thus be integrated with the previous techniques by integrating the conceptual similarity produced by ECA with the similarity estimated based on other parts of FAQs (e.g., the answer parts of the FAQs and the webpage containing the FAQs).

Given q as a query (question) and f as the question part of a FAQ, previous FAQ retrievers estimated the similarity between q and f by two typical methodologies: term matching and semantic analysis. Term matching methods often considered four types of information for the similarity estimation: (1) overlap of the words in q and f (i.e., q and f may be more similar to each other if many words co-occur in them, Bernhard and Gurevych, 2008), (2) cosine similarity based on the vectors of q and f (Bernhard & Gurevych, 2008; Burke et al., 1997; Wu et al., 2005), (3) relatedness of the words in q and f (e.g., measured by the distance of the words on an ontology, Burke et al., 1997; Wu et al., 2005; Wu et al., 2006, or the handling of the spelling errors of words in q, Bernhard & Gurevych, 2008; Contractor et al., 2010; Kothari et al., 2009), and (4) mapping or translation between the words in q and f (to tackle the problem of word mismatch between q and f, Jeon et al., 2005; Lee et al., 2008b; Riezler et al., 2007; Xue et al., 2008). However, none of the term matching methods considered essential concepts (i.e., event, condition, and aspect) in Chinese healthcare FAQs. ECA considers the essential concepts, which actually indicate a main part of semantics of the healthcare FAQs.

Semantic analysis was thus noted as a methodology for FAQ retrieval as well. However, previous semantic analysis methods were not aimed at the retrieval of Chinese healthcare FAQs. They often aimed at typical kinds of semantic information, including (1) types of questions (e.g., typical types include “what,” “how,” and “where,” Wu et al., 2005), (2) semantic categories of questions (Mishra, Mishra, & Sharma, 2013; Pan et al., 2008), and (3) syntactic or semantic structures of questions (by deeper analysis such as parsing, Casellas et al., 2007; Wang, Ming, & Chua, 2009; Wu et al., 2006; Winiwarter, 2000).

The first kind of semantic information (i.e., the question type of a FAQ) cannot indicate the essential concepts of the FAQ, and more importantly many Chinese healthcare FAQs cannot fall into any specific question types. For example, among thousands of healthcare FAQs in KingNet¹⁾ (a provider of Chinese healthcare information) we find that it is quite difficult to determine the question types of many FAQs, such as “失眠的原因與診斷” (the cause and diagnosis of insomnia). Some FAQs, such as “安樂死” (euthanasia), even consist of a single health topic without any question words.

On the other hand, the second and the third kinds of semantic information (semantic categories and syntactic and semantic structures of FAQs) are quite difficult to recognize from Chinese healthcare FAQs. To recognize the two kinds of semantic information, the previous studies often employed parsing. However, a good parser for Chinese healthcare queries is often unavailable since (1) parsing Chinese questions is still a challenging task (Lee et al., 2008a), and (2) Chinese healthcare questions are not always well-formed for parsing: they may even consist of a single term (e.g., a disease name or a treatment), multiple sentences or multiple fragments.²⁾ It is thus difficult to get the semantic structures of healthcare FAQs by parsing. ECA is concerned with the existence of essential concepts in a question rather than the semantic structure of the question, making it able to estimate the conceptual similarities without relying on parsing.

A conceptual model for the retrieval of Chinese healthcare FAQs should be both expressive (able to indicate the core intention of a healthcare question) and realizable (able to be used for similarity estimation without assuming the well-formedness of the question). To develop such an expressive and realizable conceptual model, we analyze a large number of Chinese healthcare FAQs and find that most of the FAQs have the core intentions about health promotion and disease management. This is because healthcare questions are asked by general healthcare consumers, who mainly care about the way to keep them healthy. This is also a reason why healthcare information providers for the consumers mainly focus on the topics about diseases and health for healthcare consumers. We thus identify three types of essential concepts in Chinese healthcare FAQs: (1) event: the target event under discussion, (2) condition: the condition of the discussion, and (3) aspect: the information aspect of the discussion. The core intention of a Chinese healthcare question is to ask for some aspect of information (e.g., cause) about target events (e.g., cardiovascular disease) under a certain condition (e.g., patients of the periodontal disease).

More specifically, Table 1 defines the three types of essential concepts. As healthcare questions are often quite short and specific to one to two target events, we consider at most two events E1 and E2 in the conceptual model. All concepts governing the condition of the discussion are the condition concept (C). To facilitate the recognition of the aspect concept (A), we define three high-level categories of aspects: A_cause, A_process, and A_diagnosis that are about the causal, processing, and diagnosis aspects of the health topics respectively.³⁾ The three aspects were noted as the key concepts to retrieve documents for clinical questions posted by healthcare professionals (Lin and Demner-Fushman, 2006). ECA is the first framework aiming at the conceptual retrieval of FAQs (rather than documents) for healthcare consumers (rather than professionals).

As an example, consider a FAQ from KingNet⁴⁾: “兒童常吃山藥會不會引發性早熟” (For children, will frequently eating yams   cause   precocious   puberty?). There are two event concepts: E1=山藥 (yams) and E2 = 性早熟 (precocious puberty), a condition concept: C = 兒童 (children), and an aspect concept: A = {A_cause}, which is determined based on the term 引發 (cause). It is interesting to note that a healthcare FAQ does not necessarily have all types of concepts, but it should have at least one concept, which is E1 and in that case the FAQ may be simply a term about a health topic (e.g., the name of a disease) and hence its intention is asking for all kinds of information about the topic (e.g., all information about a disease).

Based on the conceptual model, ECA is developed to provide the conceptual similarity between each query and each FAQ. As illustrated in Figure 1, ECA has two phases of tasks: (1) the offline phase for recognizing the essential concepts (event, condition, and aspect) in each FAQ, and (2) the online phase for receiving a query q and estimating the similarity between q and each FAQ. The similarity scores of q with respect to the FAQs can be used to improve the ranking performance of various kinds of FAQ retrievers.

4.1. Offline Tasks: Recognition of Essential Concepts in FAQs

Among the three types of essential concepts, event concepts are the most unrestricted, since they can be whatever health topics. Aspect concepts are the most restricted, since they can only be three categories of interest: causal, processing, and diagnosis. Therefore, the three types of essential concepts in a FAQ should be recognized by following the sequence: aspect concepts → condition concepts → event concepts. More specifically, given f as the question part of a FAQ, ECA first recognizes the aspect concepts in f (A1 in Figure 1), and then recognizes the condition concepts from f with the aspect strings removed (A2 in Figure 1). Finally ECA recognizes the event concepts from f with both the aspect strings and the condition strings removed (A3 in Figure 1).

Fig. 1. 
ECA consists of two phases of tasks: the offline phase and the online phase. The former recognizes the concepts in each FAQ (f), and given a query (q) the latter measures the similarity between q and each FAQ.

4.1.1. Recognition of the aspect concepts in a FAQ (A1 in Figure 1)

As information aspects are actually categories of interest, an aspect may be indicated by many different terms. We thus analyze the typical terms for A_cause, A_process, and A_diagnosis, and accordingly construct TA_cause, TA_process, and TA_diagnosis as their sets of corresponding terms respectively. The three sets have 74, 124, and 35 terms respectively. Example terms in TA_cause include “危險因子” (risk factor) and “引發” (incur). Example terms in TA_process include “用藥” (medication) and “抑制” (inhibit). Example terms in TA_diagnosis include “症狀” (symptom) and “檢驗” (examination and test). Obviously, the development of the sets of aspect terms is a knowledge engineering task. In empirical evaluation, we will show that the performance of ECA is robust under different settings for the sets of aspect terms (ref. Section 5.4.2).

With the three sets of typical terms for the three aspects, ECA recognizes the aspects of f (the question part of a FAQ) by checking whether f mentions the terms. More specifically, Equation 1 defines a fuzzy term matching method to estimate the similarity (StrSim) between two strings t₁ and t₂, where idf (w) is the inverse document frequency⁵⁾ of the Chinese character w. StrSim is higher if there are more matched characters with higher idf values.

StrSim(t1,t2)=∑y∊t1andy∊t2idf(y)∑z∊t1orz∊t2idf(z)×log2∣{y∣y∈t1andy∈t2}∣

(1)

Based on Equation 1, the strengths of classifying a FAQ f into A_cause, A_process, and A_diagnosis are estimated by Equation 2 ~ Equation 4, respectively.

ASF(f,Acause)=maxt∈TAcausemaxc∈f{StrSim(t,c)}

(2)

ASF(f,Aprocess)=maxt∈TAprocessmaxc∈f{StrSim(t,c)}

(3)

ASF(f,Adiagnosis)=maxt∈TAdiagnosismaxc∈f{StrSim(t,c)}

(4)

We employ 0.5 as the threshold for the strength: f is classified into an aspect only if its strength corresponding to the aspect is higher than 0.5. It is interesting to note that, if f is not classified into any aspect (e.g., f may simply consist of a single term about a health topic), a ‘don’t-care’ is assigned to each aspect for f, indicating that f does not ask for any specific aspect and hence all aspects might be related to the intention of f.

4.1.2. Recognition of the condition concepts in a FAQ (A2 in Figure 1)

By analyzing a large number of Chinese healthcare FAQs, we find that the FAQs often employ the description of time and people to indicate the context of discussion. Therefore, ECA employs pattern matching to extract the string that indicates the condition concepts in f. We define 21 patterns for specific concepts for time and people:

(5)  “春天”(spring) || “冬天”(winter) || “夏天”(summer) ||
“秋天”(autumn) || “春季”(spring) || “冬季”(winter) || “夏季”(summer) ||
“秋季”(autumn) || “男人”(male) || “女人”(female) || “小孩”(child) ||
“老人”(old people) || “兒童”(child) || “嬰兒”(infant) ||
“幼兒”(little child) || “孕婦”(pregnant woman) || “小朋友”(child) ||
“寶寶”(baby) || “男性”(male) ||“女性”(female) || “父母”(parent)

The strings (in f) that match the patterns are extracted as the strings that indicate the condition concepts in f. After applying the above patterns to f, the following 7 patterns with wildcards are applied to extract other possible condition concepts, (‘*’ denotes any string in f ):

(6)  “*期”(period) || “*時”(when) || “*後(after) || “*前”(before) ||
“*族”(some group) || “*員”(somebody) || “*者”(somebody)

Obviously, the development of the condition patterns is a knowledge engineering task. In empirical evaluation, we will show that the performance of ECA is robust under different settings for the patterns (ref. Section 5.4.2).

4.1.3. Recognition of the event concept in a FAQ (A3 in Figure 1)

After removing from f the strings about the aspects and conditions in f, ECA gets a set of separate strings, which can be treated as “string islands” as they are not consecutive strings. Punctuations are then removed to further separate the strings, and each of the resulting strings may be a candidate string that indicates an event concept in f. To further make the candidate strings more precise, 693 terms (including stop words) that are unlikely to be in event strings are defined⁶⁾. The terms are removed from the strings, and each of the resulting strings is treated as possible strings of event concepts in f. In empirical evaluation, we will also show that the performance of ECA is robust under different settings for the terms (ref. Section 5.4.2).

ECA was empirically evaluated on thousands of Chinese healthcare FAQs and queries. As the conceptual similarity (i.e., S_ECA) can indicate a kind of semantic information not considered by previous FAQ retrievers, we aimed at measuring the contribution of ECA to several kinds of the FAQ retrievers in ranking relevant healthcare FAQs.

5.4. Result and Discussion

We discuss and analyze the contribution of ECA to the ranking of relevant healthcare FAQs. Moreover, as the automatic concept recognition is a key task of ECA, we are also concerned with the performance of ECA under different settings of the knowledge for the concept recognition (i.e., patterns and terms, ref. Section 4.1.1 ~ Section 4.1.3). We investigate the performance of ECA under different amounts of the knowledge and then investigate the effect of replacing the automatic concept recognition with manual concept annotation.

5.4.1. Ranking of Relevant Healthcare FAQs

Figure 2 shows the ranking performance of each underlying retriever before and after it is integrated with ECA. The results showed that the conceptual similarity information provided by ECA was helpful for all the retrievers, which have considered different kinds of similarity information employed by state-of-the-art FAQ retrievers. To verify whether the performance improvements were statistically significant, we conducted two-sided and paired t-test with 95% confidence level. The results showed that ECA helped all the retrievers to achieve significantly better performance in MAP. Among the retrievers, LM and IntALL had the best performance, and ECA significantly improved them as well.

Fig. 2. 
Results on ranking of relevant FAQs: Each retriever with ECA has better MAP performance than the retriever without ECA, and all the performance differences are statistically significant.

Table 3 shows an example to illustrate the contribution of ECA. The query q asks for the advice about whether seasonings can be added to non-staple food for infants. Among the FAQs, there is a FAQ (f1) that talks about the possible problems that should be considered when feeding non-staple food, and hence the FAQ is judged to be relevant to q. Unfortunately, both the best FAQ retrievers (LM and IntALL) make an error of preferring a non-relevant FAQ (f 2) to f 1. The non-relevant FAQ talks about the way to deal with the case where the body temperature of an infant is too low. LM and IntALL prefer the non-relevant FAQ because it shares a term “嬰兒” (infant) with q and the term has higher idf. ECA successfully gives the non-relevant FAQ a smaller similarity score (S_ECA) as e1_f2 (“體溫太低,” body temperature is too low) in the non-relevant FAQ cannot be found in q. Also note that in the example, recognition of essential concepts by ECA is not perfect, especially for the recognition of aspects (see the underlined parts in Table 3). ECA misclassifies f 2 into the cause aspect (i.e., A_cause = True) because “怎麼來” (how does it happen) is a term for the cause aspect (i.e., the term is listed in TA_cause) and the term happens to match “怎麼” (by Equation 1), which is simply a substring of “怎麼辦” (what to do) in f2 (i.e., f2 should be classified into process aspect rather than the cause aspect). Even the recognition of concepts cannot always be perfect, ECA can still provide significant contribution to the ranking of relevant healthcare FAQs. We are also exploring the possible ways to further improve the recognition of the concepts in FAQs.

Table 3. 
An example to illustrate the contribution of ECA: Given the query q, both the best FAQ retrievers (LM and IntALL) make an error of preferring the non-relevant FAQ (f 2) to the relevant one. Although the recognition of essential concepts by ECA cannot be perfect (see the underlined parts), ECA successfully gives the non-relevant FAQ a smaller similarity score (S_ECA) as e1_f2 (“體溫太低”) in the non-relevant FAQ cannot be found in q.

		Query (q): 嬰兒副食品是否可以添加調味料? (Can seasonings be added to non-staple food for infants?)
Question of the FAQ	Essential concepts recognized by ECA	Aspects in q	Events and conditions in q w.r.t. the FAQ
f1 (Relevant FAQ):餵食副食品可能會遇到的問題? (The problems that should be considered when feeding non-staple food?)	(E)e1f1 =餵食副食品 (feeding non-staple food); e2f1 =題; (C) Cf1: None; (A) Acause = Aprocess = Adiagnosis = don’t-care.	Acause = True (by “可以讓” (can cause)); Aprocess = True (by “調理” (recuperate)); Adiagnosis = False.	e1q,f1 =副食品 (non-staple food); e2q,f1: None; Cq,f1: None. → SECA = 0.7243 ( = average of the scores on e1q,f1 and aspects)
f2 (Non-relevant FAQ):嬰兒體溫太低怎麼辦? (What to do for an infant whose body temperature is too low?)	(E)e1f2=體溫太低(body temperature is too low); (C) Cf2 =嬰兒(infant); (A) Acause = True (by “怎麼來” (how does it happen); Aprocess = True (by “怎麼辦” (what to do)); Adiagnosis = False.		e1q,f2: None; e2q,f2: None; Cq,f2 =嬰兒(infant). → SECA=0 (since e1f2 is not found in q)

5.4.2. Effect of Different Knowledge Engineering Strategies for Concept Recognition

As described in Section 4.1.1 ~ Section 4.1.3, several terms and patterns are defined for ECA to recognize essential concepts in FAQs so that conceptual similarity between the FAQs and the queries can be estimated. The development of these patterns and terms is actually a knowledge engineering task, and hence we are concerned with the performance of ECA under different strategies for the knowledge engineering task.

We first investigate the robustness of ECA when less knowledge is defined for the concept recognition. We tested different degrees of the removal by randomly removing 5%, 10%, 20%, and 30% of the terms and patterns for concept recognition. The result is shown in Figure 3. It indicated that as more of the terms and patterns were removed (e.g., 30% of them are removed), MAP of ECA tended to drop slightly. However, after conducting significance tests on the performance differences, we found that most of the differences were not statistically significant. Significant differences occurred only in three cases: BM25 with 5% removal, FAQFinder with 20% removal, and IntALL with 5% removal. The result shows that ECA is robust in facing the different sets of terms and patterns for concept recognition. A detailed analysis shows that the robustness is contributed by the fuzzy term matching method (i.e., StrSim in Equation 1) that is employed by ECA to measure the similarity between each query and FAQ (ref.. Equation 8 ~ Equation 16). When less knowledge is defined to recognize the concepts in each FAQ, more errors might be incurred in concept recognition but the fuzzy string method helps to reduce the effect of the errors by producing robust similarity scores.

Fig. 3. 
Robustness of ECA under different settings of the knowledge (terms and patterns) for concept recognition: Although MAP of ECA tends to drop when a part of the knowledge is removed, most of the MAP differences are not statistically significant.

Moreover, we are also interested in the performance of ECA when the concept recognition is conducted manually (i.e., disabling the offline concept recognition tasks of ECA and replacing it with manual annotation). As noted in the example discussed above (ref. Table 3), recognition of essential concepts by ECA cannot always be perfect. Therefore, manual annotation for the essential concepts could be helpful for the ranking of relevant FAQs. Another motivation of considering the manual annotation is that the manual annotation should be both feasible and helpful for FAQ retrieval, based on two reasons: (1) concept annotation to a FAQ is conducted only once (e.g., conducted when the FAQ is edited and entered to the database), and (2) the annotation can provide higher-quality information to ECA. Therefore, following the definition of the essential concepts, each FAQ was manually annotated with essential concepts.

Figure 4 compares the performance of the two versions of ECA: ECA with automatic concept recognition and ECA with manual concept annotation for FAQs. We surprisingly found that manual annotation did not always help ECA to achieve better MAP. All performance differences between the two versions of ECA were not statistically significant. Therefore, the imperfect concept recognition by ECA did not significantly deteriorate its performance in the experiment. The result reconfirms the robustness of ECA under different strategies of concept recognition for FAQs. As the concept annotation task is the offline task of ECA, the result also suggests that future extension of ECA should be directed to the online modules of ECA (i.e., similarity measurement, Section 4.2). We are interested in extending ECA by text classification (for aspects), and otology-based and translation- based term matching (for event and condition concepts).

Fig. 4. 
Possible improvement of ECA by manual annotation of the concepts in FAQs: Manual annotation is not always helpful for ECA, and all MAP differences are not statistically significant.

ECA can be extended in two ways. The first way is to expand the conceptual model employed by ECA (ref. Table 1). Currently ECA considers three aspect categories, which are respectively about the cause, diagnosis, and process of disorders. As users (healthcare consumers) are often concerned with the information about different stages of a disorder, it would be interesting to expand the conceptual model by considering four typical stages of a disorder, including: (1) before the disorder is diagnosed, (2) when the disorder is being diagnosed, (3) after the disorder is confirmed, and (4) after the disorder is treated. Typical categories may thus include one most-general category (general description), four general categories (prevention, diagnosis, treatment, and prognosis), and seven specific categories (risk factors, symptoms and signs, lab test, homecare, medicine, mortality, and recurrence). It is thus interesting to investigate whether the retrieval of healthcare FAQs can be improved by the expanded conceptual model.

Another interesting way to extend ECA is to develop a machine learning technique to build a classifier to determine the aspect category of a given healthcare FAQ. Currently ECA determines the aspect category by string matching (ref. Section 4.1.1). Although the string matching method performs well in the experiments, it may be costly to construct and maintain a complete set of strings for each aspect category, especially when more aspect categories are considered. Therefore, given a set of healthcare FAQs that are labeled with suitable aspect categories, a machine learning technique can be developed to automatically train a classifier, without needing to manually construct and maintain the strings for each category. It is thus interesting to investigate whether the retrieval of healthcare FAQs can be improved by the machine-learning-based classifier.

It is also interesting to apply ECA to retrieval of healthcare FAQs in languages other than Chinese. ECA is based on a healthcare conceptual model, which has been shown to be helpful for healthcare FAQ retrieval. The conceptual model should be applicable to healthcare FAQs in different natural languages, since healthcare consumers are often concerned with disease management and health promotion, no matter how they express their concerns in different languages. Technically, recognition of the three kinds of concepts (i.e., event, aspect, and condition) in a FAQ should call for different techniques when the FAQ is expressed in different languages.

Healthcare FAQs are a valuable source of readable and reliable healthcare information for healthcare consumers. Given a Chinese healthcare question as a query, semantic similarities between the query and FAQs are essential for the ranking of relevant FAQs. To estimate the semantic similarities, semantic analysis on Chinese healthcare FAQs is required, however it is challenging. As healthcare FAQs are actually specific questions about health promotion and disease management, their semantic structure is quite different from those of the FAQs in other domains. Therefore, we identify three types of essential concepts (event, condition, and aspect) as the key semantic elements in Chinese healthcare FAQs, and show that based on the essential concepts, the conceptual similarities between Chinese healthcare queries and FAQs can be estimated by a technique ECA.

The conceptual similarities provided by ECA actually indicate a kind of semantic information that is not considered by previous FAQ retrievers and can serve as a supplement to the FAQ retrievers. An empirical evaluation on thousands of Chinese healthcare FAQs confirms that the similarity scores produced by ECA can be used to significantly enhance several kinds of FAQ retrievers in ranking the FAQs for the input queries. Performance of ECA is also robust under different settings of the knowledge (terms and patterns) for concept recognition. The results thus confirm the expressive power of the conceptual model and the significant contribution of ECA to the ranking of relevant healthcare FAQs. The contribution is of technical significance to the studies of FAQ retrieval in the healthcare domain. It is also of practical significance to the utility of healthcare FAQs, which provide both reliable and readable healthcare information for specific questions of healthcare consumers. We thus make ECA online (http://203.64.84.94:126/). Given two Chinese healthcare questions, the online service returns a conceptual similarity score. It can thus provide similarity estimation for various kinds of FAQ retrievers, facilitating further technical studies and practical applications in the healthcare domain.