Online First

According to Kostoff (1995, p. 869), in research sponsoring organizations, the selection of research programs is based on the quality of science and potential contribution to the organization's mission. In other words, funders expect outcomes and impact from research projects. In the last two decades, the shift in research evaluation has moved from output to outcome and impact. However, measuring outcome and impact of research is not as easy as measuring output, and it is a challenging task. Several methods have been used for the research impact assessment and one can find a good review of them in Boaz, Fitzpatrick, and Shaw (2008). Also several frameworks have been developed for research impact assessment, especially in health sciences (Brutscher, Wooding, & Grant, 2008; Marjanovic, Hanney, & Wooding, 2009). However, most of these methods and frameworks are mainly based on qualitative methods and data such as interviews and so on. This makes implementing these methods and frameworks expensive and difficult. As a result, scientometricians focus on quantitative metrics such as citation‐based metrics as proxies for impact. For example if an article is cited in another article we can assume that the cited article has had some scientific impact, or if it is cited in a legislative or government document, it has had impact on policy‐making or legislation. However, citation is limited to specific types of use (research purposes, mainly) and specific types of users (authors only).

With the growth of Web 2.0 and social media, altmetrics has emerged as an alternative set of metrics. Some of these alternative metrics include numbers of social bookmarks, usage data or numbers of downloads, numbers of links received, and numbers of microblogging instances (e.g. Twitter).

One of these metrics is the number of times a journal article is mentioned or cited in weblog posts. The assumption is that the more people write about an article in their blogs, the more likely that article is to be popular, interesting, and probably useful at least for the community to which bloggers belong. This could be taken as an indication or proxy for the societal impact of a study. Societal impact or benefits refers “to the contribution of research to the social capital of a nation, in stimulating new approaches to social issues, or in informing public debate and policy‐making” (Bornmann, 2012, p. 673). Therefore, one can argue that if blog posts use research studies to help promote and enhance the discussion of social issues and public debates, they are increasing the societal impact of those studies.

During the last few years an increasing number of studies (Priem, Piwowar, & Hemminger, 2011; Priem et al., 2012) have dealt with altmetrics. However, few studies have dealt with the context in which articles are mentioned or cited in blogs and the actual meaning of a citation to an article in a blog post. The only content analysis study in this area is the study by Shema, Bar‐Ilan, and Thelwall (2014b) on health research blogs.

This article aims to analyze motivations behind social sciences blog posts citing journal articles in order to find out whether blog citations of scholarly journal articles are good indicators for the societal impact of research. We chose social sciences because, according to Costas, Zahedi, and Wouters (2014), publications from the social sciences, humanities, and the medical and life sciences show the highest presence of altmetrics, indicating their potential value and interest for these fields. Additionally, research in social science is supposed to have an impact on and be beneficial for society as society is the main subject of study in social sciences. With the aid of content analysis of blog posts citing journal articles, we hope to shed some light on the context in which journal articles are used in weblogs. Specifically, we seek to answer the following questions:

∙ What are the motivations for social sciences blog posts citing journal articles?
∙ What journal titles are frequently cited in social sciences blog posts?
∙ What are the subject categories of journals cited in social science blogs?
∙ What is the average age of the articles cited in social sciences blogs?

Blogs, as a kind of social media, are used by researchers to support the research lifecycle (Nicholas & Rowlands, 2011). Blogs were studied as a source of information before they were considered a source of data for altmetrics. For instance Bar‐Ilan (Bar‐Ilan, 2005) studied library and information science blogs as sources of information, and Kim (2009) did a content analysis of 485 cancer blog posts to see if they are good sources of information for patients.

After blogs and microblogs were introduced as a source of data for altmetrics, there were a few quantitative studies to show if they correlate with the well‐established citation metrics. Eysenbach (2011) studied tweets citing medical articles and showed that tweets can predict highly cited articles within the first 3 days of article publication. He maintained that social media activity either increases citations or reflects the underlying qualities of the article that also predict citations, but the true use of these metrics is to measure the distinct concept of social impact. Shuai, Pepe, and Bollen (2012) studied the correlation between citations, downloads, and Twitter mentions for a set of arXiv preprints. They found that the volume of Twitter mentions was statistically correlated with arXiv downloads and early citations just months after the publication of a preprint, with a possible bias that favors highly mentioned articles. Thelwall et al. (2013a) compared 11 altmetrics (including blog mentions) with Web of Science citations for a set of PubMed articles. Their results provided strong evidence that six of the eleven altmetrics (tweets, Facebook wall posts, research highlights, blog mentions, mainstream media mentions, and forum posts) associate with citation counts, at least in medical and biological sciences and for articles with at least one altmetric mention. However, they admitted that their methods did not shed light on the magnitude of any correlation between the altmetrics and citations (i.e., the correlation effect size was unknown). They also concluded that the coverage of all the altmetrics except for Twitter seems to be low and so it is not clear if they are prevalent enough to be useful in practice.

Shema, Bar‐Ilan, and Thelwall (2014a) showed a correlation between blog citations and future citations. An analysis of blog posts aggregated at ResearchBlogging.org, which cited peer‐reviewed articles published in 2009 and 2010 revealed statistically significant evidence that articles receiving blog citations close to their publication time receive more journal citations later on than the articles in the same journal published in the same year that did not receive such blog citations. Another study on the correlation of altmetrics and citations (Costas et al., 2014) confirmed previous claims of positive correlations but relatively weak, thus supporting the idea that altmetrics do not reflect the same concept of impact as citations. They also maintained that altmetric counts do not always present a better filtering of highly cited publications than journal citation scores. Altmetrics scores (particularly mentions in blogs) are able to identify highly cited publications with higher levels of precision than journal citation scores (JCS), but they have a lower level of recall. Finally, a recent meta‐analysis of altmetric studies by Bornmann (2015) revealed that the correlation with traditional citations for micro‐blogging counts is negligible (pooled r = 0.003) and for blog counts it is small (pooled r = 0.12).

Once the correlation of blog citations and traditional metrics were partially confirmed, some studies focused on the context in which a journal article is mentioned or cited in blogs and microblogs. Analysis of a sample of 270 tweets by Thelwall et al., (2013b) showed that 42% of the tweets echoed an article title and another 41% presented a brief summary. One reason for summarising an article seemed to be to translate it for a general audience. Few tweets explicitly praised an article and none were critical. Most tweets did not directly refer to the article author, but some did, and others were clearly self‐citations. They maintained that “tweets containing links to scholarly articles generally provide little more than publicity, and so whilst tweet counts may provide evidence of the popularity of an article, the contents of the tweets themselves are unlikely to give deep insights into scientists' reactions to publications, except perhaps in special cases”. Shema et al. (2014b) used content analysis to discover health research bloggers’ motivations for citing journal articles. A random sample of 391 blog posts from ResearchBlogging.org was analyzed using a coding scheme they developed which included ten categories (discussion, criticism, advice, trigger, extensions, self, controversy, data, ethics, and other). Multidisciplinary and general medical journals were the main publications referred. Few self‐citations were identified, and 90% of posts provided a general report of the research cited. About 30% of bloggers also gave health advices. The same number tended to criticize findings of studies cited. Results could be interpreted as the motivation for widening health debates and popularizing medical issues on the one hand, and on the other hand, gaining more readers.

Past studies also tell us about the characteristics of research blogs and bloggers. Shema, Bar‐Ilan, and Thelwall (2012) studied 135 bloggers of ResearchBlogging.org and their blogs. The study revealed that Life Science blogs were the most popular in the sample, followed by the Psychology, Psychiatry, and Neurosciences & Behavioral Science blogs. Blogs about Social Sciences, Humanities, Computer Science, and Engineering were the least represented in the sample. About two‐thirds of the blogs had one male author, 18% had one female author, 5% had two male authors, and 4% had one female and one male author. Out of the 126 blogs in the sample, ninety (72%) had at least one active, public Twitter account. References in blog posts included 913 references to articles of 429 journals, 9 references to articles uploaded to arxiv.org, 3 references to conference proceedings, and 2 references to books. Science, Nature, and PNAS were the most cited journals. The science bloggers were highly educated. In general, most of the bloggers (59%) were either students or researchers in an academic institute. Less than a third (30%) were not affiliated with an academic institute, and 10% remained unknown. Holmberg and Thelwall (2014) studied tweeting behavior of researchers in ten disciplines and found subject differences. For example researchers in digital humanities and cognitive science used Twitter more for conversations, while researchers in economics shared the most links. Another study on tweeting behavior by Haustein et al., (2014a) focused on 37 astrophysicists and showed that different user groups had different tweeting behaviors.

Overall, the number of content analysis and qualitative studies on research blogs are limited and only one study (Shema et al., 2014b) exists in this area. Most of the altmetric studies also focus on health or basic sciences and rarely deal with social sciences.

ResearchBlogging.org (launched in 2007) is an aggregator of blog posts referencing peer‐reviewed research in a structured manner, similar to the references in scholarly discourse. Research bloggers can register their blogs with ResearchBlogging, however, editors of ResearchBlogging make sure that the posts of participating blogs are about research and are not commercial in nature. ResearchBlogging shows a snippet of blog posts with the cited articles and readers can click and read the full post on the blog sites. Blogs registered in ResearchBlogging are categorized by topic (e.g. social sciences, health etc.)

We used the advanced search of ResearchBlogging, restricting the topic to Social Sciences, and searched for all blog posts between 01/01/2012 to 18/06/2014. This time frame was chosen because we wanted to study recent blog posts, and therefore, we chose a 30‐month time frame up to the time of data collection, which was mid‐2014. In, total 1,233 blog posts were retrieved. The results were presented in 62 pages (20 posts in each page). Using the Cochrane (1977) formula, the sample size with 95% confidence level and confidence interval of 5 was calculated to be 297 blog posts. However we made the number round and added 3 to the needed sample size to make it 300. A systematic random sampling method was used. One of the authors went through all pages of the results and chose every fourth blog post and collected the information of 300 blog posts including URL of the post, cited references, blog title, and post date. The 300 posts belonged to 103 unique blogs. Using the URL of the post, every single blog post was visited for further data collection and content analysis. The dates of all cited articles were recorded along with the cited journal titles. Scopus second level subject categories were used to check the subject of the cited journals.

Citation motivations can be done using a few different methods including surveys of authors (e.g. Kapseon, 2004), critical incident interviews (e.g. Thornley et al., 2015), and content analysis of citing documents (e.g. Case and Higgins, 2000). A review of citation motivation studies (Tohidinasab & Jamali, 2013) has shown that content analysis is a common method for such studies. This study uses content analysis, especially as it is difficult to access bloggers located around the world for survey or interview purposes.

For content analysis we used the categories in the coding scheme developed by Shema et al.(2014b) as the base. However, since their scheme was developed for analyzing health blogs and was partly inapplicable to social sciences blogs, we inductively created a few other categories. For example, we did not find any post related to the ‘ethics’ category or any post related to ‘advocacy against a certain treatment’ motivation. Therefore, we did not use these. Instead we came across motivations such as introducing or criticizing a book, and so we added these to the original schema. The lead author did the content analysis and the second author also coded 30 blog posts (10% of the sample) to check the inter‐coder reliability which turned out to be 83%. The measure for inter‐coder reliability was the similarity and overlap between the two coders’ analyses.

There are nine categories which include 35 motivations. To find out about the motivation for citing an article, we read all blog posts and looked at the context in which articles were cited. Some of the blog posts had a list of structured references but did not have any in‐text citation so it was not clear what the role of the specific paper was in the content of the blog post. In‐text citation did not have to be an ‘Author‐Date’ sign of the reference. Some bloggers used phrases such as ‘a new study…’ or ‘a recent article…’ in the post as a reference to the paper in their reference list. There were 14 of those references. It should be noted that some of the blog posts could be categorized under more than one category, but only the main category was used in the analysis. The nine categories and sub‐categories (motivations) are presented below:

• ∙ Discussion : consideration and examination of an issue.
  • ∘ Discussing a social phenomenon
  • ∘ Praising and giving detailed description of an article
  • ∘ Review of current knowledge about a subject
• ∙ Criticism : finding fault with a research‐related issue.
  • ∘ Disputing a common belief
  • ∘ Raising methodological issues
  • ∘ Overall criticism
  • ∘ Criticism of a practice
  • ∘ Criticism of a book
  • ∘ Criticism of a third work
  • ∘ Criticism of a public policy
  • ∘ Criticizing an article's conclusion/findings
  • ∘ Criticizing an article for missing references
  • ∘ Criticizing an article for not sharing data
• ∙ Advice : recommending actions for the readers.
  • ∘ Providing advice
  • ∘ Providing a solution to a social problem
  • ∘ Suggesting policy
  • ∘ Recommending a behavior
• ∙ Trigger : a direct stimulus for post writing that was mentioned in the post.
  • ∘ Neutral detailed description of an article
  • ∘ Reaction to another blog post
  • ∘ Reaction to a topic in news/media
  • ∘ Announcing the publication of a new article
  • ∘ Invitation for further discussion in the comments
  • ∘ Answering a common question
  • ∘ Introducing a book
• ∙ Extension : suggesting possibilities beyond the post’s scope.
  • ∘ Suggesting further reading or other material
  • ∘ Suggestions for possible future research
• ∙ Self : the bloggers added post content which was specifically related to them.
  • ∘ Self‐citation
  • ∘ Sharing personal experience
  • ∘ Used as the basis of own work
• ∙ Controversy : discussing controversy; explaining and/or discussing disagreements.
• ∙ Data : providing data and facts with practical implications; background factual information about the blog post subject.
• ∙ Unclear : the motivation for citation was not clear.

In this study we only analyzed structured references, i.e. references that are presented in a structured manner similar to the references in journal articles. We did not include unstructured references (also called blog mentions) in the study.

From 300 blog posts, 145 (48%) belonged to 2012, and 39 (13%) posts were from 2014. The lower number of posts from 2014 was because we only included the first six months of 2014 in our analysis (See Figure 1.) The 300 blog posts had 472 references (on average, 1.57 references per blog post). Links to other blog posts or news websites or other general websites were not included in this count. From 472 references, 424 were journal articles (on average, 1.41 references to journal articles per blog post). The rest of references were books (35), book chapters (5), theses (3), conference papers (4), and e‐print (1).

Fig. 1.  
Number of blog posts analyzed by publication year

4.2. Cited journals

The 424 journal articles cited in blogs appeared in 269 unique journal titles. The list of journal titles with the frequency of 4 or more is presented in Table 2. It should be noted that higher frequency does not always mean diverse usage of that journal in blog posts, since one post might cite several articles of a single journal. For instance, all four references to the ‘Australian Veterinary Journal’ were in a single blog post. The subject category of journals is presented in the last column and it is clear that none of the top 13 journals are from the category of Social Sciences.

The content analysis showed that although social science blogs cite journal articles from domains such as medicine or neuroscience, they are cited because the research presented in those articles has social aspect. The list of Social Sciences journals with the frequency of two or more is given in Table 3.

Figure 2 shows the cumulative percentage of unique journals (green line) and the cumulative percentage of citations (blue line). The horizontal axis represents the number of references to journals. From 269 journals, 203 journals (75.5% of all journals) were referenced only once, 35 journals (13%) were referenced twice, and 18 journals (6.7%) appeared among the references three times. In other words, journals with 2 or more references accounted for 24.5% of all cited journals and 87.7% of references. Most of the Social Sciences journals were among those journals that have been cited only once.

Table 2.  
List of journals cited 4 or more times in the blog posts

	Journal title	Frequency	%	Subject category
1	PLoS ONE	19	4.48	Agricultural and Biological Sciences
2	PNAS	18	4.24	General
3	Science	9	2.12	General
4	Animal Behavior	7	1.65	Agricultural and Biological Sciences
5	Nature	7	1.65	General
6	Psychological Science	7	1.65	Psychology
7	Journal of Human Evolution	5	1.18	Agricultural and Biological Sciences
8	Journal of The Interactions of People & Animals	5	1.18	Agricultural and Biological Sciences
9	Applied Animal Behavior Science	4	0.94	Veterinary
10	Australian Veterinary Journal	4	0.94	Veterinary
11	New England Journal of Medicine	4	0.94	Medicine
12	Philosophical Transactions of the Royal Society B: Biological Sciences	4	0.94	Biochemistry, Genetics and Molecular Biology
13	Trends in Cognitive Sciences	4	0.94	Neuroscience

Table 3.  
Social Sciences journals cited in blog posts

	Journal title	Frequency	%
1	Current Anthropology	3	0.71
2	Public Understanding of Science	3	0.71
3	American Anthropologist	2	0.47
4	Death Studies	2	0.47
5	Evolutionary Anthropology: Issues, News, and Reviews	2	0.47
6	Journal of Politics	2	0.47
7	Tobacco Control	2	0.47

Fig. 2.  
Percentage of journal titles and references by number of references to journals

4.4. Age of cited articles

In terms of the age of the cited articles, the oldest cited paper was published in 1951. Table 5 shows the distribution of the cited articles by their publication year for blog posts 2012‐2014. Figure 3 shows all of the cited papers since 1951 by their publication year.

Taking into account the time period between the date of blog posts and the publication year of cited articles, the age of each reference was calculated and on average, references were 5.8 years old. Removing outliers (papers published during the 1950s and 1960s), the average age was 5.27. In this calculation, if a blog post in 2012 cited a paper published in 2012, the age was zero and if the cited paper was published in 2011, the age was 1 year. If the cited paper was 2013 (as some journals are published ahead of time) the age was ‐1. There were 8 references with the age of minus one (‐1) (2012 blog posts that cited papers published in 2013). There were also 150 cases where blog posts and cited articles were both from the same year. 13.7% of articles cited in blogs were at least 14 years old (this figure was 7% for health blogs (Shema et al., 2014b).

Table 5.  
Blog post references by cited article publication year (rows) and blog post publication year (columns)

2012			2013			2014
Year	n	%	Year	n	%	Year	n	%
2013	8	3.69	2014	0	0	2015	0	0
2012	66	30.41	2013	52	33.99	2014	19	35.19
2011	34	15.67	2012	25	16.34	2013	4	7.41
2010	17	7.83	2011	10	6.54	2012	5	9.26
2009	13	5.99	2010	6	3.92	2011	2	3.70
2008	11	5.07	2009	10	6.54	2010	3	5.56
2006‐2007	12	5.53	2007‐2008	12	7.84	2009	4	7.41
2004‐2005	6	2.76	2005‐2006	8	5.23	2004‐2008	6	11.11
2000‐2003	15	6.91	2000‐2004	12	7.84	2000‐2004	4	7.41
1996‐1999	10	4.61	1996‐1999	5	3.27	1996‐1999	1	1.85
1990‐1995	7	3.23	1990‐1995	5	3.27	1990‐1995	3	5.56
1980‐1989	6	2.76	1980‐1989	2	1.31	1980‐1989	2	3.70
1971‐1979	8	3.69	1971‐1979	5	3.27	1971‐1979	1	1.85
1955‐1970	4	1.84	1951‐1970	1	0.65	1951‐1970	0	0
Total	217	100		153	100		54	100

Fig. 3.  
Number of articles cited by publication year

This article was a content analysis of a sample of 300 posts (2012‐2014) from 103 blogs registered as social sciences blogs in ResearchBlogging.org. Going back to the main aim of the study, we can conclude that blogs can indeed serve as proxies for the societal impact of research. However, their limitations and context should be taken into account. Looking at the bloggers’ motivations for citing journal articles (Table 1, above), one can see that a considerable number of them serve purposes that could be linked to societal benefits. As Bornmann (2012) defined it, societal impact is partly about stimulating new approaches to social issues, or informing public debate and policymaking. Discussion and criticism were the two main categories of motivations. Motivations such as disputing a belief, suggesting policies, providing a solution to a problem, reacting to media, criticism and the like, all mean that blogs serve that definition of societal benefits. Moreover, the discourse that takes place on blogs as a form of social media also helps improve the societal benefits of research.

The other reason blogs are potentially useful as an indicator of research impact is that unlike some other altmetric measures where users do not engage in any form of intellectual process and simply do activities such as clicking, liking, linking, or bookmarking, bloggers engage in an intellectual process, i.e. generating content. Even tweets, although including contents, have not proved to be very useful as a data source for altmetrics (Thelwall et al., 2013b). However, since blog posts are usually longer and providing a context, they are more helpful as a source of data for altmetrics. Moreover, the low rate of self‐citation makes blogs even more attractive as a proxy measure for research impact. We also know from past studies (Groth & Gurney, 2010) that scientific discourse on blogs is more immediate, contextually relevant, and has a larger non‐technical focus than the academic literature.

Comparing the motivations for citations in blogs with the citation motivations of authors of journal articles (of which a review and classification was presented in Tohidinasab and Jamali (2013)) one can see the differences are probably more than the similarities. The reason might be that the objective and function of a blog post is different from that of a journal article. Blogs are mainly discussion about others’ works while articles are meant to present authors’ own research. These differences imply that citations in blogs and in journal articles capture and manifest two different kinds of research impact and are complementary to each other. Citation in journal articles are considered as signs of scientific impact of research as they show the benefit one paper has had for the scientific community. We argued that blog citations, on the other hand, could show societal impact.

However, as mentioned above, limitations and contextual issues should not be ignored when using blogs as a data source for altmetrics. One issue is disciplinary differences. Comparing the findings about social sciences blogs with those of health blogs (Shema et al., 2014b), some subject differences are revealed. For instance, self‐citation is lower in social science blogs (1%) than in health blogs (3.1%); the number of posts including a general discussion of the issue covered in the article is much lower in social sciences compared to health sciences blogs; books are more frequently cited in social sciences blogs than in health blogs; and on average, articles cited in social sciences are older compared to health blogs. Other studies (e.g. Holmberg & Thelwall, 2014) have also confirmed the existence of subject differences.

Another issue is that altmetrics themselves are still in their infancy. In spite of the increasing number of studies on altmetrics, the scientific evidence about their reliability and validity is not yet robust. Previous studies such as those by Thelwall et al. (2013b), Costas, Zahedi, and Wouters (2014), and Haustein et al., (2014b) generally confirmed low uptake of altmetrics in scholarly publications. Therefore, any use of blogs (and all other altmetric data) for research assessment purposes should be done cautiously, until they become prevalent enough to be useful in practice.

Overall, social science blogs are rather subject‐oriented than article oriented. This means a considerable number of blog posts are not driven or triggered simply by writing about an article. In other words, similar to references in journal articles, bloggers write about their subject of interest and in that process use and cite articles they find relevant or helpful for their discourse. Although, some of the blog posts are triggered by the publication of new articles, this is not the case in many posts. The fact that the average age of articles cited in blogs are 5.8 years confirms this.

Finally, a word of caution about the limitations of this study, and future directions. One limitation was that the sample included only English blogs, and only blogs that are registered in ResearchBlogging. This might not be representative of all research‐related blogs in social sciences. ResearchBlogging is very limited in nature, and studies should also be done on other blogs. For example, it would be helpful to see how research is reflected in blogs by non‐researchers and the public.

We did not do any systematic analysis of the characteristics of journal articles cited in blog posts. However, while reading the list of the articles it was clear that at least most of the article titles sounded attractive. Titles such as ‘The Neurobiology of Love’, ‘Has the tobacco industry evaded the FDA’s ban on Light cigarette descriptors?’, ‘You see, the ends don’t justify the means: visual imagery and moral judgment’, and ‘The illusion of saving face: how people symbolically cope with embarrassment’. Characteristics of article titles cited in blogs should be studied to see if they are any different from those not cited in blogs. Article title typing presented in Jamali and Nikzad (2011) might be helpful for this matter. Surveys also can be done to find out on what basis bloggers decide to blog about an article or to cite one.