12345672 of 14
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
On using grey literature and google scholar in systematic literature reviews in software engineering
Tsinghua University, CHN.
Tsinghua University, CHN.
Tsinghua University, CHN.
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering. Mälardalens högskola, SWE.
Show others and affiliations
2020 (English)In: IEEE Access, E-ISSN 2169-3536, Vol. 8, p. 36226-36243, article id 8984351Article in journal (Refereed) Published
Abstract [en]

Context: The inclusion of grey literature (GL) is important to remove publication bias while gathering available evidence regarding a certain topic. The number of systematic literature reviews (SLRs) in Software Engineering (SE) is increasing but we do not know about the extent of GL usage in these SLRs. Moreover, Google Scholar is rapidly becoming a search engine of choice for many researchers but the extent to which it can find the primary studies is not known. Objective: This tertiary study is an attempt to i) measure the usage of GL in SLRs in SE. Furthermore this study proposes strategies for categorizing GL and a quality checklist to use for GL in future SLRs; ii) explore if it is feasible to use only Google Scholar for finding scholarly articles for academic research. Method: We have conducted a systematic mapping study to measure the extent of GL usage in SE SLRs as well as to measure the feasibility of finding primary studies using Google Scholar. Results and conclusions: a) Grey Literature: 76.09% SLRs (105 out of 138) in SE have included one or more GL studies as primary studies. Among total primary studies across all SLRs (6307), 582 are classified as GL, making the frequency of GL citing as 9.23%. The intensity of GL use indicate that each SLR contains 5 primary studies on average (total intensity of GL use being 5.54). The ranking of GL tells us that conference papers are the most used form 43.3% followed by technical reports 28.52%. Universities, research institutes, labs and scientific societies together make up 67.7% of GL used, indicating that these are useful sources for searching GL. We additionally propose strategies for categorizing GL and criteria for evaluating GL quality, which can become a basis for more detailed guidelines for including GL in future SLRs. b) Google Scholar Results: The results show that Google Scholar was able to retrieve 96% of primary studies of these SLRs. Most of the primary studies that were not found using Google Scholar were from grey sources. © 2013 IEEE.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc. , 2020. Vol. 8, p. 36226-36243, article id 8984351
Keywords [en]
empirical evaluation, Google scholar, gray, Grey literature, quality checklist, software engineering, systematic mapping, tertiary study, Search engines, Empirical evaluations, Mapping
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-19337DOI: 10.1109/ACCESS.2020.2971712Scopus ID: 2-s2.0-85081118214OAI: oai:DiVA.org:bth-19337DiVA, id: diva2:1416359
Note

Open access

Available from: 2020-03-23 Created: 2020-03-23 Last updated: 2020-03-23Bibliographically approved

Open Access in DiVA

On using grey literature and google scholar in systematic literature reviews in software engineering(1472 kB)4 downloads
File information
File name FULLTEXT01.pdfFile size 1472 kBChecksum SHA-512
060fb517f00130da9042705880024f34092298711afb49628eec48fbec3005ab62b1e4a935010bafb144eb613b05f7482cc28ee066e9e5befe79042e58ea3fe2
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records BETA

Afzal, WasifTorkar, Richard

Search in DiVA

By author/editor
Afzal, WasifTorkar, Richard
By organisation
Department of Software Engineering
In the same journal
IEEE Access
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 4 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 14 hits
12345672 of 14
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf