Technical Documentation

WHAT IS A SYSTEMATIC MAP?

According to the Collaboration for Environmental Evidence, systematic maps are “overviews of the distribution and abundance of evidence in relation to multifaceted elements of a broad question of policy or management relevance.” The question for our systematic review states: “What is the empirical evidence for breeding forest bird species-vegetation relationships in eastern and boreal North America?” Systematic maps often differ from other types of reviews via the development of interactive data visualization systems, like the forest bird literature gateway, to facilitate deep exploration of review outcomes. Detailed guidelines and standards for evidence synthesis in environmental management provide a structured approach for conducting systematic maps, and other types of systematic reviews, that place a premium on comprehensiveness and controlling for biases that are common to narrative reviews. The final products of a systematic map should: a) summarize all potential evidence on a clearly defined review topic, b) be free from biases associated with less robust evidence search and synthesis strategies, and c) provide an open and accessible resource to others who do not have the time to conduct such a thorough review but wish to explore the full richness of the evidence base presented by a systematic map.

SEARCHING FOR REFERENCES

After a review question has been defined, the first step in the creation of a systematic map, or any type of systematic literature review, is to develop a search strategy. A search strategy includes the specification of a search string and a list of sources that will be searched. A search string is a set of search terms, typically separated by Boolean operators, strategically selected to locate all references that may have relevance to the review question. For systematic reviews, the search objective is to be comprehensive and to miss few potentially relevant references. For this reason, search strings for systematic maps that summarize evidence bases for large subject areas tend to be quite long.

For this review, our final search string (for search tools that allowed for Boolean searches and did not place limits on the number of search terms) was arranged according to the basic formulation: A list of terms related to forests or forest management (separated by OR operators) AND a list of terms related to birds (separated by OR operators). This search string, or shorter versions that were necessary for search engines that limited the number of search terms, was then used to search > 60 different sources of information. This included a number of bibliographic databases accessed through 3 academic search engines (Web of Science, EBSCO Host, and ProQuest); government agency sources (US and Canada); websites for non-government organizations, publisher websites, and 3 web-based search engines (Google Scholar, Microsoft Academic, and Dimensions).

Searching this many sources with a lengthy search string results in many search results (in this case, >180,000 across all 61 sources), which then need to be de-duplicated to create the set of unique references that will be screened further during the systematic review. After de-duplication, this search strategy yielded ~113,000 references with potential relevance to our review question. Evaluation of references for inclusion in a systematic map is typically done in two stages: 1) title/abstract screening and 2) full-text screening.

TITLE/ABSTRACT SCREENING

During title/abstract screening, references are evaluated as to whether they MAY meet the final eligibility criteria for inclusion in the systematic map. It is often difficult to determine, conclusively, if a reference will include a direct presentation of empirical data based on the short and often generalized text of titles and abstracts. For this reason, all references that appear like they MAY include relevant information are typically brought forward into the next stage of full-text screening, where full-text versions of articles are acquired to facilitate detailed examination of results sections of the potentially eligible references that were identified during title/abstract screening. For our systematic review, the following criteria were used during title/abstract screening to determine which references were subjected to full-text screening:

1. Data should be available for at least one target bird species for…
2. At least one target bird metric AND at least one target vegetation metric.
3. Data should be presented for at least one forested study site within the target spatial extent (Figure 1).

Systematic maps typically summarize only primary literature. However, since we knew from the start that we wanted to summarize existing review papers on our topic, we passed both primary references and review papers that were identified during title/abstract screening along to the full-text screening phase.

FULL-TEXT SCREENING

We were able to acquire full-text versions for >96% of the >4,100 primary literature references that made it through title/abstract screening. We used the same general inclusion criteria for full-text screening as we did for title/abstract screening. However, full-text screening allowed us to assess whether each reference truly contained empirical data that met each of the criteria below as quantitative results.

1. Data should be available for at least one target bird species for…
2. At least one target bird metric AND at least one target vegetation metric.
3. Data should be presented for at least one forested study site within the target spatial extent.
4. References should not meet any of the “additional exclusion criteria” described below.

FULL-TEXT EXCLUSION CRITERIA FOR THE SYSTEMATIC MAP

Since our systematic map was intended to summarize primary literature, our first category for exclusion identified review papers, which were retained and summarized in a separate publication titled: Do review papers on forest bird-vegetation relationships provide actionable information to forest managers in the eastern United States?.

ADDITIONAL EXCLUSION CRITERIA

Each primary literature reference that did not meet all 4 of our inclusion criteria were assigned a “reason for exclusion” from the following list. Only a single reason for exclusion was recorded per reference, even though some references could have been excluded for more than one reason.

NO TARGET BIRD SPECIES- We initially identified 153 bird species that occur widely across eastern temperate and/or boreal forests of North America as candidates for inclusion in this systematic map. Given time and resource constraints for this review, we chose to exclude references that present single-species research on 6 different species of game birds (Ruffed Grouse, Wild Turkey, Northern Bobwhite, American Woodcock, Spruce Grouse, and Sharp-tailed Grouse), and 3 species currently or formerly included on USFWS list of Threatened and Endangered Species (Kirtland’s Warbler, Red-cockaded Woodpecker, and Peregrine Falcon). This resulted in a final list of 144 target bird species for inclusion, where empirical data had to be presented for at least one of these species for a reference to be included in the systematic map.

NOT IN BREEDING SEASON- This systematic map was restricted to forest bird-vegetation relationships during the breeding season. Consequently, we excluded references that presented only species-vegetation relationships during migration or winter. We defined the breeding season to be inclusive of all events/stages staring with settlement in breeding areas after spring migration through post-fledging dispersal. This definition of the breeding season ends with directional movement for southbound migration. These criteria also resulted in the exclusion of papers that reported only on demographic metrics that extend beyond a single breeding season (e.g., inter-annual survival, inter-annual dispersal, population trend/growth, or carry-over effects).

NOT IN FORESTED LANDSCAPE- We excluded references that addressed only bird use of forest patches within predominantly rural agricultural, grassland, urban, or suburban-dominated landscapes. We did include some papers that addressed these contexts, if they also presented data from at least one forested study site, which we defined as having >60% forest within 1km.

GUILD OR SPECIES GROUP-BASED ANALYSES ONLY- We excluded references that provided information for any combination of our target species grouped together (by any criteria), that did NOT also provide species-specific information (e.g., a paper that included information on “ground foraging birds” but no specific information for Ovenbirds or Swainson’s Warblers). Papers that included information on individual species AND on groups were retained. We took this same approach with references that presented information on vegetation relationships with a wide range of bird diversity metrics (e.g., species richness, evenness, or diversity indices), retaining only those papers that also presented species-specific information on at least one of our target bird species.

INTRINSIC MARKERS- We included references that included a pre-defined set of target bird metrics related to bird abundance, occurrence, reproductive performance, or habitat use/selection. We excluded, however, papers that described vegetation relationships with intrinsic markers, such as stable isotope ratios, genetic variables, or contaminant loads unless these same papers provided information on species-specific bird-vegetation relationships using at least one of our target bird metrics.

NON-FORESTRY FIELD EXPERIMENTS- We excluded all references that presented results from non-forestry related field experiments. For example, we excluded all references that used artificial eggs or nests as the study unit, studies that used foraging exclosures, or studies that used any type of supplemental feeding.

ECOSYSTEM SERVICES- We excluded references that presented information on bird-provided ecosystem services only (e.g., fruit dispersal, vectors for disease, predation on insects) and did not present information on any our target bird metrics.

MODEL OUTPUTS ONLY- We excluded all references that presented information generated exclusively from model outputs and not empirical data.

SPATIALLY IMPRECISE BIRD DATA- We excluded references with spatially imprecise bird observations which could not be consistently mapped to vegetation data at the plot or stand scale. This included many references that presented analysis of bird species-vegetation relationships using that compared North American Breeding Bird Survey with remotely sensed vegetation metrics, as well as some references that analyzed eBird data without removing spatially imprecise records.

ROADS/RECREATION/NOISE- We excluded references that only included variables related to roads, trails, recreational use, noise, and other human structures in the forest.

NO BIRD METRICS- This was a more general category than the specific exclusion criteria listed above where we simply excluded all references that did not include data on at least one target bird metric type.

NO VEGETATION METRICS- This was a more general category than the specific exclusion criteria listed above where we simply excluded all references that did not include data on at least one of our target vegetation metrics.

DATA EXTRACTION

Due to time and resource limitations, there were differences in the types of reference metadata that were extracted from all references within our full eastern and boreal forest study extent (Figure 1a) and our smaller focused study extent, covering the Appalachian Mountains, Northeastern Highlands, and the Great Lakes (Figure 1b), where metadata extraction was more extensive. These two different subsets of references have different data summary and search capabilities, as evidenced by the two different search tabs of the web-based literature gateway for the systematic map (Extracted Data and Keyword Search).

Figure 1. A) The full spatial for this systematic map, covering all eastern and boreal forests of North America. B) A smaller focal extent, covering the Appalachian Mountains, Northeastern Highlands, and Great Lakes forests, for which more detailed reference metadata were extracted (e.g., bird species lists, bird and vegetation metric types, and study characteristics). For publications with study sites in the map a area, which were outside of our focused extent (map b), we collected only study location, citation information, and curated lists of keywords) for each reference. We acquired this same basic information, as well as the more detailed extracted data listed above, for all references in map b.

METADATA EXTRACTED FOR ALL REFERENCES

LATITUDE AND LONGITUDE IN DECIMAL DEGREES- We recorded latitude and longitude coordinates for each of the primary references that passed our title/abstract and full-text screening stages. This allowed us to display study site locations in our data visualization tool for the systematic map. However, the accuracy and precision of study site descriptions was exceptionally variable among publications. Coordinates were presented directly in papers with anything from 0 to >5 decimal places of precision (e.g., location accuracy ranging from ± 111 kilometers to a single meter). Many publications did not present coordinates at all, providing only narrative study site descriptions that ranged from the mention of specific properties that could be located on maps to references to entire states, provinces, or ecoregions. While we used this variable data on study locations to present of a coarse distribution map of all references in our data visualization, the variable precision (and often imprecision) of study site descriptions should preclude the analysis of any reference metadata related to our recorded geographic coordinates. To discourage this type of analysis, we do not provide latitude and longitude coordinates in download files. The inconsistent (and often detail-poor) description of study sites is a frequent flaw in the ornithological, ecological, or forest management literature.

TITLE AND ABSTRACT TEXT- To facilitate topical keyword searches and subsetting of references in the web-based data visualization for this systematic map, we extracted complete title and abstract text for all references (for both the full and limited spatial extents described above). This information was used as the raw material for additional text-based analyses, which are the topic of a manuscript in preparation.

FULL CITATION- For each reference we extracted citation information, using a standard citation export format in EndNote that covered each of the 7 different reference types that were included in this systematic map.

REFERENCE TYPE- Our search strategy covered many peer-reviewed and grey literature information sources, resulting in the presence of 7 different reference types in our systematic map: journal article, book, book section, thesis, government document, conference paper, and report.

OUTLET TYPE- This field provides the name of the specific journals, books, conference proceedings, universities, government agencies, or organizations associated with each reference.

AUTHOR(S)- This includes the full list of coauthors for every publication.

PUBLICATION YEAR- A single, four-digit year.

REFERENCE METADATA EXTRACTED BY SUBJECT EXPERTS

TARGET BIRD SPECIES- After the exclusion of 6 game bird species and 3 ESA-listed species (see NO TARGET BIRD SPECIES section above), we generated a list of 144 target bird species for which species-specific data on at least one bird metric and at least one target vegetation metric needed to be presented in the results section (or appendices) for a reference to be included in the final systematic map. For each reference within our focal data extent (Figure 1b above) that met all full-text eligibility criteria, subject experts on the systematic map team extracted a complete, comma-separated list of all species for which data was presented. For some references, this list contained only a single species. For others, it contained >100 species.

TARGET BIRD METRIC TYPES- To meet final eligibility criteria for the systematic map, references had to present data for at least one of 3 general bird metric types. The following text lists which specific metrics might be included in each of these general bird metric types. Each reference was coded by a subject expert from the systematic map team as to whether it included data for one, two, or all three of these metric types.

Abundance/Occurrence- Any metric that specifies whether more than 1 individual of a species has been recorded. This includes “abundance”, “count”, “relative abundance”, “density”, “presence/absence,” “prevalence,” “incidence”, “occupancy”, “occurrence probability” and related metrics.

Reproductive performance- Any metric related to any stage in the production of young. This includes “clutch size”, “hatching success”, “nest success”, “nest survival”, “pairing success”, “fecundity”, “productivity”, “reproductive success”, “nest survival”, “chick survival”, or any other demographic metric that can be calculated as a component of annual reproductive success.

Resource use/selection/preference- Any metric related to whether a bird “uses”, “prefers”, or “selects” a particular environmental attribute. This may include any number of metrics across a wide range of terminology related to habitat use, resource use, habitat selection, habitat preference, resource selection, foraging preferences, niche, and others.

TARGET VEGETATION METRIC TYPE- To meet final eligibility criteria for the systematic map, references had to present data for at least one of 5 different target vegetation metric types. The following text lists which specific metrics might be included in each of these general vegetation metric types. Each reference was coded by a subject expert from the systematic map team as to whether it included data for one, two, three, four, or all five of these metric types.

Forest composition- Any metric related to plant/tree species composition where the identity or type of species is important. This could be basal area or importance values for individual tree species, specific forest classifications, basal area of mast producing trees, dominant tree species, stems per acre of particularly important understory species, etcetera.

Forest structure- Any metric related to local or small plot variation in the vertical or horizontal arrangement of vegetation irrespective of vegetation composition. This includes a variety of metrics such as: shrub-stem density, number of vertical layers, % canopy cover, forest age classes, tree size distributions, percent cover estimates for different growth forms, and more.

Habitat elements- This refers to the presence, absence, or abundance of discrete features or resources that may be used by birds, such as: cavities, perches, rock outcrops, soft mast, dead trees, streams, caves, etc.

Landscape metrics- Metrics that are calculated at a larger scale than a points or small plots that indicate aspects of spatial arrangement or composition of features. This includes edge-related metrics; many different patch shapes, size, or configuration metrics; land cover compositions at variable scales, and distances to important features like roads or water.

Management Treatment- samples are grouped by different categorical management types (e.g., single-tree selection/clear-cut, burned/unburned, deer fencing/no deer fencing, etc.), regardless of whether a study presents explicit measurements of environmental attributes that describe different levels of the management type.

SAMPLING SCALE- Studies are categorized by subject experts of the systematic map team as to whether information is presented for one, two, or all three of the following spatial scales.

Within-stand or plot- Measurements from points, plots, or transects within stands.

Stand- Measurements collected at the stand scale AND/OR within-stand measurements that are summarized at the stand scale.

Landscape- Measurements from areas larger than a single stand. This could include distance to feature measurements, analyses done in circular or square neighborhoods surrounding a sampling point that extend beyond the boundaries of a single stand, measurements that summary landscape attributes from within a polygon shape that is larger than a single stand, or adjacent stand/land-use types.

GROUP COMPARISON TYPE- Studies are categorized by subject experts of the systematic map team as to whether information is presented for one, two, three, four, five, or all six of the following group comparison types.

Before/After- Samples collected from the same locations before and after an event.

Control/Impact- Samples collected from both impact sites and control sites.

Temporal Effects (space for time substitution)- Samples collected from multiple study sites of different successional stages to sample temporal dynamics (e.g., variation in years since harvest) over a shorter (usually 1-4 year) sampling period than would be required to sample a single site over a longer time period.

Temporal Effects (time Series/chronosequence)- Longitudinal sampling for >3 years at a single site to explore temporal dynamics at the same site.

Treatment Comparison- Samples grouped to categorical management treatment types, or levels of a management treatment within the same type, in any way prior to analysis.

Forest Type/Tree Species- Samples stratified to compare bird and/or vegetation metrics across different forest types or gradients of tree species composition.

N BREEDING SEASONS- The number of breeding seasons for which a reference presents empirical data.

CURATED KEYWORD LISTS- We identified >1,300 keyword terms (either single words or multi-word phrases), grouped into 18 different topical categories, with practical relevance for searching the literature on forest bird vegetation relationships. The occurrence of each of these keywords in titles or abstracts for each of the primary references (and in the full text of each of the review papers) has been saved in document-term matrix that allow users to search all references by curated keyword lists in addition to metadata extracted by the systematic map team. Full methods for keyword analyses are the topic of a manuscript that is currently in preparation.   

ACKNOWLEDGMENTS

This data visualization tool was inspired by, and is dedicated to, several remarkable groups. First, we acknowledge that without the herculean efforts of library scientists and bibliographic database managers over the past half century, it would simply be impossible to access such a large body of literature, on any topic, much less one as specific as bird-vegetation relationships. As the volume of information produced by scientists increases exponentially, we encourage all users of this tool to recognize the importance and value of maintaining and expanding the data storage, extraction, and exploration infrastructures that support the continued growth of science as an enterprise that continues to build on its own achievements. Given the challenges of searching for, compiling, and evaluating literature for this broad topic systematic map, we will never take this infrastructure for granted again! Second, we recognize that without insights from key figures in the field of information science and data visualization, particularly Edward Tufte, Ben Schneiderman, Steven Few, and Miguel Cairo, we would have never stumbled upon the core design principles that support effective interactive data visualizations. We hope that they stumble upon this tool and criticize us soundly! Third, we are grateful to the many software engineers and developers who have created sophisticated tools, like JMP (which we used extensively for text analyses) and R shiny (which was used to create this web-based product) that allow for the exploration of rich, and highly dimensional data sets, like systematic maps. Fourth, we are grateful to the many scientists associated with the Collaboration for Environmental Evidence, who have put forth, and regularly updated, robust standards for the creation and documentation of information synthesis via systematic literature review. In the fight against bias, which becomes more important every day, these are the giants whose shoulders we stand on. Finally, we thank the USDA’s Natural Resources Conservation Service’s Working Lands for Wildlife Program for their interest in supporting a project with the challenging objective of making such a large body of scientific information available to a broad user community of forest managers, policy makers, scientists, journal editors, conservation planners, and other stakeholder groups that have a vested interest in making sure that the best available science is consulted to inform decisions in the intersection between species conservation and natural resource management.