Automated Workflow for Soil Journal Retrieval and Summarization

Automated Workflow for Soil Journal Retrieval and Summarization

Introduction

Land resource management researchers need to stay up-to-date with rapidly emerging literature on soil health and related topics. However, manually tracking new papers across multiple journals is time-consuming and prone to missing important updates (Using RSS to keep track of the latest Journal Articles – Elephant in the Lab). An automated workflow can regularly fetch the latest research from top soil science journals and summarize key findings, helping staff remain informed without overwhelming manual effort. This workflow will focus on journals like Soil Biology & Biochemistry, Geoderma, Catena, and similar publications, targeting content on soil health, soil quality, soil function, and related themes. Below, we outline a step-by-step strategy to automate article retrieval via institutional access and to generate structured summaries using a large language model (LLM). We also discuss recommended tools, implementation steps, and potential challenges in setting up this system.

Focus Journals and Keywords

To ensure relevant coverage, the workflow will target leading soil science journals and search for specific keywords:

• Target Journals: Soil Biology & Biochemistry (SBB), Geoderma, Catena, and other high-impact soil science journals (e.g. Soil & Tillage Research, Applied Soil Ecology). These titles rank among the top outlets in the soil science field (Soil Science: Journal Rankings | OOIR), so monitoring them captures a large share of significant research.
• Key Topics: Filter for papers discussing soil health, soil quality, soil function, soil fertility, soil biology, and related terms. These keywords align with core interests in land resource management and will help narrow the feed to articles about soil sustainability, ecosystem functions, and soil management practices.

By focusing on these journals and terms, the automated system will retrieve papers most pertinent to soil health and quality research, rather than all published articles. This targeted approach reduces information overload and zeroes in on relevant studies.

Using Institutional Access for Retrieval

Many high-quality journal articles are behind paywalls, so leveraging institutional access is crucial. The workflow should be executed on a machine within the campus network or via the institution’s VPN to automatically bypass paywalls using the library’s subscriptions (Getting started with Elsevier APIs | Augustus C. Long Health Sciences Library). Accessing the content in this way ensures that full-text retrieval (PDF or HTML) is possible for subscribed journals without manual login. Key considerations include:

• Authentication: If using a script, ensure it either runs on-campus (recognized by IP) or uses the library’s proxy. Some publisher APIs also allow an Institutional Token or API key linked to the university’s account for authentication (Getting started with Elsevier APIs | Augustus C. Long Health Sciences Library). For example, Elsevier’s APIs (which cover journals like SBB, Geoderma, Catena) require users to be on a subscribing institution’s network and to use an API key tied to their account (Getting started with Elsevier APIs | Augustus C. Long Health Sciences Library).
• Terms of Use: Check publisher policies on text and data mining. Elsevier and others often permit non-commercial text mining through official APIs with an API key, as long as usage is within limits (Getting started with Elsevier APIs | Augustus C. Long Health Sciences Library). It's best to register for any required developer access (e.g. Elsevier’s Developer Portal for ScienceDirect/Scopus APIs) and include a valid email in API queries to avoid being blocked (Best way to extract list of articles related to something? - Metadata Retrieval - Crossref community forum).
• Alternate Access: If API access is restricted, consider using the DOI or stable URLs via the library proxy. For instance, retrieving PDFs by constructing URLs with the library’s proxy prefix or using tools like Zotero (with proxy settings) can automate the download of full texts when on the campus network.

Using institutional access in the retrieval process ensures the workflow can fetch the complete papers (not just abstracts) needed for thorough summarization.

Data Sources: RSS Feeds and APIs for Article Retrieval

To automate discovery of new papers, the workflow can leverage two primary data sources: journal RSS feeds and scholarly APIs.

• Journal RSS Feeds: Most academic publishers provide RSS feeds listing newly published articles for each journal (Using RSS to keep track of the latest Journal Articles – Elephant in the Lab). These feeds are XML files that include recent article metadata (title, authors, publication date, and often the abstract and a link to the article) (Using RSS to keep track of the latest Journal Articles – Elephant in the Lab). For each target journal (SBB, Geoderma, Catena, etc.), find the RSS feed URL (often indicated by the RSS icon on the journal homepage (Using RSS to keep track of the latest Journal Articles – Elephant in the Lab)). An automation script can periodically poll these feeds to detect new entries. The feed entries can be filtered by keywords (e.g., checking if "soil health" or "soil quality" appear in the title or abstract) to focus only on relevant articles. Using RSS is convenient because publishers update these feeds in real-time when new articles or issues are released (Using RSS to keep track of the latest Journal Articles – Elephant in the Lab). Libraries like Python’s feedparser can read and parse RSS feed entries easily.

• Scholarly APIs (CrossRef & Publisher APIs): For more advanced or flexible searching, APIs can be used to query for new papers:

  • CrossRef API: CrossRef’s REST API allows querying the global DOI registry for articles by keywords, journal names, dates, etc. For example, one can query CrossRef for articles published in Soil Biology & Biochemistry in the last month that mention "soil health" in their metadata. An example query (for COVID-related papers) is shown below, which can be adapted for soil keywords and date ranges (Best way to extract list of articles related to something? - Metadata Retrieval - Crossref community forum):

    https://api.crossref.org/works?query.bibliographic=soil+health&filter=from-pub-date:2024-01,until-pub-date:2024-12,container-title:Soil+Biology+%26+Biochemistry&type=journal-article  
    

    This would return JSON metadata for all matching articles (title, authors, DOI, etc.). The script can parse this response to get DOIs of new papers. Keep in mind CrossRef may return a large set if the query is broad, so applying filters (journal, year) is useful to narrow results (Best way to extract list of articles related to something? - Metadata Retrieval - Crossref community forum).
  • Publisher APIs: Many journal publishers offer APIs or feeds for programmatic access. For Elsevier journals (which include SBB, Geoderma, Catena), the ScienceDirect API and Scopus API are relevant. Using Elsevier’s APIs, one can search for articles by keywords and journal source, and retrieve article metadata or full text in XML/JSON form (Getting started with Elsevier APIs | Augustus C. Long Health Sciences Library). This requires obtaining a free API key from Elsevier’s developer portal and using an institutional login/token (