FSE 2026 Preview

Foundations of Software Engineering (FSE) is one of the leading venues for software engineering research. We are excited to share several contributions that will be presented at the conference and co-located events this year:

Beyond Code Snippets: Benchmarking LLMs on Repository-Level Question Answering [PROMISE]

Yoseph Berhanu Alebachew, Hunter Leary, Swanand Vaishampayan, Chris Brown

• 🔍 Problem: Large Language Models (LLMs) have demonstrated strong capabilities in software engineering tasks, such as question answering (QA). However, existing benchmarks primarily focus on isolated code snippets, neglecting the complex challenges of real-world program comprehension, which often involves multiple files and intricate system-level dependencies.

• 🧪 Study: We introduce StackRepoQA, a repository-level dataset of 1,318 real developer StackOverflow questions and accepted answers from 134 open-source Java projects. We evaluate two LLMs (Claude 3.5 Sonnet and GPT-4o) across prompting and retrieval-augmented strategies.

• 📊 Findings: We found that LLMs achieved moderate baseline accuracy, with performance improving with graph-based retrieval methods. However, much of their success came from memorization rather than genuine reasoning about repositories.

• 💡 Implication: Our findings emphasize the need for benchmarks and retrieval strategies that prioritize reasoning over memorization in repository-level comprehension. Moreover, retrieval methods show promise but require refinement to address the challenges of evolving repositories and repository-level complexity.

TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation [AIWare arxiv]

Hunter Leary, Luke Hanuska, Chris Brown

• 🔍 Problem: Foundation models (FMs) are increasingly used in software development, including automated test case generation. However, there is a critical need to validate the quality and correctness of generated tests to ensure they provide actionable evidence for verifying code and are maintainable over time.

• 🧪 Study: We introduce TestMap, an open-source evidence infrastructure designed for foundation-model-assisted test generation. TestMap automatically validates, measures, and records generated test candidates across repositories, using evidence-based practices to track the lifecycle of generated tests, from failures to repairs and validation.

• 💡 Implication: We outline our TestMap approach, highlighting the importance of evidence-based validation for FM-assisted test generation and emphasizing the need for rigorous evaluation, repository-specific experimentation, and strategies to improve test maintainability and reliability. TestMap serves as a practical tool to support developers and researchers in adopting foundation models for robust, repeatable test generation workflows.

Practitioner Perspectives of DAST Integration in Agile Development Workflows: An Experience Report [FSE Industry]

Arpit Thool, Chris Brown

• 🔍 Problem: Dynamic Application Security Testing (DAST) tools are crucial for detecting vulnerabilities in web applications, but their integration into Agile and CI/CD workflows often conflicts with the fast-paced nature of modern software development.

• 🧪 Study: We provide an experience report of a real-world development team that undertook an initiative to integrate DAST in their Kanban-based Agile process and CI/CD pipelines. We conducted semi-structured interviews with 10 practitioners from the organization to understand their their perceptions, challenges, and opportunities for improvement.

• 📊 Findings: Practitioners reported that while DAST added value by identifying vulnerabilities, its integration introduced challenges such as time management constraints, bandwidth limitations, and difficulty interpreting technical reports. Despite these hurdles, the team recognized DAST’s benefits in improving system security and expressed willingness to continue using it, emphasizing the importance of automation and improved reporting.

• 💡 Implication: We provide lessons learned from the team's experience, highlighting the need for better automation, streamlined reporting, and cultural shifts to prioritize security awareness in Agile teams.

SafeAIMerge: A Tool for Integrating DAST and LLM-generated Security Feedback into GitHub Actions Workflows [SecDev]

Arpit Thool, Justin Smith, Chris Brown

• 🔍 Problem: While Continuous Integration and Delivery (CI/CD) pipelines streamline software development, integrating security measures like Dynamic Application Security Testing (DAST) tools remains challenging. DAST tools are often complex, produce overly technical reports, and are difficult for developers to integrate into their workflows. These challenges hinder effective vulnerability remediation and security adoption in modern development pipelines.

• 🧪 Study: We developed []SafeAIMerge](https://github.com/arpitthool/SafeAIMerge), a tool that integrates DAST scans with LLM-generated security feedback into GitHub Actions workflows. Using a two-phase study, we gathered formative feedback from 46 developers via a formative survey and conducted a user study with 12 participants to evaluate the tool’s impact on developer workflow and vulnerability remediation.

• 📊 Findings: SafeAIMerge received positive early perceptions and significantly reduced cognitive load and improved developers' ability to remediate vulnerabilities compared to conventional DAST workflows. The tool also was viewed as usable and reduced mental effort in security-related tasks.

• 💡 Implication: We demonstrate the potential of integrating LLM-driven security feedback into CI/CD pipelines, providing actionable insights that align with developer workflows.

Beyond Text: Understanding How Multimodal Generative AI Impacts Students Learning Software Development (Extended Abstract) [FSE SEET]

Huayu Liang, Chris Brown

• 🔍 Problem: Multimodal generative AI tools that integrate text, audio, images, and real-time screen sharing have the potential to transform how students learn software development. However, understanding their impact on learning outcomes in software engineering education is crucial.

• 🧪 Study: We propose a study plan to explore of how multimodal AI tools influence student learning during software development tasks. Our evaluation, leveraging Google AI Studio, will compare students completing bug fixing, feature implementation, and test writing tasks across multimodal and text-based LLMs---analyzing student interactions, modality usage patterns, perceptions, and the efficiency and accuracy of solutions.

We are looking forward to sharing these efforts and discussing with the research community at FSE 2026 in Montreal, QC, Canada!