SSP Forum: Kunal Sinha and Ayush Chakravarthy (M.S. Candidates)

The
Symbolic Systems Forum
(community sessions of SYMSYS 280 - Symbolic Systems Research Seminar)
presents

End-to-end GNN-RAG: retrieval and reasoning on black-box knowledge graphs
Kunal Sinha (M.S. Candidate)
Symbolic Systems Program

Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
Ayush Chakravarthy (M.S. Candidate)
Symbolic Systems Program

Monday, April 21, 2025
12:30-1:20 pm PT
Margaret Jacks Hall (Bldg. 460), Room 126
In-person event, not recorded
(see below for entry instructions if you are not an active Stanford affiliate)

Note: Lunch is provided, if pre-ordered, only for members of SYMSYS 280, but others are welcome to bring a lunch and eat during the presentation.

Abstracts:

(1) Kunal Sinha, "End-to-end GNN-RAG: retrieval and reasoning on black-box knowledge graphs" (Primary Advisor: Chris Potts, Linguistics)
     Retrieval-Augmented Generation (RAG) has emerged as a powerful framework for combatting hallucination in LLMs, allowing models to retrieve relevant information from external sources before answering questions. However, many real-world enterprise settings present challenges for traditional RAG systems. First, companies' internal data is often organized in a graph structure, making the retrieval task more complicated. Second, companies typically do not have "training labels" indicating which nodes or paths in their graph contain the answer for a given question, despite the fact that existing state-of-the-art methods need these labels during training. We present a novel end-to-end GNN-RAG method to address these challenges. We first show that our method matches state-of-the-art performance on existing benchmarks. Next, we curate a synthetic dataset with missing training labels, demonstrating that our method can perform well in settings where others cannot.

(2) Ayush Chakravarthy, "Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs" (Primary Advisor: Noah Goodman, Psychology)
     Test-time inference has emerged as a powerful paradigm for enabling language models to ''think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors --- verification, backtracking, subgoal setting, and backward chaining --- that both expert human problem solvers and successful language models employ. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.

Notes on entry to the meeting room:

Entry to the building is open to anyone with an active Stanford ID via the card readers next to each door. If you do not have a Stanford ID, you can gain entry between 12:15 and 12:30pm ONLY by knocking on the exterior windows of room 126. These windows are to the left of the west side exterior door on the first floor of Margaret Jacks Hall, which faces the back east side of Building 420. Please do not knock on these windows after 12:30pm when the talk has started. We will not be able to come out and open the door for you at that point.