PhD student, Multilingual LLMs for Software Engineering @serg_delft @AISE_TUDelft
Joined May 2022
- Tweets19
- Following50
- Followers40
- Likes73
Jonathan Katzy retweeted
Closed-source LLMs hide training data. The Poisoned Chalice competition asks: was this code file in the model’s training set? Build membership-inference methods, compete, and present at FSE/AIWare 2026 (Montréal).
razvain.github.io/miacomp/ #LLM4Code #FSE2026 #MembershipInference
Jonathan Katzy retweeted
📣 Exciting competition for the AI & Software Engineering communities! 🎯
Large language models (LLMs) for code are powerful but they often memorize what they’ve seen. When evaluation datasets overlap with training data, reported performance can be misleading
🧵Introducing "The Heap" - A contamination-free code dataset for fair LLM evaluation!
🔍 The Problem: 90% of code LLM studies don't deduplicate evaluation data, leading to inflated performance metrics
@MalihehIzadi @razvanmp27 @avandeursen
Thread 👇
🎯 Impact: Enables reliable LLM performance assessment without the data cleaning overhead. Perfect for researchers who want to focus on evaluation, not data preprocessing!
Languages covered: Assembly to Mathematica, C to Haskell, Python to Rust 🌍
Jonathan Katzy retweeted
Great insights from @katzy_jonathan!
Jonathan Katzy from @serg_delft @AISE_TUDelft just presented "A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics". Exciting work on #LLM #CodeComments #SoftwareEngineering
Jonathan Katzy retweeted
@katzy_jonathan presenting our #FORGE24 paper on “An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets”!
Talk to him if U are interested in the data used for training a wide range of #llm4code and their implications!
@ConfForge
Are foundation models vulnerable to lawsuits regarding the data they used to train?
We analyzed the state of LLMs wrt licenses in public code datasets.
Checkout the preprint: arxiv.org/abs/2403.15230
and discuss the implications with me at FORGE/@ICSEconf in April.
@MalihehIzadi
We analyzed the prevalence of exact duplicates of code files with strong copyleft licensed code in public code datasets, and searched the comments of code files to identify strong copyleft licenses.
Jonathan Katzy retweeted
resulted in identifying 38M exact duplicates in our strong copyleft dataset. This indicates the pervasive issue of #license inconsistencies in #LLMs4Code. We call upon the community to prioritize developing and adopting best practices for #legal & #ethical dataset creation... 3/n
Jonathan Katzy retweeted
1. "An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets" 📊
w/ @katzy_jonathan, Razvan Popescu, Arie van Deursen, & @MalihehIzadi
Our deep dive into 514 million code files used for training various #LLMs trained on code...
2/n
Jonathan Katzy retweeted
🎆We're happy to share that 2 full papers from our lab, have been accepted at the International Conference on AI Foundation Models and Software Engineering, co-located with @ICSEconf
cc @katzy_jonathan, Razvan, Arie, Tim, Frank, Philippe, Berend, Marc, & @MalihehIzadi
#FORGE
1/n
Jonathan Katzy retweeted
At @AISE_TUDelft, we're happy to wrap up the year with 2 papers accepted at @ICSEconf (ResearchTrack)🎉
-Language Models for Code Completion: A Practical Evaluation
-Traces of Memorisation in #LLMs for Code
@aalkaswan1, @katzy_jonathan, @timvandamdev, Marc, Razvan & @avandeursen
Jonathan Katzy retweeted
Heartfelt thanks to Prem @devanbu for spending an enlightening week w @serg_delft & delivering a thought-provoking lecture on quality of code generated by #LLMs in our course (ML4SE'23)!
I am sure your expertise added a rich dimension to the learning experience of our students.
ALT Lunch at TU Delft X - Prem visit
Why do #LLMs perform better on some programming tasks than others?
In our #IEEESCAM2023 paper, we show early evidence for an inherent difference in representations of tokens between languages.
arxiv.org/abs/2308.13354
@MalihehIzadi
@avandeursen
@ieeescam
#ML
#LLM
#ML4SE