Abdalghani Abujabal (@ajabal4) profile picture

Scientist @SDAIA_SA, ex-@amazon AGI. NLP and AI. PhD at MPI for Informatics.

Germany
Joined March 2015
Abdalghani Abujabal retweeted
Language models handle a single question well. Keeping track of a whole conversation is harder, especially in languages like Arabic that are underrepresented in training data. A user mentions early on that a medical condition stops them from flying. A few turns later they ask how to reach another country, and the model suggests a flight. The last question wasn't hard. The model just didn't carry forward what it had been told. @scale_AI's MultiChallenge benchmark (Sirdeshmukh et al., 2025) was built to measure exactly this kind of failure. When it was released, every frontier model tested scored below 50%. Its data is in English, though, and an automatic translation wouldn't be a reliable Arabic test. So, building on @scale_AI's framework, we created an Arabic version: 1,600 five-turn evaluation conversations in Modern Standard Arabic, covering 16 domains and 164 subdomains. It uses five challenge types: 1. Instruction Retention: does the model still follow an instruction from the first prompt? 2. Inference Memory: does it apply a detail the user mentioned earlier, even when the final request doesn't ask for it? 3. Reliable Versioned Editing: can it go back to an earlier version of a draft and edit that one? 4. Self-Coherence: does it stay consistent with what it said in earlier turns? 5. Contextual Anchoring (a new type we added): can it work out which topic the user means when they don't name it? For example, "the third architect we haven't talked about yet." The full post includes one Arabic sample for each challenge type 👇 aiastrolabe.com/blog/multich…
2
3
3
213
Abdalghani Abujabal retweeted
Real users do not stop at a single prompt. They hold long conversations and ask models to work over long documents, and that is where things break. A model can nail an isolated answer and still lose instructions, facts, or source constraints introduced 20 turns earlier. We built an Arabic dataset to push exactly on that, spanning conversations up to 30 turns and long source documents, across Modern Standard Arabic and the Najdi Saudi dialect. Two tracks: - Long conversations: holding context and resolving references across dozens of turns. - Document-grounded chats: staying faithful to a source, and saying "that's not in the document" instead of guessing. Every conversation is written by native speakers and reviewed for factual accuracy, dialectal naturalness, source faithfulness, and cross-turn consistency. We also tested three Arabic models on it, and they stumble on long, source-bound, dialectal conversations. Read the full breakdown: aiastrolabe.com/blog/extreme…
3
3
118
Abdalghani Abujabal retweeted
AI conversations rarely happen in a single turn. Users ask follow-up questions, refer back to earlier information, introduce new requirements, and expect models to keep track of the conversation as it evolves.
1
3
5
199
Abdalghani Abujabal retweeted
Most AI systems don’t fail safety benchmarks. They fail when users show up. Red teaming, trying to break the system and surface unsafe behavior, becomes a checklist. Blue teaming, ensuring the model responds to safe requests, becomes cautious. Both miss how systems are used.
2
4
1
4
242
Abdalghani Abujabal retweeted
Most Arabic AI demos look great, until real users show up. Because users don’t speak in clean, structured Arabic. They speak in dialects, interrupt each other, talk over noise, and switch languages mid-sentence. And this is where models break.
1
4
6
236
Abdalghani Abujabal retweeted
Can today’s AI models reliably read charts in Arabic? We tested them. We ran a simple probe: Two Arabic charts, two objective questions, tested across: @OpenAI - GPT, @AnthropicAI - Claude, @Google - Gemini, @QatarComputing - @Ai_Fanar, and Z.ai - GLM-5
1
2
2
248
Abdalghani Abujabal retweeted
Incidents like this highlight a core problem we see every day: agents behave differently once they touch real systems. Testing, red-teaming, and alignment in realistic environments isn’t optional anymore - it’s foundational
Nothing humbles you like telling your OpenClaw “confirm before acting” and watching it speedrun deleting your inbox. I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb.
2
1
166
Abdalghani Abujabal retweeted
Ramadan Kareem / رمضان كريم 🌙 As people and brands use AI to generate greeting images, it is critical that the images make sense linguistically and culturally.
1
2
4
111
Abdalghani Abujabal retweeted
In the recent paper Simple Test-Time Scaling: arxiv.org/pdf/2501.19393, researchers from @Stanford, @UW, @allen_ai , and @ContextualAI show that even 1,000 curated samples can significantly improve model performance.
1
1
1
360
Abdalghani Abujabal retweeted
We recently published a blog post redteaming leading LLMs on Arabic: aiastrolabe.com/post/redteam… Building on that assessment, we zoomed in on safe responses and categorized them further: (1) Helpful Response, and (2) Refusals.
1
1
2
279
ASAS shows where frontier LLMs stand, and where they must improve on Arabic safety. Essential reading for building safer AI. #ICLR2025 #ICLR
🚀 The @aiastrolabe Arabic Safety Index (ASAS - أساس) is now live. We are officially launching the first-ever benchmark focused on Arabic LLM safety, and the results demand attention.
2
425
ASAS is eye-opening. Even top models struggle with basic safety in Arabic. Redteaming does not transfer well when context is this complex; dialect, religion, politics, all layered.
Arabic AI is being left behind. We are here to change that. At @aiastrolabe, we’re building the gateway to #ArabicAI; pushing the frontier across dialects, domains, and capabilities, and making sure models are safe, culturally grounded, and built for our communities.
1
1
258
Thrilled to share that our work on ALLaM has been accepted to ICLR! Check out our paper!
🌍 Ensuring AI models preserve the full spectrum of human language is more critical than ever. As LLMs increasingly reflect the linguistic homogenization of the internet, we must actively push back to bring the world's languages in AI-driven communication. This is challenging but essential work. The ALLaM team's incredible efforts—accepted to ICLR 🎉—in adding world-class Arabic language capabilities for LLMs primarily trained on English is one example of this (proud to have been part of the team!). Stay tuned for more work coming in this area, particularly high quality non-English datasets to train the best AI models...
1
1
4
488
Abdalghani Abujabal retweeted
Exciting news ahead! In Arabic, (ALLaM) means 'knowledge'. Get ready for our first launch of LLM, ALLaM - stay tuned for more updates soon. (Tweet Written by ALLaM) With - @areebsa @haidarkk1 @twairesh @ajabal4
5
5
18
2,034
Abdalghani Abujabal retweeted
4,094
17,395
3,420
65,617
6,452,308
Abdalghani Abujabal retweeted
أمجد مسعد @amasad مؤسس ريبليت @replit - ستارتاب تقدر قيمتها بمليار دولار و تتيح لمستخدميها البرمجة من خلال المتصفح دون الحاجة لأجهزة أو تطبيقات برمجية متخصصة. وصل عدد مستخدمي @replit إلى ١٠ ملايين مستخدم حول العالم..🧵 #replit #العنصر14 #برمجة #startup youtu.be/AYiBBq22N_U
1
7
1
32
Abdalghani Abujabal retweeted
في هذه الحلقة نتحدث مع @AbbadDira عن تجربته مع الطويلة مع @ArabicWikipedia. عبّاد يجيبنا هنا عن أسئلة تتكرر حول ويكيبيديا مثل من يديرها وينظم محتواها؟ وهل هي مصدر موثوق؟ من أين يبدأ المرء إذا أراد المساهمة في ويكيبيديا؟ وما الفوائد المرجوة من المساهمة؟ youtu.be/7kaqW0VLCFA
2
3
6
Abdalghani Abujabal retweeted
حوار شيق مع @DimaDamen من @BristolUni عن أبحاثها في مجال الرؤية الحاسوبية📺🤖 (#ComputerVision) و تعلم الآلة (#MachineLearning) و عن مسيرتها العلمية بما في ذلك اختيار التخصص في مراحل التعليم المختلفة و الفروقات بين أنظمة التعليم العربية و الغربية. youtube.com/watch?v=tJBs0O8x…
1
5
11
Abdalghani Abujabal retweeted
حوار شيّق مع @Walid_Magdy من @EdinburghUni و @turinginst عن العلوم الاجتماعية الحاسبيّة (Computational Social Science) youtube.com/watch?v=soKbFF-3… #ArabicNLP #NLProc #ComputationalSocialScience