
Associate Director, Digital Licensing @wileyinresearch
Oxford/Bristol, England
Joined July 2013
- Tweets271
- Following577
- Followers146
- Likes235
"Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information." (via @DWFlanagan)
Awesome and highly useful: FineWeb-Edu 📚👏
High quality LLM dataset filtering the original 15 trillion FineWeb tokens to 1.3 trillion of the highest (educational) quality, as judged by a Llama 3 70B. +A highly detailed paper.
Turns out that LLMs learn a lot better and faster from educational content as well. This is partly because the average Common Crawl article (internet pages) is not of very high value and distracts the training, packing in too much irrelevant information. The average webpage on the internet is so random and terrible it's not even clear how prior LLMs learn anything at all. You'd think it's random articles but it's not, it's weird data dumps, ad spam and SEO, terabytes of stock ticker updates, etc. And then there are diamonds mixed in there, the challenge is pick them out.
Pretraining datasets may also turn out to be quite useful for finetuning, because when you finetune a model into a specific domain (as is very common), you slowly lose general capability. The model starts to slowly forget things outside of the target domain. But this is not only restricted to knowledge; You also lose more general "thinking" skills that the original data demanded, but your new domain might not exercise. i.e. in addition to the broad knowledge fading, those computational circuits also slowly degrade. So there are likely creative ways to blend the pretraining and finetuning stages.
53
Attending the London Book Fair this year?
Visit the main stage on Wednesday, March 13th to watch Jay Flynn, Wiley's Executive VP & General Manager, Research & Learning, take part in a panel for 'Academic Publishing Landscape in 2030 and Beyond.'
ow.ly/f9Z250QQFqh
#LBF24
95
Really pleased to see a Wiley mention in this week's Scholarly Kitchen article on the influence of Wikipedia!
ow.ly/nxfE50LrSCt
We've just launched a new division: Wiley Partner Solutions. 🙌
The new division supports societies and academic publishers’ plans for a successful transition to #OpenResearch.
Find out more: ow.ly/wS8h50Lgsf5
Check out this upcoming webinar about applications in model-based drug discovery & development, featuring speakers from the editorial team of the journal CPT: Pharmacometrics & Systems Pharmacology.
ow.ly/ZM0e50L0xs5
My colleagues recently sat down with Dr. Charles Young, founding editor-in-chief of the journals Clinical Case Reports and Health Science Reports and practicing emergency room physician. Find out what he sees as emerging trends in healthcare. ow.ly/w5VG50Kitqx
I'm bragging on my colleagues in Wiley's rights & licensing team today: check out the different industry groups they work with to support digital preservation, copyright, industry standards, and more. ow.ly/6Lir50KbhF3
Really pleased to see Wiley featured in CCC's Velocity of Content blog this week, highlighting a partnership that offers small and mid-size enterprise affordable access to the Wiley portfolio
ow.ly/y7yX50JY1p1
3
Quality is at the center of Wiley's content programs. Check out this post on the Couranto blog by my colleague Greta Boonen, about the partners Wiley relies on to deliver trusted content across our publishing portfolio. ow.ly/W44A50JUAlM
Have you heard? Wiley signed DORA! 👇
We’re formally committing to diverse measures of impact and contribution. For researchers, that means more equitable recognition.
Here's the info: ow.ly/TFVs50J2t2I
Wiley has joined 58 other publishers on the Education Platform, a tool from Copyright Licensing Agency that helps UK school teachers integrate library resources into their teaching while staying copyright compliant. Learn more: ow.ly/tneE50IZqYS
Wiley has added over 22,000 reference books to our preservation agreement with the CLOCKSS digital archive! Learn more: ow.ly/qbjQ50IBPo3
Congratulations to the recipients of the 20th annual Wiley Prize in Biomedical Sciences: David Baker, Demis Hassabis, and John Jumper.
Join this webinar to celebrate their achievement and learn about their work.
ow.ly/qBSm50IrgUB
The Wiley Foundation 20th annual Wiley Prize in Biomedical Sciences will be awarded to David Baker, @demishassabis & John Jumper for procedures to predict highly accurate three-dimensional structures of protein molecules from their amino-acid sequences.
secure.wiley.com/TheWileyPri…
17 journals from the Wiley portfolio are expected to receive their first Journal Impact Factor in the 2022 Journal Citation Reports (JCR)! Congratulations to the teams behind these publications.
ow.ly/Ewtt50Iaf8a
1
Dale Morgan retweeted
S&P Global and IHS Markit Announce Final Regulatory Approval for Merger and Expected Close on February 28, 2022: ow.ly/8vZB50I4hhe
Dale Morgan retweeted
Wiley is now available to active @Wikipedia editors through The Wikipedia Library!
We’re excited to announce a new partnership with the Wikipedia Library, a project of the nonprofit @Wikimedia, which will connect @Wikipedia editors with full-text access to Wiley Online Library.
Find out more here: ow.ly/YPpl50I1AH3
Dale Morgan retweeted
We’re excited to announce a new partnership with the Wikipedia Library, a project of the nonprofit @Wikimedia, which will connect @Wikipedia editors with full-text access to Wiley Online Library.
Find out more here: ow.ly/YPpl50I1AH3
Dale Morgan retweeted
What is #indexing, and why is it important for journals?
Find out more in this great intro piece:
wiley.com/network/societylea…
Francisco Partners to Acquire IBM’s Healthcare Data and Analytics Assets newsroom.ibm.com/2022-01-21-…
1




















