weekly update

• no updates this week as I was attending ACL 2024 conference.

weekly update

• no updates this week as I was attending ACL 2024 conference.

weekly update:

• no updates this week as there were no requests for additional support so far

weekly update:

• Updated the project page on meta with current status: https://meta.wikimedia.org/wiki/Research:Develop_a_model_for_text_simplification_to_improve_readability_of_Wikipedia_articles/FY24-25_WE.3.1.3_content_simplification
• Identified informative/interpretable metrics to evaluate performance of simplification models. For the manual evaluation, the most common approach is to judge 3 dimensions: simplicity (is the text simpler), fluency (is the text grammatical), and adequacy (does the text preserve the meaning). Looking into the literature, we can define automated metrics which approximate judgements along these dimensions.
  • Simplicity: Measure the change in readability score using our readability scoring model for Wikipedia articles.
  • Fluency: Count the number of grammatical errors using LanguageTool
  • Adequacy: Measure the factual consistency between the original and simplified text to detect, e.g., “model hallucination”. This is a very active field of research and several methods have been proposed recently such as FactCC (based on a trained classification model), SummaC (based on textual entailment), or QuestEval (based on question generation and answering).
• The advantage of these metrics is that they are more interpretable and that they dont require reference samples from a ground truth dataset. This will hopefully make it easier to obtain confidence about whether models are “good enough” for potential use in production.
• Started to implement metrics so we can automatically measure model performance.
• Next step: Finalize implementation of metrics for evaluating model performance and run on representative sample with existing model.

Update: I added a draft section about this to the handbook https://office.wikimedia.org/wiki/Research/Handbook/Communication#Communicating_with_public_media

weekly update:

• I put together documentation for article recommendations with different tools from Research for experiments WE.3.1.1 (doc)
• Shared documentation with hypothesis owner for feedback.

weekly update:

• Built two working model prototypes for content simplification/summarization
  • 1) Simplification: Generate a simplified version of a section/paragraph using simpler language.
  • 2) Section-gist: Generate a plain language summary of a section (i.e. combining simplification and summarization)
• Put together detailed documentation about the two models with examples and tutorial notebooks on how the models can be run (doc). I will add these updates to the project-meta page too in the next week or so.
• Exploring alternatives for automatic evaluation of models to make it easier to iterate through different model variants without the need for manual evaluation.
• Experimenting with the test-deployment in the staging instance of LiftWing (available thanks to the ML-Team) of the Aya-23 model (Aya-23-8B) which is used for the section gist model. The model works and returns requests in a reasonable time. Though the model quality seems substantially lower than the larger model I used in the experiments using the external API. ML Team is planning to test-deploy the larger model version (Aya-23-35B) in the next weeks.

Update: Fabian, Isaac, and I coordinated and left detailed feedback and comments in the design doc. Resolving the task as the request in the task description is completed. We will of course continue to engage in any follow-up discussions in the doc.

For the next round of updates, could you add the following items:

• Add to Publications page and Knowledge Gaps publications:
  • Mykola Trokhymovych, Indira Sen, Martin Gerlach. 2024. An Open Multilingual System for Scoring Readability of Wikipedia. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL '24).
• Also add blurb to Knowledge Gaps updates:
  • Date: Aug 2024
  • Title: A multilingual model for measuring readability
  • Blurb: We published a new paper at ACL ‘24 where we develop a multilingual model to score the readability of Wikipedia articles across languages.
  • Link: https://arxiv.org/abs/2406.01835

weekly update

• Shared results on section-gists of Wikipedia articles (T369288#10018291) with Web team as one potential approach for experiments on summarization and simplification for readers.

weekly update:

• Ran small-scale experiments to automatically generate section-gists (i.e. plain-language summaries of a section) for Wikipedia articles using different models.
  • Test-dataset with original and simplified text from only the lead section of 10 articles in English, German, Portuguese (see spreadsheet)
  • Sample-dataset of several sections from the same article without the reference-simplification (see spreadsheet)
• We are able to automatically generate section-gists (plain-language summaries of a section) of Wikipedia articles across languages using the LLM Aya-23. Based on qualitative evaluation of small sample (<100), the results are promising that the model can provide a concise and easy-to-understand overview of a long piece of text. I successfully tested English, German, and Portuguese, but the model supports 23 languages supposedly covering nearly half the world’s population in terms of speakers (which is way more than any other comparable LLM that I am aware of).
• Evaluation of text simplification is challenging. Automated metrics (such as SARI) dont seem to be very useful to rigorously assess model performance for specific use cases. Simple baselines such as returning a truncated version of the input-text can yield surprisingly good results. As a result, we should probably not rely on these metrics and, instead, resort to manually judging (samples of) the model output which is, however, much more resource-intensive.

weekly update:

• Android reached out to understand more about recommended content within search. I shared some resources from research on search and articles recommendations (e.g. from list building)
• Web is starting with first experiments on recommendations in search. Providing support for using the article-similarity search from the list-building tool https://list-building.toolforge.org/
• Web is preparing to start thinking about experiments on simplifications which will happen later in the quarter. Ongoing discussions about what simplification would be useful and how to evaluate.

weekly summary:

• Clarified the criteria and constraints for the model
  • Multilingual: support (some) languages other than English
  • Openness: it needs to be open.
  • Resources: We need to be able to host the model in our infrastructure in LiftWing with reasonable inference time.
  • Use-case: We need to define the use-case; for example, should simplification be on the level of sentence, paragraph, section, or the full article?
  • Quality: Ensure the output is useful in practice according to some metric
• Did a deep-dive on recent works on text simplification with language models (reviewing 26 papers from the past 2-3 years see below [1]). This helped me to understand the most common and promising strategies to approach the task and to identify challenges.
• Some of the main learnings came from a paper (Paper Plain) which aims to improve access to medical papers. Based on interviews with readers about barriers for interacting with content and usability testing, they identify section gists as a valuable and most frequently-used feature by non-expert readers.
  • Operationalize simplification as a plain-language summary of a section
  • For example, use a prompt: “Summarize content you are provided with for a fifth-grade student.”
  • This approach seems highly effective as most LLMs are explicitly trained on the task of summarization across languages (e.g. XLSum)
• I also reviewed some of the recent large language models that could be good candidates. I identified the Aya-23 model as a promising candidate model
  • Multilingual: It is multilingual supporting 23 languages (these languages cover approximately half the world’s population)
  • It is an open-weight model and available in Hugging Face
  • Given the successful test-deployment of the similarly-sized Gemma2-27B-it on LiftWing (T369055), it seems that this model could in principle be also hosted.
  • The model can be prompted to generate plain-language summaries of sections without fine-tuning
  • Some initial tests with the API-endpoint looked promising across several languages.

weekly update:

• we succesfully ran the second part of the pilot survey
• we will start analyzing the data next week. based on the results we will decide on next steps: either continue and scale, or re-assess the general approach. we will post results on the meta-page https://meta.wikimedia.org/wiki/Research:Understanding_perception_of_readability_in_Wikipedia#Pilot_survey:_version_3
• for now I am closing the task as the goal of running the survey was accomplished.

weekly update:

• I reached out to all hypothesis owners in WE.3.1 individually We also had a joint meeting to support Support WE.3.1. From this, I obtained a much clearer picture about the needed support
• WE.3.1.1: might need light support (consulting) for options to generate recommendations. Substantial support needed for experiments on simplification/summarization. Web Team is starting to think about specifications in more detail. So these are ongoing discussions at the moment.
• WE.3.1.4: No support needed at this point. They will focus on figuring out what work would need to be done for scaling search (e.g. morelike). They also want to start looking into vector search which would likely require some support from Research (e.g. creating vectors/embeddings). However, they are starting from scratch and during Q1 will start to figure out what they want and what support would be needed in the future.
• WE.3.1.5: No support needed at this point. They want to skip the use of orphans for the first round of experiments.

Weekly update:

• Systematizing open questions for successful model development
  • Infrastructure requirements: size, performance (inference time)
  • Product requirements: Languages, Quality, Use-case
  • Model requirements: Openness (can we host), Effectiveness (does the model have a chance to perform well), training (supervised, in-context learning, zero-shot as is), evaluation
• Gathering input from product team (Web) on intended use to tailor model specifications (languages, input/output format, quality boundaries). These are ongoing discussions but should become more clear in the next week or so
• Learning about updates in ML infrastructure which expands the potential set of candidate models. The ML team announced that we will have new servers will GPUs available for model training and hosting. This will allow us to use larger (and potentially better) models. Similarly, I am following the test-deployment of the Gemma2-model on LiftWing T369055. If successful, this could constitute a promising candidate model for the simplification task (or models with similar size/architecture).
• Reviewing scientific literature to compile list of candidate models for the task. I have identified more than 10 recent papers (2023/2024) on using LLMs for text simplification approach. This is very insightful to understand which models are most promising for the specific task. For example, a promising model is Aya-23, an open model with dedicatedly multilingual support and, in principle, compatible with out infrastructure constraints (see above)..

weekly update:

• ran the first part of the survey this week. next week will be the second part.

@MGerlach if you have any other observations, please add them to this task. We'll review this as part of the retro and it's good to have as we prepare for next year and know what we need to address in advance. I'll resolve the task after you're done :)

@KinneretG A gree with everything you wrote. In general though I did like OpenReview. One additional comment:

• setting of deadlines was public (e.g. for submission deadline of reviews for PC members). We often set the deadline in OpenReview to some time (e.g. a few hours or a day) after the officially communicated deadline in order to accommodate also those who run into technical or other issues. However, the internally set deadline is also visible publicly for the relevant folks. This lead to some confusion as to what the actual deadline is.

Moving this to the next quarter (FY2024-25-Research-July-September) as the work is not yet fully completed

• the MR for the pipeline in airflow is submitted.
• this needs some code-review by research-engineering before it can be merged. this should be completed by mid-July (see T361929#9935541)
• we expect to resolve the task in the next 1-2 weeks

We succesfully ran the research track at wikiworkshop2024.
Closing this task as all sub-tasks have been completed.

Moving to FY204-25-Research-July-September for now as we need 1 more week to run actually run the survey.

weekly update:

• finished internal testing
• implemented two changes based on feedback: i) make the survey shorter by removing a few pairs, ii) add a question about how regularly participants use Wikipedia.
• figuring out available resources on prolific
• planning to deploy next week

weekly updates:

• finally I managed to spend some time on this.
• I figured out that one of the main bottlenecks was to calculate the indegree for each potential link-target (this is crucial since we want to prioritize articles with low indegree such as orphans). My initial approach was to use the Linkshere-API. However, this requires a separate call for each individual article. A much cheaper alternative is to query the replicas, with only a single query for potentially hundreds of articles for which we want to get the indegree (see the example in quarry). The replicas can be easily queried from toolforge (wikitech-documentation, example script in PAWS). For an example article, I could reduce the query-time 10-fold.
• I integrated this (and a few other fixes to improve the recommendations) into the latest version of the tool. Example: https://linkrec.toolforge.org/readmore?lang=en&title=Tiwanaku
• the example still takes some time but its much less likely to just timeout since the number of API calls is much smaller.
• in case, we need much more substantial speedups, we might want to resort to other heuristics such as the one mentioned above (T361944#9809372) using morelikethis in cirrussearch combined with the orphans-template

weekly update:

• finalized the mulitlingual experiments beyond English
• wrote up the results on the meta-page https://meta.wikimedia.org/wiki/Research:Develop_a_model_for_text_simplification_to_improve_readability_of_Wikipedia_articles/First_round_of_experiments#Multilingual_experiments
• Main takeaway is that the model seems to work well for some language (but not all for others). "For a few languages scores were similar to English (German, Italian, Catalan); for many languages scores were slightly lower (Spanish, Basque, French , Portuguese, Dutch); and for some languages performance was substantially worse (Greek, Armenian, and Russian)"

weekly updates:

• Ran and evaluated experiments with fine-tuned Flan-T5 on other languages. The main problem is that if the model is only trained on article pairs (original/simplified) in English, the model output (simplified) will most often be in English, even if the model input (original) is in, say, German or French. Thus, the model will perform a simplification AND translation in to English since it has seen only English samples of simplfiied text.
• As a potential solution I am experimenting with the following setup: i) fine-tune the model with additional samples from the other languages, ii) add a prefix "Simplify in <LANG>: " to the model-input. This will provide additional instructions for the model treating the simplifications in different languages as different but related tasks.

weekly update:

• internal testing of the limesurvey (including integration with prolific)
• next week: deployment on prolific

In T349774#9913345, @DDeSouza wrote:

@MGerlach done.

@DDeSouza could you please remove the pdf for the paper (the sooner the better so it doesnt get indexed by GoogleScholar etc):
"Structural Evolution of Co-Creation in Wikipedia [PDF]
Negin Maddah and Babak Heydari"
They opted out of having the pdf put on the website.
Thank you

weekly update:

• no updates

weekly update:

• finalized the Limesurvey
• will test the Limesurvey internally next week (after that can be deployed)

weekly update:

• started setting up the experiments for evaluating fine-tuned Flan-T5 on other languages
• will run experiments next week

weekly update:

• finalized the program for the research track (sessions, session chairs) https://docs.google.com/spreadsheets/d/1KXCIitFfd57bRwuL30jciJEhipq4-wMI_hzvW0JmBCo/edit?gid=0#gid=0
• wrote a doc with instructions for session chairs

weekly update:

• no update. main focus was preparing for attending ICWSM T362416

weekly update:

• no update. main focus was preparing for attending ICWSM T362416

weekly update:

• no update. main focus was preparing for attending ICWSM T362416

weekly update:

• iterating on the sessions and the session chairs
• planning to finalize next week

FInal update:

• send notifications to authors
• website will be updated with accepted extended abstracts and the members of the program committee
• also send thank-you email to reviewers and inviting them to register to attend the event

weekly update:

• no update since my main focus was on Wiki Workshop T352543

weekly update:

• no update since my main focus was on Wiki Workshop T352543

weekly update:

• working on finalizing the limesurvey with manually verified samples

weekly update:

• finalized session format and alignment with workshop schedule
• created first draft of organizing papers into sessions
• created shortlist of session chairs

weekly udpate:

• made decisions about all submissions
• will be sending out notifications to authors next week (May 27)

weekly update:

• investigated how we could take advantage of existing recommendations from the RelatedArticles feature. the feature obtains recommendations from queries to cirrussearch' morelike. There seem to be two interesting options from the options available in the API
  • filtering or boosting with the number of inlinks (i.e. prioritizing articles with low indegree), e.g., via Srsort:incoming_links_asc
  • using the m̀orelikethis`option which allows to specify specific templates. In our case, we could then require recommendations to have the Orphans-template. The query then yields related articles that are marked as orphans via the corresponding template. Example query: https://en.wikipedia.org/w/api.php?action=query&list=search&srsearch=morelikethis:Tiwanaku%20hastemplate:%22orphan%22&srlimit=10
  • especially, the latter approach seems promising (as it could also be adapted to other languages) but would require some evaluation of the quality of the recommendations. while the recommendations are orphans, it is not clear how "well" they are related to the query-article. however, in contrast to link translation, the queries via morelike/morelikethis are much faster.

weekly update:

• finishing manual validation of the selected snippets that will be shown to participants

weekly update:

• no update. ready to pick up next week since T354653 was completed

weekly update

• Finished write-up of first exploratory experiments https://meta.wikimedia.org/wiki/Research:Develop_a_model_for_text_simplification_to_improve_readability_of_Wikipedia_articles#First_exploratory_experiments

weekly update:

• Started to plan format and schedule and candidates for sessions
• Started to group papers into themed sessions

weekly update

• Sending reminders to reviewers
• Made sure we have 3 emergency reviewers available for short-notice reviews during next week
• Drafted decision emails
• Preparing decision stage (to be completed next week)

@Htriedman this is super useful. thanks for looking into this and sharing your results.
For future reference: The above code looks for all pages that use the template {{Closed_rfc_top}}. This template is used to close a Request for comment (RfC) on a talk page or a noticeboard. Thus, we can easily identify pages that contain relevant (i.e. closed) discussions. The above code then goes a lot further by parsing the content of the page to extract the discussion in a tree-like structure (capturing comments etc).

weekly update:

• finalized design of the Limesurvey
• next step: add selected snippets to survey

Ran session at Wikimedia Hackathon (see program)

Unassigning myself as I am not planning to work on this in the near future (next 3 months)

I ran a session on ML models developed by the Research Team that are hosted in LiftWing (T362751). There was a lot of interest with ~20 attendees and a good discussion. Several developers were interested in exploring how to use the models in their projects. For example, folks from Maroccon Arabic Wikipedia were interested to use the revert-risk model for a bot for vandalism detection. There was also questions about how to use the models for third-party wikis. More generally, there was an interest in using larger prompt-based LLMs in different applications.

I didnt get to work a lot on this during the hackathon as I was mostly focusing on T362805.

set up an API endpoint on cloud-vps as a backend from where we can query the dataset for individual pages.

• repo https://gitlab.wikimedia.org/htriedman/dp-pageviews-tool-backend
• example query https://dp-pageviews.wmcloud.org/api/v1/en.wikipedia/38375/2024-04-02

weekly update:

• finalized assignment of reviewers to papers. mainly relied on bids from reviewers (90% entered their bids) with some manual adjustments
• sent review assignments to reviewers (deadline is May 20)

update:

• scheduled the session on ML models in the official program
• started preparing the content of the sessions (which slides and what I could demo)

weekly update:

weekly updates:

• progressing on the write-up of the results. I spent some time to collect some representative examples for complementing the results with some qualitative evaluation. Looking at the output of different models in more detail, it seems there are strong differences that are not well captured by the automatic evaluation metrics (e.g. SARI)

weekly update:

• no update this week because I didnt manage to free up time for this task (shorter week, annual planning deadlines, wiki workshop submission deadline)

weekly update:

• prepared and submitted an extended abstract to wikiworkshop with preliminary results. this is not directly related to the survey but will give us additional feedback for improving later versions of this survey.
• Decided on final sampling of snippets and design details (buttons etc)
• next step: implement changes into the Limesurvey

weekly update:

• submission deadline passed: we received 50 submissions
• reviewer bidding is ongoing until end of this week
• next step: finalizing review assignment early next week

Request for a small (but important) change:

• on https://wikiworkshop.org/call-for-papers/ the email at the very bottom has a typo and should read wikiworkshop@googlegroups.com

Thanks

weekly update:

• no update this week

weekly update:

• extracted 100 suitable pairs of original/simplified snippets for comparison (sheet)
• from this we will select two groups of each 10 pairs for the pilot: i) "treatment group": the two version of the pair are very different in their automatic readability score; ii) "control group": the two versions of the pair are very similar in their automatic readability score.

weekly update:

• no update (still working on T354653)

weekly update:

• finished experiments with Flan-T5 model in English
• writing up results for the meta-page (incomplete draft as doc)

weekly update:

• checked the review form
• testing the bidding interface for reviewers and prepared upcoming comms with PC members
• monitoring submissions (deadline next week) and answering questions by authors