Thank you so much for your interest in the QCRI Arabic Data Annotation Project, and for your continued support to TWB. We are very excited to be working together on this pilot in partnership with the Qatar Computing Research Institute to evaluate a large language model for Modern Standard Arabic, and to contribute to our shared mission to address communication gaps.
Many thanks to those of you who joined the introductory webinar. For those of you unable to attend or looking to rewatch the presentation, here’s a link to the recording to watch back
The webinar addressed these topics:
Project background, aims and partnership with QCRI,
How to access and use the Label Studio platform for annotation tasks,
Task instructions, tips, and quality guidelines, including worked examples,
Guidelines for handling sensitive content,
Recognition program for this project,
Where to ask questions or seek help.
So what comes next?
To help prepare for the first batch of tasks in Label Studio, we have prepared a short quiz with questions from the webinar and practice examples of texts to annotate with a relevant tag. After watching the webinar, we ask you complete this quiz before we assign your first project in Label Studio, ideally by this Friday August 23rd, but let us know if you need more time. We do not expect a perfect score and also want to hear your feedback. The last question is an open response for your questions. We think the quiz should take up to 20 minutes to complete. @Polly will be the dedicated Project Officer supporting you. You can reach her over email at polly.harlow@clearglobal.org or tag her here. She will also be in touch to share any feedback from the quiz and coordinate the first tasks in Label Studio.
Thank you all for your attendance at the webinar and to all team members who have completed the practice quiz and share valuable feedback. It is great to have you on board! We are waiting for QCRI to finalize the Label Studio platform setup and will be in touch once ready to coordinate the first batch of tasks
In the meantime, we want to share your feedback from the quiz, let you know how we are responding to this, and to answer a couple of questions shared.
Feedback
It was hard to apply tags correctly in the practice texts without viewing the original prompt. Sorry to make the practice more difficult than it needed to be! Thankfully you will be able to see the main prompt when working in Label Studio. We will also share very soon task guidelines which include longer examples with explanations.
Some responses contained multiple issues and could have more than one tag added. We have shared this feedback with QCRI. It will now be possible to apply more than one tag to a response, up to a maximum of 3 tags. You will also be able to highlight the particular phrase in the response which contains each issue. You should still stop at the response where you first notice an issue, review just this response, and apply up to 3 tags. Guidelines and video guidance coming soon to demonstrate this update!
It would be helpful to have a ‘severity hierarchy’ to know which tags to apply first if it is possible to add multiple tags. We have shared this with QCRI too. They have now applied a severity hierarchy, splitting the tags into three different levels of seriousness. If you find more than one type of issue in a response, you should prioritize first the most serious tags. Again, more guidance coming to demonstrate this!
The tag explanation is not always clear and it would be useful to have an additional resource with more information about the tags and when to apply these. In addition to task guidelines, we are working on a one pager which you can have on hand while annotating. This will include more detailed description of each tag and when to apply this. We hope this will be useful. This and task guidelines will be shared in this forum thread.
Questions
How should I handle assistant responses with minor grammatical issues? Should I apply a tag to these? Yes, you should apply a tag, select submit, and stop reading the rest of the conversation if you notice grammatical errors in an assistant response. With the new interface update, you will be able to highlight the error. You should not tag a response if you have preferential feedback on the phrasing, only if the error is a clear grammatical inaccuracy, including if minor.
Will I receive feedback on my tasks? Yes! @Muhannad, our Arabic Language Lead will be reviewing some of the annotations submitted. We will aim to share individual feedback on your task and any tips over email. General team feedback will be shared in this dedicated QCRI feedback and questions thread. You can use this too for any questions or feedback you have.
Hello QCRI team,
I am facing a problem, that whenever I try to submit a conversation after annotating it, a speech box pops up:
Checkbox “stop_condition” is required.
I am wondering what does this mean.
Hi
I had the same issue when trying to submit a conversation after annotating it. I reached out to the project officer regarding this problem. Let’s see what happens!
Hello @khedribrahim174 and @lama_saleem . Thanks for sharing this issue. I’ve already forwarded this to our partner and will update you once as soon as I hear from them. Thanks for letting us know!
Hello @khedribrahim174@lama_saleem. Our partner just replied and mentioned that the issue stems for a missing required annotation. When you work on the annotations, you must mark one of these before submitting:
If there are no problems and no annotations to be made, you mark “No problem with assistant detected” and you submit the conversation.
If, however, there’s an issue to report, you mark “User dangerous, inappropriate, sensitive text”, you make the necessary annotations and then you submit the conversation.
Thank you for the clarification; however, if there is a problem with the assistant’s reply, shall I mark it “User dangerous, inappropriate, sensitive text”?
Thank you so much for your interest in the QCRI Arabic Data Annotation Project, and for your continued support to TWB. We are very excited to be working together on this pilot in partnership with the Qatar Computing Research Institute to evaluate a large language model for Modern Standard Arabic, and to contribute to our shared mission to address communication gaps.
So what comes next?
For questions about the dedicated recognition program for this project, please contact recognition@translatorswithoutborders.org.
