Thank you for being part of the team! We will update this topic shortly to include useful materials for reference while working on the QCRI project.
Hello QCRI annotation team! 
With thanks for your patience in us kicking off tasks in Label Studio, I am coming with an important update today and to share further resources to support your work.
Firstly, we are finalizing our planning with QCRI and will be in touch as soon as we are ready to go to provide details of the first batch of tasks. We will contact all contributors who joined or watched the introductory webinar and completed the practice quiz. For the first trial batch of tasks, we are aiming to work with around 15 team members – please rest assured there will be ample opportunity to work on further tasks if you are not available for the first trial batch!
In the meantime, we would like to share a few key updates which come in response to your feedback on the quiz. Thank you very much for your helpful insights! 
QCRI has now made several important changes to the interface where you will work in Label Studio.
What are the changes?
- It is now possible to add more than one tag to an assistant response Label Studio, up to a maximum of three tags. There is a highlight function to highlight the specific part of the response to which each tag applies. You should still stop reading further into the conversation and submit your annotations once you reach the first assistant response which includes any issues, but you can add multiple tags to this response. This does not mean you have to or should add more than one tag – a single tag is absolutely fine if there is only one issue type to annotate!
- To help you decide which tags to apply when there are more than three types of issue within a single response, QCRI has implemented a ‘severity hierarchy’. This shows which issues are considered more serious and should be tagged in preference. You can read more about this and see the hierarchy in the task instructions shared below.
- There is a new checkbox to apply when a user response violates safety (such as by asking the assistant for dangerous information). The annotation tags, however, are for assistant responses only. You should never stop or submit on a user response and ONLY on an assistant message. The only thing to check for in the user responses is safety; if the user response violates safety you should use the dedicated checkbox for this, but keep reviewing the conversation until the first assistant response with issues.
- You can now delete your annotation if you make a mistake while working using a delete function, and skip a task if you do not feel comfortable working on this. To use the skip function, you will need to access your Label Studio project using the “Label” pathway, as shown in the task instructions.
What does this mean? New resources 
- Some of the information provided in the introductory webinar is now outdated. The details about QCRI, the aims of the project, recognition, the categories of tags, and the possibility to skip sensitive content remains the same, but the interface now looks a little different and some changes to the process have been made as detailed above.
- For the latest guidelines, please always refer to this document with TASK INSTRUCTIONS and this accompanying CRIB SHEET which we recommend to keep to hand while annotating. The instructions outline the project, process, tips for the task, and provide a pre-delivery checklist. The crib sheet shows you just the different tags with a short explanation, as well as keyboard shortcuts you can use to make your work easier.
- Our wonderful Language Lead @Muhannad has also created a SHORT VIDEO INSTRUCTION to demonstrate the updated process.
- As many of you requested further examples to practise with, we have created an additional practice quiz
. This is not required and we will not evaluate this. You are also welcome to respond multiple times. This is only for you to use if helpful to you and we will keep the quiz open for responses without a set deadline. An answer key is provided. If you take the quiz and have doubts or specific questions, you can share your feedback and questions with other team members and the TWB team in this forum thread.
Let us know if you have questions and thank you again for your support to this project!
2 Likes