Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit 223dbbd

Browse filesBrowse files
authored
Update quality docs (#9390)
Added project quality estimation details in the documentation
1 parent 44603db commit 223dbbd
Copy full SHA for 223dbbd

8 files changed

+68
-20
lines changed

‎site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md

Copy file name to clipboardExpand all lines: site/content/en/docs/manual/advanced/analytics-and-monitoring/auto-qa.md
+68-20Lines changed: 68 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,7 @@ and calculates annotation quality based on this comparison.
1313
> **Note** that quality estimation only supports
1414
> 2d tasks. It supports all the annotation types except 2d cuboids.
1515
16-
> **Note** that quality estimation is currently available for tasks and jobs.
17-
> Quality estimation in projects is not supported.
16+
> **Note** that quality estimation is available for projects, tasks, and jobs.
1817
1918
CVAT has the following features for automated quality control of annotations:
2019
- Validation set configuration for a task
@@ -67,7 +66,15 @@ Read more about Ground Truth management [here](#ground-truth-job-management).
6766

6867
## Configuring quality estimation
6968

70-
Quality estimation is configured on the Task level.
69+
There are 2 key components related to quality estimation configuration:
70+
Ground Truth jobs and Quality settings. Ground Truth jobs are configured at the Task level.
71+
In this section, we explain how to set up a Ground Truth job.
72+
Read more about quality settings [here](#annotation-quality-settings).
73+
74+
> A **Ground truth** job is considered **configured**
75+
> if it is at the **acceptance** stage and in the **completed** state.
76+
77+
A _configured_ Ground Truth job is required for all quality computations in CVAT.
7178

7279
{{< tabpane text=true >}}
7380

@@ -125,11 +132,6 @@ The **Ground truth** job will appear in the jobs list.
125132

126133
{{< /tabpane >}}
127134

128-
> A **Ground truth** job is considered **configured**
129-
> if it is at the **acceptance** stage and in the **completed** state.
130-
131-
A _configured_ Ground Truth job is required for all quality computations in CVAT.
132-
133135
## Validation modes
134136

135137
Currently, there are 2 validation modes available for tasks: **Ground Truth** and **Honeypots**.
@@ -349,23 +351,28 @@ Import, Export, and Delete options are available from the Ground Truth job Actio
349351

350352
### Annotation quality settings
351353

352-
If you need to tweak some aspects of comparisons, you can do this from
353-
the **Annotation Quality Settings** menu.
354+
Quality settings provide options to tweak some aspects of annotation comparisons and
355+
quality estimation in general. For instance, you can configure which annotation overlap
356+
should be considered good enough or how specific annotation types must be compared.
357+
358+
Quality settings can be set up at the Task or the Project level.
359+
If a task is not bound to a project, it uses its own quality settings.
360+
Tasks inside a project can use individual quality settings or inherit settings
361+
from the project they belong to. Read more about quality settings in projects
362+
[here](#project-quality-settings).
354363

355-
You can configure what overlap
356-
should be considered low or how annotations must be compared.
364+
To set up quality settings, open the **Quality Settings** tab on the **Quality Control** page
365+
for a task or project, available in the **Actions** menu.
357366

358-
The updated settings will take effect
359-
on the next quality update.
367+
![Quality control button in the task actions menu](/images/quality-control-actions-button.png)
360368

361-
To open **Annotation Quality Settings**, find
362-
**Quality report** and on the right side of it, click on
363-
three dots.
369+
There is a number of parameters available for configuration.
370+
Hover the mouse over a **?** mark to display a description for a setting.
364371

365-
The following window will open.
366-
Hover over the **?** marks to understand what each field represents.
372+
![Quality settings page](/images/quality-settings-overview.png)
367373

368-
![Quality settings page](/images/honeypot08.jpg)
374+
After the settings are updated, remember to save the values using the **Save** button.
375+
The updated settings will take effect on the next quality update.
369376

370377
Annotation quality settings have the following parameters:
371378

@@ -375,6 +382,7 @@ Annotation quality settings have the following parameters:
375382
| - | - |
376383
| _General reporting_ |
377384
| Target metric | The primary metric used for quality estimation. It affects which metric is displayed in the UI and used for overall quality estimation. |
385+
| Job selection filter | The filter for the jobs included in quality computation. Only jobs matching the filter criteria will be included in the quality results. |
378386

379387
| _Immediate feedback_ | |
380388
| - | - |
@@ -414,6 +422,46 @@ Annotation quality settings have the following parameters:
414422

415423
<!--lint enable maximum-line-length-->
416424

425+
### Project quality settings
426+
427+
In CVAT, it is possible to group tasks into projects to share common configurations or establish
428+
a logical grouping for datasets. In this section, we explain options for quality management
429+
inside projects.
430+
431+
When tasks are inside a project, it can be convenient to reuse the same quality setup for
432+
all the project tasks. There is an option to use quality settings from the parent project
433+
for all or only for specific tasks inside a project. This is controlled by the corresponding
434+
toggle on the **Quality control** page of the task:
435+
436+
![Task quality settings - inherit project settings](/images/quality-settings-inherit.png)
437+
438+
If some of the project tasks have individual settings, a notification is displayed
439+
on the project settings page. You can enforce project settings for all the project tasks
440+
by clicking the **Force project settings** button.
441+
442+
![Force projects settings button](/images/quality-settings-force-project-settings-button.png)
443+
444+
By default, new tasks inside a project inherit the quality settings of the project.
445+
You can freely switch between these 2 modes without losing the individual configuration
446+
for the task.
447+
448+
Updating project quality includes quality computation for all the nested tasks. If the task
449+
quality is updated in a specific task manually, project quality has to be recomputed to display
450+
relevant values.
451+
452+
### Job filtering
453+
454+
Depending on the situation, you may need or don't need to include specific jobs in
455+
the quality report. For example, your workflow may require quality checks only for
456+
completed jobs or maybe you want to exclude jobs from a specific task from
457+
a project quality report. There is an option to filter which jobs are included in quality
458+
computations. You can configure this by changing the _Job selection filter_
459+
in quality settings. Only jobs matching the filter criteria will be included
460+
in the quality results. If a filter is changed, quality must be recomputed for the filter
461+
to take effect.
462+
463+
![Job selection filter](/images/quality-settings-job-selection.png)
464+
417465
## Comparisons
418466

419467
### Tags

‎site/content/en/images/honeypot08.jpg

Copy file name to clipboard
-20.2 KB
Binary file not shown.
Loading
Loading
Binary file not shown.
Loading
Loading
Loading

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.