Skip to content

Migrate Gentoo importer to advisory V2#2090

Open
ziadhany wants to merge 12 commits intoaboutcode-org:mainfrom
ziadhany:gentoo-migration
Open

Migrate Gentoo importer to advisory V2#2090
ziadhany wants to merge 12 commits intoaboutcode-org:mainfrom
ziadhany:gentoo-migration

Conversation

@ziadhany
Copy link
Collaborator

@ziadhany ziadhany commented Dec 30, 2025

@ziadhany ziadhany changed the title Add initial migration to Gentoo importer v2 Migrate Gentoo importer to advisory V2 Dec 30, 2025
@ziadhany ziadhany marked this pull request as ready for review January 1, 2026 13:58
@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026

gentoo importer V2 logs:

/home/ziad-hany/PycharmProjects/vulnerablecode/venv/bin/python /home/ziad-hany/PycharmProjects/vulnerablecode/manage.py import localhost:8000 --all 
INFO 2026-02-18 11:08:30.171924 UTC Pipeline [GentooImporterPipeline] starting
INFO 2026-02-18 11:08:30.172113 UTC Step [clone] starting
INFO 2026-02-18 11:08:30.172208 UTC Cloning `git+https://anongit.gentoo.org/git/data/glsa.git`
Importing data using gentoo_importer_v2
INFO 2026-02-18 11:08:37.365248 UTC Step [clone] completed in 7 seconds
INFO 2026-02-18 11:08:37.365399 UTC Step [collect_and_store_advisories] starting
INFO 2026-02-18 11:08:37.409970 UTC Collecting 3,814 advisories
INFO 2026-02-18 11:08:50.000170 UTC Progress: 10% (382/3814) ETA: 113 seconds (1.9 minutes)
INFO 2026-02-18 11:09:00.487704 UTC InvalidVersion constraints version: 1.3* error:'1.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:09:02.818338 UTC Progress: 20% (763/3814) ETA: 102 seconds (1.7 minutes)
INFO 2026-02-18 11:09:16.146918 UTC Progress: 30% (1145/3814) ETA: 90 seconds (1.5 minutes)
INFO 2026-02-18 11:09:22.929879 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:09:22.930098 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:09:28.732313 UTC Progress: 40% (1526/3814) ETA: 77 seconds (1.3 minutes)
INFO 2026-02-18 11:09:41.584200 UTC Progress: 50% (1907/3814) ETA: 64 seconds (1.1 minutes)
INFO 2026-02-18 11:09:46.868690 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:09:46.868922 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:09:54.568607 UTC Progress: 60% (2289/3814) ETA: 51 seconds
INFO 2026-02-18 11:09:55.615427 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:10:03.900624 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:10:03.900804 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:10:06.323881 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:10:06.324103 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:10:07.005596 UTC Progress: 70% (2670/3814) ETA: 38 seconds
INFO 2026-02-18 11:10:12.655711 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:10:12.655953 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-18 11:10:19.674850 UTC Progress: 80% (3052/3814) ETA: 26 seconds
INFO 2026-02-18 11:10:32.068708 UTC Progress: 90% (3433/3814) ETA: 13 seconds
INFO 2026-02-18 11:10:44.422469 UTC Progress: 100% (3814/3814)
INFO 2026-02-18 11:10:44.458357 UTC Successfully collected 3,814 advisories
INFO 2026-02-18 11:10:44.458538 UTC Step [collect_and_store_advisories] completed in 127 seconds (2.1 minutes)
INFO 2026-02-18 11:10:44.458627 UTC Step [clean_downloads] starting
INFO 2026-02-18 11:10:44.458699 UTC Removing cloned repository
INFO 2026-02-18 11:10:44.567000 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-02-18 11:10:44.567182 UTC Pipeline completed in 134 seconds (2.2 minutes)

Process finished with exit code 0
from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[2]: 0
AdvisoryV2.objects.count()
Out[3]: 3814

gentoo importer V1 logs:

Importing data using vulnerabilities.importers.gentoo.GentooImporter
Invalid safe_version 3.24.48:3 - error: '3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
Invalid safe_version 6.9.3:6 - error: '6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
Successfully imported data using vulnerabilities.importers.gentoo.GentooImporter

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026

Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany Thanks, see some feedback below.

Comment on lines 144 to 145
if invert:
constraint = constraint.invert()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany why do we invert fixed range, we should report fixed range as is.

Copy link
Collaborator Author

@ziadhany ziadhany Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@keshav-space we have two type of package version

  • unaffected
  • vulnerable
GLSA-201405-01 , CVE-2014-0004
  <affected>
    <package name="sys-fs/udisks" auto="yes" arch="*">
      <unaffected range="rge">1.0.5</unaffected>
      <unaffected range="ge">2.1.3</unaffected>
      <vulnerable range="lt">2.1.3</vulnerable>
    </package>
  </affected>

we invert the unaffected (safe_version) to get the affected version if it isn’t specified.
see line: 159, 162

the main question is that does the unaffected mean fixed range, if yes I should update this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the main question is that does the unaffected mean fixed range, if yes I should update this

@ziadhany Yes, unaffected is fixed range see resolution section here https://security.gentoo.org/glsa/201405-01 so if we get unaffected range it should be treated as fixed range.

Also rge means revision greater than equals and rgt means revision greater than lets mention this in comments.

<unaffected range="ge">2.1.3</unaffected> is pretty straight forward it translates to fixed range vers:ebuild/>=2.1.3. But revision range are bit tricky here <unaffected range="rge">1.0.5</unaffected> translates to vers:ebuild/>=1.0.5|<1.1 see the interpretation here https://security.gentoo.org/glsa/201405-01

similarly for this advisory https://security.gentoo.org/glsa/202004-13 we have these unaffected range

<unaffected range="rge">2.23.3</unaffected>
<unaffected range="rge">2.24.3</unaffected>
<unaffected range="rge">2.25.4</unaffected>
<unaffected range="rge">2.26.2</unaffected>

and these would be interpreted as fixed range

vers:ebuild/>=2.23.3|<2.24
vers:ebuild/>=2.24.3|<2.25
vers:ebuild/>=2.25.4|<2.26
vers:ebuild/>=2.26.2|<2.27

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany I went through your pr aboutcode-org/univers#181 and I also reviewed the gentoo version documentation https://projects.gentoo.org/pms/8/pms.html#x1-250003.2. It turns out that gentoo packages do not follow any particular versioning standard, so it becomes difficult to create a range from a single version such that the range includes only revisions of that version.
The only reliable way to generate a bounded range for a revisione version is to retrieve all versions of a particular gentoo package and sort them as described here https://projects.gentoo.org/pms/8/pms.html#x1-260003.3 and then select the last revision of that version as the terminating bound. We cannot do that in this pipeline and it should be handled separately in an improver pipeline.

For now we can only capture what we are presented with, in the above example.

<unaffected range="rge">2.23.3</unaffected>
<unaffected range="rge">2.24.3</unaffected>
<unaffected range="rge">2.25.4</unaffected>
<unaffected range="rge">2.26.2</unaffected>

we should create impacted package with these fixed version range.

vers:ebuild/>=2.23.3
vers:ebuild/>=2.24.3
vers:ebuild/>=2.25.4
vers:ebuild/>=2.26.2

Comment on lines 37 to 50
{
"package": {
"type": "ebuild",
"namespace": "dev-vcs",
"name": "subversion",
"version": "",
"qualifiers": "",
"subpath": ""
},
"affected_version_range": "vers:ebuild/<=1.8.18",
"fixed_version_range": null,
"introduced_by_commit_patches": [],
"fixed_by_commit_patches": []
}
Copy link
Member

@keshav-space keshav-space Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not right, for the unaffected range <unaffected range="ge">1.9.7</unaffected> and <unaffected range="rgt">1.8.18</unaffected> we should report vers:ebuild/>1.8.18 and vers:ebuild/1.9.7 as fixed version range and do not invert these and report it as affected vers.

@ziadhany
Copy link
Collaborator Author

@keshav-space Is this correct implementation?

https://security.gentoo.org/glsa/201709-09

<package name="dev-vcs/subversion" auto="yes" arch="*">
    <unaffected range="ge">1.9.7</unaffected>      <---- fixed_version_range  >=1.9.7
    <unaffected range="rgt">1.8.18</unaffected>   <---- fixed_version_range >1.8.18|<1.9.0
    <vulnerable range="lt">1.9.7</vulnerable>
    <vulnerable range="eq">0.1.1</vulnerable>
</package>
"affected_version_range": "vers:ebuild/0.1.1|<1.9.7",
"fixed_version_range": "vers:ebuild/>1.8.18|<1.9|>=1.9.7",

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Update the Gentoo get_safe_and_affected_versions function in advisory v2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
… and fixed_version_range for non-revision versions

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Update the pipeline to use the new AdvisoryDataV2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Copy link
Collaborator Author

@keshav-space I am also getting an InvalidConstraintsError even after using the new implementation.
aboutcode-org/univers#181

see:
gentoo.zip

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
@ziadhany
Copy link
Collaborator Author

As we discussed in the weekly VulnerableCode meeting, we decided to have an affected_package for every index in package (unaffected, vulnerable). to avoid conflict on version range constraints.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments