Add Openstack Importer V2 Pipeline#2154
Conversation
Samk1710
force-pushed
the
add-ossa-importer
branch
from
e2ef492 to
4111f98
Compare
|
|
||
| def parse_version_range(self, version_str: str) -> PypiVersionRange: | ||
| """ | ||
| Normalizes the version string and extracts individual constraints to create a PypiVersionRange object. | ||
| """ |
There was a problem hiding this comment.
I think this function should be part of https://github.com/aboutcode-org/univers
There was a problem hiding this comment.
See: https://gist.github.com/Samk1710/efe6a7284cdddd50c4209e63e3d82215
The parse_version_range is more of a cleanup function to handle messy version strings provided by OSSA. It is bound to cleaning up inconsistent patterns in OSSA only.
| ossa_id = data.get("id") | ||
|
|
||
| date_str = data.get("date") | ||
| date_published = dateparser.parse(str(date_str)).replace(tzinfo=UTC) |
There was a problem hiding this comment.
what if we don't have a date_str ?
There was a problem hiding this comment.
def fetch(self):
ossa_dir = Path(self.vcs_response.dest_dir) / "ossa"
self.processable_advisories = []
skipped_old = 0
for file_path in sorted(ossa_dir.glob("OSSA-*.yaml")):
data = load_yaml(str(file_path))
date_str = data.get("date")
date_published = dateparser.parse(str(date_str)).replace(tzinfo=UTC)
if date_published.year < self.cutoff_year:
skipped_old += 1
continue
self.processable_advisories.append(file_path)
if skipped_old > 0:
self.log(f"Skipped {skipped_old} advisories older than {self.cutoff_year}")
self.log(f"Fetched {len(self.processable_advisories)} processable advisories")It is being checked in the fetch function.
| self.log(f"Cloning `{self.repo_url}`") | ||
| self.vcs_response = fetch_via_vcs(self.repo_url) | ||
|
|
||
| def fetch(self): |
There was a problem hiding this comment.
I think this should be merged in collect_advisories function
| skipped_old += 1 | ||
| continue | ||
|
|
||
| self.processable_advisories.append(file_path) |
There was a problem hiding this comment.
Also, avoid using global variables to store all the paths, as this takes up some amount of memory.
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
Samk1710
force-pushed
the
add-ossa-importer
branch
from
4111f98 to
80028b6
Compare
2 participants
Closes
Data Source: https://github.com/openstack/ossa
Research Notes: #374 (comment)
Importer logs: