Make calculating of Map Analysis Faster and less resource intensive

[northdpole]

Currently gap analysis requires 64gb of ram on a chunky server with an external neo4j cluster and an external redis in order to calculate ~10gb worth of graph shortest paths.

This crashes most commercial laptops and takes more than 24hours on a GCP medium machine.

There are many micro-optimizations we can do to make the gap analysis faster and less resource intensive such as:

• preload the relevant subgraphs only in neo4j
• re-use precalculated paths
• optimize the cypher queries and the redis usage
• for any standard pair avoid trying to calculate a path between every node of standard A and every node of standard B
• experiment with cutting out the standards and calculating a gap analysis between relevant CREs, since we only have 400 CREs this should be much faster than calculating gaps between thousands of standard nodes.
• Optimize the python code to not access memory repeatedly
• Reduce the information reported as the final result
  etc.

[Hardik301002]

Hi @northdpole
"Here are some potential optimizations to improve efficiency and reduce resource usage:"

1. Preload Relevant Subgraphs – Load only necessary subgraphs in Neo4j instead of the entire graph to reduce memory usage.
2. Reuse Precomputed Paths – Implement caching to store and reuse previously calculated paths, avoiding redundant computations.
3. Optimize Cypher Queries & Redis Usage – Improve query efficiency and ensure Redis is used effectively to reduce latency.
4. Limit Pairwise Node Calculations – Focus only on critical node pairs instead of computing paths between all nodes.

Let me know if you’d like me to explore any of these solutions in more detail !

[northdpole]

Hey @Hardik301002 , i think i've tried to do several of these but i'm by no means an expert in graph dbs, if you want you can take a stab at it.

Btw, if you used an LLM to get these suggestions some details:

1. the query is matching subgraphs already -- if you know cypher in any depth, perhaps you can optimise this further
2. the whole gap analysis is cached when calculated once, i'm not sure how to cache precomputed paths in neo4j
3. this is too vague to be able to give you more pointers
4. i think this is the same as 1.

[Hardik301002]

Please assign this issue to me, i want to work on this issue further under your guidance.

[northdpole]

T
@Hardik301002 thanks for volunteering, issue assigned

[sd2604]

Hi, I’d like to work on this issue. I have experience with performance optimization and efficient data handling, and I believe I can significantly improve the speed and resource usage of the Map Analysis module.
Could you please assign this issue to me? I’ll make sure to document all changes and provide clear benchmarks for comparison.

[moizulfiqar]

Hi @northdpole, @Pa04rth and @viyatb,

Could you please let me know if this issue has been resolved, or if it is still pending?

If it is still pending, I would kindly like to request to be assigned to it. I am a contributer on the Nest project and would like to work on resolving this issue immediately.

[muhammad7865]

Hi @northdpole , I’m Awais, preparing for GSoC 2026 and I’d like to contribute to OWASP OpenCRE.
I’ve already contributed to OWASP Cornucopia (merged PRs: #2145 and #2164).

I’m interested in the Scraper/Indexer direction, especially Module D (human-in-the-loop review UI) or Module A (nightly pipeline/automation).
Could you suggest the best starter issue aligned with those modules, and share your expectations for GSoC applicants (PR count/type, tests, preferred communication channel)?

Thanks!

[Arunodoy18]

Hi, I’m interested in contributing to performance improvements in the gap analysis module.

Before proposing large-scale changes, I’d like to start by profiling the current implementation and identifying the primary bottleneck (query, memory usage, or Python layer).

Is there an existing benchmark or profiling baseline I should use?