Skip to content

Navigation Menu

Appearance settings

View all features
- BY COMPANY SIZE
  Enterprises
  Small and medium teams
  Startups
  Nonprofits
- BY USE CASE
  App Modernization
  DevSecOps
  DevOps
  CI/CD
  View all use cases
- BY INDUSTRY
  Healthcare
  Financial services
  Manufacturing
  Government
  View all industries
View all solutions
- EXPLORE BY TOPIC
  AI
  Software Development
  DevOps
  Security
  View all topics
- EXPLORE BY TYPE
  Customer stories
  Events & webinars
  Ebooks & reports
  Business insights
  GitHub Skills
- SUPPORT & SERVICES
  Documentation
  Customer support
  Community forum
  Trust center
  Partners
View all resources
- COMMUNITY
  GitHub SponsorsFund open source developers
- PROGRAMS
  Security Lab
  Maintainer Community
  Accelerator
  GitHub Stars
  Archive Program
- REPOSITORIES
  Topics
  Trending
  Collections
- ENTERPRISE SOLUTIONS
  Enterprise platformAI-powered developer platform
- AVAILABLE ADD-ONS
  GitHub Advanced SecurityEnterprise-grade security features
  Copilot for BusinessEnterprise-grade AI features
  Premium SupportEnterprise-grade 24/7 support
Pricing

Search code, repositories, users, issues, pull requests...

Search

Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

Saved searches

Use saved searches to filter your results more quickly

Name

Query

To see all available qualifiers, see our documentation.

Appearance settings

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

als-computing / splash_flows Public

Notifications You must be signed in to change notification settings
Fork 6
Star 7

Code
Issues 26
Pull requests 15
Actions
Projects
Security and quality
Insights

Additional navigation options

Code
Issues
Pull requests
Actions
Projects
Security and quality
Insights

Issue 59: Transfer Flows between CFS and HPSS#62

Open

davramov wants to merge 128 commits into

als-computing:mainals-computing/splash_flows:mainfrom

davramov:issue_59davramov/splash_flows:issue_59Copy head branch name to clipboard

Conversation Commits128 (128)Checks Files changed

Open

Issue 59: Transfer Flows between CFS and HPSS#62
davramov wants to merge 128 commits into
als-computing:mainfrom
davramov:issue_59

Commits

Commits on Nov 4, 2025

Initial commit for issue #59. This is still a WIP and requires thorough testing. Includes a new transfer controller CFSToHPSSTransferController() with logic for handling single files vs directories…
davramov
committed
Adding a generic BeamlineConfig(ABC) class in orchestration/config.py that is inherited by specific Config implementations such as orchestration/flows/bl832/config.py: Config832(). Updating typing …
davramov
committed
Added logic for HPSSToCFSTransferController() copy() method. Now it will launch an SFAPI Slurm job script that handles each case: a single file is requested (non-tar), an entire tar file, or specif…
davramov
committed
Moving endpoint definitions to orchestration/transfer_endpoint.py so they can be easily used by both transfer_controller.py and the new prune_controller.py file. Prune_controller.py follows the sam…
davramov
committed
Added a get_prune_controller() method.
davramov
committed
Getting ready to test CFSToHPSSTransferController flow. Created a Prefect deployment script for the HPSS flow. Added logic to structure the filename in the htar according to year and beamcycle, add…
davramov
committed
Fixed syntax errors
davramov
committed
Cleaned up logging for the CFSToHPSSTransferController slurm job script to clearly show what files are in each tar bundle. Ready to test with a real project.
davramov
committed
adding support for dynaconf for handling beamline configurations
davramov
committed
Adding logic for the prune_filesystem_files flow in prune_controller.py
davramov
committed
Moving specific prune implementations as private/internal methods in their respective PruneControllers, with Prefect Flow decorators.
davramov
committed
Adding documentation outlining shared infrastructure between beamline implementations
davramov
committed
Linting and adding a few TODO comments
davramov
committed
Fixed get_prune_controller() method to accept an Enum for supported types. Updated documentation. Added a new file orchestration/flows/bl832/move_refactor.py that will replace orchestration/flows/b…
davramov
committed
Working on a SciCat Ingestor Controller ABC, and a BL832 implementation, including how to append a new dataset location to an existing dataset in scicat.
davramov
committed
sourceFolder expects a string
davramov
committed
generalizing relevant code from ingest_tomo832.py, refactoring into a bl832/scicat_ingestor.py, udpate documentation
davramov
committed
Refactoring scicat ingestion to be more modular and structured
davramov
committed
Adding a custom Prefect Block to keep track of pending project names to be transferred to HPSS Tape.
davramov
committed
Added three dispatchers for bl832 to handle archiving data to HPSS: 1) specify project(s) to copy, 2) schedule a flow every 6 months to copy the previous cycle's data, and 3) archive everything. Al…
davramov
committed
Updating documentation and comments
davramov
committed
Testing HPSS flow, verified that I can create new directories on HPSS from Slurm/SFAPI.
davramov
committed
Updating documentation:
davramov
committed
Verified that htar bundling and building on HPSS works
davramov
committed
Updating HPSS documentation typo
davramov
committed
Adding a _find_dataset() method to ingestor_controller base class, in case a SciCat ID is not known, but the proposal_id and file_name are known. I anticipate a case where we move data later, but d…
davramov
committed
Successfully extracted files from a .tar archive on HPSS back to CFS! One thing to change is the file_path stored in the .tar (currently each file in the tar has the full cfs path, which seems sill…
davramov
committed
Moving all HPSS related transfer/prune implementations into orchestration/hpss.py. Working on an HPSS pruning implementation
davramov
committed
Adjusting hpss imports in the transfer_controller pytest
davramov
committed
Verified that the HPSSPruneController successfully pruned from HPSS
davramov
committed
Simplified file paths within the .tar archives on HPSS so it references only the file name, not the entire cfs path
davramov
committed
Updated the HPSS flows in bl832/dispatcher.py to include updating the filepath and filesystem location in SciCat. To be tested...
davramov
committed
Updated docstrings, logging, error handling
davramov
committed
Updated docstrings, logging, typing, error handling
davramov
committed
Improved error logging and exception handling for tape flows
davramov
committed
Verified SciCat login to the latest Docker version of scicatlive. It first tries the currently implemented way of logging in (for backwards compatability), but if that fails, it falls back to a new…
davramov
committed
Fixing pytest errors and failures
davramov
committed
Testing scicat ingestion locally on a small test h5 dataset from 832. Able to ingest using the default admin account on a local scicatlive instance.
davramov
committed
Testing scicat ingestion locally on a small test h5 dataset from 832. Able to ingest using the default admin account on a local scicatlive instance.
davramov
committed
updated and tested add_new_dataset_location() in orchestration/flows/scicat/ingestor_controller.py, which now creates a new OrigDataBlock with the updated filesystem host and path. The new path app…
davramov
committed
Addedsupport for linking tomography reconstructions as derived datasets in SciCat. Tested it and it works, Tiffs and Zarrs are findable under the Related Datasets tab in the UI. Need to fix the th…
davramov
committed
Fixed thumbnails uploaded for derived tiff/zarr datasets in SciCat
davramov
committed
Adding logic to the remove_dataset_location() function in ingestor_controller.py. For some reason, none of the default accounts that are configured with scicatlive seem to have deletion permissions…
davramov
committed
Adding a test script (not pytest) for end-to-end validation of the controller classes (orchestration/tests/test_controllers_end_to_end.py). I kept it as minimal/generic as possible as a way to vali…
davramov
committed
Moved test_controllers_end_to_end.py to the scripts/ folder does it does not conflict with pytest.
davramov
committed
Addressing Dylan and Raja's comments
davramov
committed
Updating end-to-end tests
davramov
committed
For the HPSS->CFS controller, convert slashes to underscores for the log file name to make it easier to find
davramov
committed
Added filter for files >65GB to be moved with HTAR. Verified that this works.
davramov
committed
Updating .env.example and README with current environment requirements.
davramov
committed
Added comment that this will be moved to the scicat_beamline repo at some point
davramov
committed
Updating documentation for HPSS
davramov
committed
Updating documentation for HPSS
davramov
committed
Adding ability to hpss controllers for ls command to see what's on tape
davramov
committed
Fixed commenting
davramov
committed
fixed pytest after rebasing
davramov
committed
bumping python from 3.11->3.12.5 to see if it fixes a TypeError where type object is not iterable. I could not reproduce the pytest error locally, so I am bumping to the same version
davramov
committed
Fix error message grammar
davramov
committed
Fixing which flow the globus prune controller calls
davramov
committed
making the days_from_now parameter a float, which is converted into datetime.timedelta within the functions
davramov
committed
adding try and except to build_thumbnail, in case of edge cases (like divide by 0)
davramov
committed
Updated documentation regarding hsi mkdir -p
davramov
committed
Addressing Garrett's comment about the add_new_dataset_location method. Making it less brittle by requiring the caller to compose the full path when calling the function
davramov
committed
Removing redunant datafile.path = file_path lines, as it is now configured at the top of the method, before calling datafile = DataFile(...)
davramov
committed
updating requirements.txt
davramov
committed
Adding a TODO comment to _find_dataset to support searching dataFileLists. This method isn't currently called anywhere, so we can build this feature out as needed.
davramov
committed
Adjusting sfapi pytest to match new expected value
davramov
committed
Adjusting prune_controller.prune() calls to use a float for days_from_now, rather than datetime, for simpler calls
davramov
committed
Renamining dummy to mock
davramov
committed
linting
davramov
committed
Fixed type (tranfer_client -> transfer_client)
davramov
committed
Fixed typo (tranfer_client -> transfer_client)
davramov
committed
Fixed typo (tranfer_client -> transfer_client)
davramov
committed
Fixed typo (tranfer_client -> transfer_client)
davramov
committed
Adding a comment at the top
davramov
committed
Updating test_prune_controller.py based on Dylan's comments. Ensuring use of the term mock over other synonyms, creating fixtures as needed, and ensuring core functionality of the prune controller …
davramov
committed
Removing redundant logger.debug messages when initializing transfer controllers
davramov
committed
Updating 733 docs sequence diagram
davramov
committed
Using the tmp_path fixture for pytest
davramov
committed
Adding checks in prune_controller (Globus and Filesystem) for days_from_now < 0, which throws an error if it is negative
davramov
committed
Adding a new pytest functino called test_globus_prune_schedules_when_days_from_now_negative
davramov
committed
updating test_prune_controller with a new function test_fs_prune_schedules_when_days_from_now_negative
davramov
committed
Fixing tests for when days_from_now == 0
davramov
committed
Using Pathlib instead of os.path
davramov
committed
removing redundat MockClient assertion
davramov
committed
Rewriting the filesystem transfercontroller to follow Kate's suggestions to make it less brittle.
davramov
committed
Rewriting the filesystem transfercontroller exception handling to follow Kate's suggestions to make it less brittle.
davramov
committed
Linting
davramov
committed
Set self.config = None after assigning beamline-specific configs
davramov
committed
Enforcing consistency for beamline_id to be a str with numbers delimited by a period
davramov
committed
Adding env var check to the start of the main method
davramov
committed
Making the check_required_envars() function more verbose with which variables are missing
davramov
committed
Removing unused imports
davramov
committed
Adding better checks to make sure env variables are set
davramov
committed
Using pathlib to build the full path
davramov
committed
Updating docstring
davramov
committed
Including comment about circular dependencies
davramov
committed
Removing main block (tests are handled in scripts/test_controllers_end_to_end.py)
davramov
committed
Prefect/Globus env variable checks throw warnings rather than errors to prevent tests from failing
davramov
committed
Elevating pruning flows out of the controller classes.
davramov
committed
Moving HPSS slurm jobs into a folder orchestration/slurm/
davramov
committed
Ensuring all variables are passed into hpss_to_cfs.slurm
davramov
committed
moving hpss main method to it's own check_hpss script
davramov
committed
Adding a note about encode_thumbnail being part of scicat
davramov
committed
capitalizing severity enum options
davramov
committed
Adding comment about cput not overwriting
davramov
committed
Deleting unused sfapi key variable
davramov
committed
Docstring
davramov
committed
Docstring
davramov
committed
renaming login_to_scicat to get_scicat_client
davramov
committed
removing redundant error message
davramov
committed
Docstring
davramov
committed
removing test code
davramov
committed
Fixing project path endpoint to nersc cfs endpoint. Updating the log to use that path
davramov
committed
adding log paths to error message
davramov
committed
Addressing comments regarding hpss dispatcher flows.
davramov
committed
Adjusting comments to be clearer
davramov
committed
Adding typing throughout
davramov
committed
Docstrings
davramov
committed
replacing old move.py with move_refactor.py
davramov
committed
Documentation
davramov
committed
Adding screenshots to SFAPI documentation
davramov
committed
hyperlink to sfapi example
davramov
committed
Reducing the memory from 20GB to 2GB due to sbatch: error: More resources requested than allowed for logical queue xfer (0.0390625 requested core-equivalents > 0.025)
davramov
committed
Check if scicat's base url includes localhost, and pass in the backend.localhost header when logging in if true.
davramov
committed
removing main method used for testing
davramov
committed
replacing run_specific_flow with run_deployment after rebase
davramov
committed
making sure process_new_832_file_task is called by dispatcher rather than the flow version
davramov
committed

Footer

© 2026 GitHub, Inc.

Footer navigation

Terms
Privacy
Security
Status
Community
Docs
Contact

You can’t perform that action at this time.