Skip to content

Use filer in sync command#424

Closed
pietern wants to merge 4 commits into
mainfrom
sync-use-filer
Closed

Use filer in sync command#424
pietern wants to merge 4 commits into
mainfrom
sync-use-filer

Conversation

@pietern
Copy link
Copy Markdown
Contributor

@pietern pietern commented Jun 1, 2023

Changes

This deprecates usage of the repofiles package in favor of the filer package and consolidates the code paths into WSFS.

Note: one breaking change here is the following. If a file at foo/bar.txt is created and removed, the directory foo is kept around because we do not perform directory tracking. If subsequently, we need to write a file at foo, it will result in an fs.ErrExist because it is impossible to overwrite a directory.

The previous implementation performed a recursive delete of the path if this happened, where this implementation will return the fs.ErrExist error to the user.

How to solve directory handling:

  1. Maintain set of directories needed for current set of files. Compare to previous set of files. This results in mkdir of added directories and rmdir of removed directories.
  2. Making new directories should happen prior to dealing with files. Removing existing directories should happen after dealing with files.
  3. Making new directories can be deduped across common prefixes where only the longest prefix is created recursively.
  4. Removing existing directories must happen sequentially, starting with the longest prefix.
  5. If removal of a directory fails, it must not be empty. We then recursively look for nested empty directories, remove those, and continue. This is necessary to remove empty directories left around by prior sync runs.

Tests

Existing integration tests pass.

pietern added 2 commits June 1, 2023 22:44
This deprecates usage of the `repofiles` package in favor
of the filer package and consolidates the code paths into WSFS.

Note: one potentially breaking change here is the following.
If a file at `foo/bar.txt` is created and removed, the directory
`foo` is kept around because we do not perform directory tracking.
If subsequently we need to write a file at `foo`, it will result
in an `fs.ErrExist` because it is impossible to overwrite a directory.

The previous implementation performed a recursive delete of the path
if this happened, where this implementation will return the `fs.ErrExist`
error to the user.

We can mitigate this in one of two ways:
* Track directories to remove as part of a `diff` and remove them
* Attempt to remove an empty directory tree if we see this error
* ...?
@pietern
Copy link
Copy Markdown
Contributor Author

pietern commented Jun 6, 2023

We can't merge this without #425. I incorporated the changes in this PR there.

@pietern pietern closed this Jun 6, 2023
@pietern pietern deleted the sync-use-filer branch June 6, 2023 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants