Skip to content
This repository was archived by the owner on Oct 9, 2023. It is now read-only.

Add a user-specifiable root to upload link request #577

Merged
wild-endeavor merged 5 commits into from
Jun 22, 2023
Merged

Add a user-specifiable root to upload link request #577

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9a04a3a
add prefix
wild-endeavor Jun 15, 2023
5f85815
save a variable
wild-endeavor Jun 19, 2023
369a68c
Update dataproxy/service.go
wild-endeavor Jun 19, 2023
b8c246a
use released idl
wild-endeavor Jun 22, 2023
c18a0ee
Merge branch 'consistent-uploads' of github.com:flyteorg/flyteadmin i…
wild-endeavor Jun 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions dataproxy/service.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,10 +77,16 @@ func (s Service) CreateUploadLocation(ctx context.Context, req *service.CreateUp
}

md5 := base64.StdEncoding.EncodeToString(req.ContentMd5)
urlSafeMd5 := base32.StdEncoding.EncodeToString(req.ContentMd5)

var prefix string
if len(req.FilenameRoot) > 0 {
prefix = req.FilenameRoot
} else {

@EngHabu EngHabu Jun 19, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One reason the previous path was "safe" without RBAC is that even if two people try to upload to the same path, unless they are uploading the same file, they won't end up with conflicting content in the same location...

I understand the motivation behind the change and maintaining a consistent structure...
Would it make sense to maintain the safety here?
Perhaps it's a client-side driven logic to include the SHA of the ordered SHAs of all files of a directory as part of the FilenameRoot?
so you end up with:
s3:////dir/name/path/filename.foo

Perhaps the SHAofSHAs logic is something like:
build a json of [MD5] = filename

"dir": {
  "filename": <md5>,
...
}

Then computing the SHA of the json string?

@wild-endeavor wild-endeavor Jun 19, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so do this computation on the client side? or are you saying change the idl to take a json string, and admin hashes the json string?

@EngHabu EngHabu Jun 20, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

client side yes... I think it'll need a whole lot more changes to incorporate that behavior on the server side...
On top of my head, a new API that's built for uploading directories:

rpc UploadDirectory(stream UploadDirectoryRequest) UploadDirectoryResponse

message UploadDirectoryRequest {
    string directory_md5 = 1;
    string filename = 2;
    bytes file_contents = 3;
}

or something like the Multipart upload semantics

rpc StartUploadDirectory(StartUploadDirectoryRequest) StartUploadDirectoryResponse
rpc UploadFile(UploadFileRequest) UploadFileResponse
rpc TerminateUploadDirectory(TerminateUploadDirectoryRequest) TerminateUploadDirectoryResponse

And then the server can continue to generate/enforce SHAs of directories... and fail the call if SHA doesn't match or if the client fails to complete the upload within a specified time...

// url safe base32 encoding
prefix = base32.StdEncoding.EncodeToString(req.ContentMd5)
}
storagePath, err := createStorageLocation(ctx, s.dataStore, s.cfg.Upload,
req.Project, req.Domain, urlSafeMd5, req.Filename)
req.Project, req.Domain, prefix, req.Filename)
if err != nil {
return nil, errors.NewFlyteAdminErrorf(codes.Internal, "failed to create shardedStorageLocation, Error: %v", err)
}
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ require (
github.com/cloudevents/sdk-go/v2 v2.8.0
github.com/coreos/go-oidc v2.2.1+incompatible
github.com/evanphx/json-patch v4.12.0+incompatible
github.com/flyteorg/flyteidl v1.5.7
github.com/flyteorg/flyteidl v1.5.11
github.com/flyteorg/flyteplugins v1.0.56
github.com/flyteorg/flytepropeller v1.1.87
github.com/flyteorg/flytestdlib v1.0.15
Expand Down
4 changes: 4 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,10 @@ github.com/felixge/httpsnoop v1.0.1 h1:lvB5Jl89CsZtGIWuTcDM1E/vkVs49/Ml7JJe07l8S
github.com/felixge/httpsnoop v1.0.1/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
github.com/flyteorg/flyteidl v1.5.7 h1:voAxMMFsKOseNFSlCyRGlpegqtQXtJjyxgsQzZg4tts=
github.com/flyteorg/flyteidl v1.5.7/go.mod h1:EtE/muM2lHHgBabjYcxqe9TWeJSL0kXwbI0RgVwI4Og=
github.com/flyteorg/flyteidl v1.5.11-0.20230615182642-01c711f0a4d7 h1:lDqtfcevqPPwcCwHxoUpFs0JiQ7U1nEMMrMNy8p8Iko=
github.com/flyteorg/flyteidl v1.5.11-0.20230615182642-01c711f0a4d7/go.mod h1:EtE/muM2lHHgBabjYcxqe9TWeJSL0kXwbI0RgVwI4Og=
github.com/flyteorg/flyteidl v1.5.11 h1:Xcb17YqNstl+dHQsK+o0Ac+1l1U154wXivg28O3C5l0=
github.com/flyteorg/flyteidl v1.5.11/go.mod h1:EtE/muM2lHHgBabjYcxqe9TWeJSL0kXwbI0RgVwI4Og=
github.com/flyteorg/flyteplugins v1.0.56 h1:kBTDgTpdSi7wcptk4cMwz5vfh1MU82VaUMMboe1InXw=
github.com/flyteorg/flyteplugins v1.0.56/go.mod h1:aFCKSn8TPzxSAILIiogHtUnHlUCN9+y6Vf+r9R4KZDU=
github.com/flyteorg/flytepropeller v1.1.87 h1:Px7ASDjrWyeVrUb15qXmhw9QK7xPcFjL5Yetr2P6iGM=
Expand Down