Release PR for Sprint ending 2021-03-29 by bjester · Pull Request #3062 · learningequality/studio

bjester · 2021-03-30T23:17:18Z

Summary

This release contains several security updates, including updates for dependent packages and a change to remove an unneeded Django feature in production. It also contains a few bug fixes:

Fixes programmatic errors arising from attempts to calculate user storage for non-existent users. Users shouldn't experience anything differently with this change.
Adds an asynchronous calculation of channel size shown in Publish modal. Users may have previously experienced timeouts for synchronous requests of the information, which would show as a forever loading spinner in the Publish modal.
Fixes to prevent floods of Sentry errors arising from attempts at creating duplicate tags. This resolves an issue that may have prevented successful syncing of user changes, resulting in temporary differences between the user's local view and the saved representation on the server. Users may have seen some of these changes actually reverted.
Fixes progress and status reporting of asynchronous backend tasks. This resolves issues where users may have encountered incorrect task progress for publishing, synchronizing imported channel contents, copying, and moving, such as that it's complete before it has actually completed, or that it never completes (forever pending)

Pull Requests

PR	Category	QA Focus
#2961	Bug fix	- File upload - User storage status
#3027	Security update	- Verify `/admin` is inaccessible
#3028	Enhancement and bug fixes	- Publishing a channel - Adding files to a channel - Adding files to an existing node - Moving files to and from trash - Copying nodes - Importing resources from another channel - Synchronizing imported resources
#3043	Bug fix	- Add and remove tags from a single node - Bulk add tags to multiple nodes
#3037	Security update	N/A
#3039	Security update	N/A
#3042	Security update	N/A
#3045	Security update	N/A
#3052	Security update	N/A

Possibly co-resolved issues

~~#2709~~ (not resolved)
#2816
#2953
~~#3003~~ (not resolved)

QA Scenarios

Case: File upload and user storage status

Open an editable channel
Click Add and Upload files
Note your current Total storage used
Upload a file and finish up editing the uploaded file's details
Click Add and Upload files again
Confirm that your Total storage used has updated properly

Case: Publishing a large channel

Open an editable channel with at least 5000 nodes
Click Add and Upload files
Select a file and upload it
Click Publish
If the channel is large enough, you should observe a loading spinner in the Publish modal until the task completes
Verify you can't proceed until the task completes, loader disappears, and the size appears

Case: Adding and deleting a file updates publish channel size

Create a new channel
Click Add and Upload files to upload a small file
Click Publish
Observe the channel size shown in the Publish modal
Close the Publish modal
Click Add and Upload files to upload a large file (like a video)
Allow your browser to finish syncing (wait ~30 seconds)
Click Publish
Verify the channel size has increased in the Publish modal
Close the Publish modal
Delete the large file
Click Publish
Verify the channel size has decreased in the Publish modal

Case: Add and remove tags from a single node

Open an editable channel
Edit an existing content node
Add one or more tags to it
Finish saving the node
Allow some time for the sync to complete (~30 seconds)
Open the channel in a fresh browser session (incognito mode works well)
Verify the tags you added are present on the node
Repeat the above for removing the tags

Case: Bulk add tags to multiple nodes

Open an editable channel
Select 2 or more nodes and click to edit them from the toolbar
Add one or more tags to them
Finish saving the changes
Allow some time for the sync to complete (~30 seconds)
Open the channel in a fresh browser session (incognito mode works well)
Verify the tags you added are present on the nodes
Repeat the above for removing the tags

First pass fix for non-existent users being passed to the calculate user storage task

Bumps [pillow](https://github.com/python-pillow/Pillow) from 8.0.1 to 8.1.1. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst) - [Commits](python-pillow/Pillow@8.0.1...8.1.1) Signed-off-by: dependabot[bot] <support@github.com>

Bumps [jinja2](https://github.com/pallets/jinja) from 2.11.2 to 2.11.3. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/master/CHANGES.rst) - [Commits](pallets/jinja@2.11.2...2.11.3) Signed-off-by: dependabot[bot] <support@github.com>

…8.1.1 Bump pillow from 8.0.1 to 8.1.1

…2.11.3 Bump jinja2 from 2.11.2 to 2.11.3

Bumps [rsa](https://github.com/sybrenstuvel/python-rsa) from 4.0 to 4.1. - [Release notes](https://github.com/sybrenstuvel/python-rsa/releases) - [Changelog](https://github.com/sybrenstuvel/python-rsa/blob/main/CHANGELOG.md) - [Commits](sybrenstuvel/python-rsa@version-4.0...version-4.1) Signed-off-by: dependabot[bot] <support@github.com>

Bump rsa from 4.0 to 4.1

…ng so

Bumps [pyyaml](https://github.com/yaml/pyyaml) from 5.3 to 5.4. - [Release notes](https://github.com/yaml/pyyaml/releases) - [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES) - [Commits](yaml/pyyaml@5.3...5.4) Signed-off-by: dependabot[bot] <support@github.com>

Merge down master

- Update resource size cache modified date when moving nodes or deleting files - Relcoate ResourceSizeCache to avoid import issues - Only allow size calculations for root nodes through the API

Create tag relations one by one, and use CTEs to minimize queries doing so

Async task fixes and new async calculation of resource size

Bump pyyaml from 5.3 to 5.4

Bumps [pygments](https://github.com/pygments/pygments) from 2.5.2 to 2.7.4. - [Release notes](https://github.com/pygments/pygments/releases) - [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES) - [Commits](pygments/pygments@2.5.2...2.7.4) Signed-off-by: dependabot[bot] <support@github.com>

radinamatic · 2021-04-05T15:12:27Z

Current Der Stand der Dinge of this release PR is as follows:

PR	Status	Comment
#2961	Not fixed	New issue filed
#3027	Fixed	-
#3028	(probably) Not fixed	publishing on `hotfixes` branch is slow and errors-prone in the console
#3043	Fixed	-
#2709	Not fixed	-
#2816	Fixed	-
#2953	(possibly) Fixed	issue was not reliably replicable to begin with
#3003	Not fixed	-

I see more misses than hits, so not sure how should we proceed 😕

cc @bjester @rtibbles

bjester · 2021-04-05T15:20:09Z

| #2961 | Not fixed | [New issue filed]

@radinamatic #2961 was not intended to resolve storage discrepancies in the UI, so we can count that as OK if it's without regressions. The bug tracked in the new issue should be present either way.

Use get_or_create_async_task for queuing user storage calc task

Disable modified_since check, since needs further query optimization

radinamatic · 2021-04-06T22:39:21Z

Let's re-assess our Der Stand der Dinge:

PR	Status	Comment
#2961	(probably) Fixed	user-facing totals still stuttering (new issue filed)
#3027	Fixed	-
#3028	Fixed	some stubborn errors on `hotfixes` branch probably due to Google cloud storage
#3043	Fixed	-
#2709	Not fixed	-
#2816	Fixed	-
#2953	(possibly) Fixed	issue was not reliably replicable to begin with
#3003	Not fixed	-

@bjester @rtibbles are we satisfied with this to merge the release?

bjester · 2021-04-06T22:44:24Z

@bjester @rtibbles are we satisfied with this to merge the release?

I'm satisfied. We just need 2 PR approvals

radinamatic

That was a nice sprint rush! 🎽
Many steps feel more streamlined and productive 💯

…03-29 Merge pull request #3062 from learningequality/sprint-release-2021-03-29

bjester and others added 30 commits March 5, 2021 13:15

Add modified field to File model, with index

36b4f19

Add resource size calculator utils to manage lookups, with caching

32e36db

Fix column name and datetime handling

fdaa781

Integrate new resource size calculator

ff7f204

Add logging and comments

e1b1151

Cleanup

5b33889

Tests for resource calculation

7c965c4

Updated task handling in viewset

1142866

Catch and log non-existent users.

9a5601a

Merge pull request #2961 from rtibbles/user_storage

300e36f

First pass fix for non-existent users being passed to the calculate user storage task

Remove Django Admin in production mode (#3027)

3da1570

Publish task handling

78b7c1b

Cleanup and fixes after updating task handling

26a5369

Don't overload __call__, breaks tests

c33b0a6

Merge pull request #3037 from learningequality/dependabot/pip/pillow-…

ec9f8dd

…8.1.1 Bump pillow from 8.0.1 to 8.1.1

Merge pull request #3039 from learningequality/dependabot/pip/jinja2-…

780bb83

…2.11.3 Bump jinja2 from 2.11.2 to 2.11.3

Linting and test fixes

c0dc4e0

Fix merge conflicts

f6039df

Yet more linting fixes...

477b8df

Fix publish test

5d9b2c8

Remove unneccesary metadata

d74e05f

Add missing import.

01adabf

Update tests to accomodate for new task API and classes

cb2a0ed

Merge pull request #3042 from learningequality/dependabot/pip/rsa-4.1

8e7a524

Bump rsa from 4.0 to 4.1

Create tag relations one by one, and use CTEs to minimize queries doi…

356f609

…ng so

Merge branch 'master' into merge-down-master

5a2eef7

rtibbles and others added 8 commits March 26, 2021 11:04

Merge pull request #3046 from bjester/merge-down-master

efaadf6

Merge down master

Code review updates

6dbbd5d

- Update resource size cache modified date when moving nodes or deleting files - Relcoate ResourceSizeCache to avoid import issues - Only allow size calculations for root nodes through the API

Fix merge conflicts

e6eb127

Merge pull request #3043 from bjester/duplicate-tags

26d8873

Create tag relations one by one, and use CTEs to minimize queries doing so

Fix incorrect cache invalidation strategy

15a145a

Merge pull request #3028 from bjester/get-total-size

3c2749b

Async task fixes and new async calculation of resource size

Merge pull request #3045 from learningequality/dependabot/pip/pyyaml-5.4

c04ed76

Bump pyyaml from 5.3 to 5.4

bjester mentioned this pull request Mar 30, 2021

[OLD] Release PR for Sprint ending 2021-03-29 #3044

Closed

bjester requested review from radinamatic and rtibbles March 30, 2021 23:18

bjester added this to the Sprint 2021-03-29 milestone Mar 30, 2021

bjester added the TODO: needs review label Mar 30, 2021

radinamatic mentioned this pull request Apr 5, 2021

'Channel syncing is in progress...' modal/message appears again while the page is refreshed after a successful sync #3003

Closed

bjester added 3 commits April 5, 2021 13:23

Use get_or_create_async_task for queueing user storage calc task

e7bbb3c

Merge pull request #3067 from bjester/once-upon-a-user-storage-task

ea1e415

Use get_or_create_async_task for queuing user storage calc task

Disable modified_since check, since needs further query optimization

be6154f

bjester mentioned this pull request Apr 6, 2021

Disable modified_since check, since needs further query optimization #3069

Merged

6 tasks

Merge pull request #3069 from bjester/disable-modified-since-check

e63c049

Disable modified_since check, since needs further query optimization

radinamatic approved these changes Apr 6, 2021

View reviewed changes

rtibbles approved these changes Apr 6, 2021

View reviewed changes

bjester merged commit ee9ff4f into master Apr 6, 2021

bjester mentioned this pull request Apr 7, 2021

File modified fix for Sprint release 2021-03-29 #3076

Merged

rtibbles added a commit that referenced this pull request Apr 10, 2021

Merge pull request #3073 from bjester/merge-down-sprint-release-2021-…

016aa65

…03-29 Merge pull request #3062 from learningequality/sprint-release-2021-03-29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Release PR for Sprint ending 2021-03-29#3062

Release PR for Sprint ending 2021-03-29#3062
bjester merged 42 commits into
masterfrom
sprint-release-2021-03-29

bjester commented Mar 30, 2021 •

edited

Loading

Uh oh!

radinamatic commented Apr 5, 2021 •

edited

Loading

Uh oh!

bjester commented Apr 5, 2021 •

edited

Loading

Uh oh!

radinamatic commented Apr 6, 2021

Uh oh!

bjester commented Apr 6, 2021

Uh oh!

radinamatic left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

bjester commented Mar 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Pull Requests

Possibly co-resolved issues

QA Scenarios

Case: File upload and user storage status

Case: Publishing a large channel

Case: Adding and deleting a file updates publish channel size

Case: Add and remove tags from a single node

Case: Bulk add tags to multiple nodes

Uh oh!

radinamatic commented Apr 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bjester commented Apr 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radinamatic commented Apr 6, 2021

Uh oh!

bjester commented Apr 6, 2021

Uh oh!

radinamatic left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bjester commented Mar 30, 2021 •

edited

Loading

radinamatic commented Apr 5, 2021 •

edited

Loading

bjester commented Apr 5, 2021 •

edited

Loading