Skip to content

Support the maximize parameter for adam when dtype is torch.half#35

Merged
a710128 merged 4 commits into
OpenBMB:mainfrom
alphaGem:param-maximize-for-adam
Jul 8, 2022
Merged

Support the maximize parameter for adam when dtype is torch.half#35
a710128 merged 4 commits into
OpenBMB:mainfrom
alphaGem:param-maximize-for-adam

Conversation

@alphaGem

@alphaGem alphaGem commented Jul 3, 2022

Copy link
Copy Markdown

Related to issue34

Comment thread bmtrain/optim/adam.py Outdated
@a710128

a710128 commented Jul 4, 2022

Copy link
Copy Markdown
Contributor
  1. AdamOffload needs to be updated.
    https://github.com/OpenBMB/BMTrain/blob/0.1.7.post1/bmtrain/optim/adam_offload.py#L131-L172
  2. The maximize parameter of torch.float also needs to be supported.
  3. A bug: grad was not passed to the adam function.

@alphaGem

alphaGem commented Jul 5, 2022

Copy link
Copy Markdown
Author

Update:

  1. Adam Offload now updated.
  2. Maximize parameter of torch.float now supported.
  3. grad now passed to the adam function.

@a710128 a710128 merged commit e6ba031 into OpenBMB:main Jul 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Supporting the maximize parameter of Adam optimizer when dtype is torch.half

2 participants