Skip to content

Adding activation functions and fast LUT implementation#177

Merged
sdatkinson merged 5 commits into
sdatkinson:mainfrom
jfsantos:activations
Jan 13, 2026
Merged

Adding activation functions and fast LUT implementation#177
sdatkinson merged 5 commits into
sdatkinson:mainfrom
jfsantos:activations

Conversation

@jfsantos

@jfsantos jfsantos commented Jan 13, 2026

Copy link
Copy Markdown
Contributor

This PR adds the SiLU (Swish) and HardSwish activations to core, as well as changes LeakyReLU into a parameterizeable function. It also adds an activation class that converts any C++ function with one float input argument into a LUT implementation with N points and linear interpolation between points.

Tests were added for the fast LUT implementation to make sure the results are close to the standard C++ implementations. With a LUT using 1024 entries, we get the following results in the tests for sigmoid(1.25) and tanh(1.25):

exact sigmoid: 0.7773
lut sigmoid:   0.777297
exact tanh: 0.848284
lut tanh:   0.848269

The LUT implementations can be activated to replace the original activations just like it is currently done for fasttanh.

Developed with support and sponsorship from TONE3000

João Felipe Santos added 5 commits January 8, 2026 12:32

@sdatkinson sdatkinson left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏻

@sdatkinson sdatkinson merged commit 6ea03b1 into sdatkinson:main Jan 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants