Skip to content

Activation function refactor #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sdatkinson merged 9 commits into from
Apr 9, 2023
Merged

Activation function refactor #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
14cb71e
Begin to refactor activation function handling
mikeoliphant Apr 3, 2023
2aa7020
Remove #pragma
mikeoliphant Apr 3, 2023
069679d
Fixed apply overrides
mikeoliphant Apr 3, 2023
a3f3ce7
More switching to new activation implementaion
mikeoliphant Apr 3, 2023
1cf8a4f
Move static function implementations into .cpp
mikeoliphant Apr 3, 2023
74b0a49
Simplify Activation apply
mikeoliphant Apr 3, 2023
26a725d
Use new activation code for WaveNet Sigmoid
mikeoliphant Apr 3, 2023
b41d172
Fixed WaveNet sigmoid
mikeoliphant Apr 4, 2023
482b69a
No need for middleCols() in full matrix apply anymore
mikeoliphant Apr 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions NAM/activations.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#include "activations.h"

std::unordered_map<std::string, activations::Activation*> activations::Activation::_activations = {
{"Tanh", new activations::ActivationTanh()},
{"Hardtanh", new activations::ActivationHardTanh()},
{"Fasttanh", new activations::ActivationFastTanh()},
{"ReLU", new activations::ActivationReLU()},
{"Sigmoid", new activations::ActivationSigmoid()}};
Comment on lines +4 to +8

@jatinchowdhury18 jatinchowdhury18 Apr 7, 2023

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this memory will be leaked?

@mikeoliphant mikeoliphant Apr 7, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it makes me uncomfortable, too - even though it is global, so should happen once and stick around for the lifetime of the program. My c++ chops are rusty - not sure what the preferred modern c++ way of handling this is.

I also thought about creating individual global instances and just taking a pointer to them in the map.

@sdatkinson sdatkinson Apr 9, 2023

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also thought about creating individual global instances and just taking a pointer to them in the map.

This seems to work?

#include "activations.h"

activations::ActivationTanh _TANH = activations::ActivationTanh();
activations::ActivationFastTanh _FAST_TANH = activations::ActivationFastTanh();
activations::ActivationHardTanh _HARD_TANH = activations::ActivationHardTanh();
activations::ActivationReLU _RELU = activations::ActivationReLU();
activations::ActivationSigmoid _SIGMOID = activations::ActivationSigmoid();

std::unordered_map<std::string, activations::Activation*> activations::Activation::_activations = {
  {"Tanh", &_TANH},
  {"Hardtanh", &_HARD_TANH},
  {"Fasttanh", &_FAST_TANH},
  {"ReLU", &_RELU},
  {"Sigmoid", &_SIGMOID}
};

Seems fine to me, though I know that this is stretching my discipline w/ C++, so either of y'all can lmk if you think there's anything wrong with this ;)


activations::Activation* tanh_bak = nullptr;

activations::Activation* activations::Activation::get_activation(const std::string name)
{
if (_activations.find(name) == _activations.end())
return nullptr;

return _activations[name];
}

void activations::Activation::enable_fast_tanh()
{
if (_activations["Tanh"] != _activations["Fasttanh"])
{
tanh_bak = _activations["Tanh"];
_activations["Tanh"] = _activations["Fasttanh"];
}
}

void activations::Activation::disable_fast_tanh()
{
if (_activations["Tanh"] == _activations["Fasttanh"])
{
_activations["Tanh"] = tanh_bak;
}
}
118 changes: 114 additions & 4 deletions NAM/activations.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,125 @@
#pragma once

#include <string>
#include <cmath> // expf
#include <unordered_map>
#include <Eigen/Dense>

namespace activations
{
float relu(float x)
inline float relu(float x)
{
return x > 0.0f ? x : 0.0f;
return x > 0.0f ? x : 0.0f;
};
float sigmoid(float x)

inline float sigmoid(float x)
{
return 1.0f / (1.0f + expf(-x));
};

inline float hard_tanh(float x)
{
const float t = x < -1 ? -1 : x;
return t > 1 ? 1 : t;
}

inline float fast_tanh(const float x)
{
const float ax = fabsf(x);
const float x2 = x * x;

return (x * (2.45550750702956f + 2.45550750702956f * ax + (0.893229853513558f + 0.821226666969744f * ax) * x2)
/ (2.44506634652299f + (2.44506634652299f + x2) * fabsf(x + 0.814642734961073f * x * ax)));
}

class Activation
{
public:
Activation(){};
virtual void apply(Eigen::MatrixXf& matrix)
{
apply(matrix.data(), matrix.rows() * matrix.cols());
}
virtual void apply(Eigen::Block<Eigen::MatrixXf> block)
{
apply(block.data(), block.rows() * block.cols());
}
virtual void apply(Eigen::Block<Eigen::MatrixXf, -1, -1, true> block)
{
apply(block.data(), block.rows() * block.cols());
}
virtual void apply(float* data, long size) {}
Comment on lines +39 to +51

@jatinchowdhury18 jatinchowdhury18 Apr 7, 2023

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting functions which will be called per-sample (or maybe multiple times per-sample depending on the network architecture) behind a virtual interface will likely have a significant performance impact, since it prevents the compiler from inlining these function calls in most cases.

In RTNeural, this is a big reason why the "dynamic" API is much slower than the "static" API.

@mikeoliphant mikeoliphant Apr 7, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was worried about inlining, too, which is why I made sure to keep everything as block operations. I did not see any slowdown over the existing implementation in my testing, but that could be compiler-specific? I'm testing using VS on Windows.

The override methods definitely make it more convenient to use, but it would be worth giving that up if there is a clearly demonstrable performance gain to be had.

@sdatkinson sdatkinson Apr 9, 2023

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The functions that apply over a block seem like a good idea. Back-of-the-envelope, this is looking at 64*16=1024 elementwise ops (buffer 64, 16-channel model). I'd ballpark the virtual table overhead around 1% or less?


static Activation* get_activation(const std::string name);
static void enable_fast_tanh();
static void disable_fast_tanh();

protected:
static std::unordered_map<std::string, Activation *> _activations;
};

class ActivationTanh : public Activation
{
return 1.0f / (1.0f + expf(-x));
public:
void apply(float *data, long size) override
{
for (long pos = 0; pos < size; pos++)
{
data[pos] = std::tanh(data[pos]);
}
Comment on lines +66 to +69

@jatinchowdhury18 jatinchowdhury18 Apr 7, 2023

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would probably be better to use Eigen's math functions for implementing some of these things, since they do some internal vectorization. (And that vectorization can be even better if the Matrix size is known at compile-time)

@mikeoliphant mikeoliphant Apr 7, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, tanh is best avoided completely...

I'm pretty sure I looked for a way to do a matrix-level tanh with Eigen, and didn't see one.

}
};

class ActivationHardTanh : public Activation
{
public:
ActivationHardTanh(){};
void apply(float* data, long size) override
{
for (long pos = 0; pos < size; pos++)
{
data[pos] = hard_tanh(data[pos]);
}
}
};

class ActivationFastTanh : public Activation
{
public:
ActivationFastTanh(){};
void apply(float* data, long size) override
{
for (long pos = 0; pos < size; pos++)
{
data[pos] = fast_tanh(data[pos]);
}
}
};

class ActivationReLU : public Activation
{
public:
ActivationReLU(){};
void apply(float* data, long size) override
{
for (long pos = 0; pos < size; pos++)
{
data[pos] = relu(data[pos]);
}
}
};

class ActivationSigmoid : public Activation
{
public:
ActivationSigmoid(){};
void apply(float* data, long size) override
{
for (long pos = 0; pos < size; pos++)
{
data[pos] = sigmoid(data[pos]);
}
}
};

}; // namespace activations
10 changes: 3 additions & 7 deletions NAM/convnet.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ void convnet::ConvNetBlock::set_params_(const int in_channels, const int out_cha
this->conv.set_size_and_params_(in_channels, out_channels, 2, _dilation, !batchnorm, params);
if (this->_batchnorm)
this->batchnorm = BatchNorm(out_channels, params);
this->activation = activation;
this->activation = activations::Activation::get_activation(activation);
}

void convnet::ConvNetBlock::process_(const Eigen::MatrixXf& input, Eigen::MatrixXf& output, const long i_start,
Expand All @@ -67,12 +67,8 @@ void convnet::ConvNetBlock::process_(const Eigen::MatrixXf& input, Eigen::Matrix
this->conv.process_(input, output, i_start, ncols, i_start);
if (this->_batchnorm)
this->batchnorm.process_(output, i_start, i_end);
if (this->activation == "Tanh")
tanh_(output, i_start, i_end);
else if (this->activation == "ReLU")
relu_(output, i_start, i_end);
else
throw std::runtime_error("Unrecognized activation");

this->activation->apply(output.middleCols(i_start, ncols));
}

long convnet::ConvNetBlock::get_out_channels() const
Expand Down
2 changes: 1 addition & 1 deletion NAM/convnet.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ class ConvNetBlock
private:
BatchNorm batchnorm;
bool _batchnorm;
std::string activation;
activations::Activation *activation;
};

class _Head
Expand Down
88 changes: 0 additions & 88 deletions NAM/dsp.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,94 +206,6 @@ void Linear::_process_core_()

// NN modules =================================================================

void relu_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end)
{
for (long j = j_start; j < j_end; j++)
for (long i = 0; i < x.rows(); i++)
x(i, j) = x(i, j) < (float)0.0 ? (float)0.0 : x(i, j);
}

void relu_(Eigen::MatrixXf& x, const long j_start, const long j_end)
{
relu_(x, 0, x.rows(), j_start, j_end);
}

void relu_(Eigen::MatrixXf& x)
{
relu_(x, 0, x.rows(), 0, x.cols());
}

void sigmoid_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end)
{
for (long j = j_start; j < j_end; j++)
for (long i = i_start; i < i_end; i++)
x(i, j) = 1.0 / (1.0 + expf(-x(i, j)));
}

inline float fast_tanh_(const float x)
{
const float ax = fabsf(x);
const float x2 = x * x;

return (x * (2.45550750702956f + 2.45550750702956f * ax + (0.893229853513558f + 0.821226666969744f * ax) * x2)
/ (2.44506634652299f + (2.44506634652299f + x2) * fabsf(x + 0.814642734961073f * x * ax)));
}

inline float hard_tanh_(const float x)
{
const float t = x < -1 ? -1 : x;
return t > 1 ? 1 : t;
}

void tanh_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end)
{
for (long j = j_start; j < j_end; j++)
for (long i = i_start; i < i_end; i++)
x(i, j) = tanh_impl_(x(i, j));
}

void tanh_(Eigen::MatrixXf& x, const long j_start, const long j_end)
{
tanh_(x, 0, x.rows(), j_start, j_end);
}

void tanh_(Eigen::MatrixXf& x)
{

float* ptr = x.data();

long size = x.rows() * x.cols();

for (long pos = 0; pos < size; pos++)
{
ptr[pos] = tanh_impl_(ptr[pos]);
}
}

void hard_tanh_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end)
{
for (long j = j_start; j < j_end; j++)
for (long i = i_start; i < i_end; i++)
x(i, j) = hard_tanh_(x(i, j));
}

void hard_tanh_(Eigen::MatrixXf& x, const long j_start, const long j_end)
{
hard_tanh_(x, 0, x.rows(), j_start, j_end);
}

void hard_tanh_(Eigen::MatrixXf& x)
{
float* ptr = x.data();

long size = x.rows() * x.cols();

for (long pos = 0; pos < size; pos++)
{
ptr[pos] = hard_tanh_(ptr[pos]);
}
}

void Conv1D::set_params_(std::vector<float>::iterator& params)
{
if (this->_weight.size() > 0)
Expand Down
28 changes: 2 additions & 26 deletions NAM/dsp.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@

#include <Eigen/Dense>

#include "activations.h"

enum EArchitectures
{
kLinear = 0,
Expand Down Expand Up @@ -138,32 +140,6 @@ class Linear : public Buffer

// NN modules =================================================================

// Activations

// In-place ReLU on (N,M) array
void relu_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end);
// Subset of the columns
void relu_(Eigen::MatrixXf& x, const long j_start, const long j_end);
void relu_(Eigen::MatrixXf& x);

// In-place sigmoid
void sigmoid_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end);
void sigmoid_(Eigen::MatrixXf& x);

// In-place Tanh on (N,M) array
void tanh_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end);
// Subset of the columns
void tanh_(Eigen::MatrixXf& x, const long i_start, const long i_end);

void tanh_(Eigen::MatrixXf& x);

// In-place Hardtanh on (N,M) array
void hard_tanh_(Eigen::MatrixXf& x, const long i_start, const long i_end, const long j_start, const long j_end);
// Subset of the columns
void hard_tanh_(Eigen::MatrixXf& x, const long i_start, const long i_end);

void hard_tanh_(Eigen::MatrixXf& x);

class Conv1D
{
public:
Expand Down
1 change: 0 additions & 1 deletion NAM/lstm.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
#include <string>
#include <vector>

#include "activations.h"
#include "lstm.h"

lstm::LSTMCell::LSTMCell(const int input_size, const int hidden_size, std::vector<float>::iterator& params)
Expand Down
23 changes: 7 additions & 16 deletions NAM/wavenet.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,13 @@ void wavenet::_Layer::process_(const Eigen::MatrixXf& input, const Eigen::Matrix
this->_conv.process_(input, this->_z, i_start, ncols, 0);
// Mix-in condition
this->_z += this->_input_mixin.process(condition);
if (this->_activation == "Hardtanh")
hard_tanh_(this->_z);
else if (this->_activation == "Tanh")
tanh_(this->_z);
else if (this->_activation == "ReLU")
relu_(this->_z, 0, channels, 0, this->_z.cols());
else
throw std::runtime_error("Unrecognized activation.");

this->_activation->apply(this->_z);

if (this->_gated)
{
sigmoid_(this->_z, channels, 2 * channels, 0, this->_z.cols());
activations::Activation::get_activation("Sigmoid")->apply(this->_z.block(channels, 0, channels, this->_z.cols()));

@sdatkinson sdatkinson Apr 9, 2023

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I wouldn't call this "hard-coded" in this case--this is the most common implementation of a gated activation (messing with the sigmoid is, to my knowledge, not really done).


this->_z.topRows(channels).array() *= this->_z.bottomRows(channels).array();
// this->_z.topRows(channels) = this->_z.topRows(channels).cwiseProduct(
// this->_z.bottomRows(channels)
Expand Down Expand Up @@ -172,7 +168,7 @@ void wavenet::_LayerArray::_rewind_buffers_()

wavenet::_Head::_Head(const int input_size, const int num_layers, const int channels, const std::string activation)
: _channels(channels)
, _activation(activation)
, _activation(activations::Activation::get_activation(activation))
, _head(num_layers > 0 ? channels : input_size, 1, true)
{
assert(num_layers > 0);
Expand Down Expand Up @@ -220,12 +216,7 @@ void wavenet::_Head::set_num_frames_(const long num_frames)

void wavenet::_Head::_apply_activation_(Eigen::MatrixXf& x)
{
if (this->_activation == "Tanh")
tanh_(x);
else if (this->_activation == "ReLU")
relu_(x);
else
throw std::runtime_error("Unrecognized activation.");
this->_activation->apply(x);
}

// WaveNet ====================================================================
Expand Down
Loading