Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
e717d93
refac(neuron_t): add kind parameter
rouson Aug 28, 2024
8efec6f
refac(layer_t): add kind parameter
rouson Aug 28, 2024
1e25cd6
refac(tensor_map): add kind parameter
rouson Aug 28, 2024
c63d9bb
refac(tensor_m): add kind parameter
rouson Aug 28, 2024
8bd7ae7
refac(inference_engine_t): add kind parameter
rouson Aug 28, 2024
dff93d9
test(inference_engine_t): use eqv op for compare
rouson Aug 28, 2024
1c1a93e
refac: define generic activation/derivative
rouson Aug 28, 2024
216e59b
chore(tensor_map_test): match file name to module
rouson Aug 28, 2024
72b6348
feat(tensor_t):dble-prec constructor/accessor/test
rouson Aug 29, 2024
dd0cb91
feat(step): add double-precision activation
rouson Aug 29, 2024
6ceabf0
feat(sigmoid):double-precision activation/derivat
rouson Aug 29, 2024
0d69cce
feat(relu): add double-precision activation/deriv
rouson Aug 29, 2024
59547e7
feat(gelu): add double-prec activation/derivatie
rouson Aug 29, 2024
fc54a18
feat: finish double-prec activation/derivs
rouson Aug 29, 2024
84d21f7
refac(tensor_map):mk type-bound proc names generic
rouson Aug 29, 2024
d878713
feat(tensor_map): add double-precision procedures
rouson Aug 29, 2024
d011f0d
refac(neuron): mk procedure names generic bindings
rouson Aug 29, 2024
c496147
refac(layer):make procedure names generic bindings
rouson Aug 29, 2024
85fba96
feat(neuron_t): add double-precision procedures
rouson Aug 29, 2024
71bbff8
feat(layer_t): add double-precision procedures
rouson Aug 29, 2024
9c99739
feat(layer_t): add double-precision procedures
rouson Aug 29, 2024
d031662
Merge remote-tracking branch 'refs/remotes/origin/parameterized-deriv…
rouson Aug 31, 2024
d919e44
feat(trainable_engine_t): add real kind parameter
rouson Aug 31, 2024
070b4c3
feat(trainable_engine_t): add generic "predict"
rouson Aug 31, 2024
de37292
refac(trainable_engine): make bindings generic
rouson Aug 31, 2024
a5636dd
feat(input_output_pair_t): add kind parameter
rouson Aug 31, 2024
d82b498
feat(input_output_pair): support double precision
rouson Sep 1, 2024
0e284a6
feat(mini_match): kind parameter +generic bindings
rouson Sep 1, 2024
0ff2681
feat(mini_match): support double precision
rouson Sep 1, 2024
7ea4e3b
feat(hyperparameters): support double precision
rouson Sep 1, 2024
d363587
fix(tensor_t): set component kind parameter
rouson Sep 1, 2024
d3da4e6
feat(training_configuration_t): add kind parameter
rouson Sep 1, 2024
0d6da76
refac(training_configuration): mk bindings generic
rouson Sep 1, 2024
533ce13
feat(dble_prec_file):extend file_t, produce dp_str
rouson Sep 1, 2024
e292cae
chore: rm global "rkind" real kind parameter
rouson Sep 1, 2024
1e38642
chore(fpm): update to julienne 1.2.1 dependency
rouson Sep 2, 2024
211f9cb
feat(metadata):dble_prec_string_t json constructor
rouson Sep 2, 2024
0e26cb0
feat(tensor_map): add double_precision_from_json
rouson Sep 2, 2024
7a1db66
chore(example): rm public non-type-bound infer
rouson Sep 2, 2024
d11a377
chore(inference_engine): generic bindings|priv tbp
rouson Sep 2, 2024
a6ccaee
feat(inference_engine_t):double-precision bindings
rouson Sep 2, 2024
cd3acd6
test(inference_engine): double-precision inference
rouson Sep 2, 2024
bf1c126
test(inference_engine): whitespace edits
rouson Sep 2, 2024
3abb069
fix(inference_engine_m): make entities public
rouson Sep 3, 2024
3558574
build(fpm): update dependency to julienne 1.2.2
rouson Sep 3, 2024
f4c0c6f
doc(example):double-precision concurrent inference
rouson Sep 3, 2024
b8468bc
Add new Github CI actions yaml file to build with flang
ktras Sep 3, 2024
71860db
Replace CI yaml file that builds with gfortran with new CI
ktras Sep 3, 2024
d134548
Update .github/workflows/CI.yml
ktras Sep 3, 2024
0221ee0
Add version check to CI
ktras Sep 3, 2024
cbf8059
Merge pull request #200 from BerkeleyLab/support-flang-ci
rouson Sep 3, 2024
867ae6f
refac(inference_engine_s): edit Cray workaround
rouson Sep 4, 2024
eb4e067
feat(pdt): merge infer_a proc from aerosol branch
rouson Sep 4, 2024
c900539
refac(inference_engine): unnormalized -> unmapped
rouson Sep 5, 2024
d06f14f
fix(unmapped_engine_t): make public
rouson Sep 5, 2024
80b9e23
Support building cloud-microphysics with LLVM Flang
ktras Sep 5, 2024
27be203
Improve setup script changes in cloud microphysics
ktras Sep 6, 2024
bfd9c17
Merge pull request #203 from BerkeleyLab/support-llvm-flang-for-cloud…
rouson Sep 7, 2024
1534ee2
chore: git mv cloud-microphysics/ demo/
rouson Sep 12, 2024
45448c1
chore: copy infer-aersol to pdt branch
rouson Sep 12, 2024
4c905b5
feat(demo/app/infer-aerosol): add double precision
rouson Sep 15, 2024
4595a8c
refac(NetCDF_file): use generic assumed-rank I/O
rouson Sep 15, 2024
a092e02
fix(infer-aerosol): handle missing req'd arg
rouson Sep 15, 2024
e7bc20a
fix(demo): revise setup.sh for Linux
rouson Sep 16, 2024
c5eb64b
fix(setup.sh): works on macOS
rouson Sep 16, 2024
2d908a6
Update filenames being read in by infer_aerosol
ktras Sep 16, 2024
5f4d8f9
Update actions/upload-artifact version for github deploy
ktras Sep 16, 2024
9f6ba89
Merge pull request #204 from BerkeleyLab/update-filenames-infer-aerosol
rouson Sep 16, 2024
47f9ef4
feat(demo): add unmapped_engine_t constructor
rouson Sep 20, 2024
98a68a4
fix(infer-aerosol): double-prec. sign() invocation
rouson Sep 20, 2024
f387290
chore: add missing copyright statements to files
rouson Sep 21, 2024
d6154f7
doc: update subproject name in demo/fpm.toml
rouson Sep 21, 2024
f4a28c2
Remove unallowed whitespace from project name in `demo/fpm.toml`
ktras Sep 23, 2024
7d383fb
Merge pull request #209 from BerkeleyLab/fix-fpm-toml-project-name
rouson Sep 24, 2024
ffd1c7c
fix(inference_engine_t): conforming real literal
rouson Sep 27, 2024
bb92ffa
Merge branch 'parallel' into parameterized-derived-types
rouson Oct 2, 2024
5a06c4d
fix(sigmoid): make real literals double precision
rouson Oct 10, 2024
08f4254
fix(gelu): make constants double-precision
rouson Oct 10, 2024
8f686f6
Merge pull request #214 from BerkeleyLab/fix-activation-precision
rouson Oct 10, 2024
f848d43
feat: make inference non_overridable
rouson Oct 9, 2024
6a1c601
refac(inference_engine): define activation_t
rouson Oct 10, 2024
3fe8fb7
feat(activation_t): add construct_from_name
rouson Oct 10, 2024
8d6f3b8
feat(tensor_map): mk map functions non_overridable
rouson Oct 10, 2024
eba29a0
feat(trainable_engine): non-polymorphic activation
rouson Oct 11, 2024
63ab041
refac: rm activation_strategy class hierachy
rouson Oct 12, 2024
1c1e487
refac(activation_s): simplify function_name getter
rouson Oct 12, 2024
d9a9073
refac(engines): make component constructors pure
rouson Oct 12, 2024
6ef2d1c
feat: non_overridble for inference & training
rouson Oct 12, 2024
d2e309c
Merge pull request #215 from BerkeleyLab/non_overridable
rouson Oct 12, 2024
9958bb7
feat(trainable_network): extend inference_engine_t
rouson Oct 14, 2024
5ef6625
feat(sat-mix-rat):adjust to use trainale_network_t
rouson Oct 14, 2024
d45c57e
fix(activation_t):add :: in default initialization
rouson Oct 14, 2024
4c76b7a
fix(unmapped_network): rm ambiguous specific proc
rouson Oct 14, 2024
e4f8ba9
chore(trainable_network):replaces trainable_engine
rouson Oct 14, 2024
e89c7b4
fix(train-cloud-micro):let write_lines() open file
rouson Oct 14, 2024
e7794c5
Merge pull request #217 from BerkeleyLab/simplify-class-relationships
rouson Oct 14, 2024
b23524d
chore(inference_engine): -> neural_network/fiats
rouson Oct 14, 2024
2fbb161
doc(ford): update project name & summary
rouson Oct 14, 2024
bada395
chore: rename asymmetric_{engine,network}_test
rouson Oct 15, 2024
be7960e
doc(README): update compiler status
rouson Oct 15, 2024
4b09cdc
refac(json): rename acceptable_engine_tag
rouson Oct 15, 2024
d69324d
chore: rm unused exhcnage_t type
rouson Oct 15, 2024
b8473d6
doc(README.md): update
rouson Oct 15, 2024
6f03491
chore: white space edits
rouson Oct 16, 2024
31ecd59
chore(training_configuration): rm unused function
rouson Oct 16, 2024
c574fd0
doc(UML): update class diagram
rouson Oct 15, 2024
7f606a4
doc(uml): add components & type-bound procedures
rouson Oct 15, 2024
5f16334
doc(README/uml): add link to UML diagram
rouson Oct 16, 2024
5292c9b
doc(uml): add classes & relationships
rouson Oct 16, 2024
48f2268
Revert "chore(training_configuration): rm unused function"
rouson Oct 16, 2024
5f844c8
Merge pull request #218 from BerkeleyLab/rename-inference_engine_t
rouson Oct 16, 2024
289dc4a
doc(README): edit compiler status
rouson Oct 16, 2024
dcc407e
doc(README): fix typo in branch name
rouson Oct 16, 2024
55c377b
doc(README): specify compiler issue
rouson Oct 16, 2024
53c8550
doc(README): fix CCE compiler status text
rouson Oct 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 11 additions & 26 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: CI
name: Build with LLVM Flang

on: [push, pull_request]

Expand All @@ -8,11 +8,12 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [macOS-12, ubuntu-24.04]
os: [ubuntu-24.04]
fail-fast: true
container: gmao/llvm-flang:latest
env:
FC: gfortran
GCC_V: 14
FC: flang-new
CC: clang

steps:
- name: Checkout code
Expand All @@ -22,28 +23,12 @@ jobs:
with:
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Install Dependencies MacOS
if: contains(matrix.os, 'mac')
run: |
brew install gcc@${GCC_V}
sudo ln -s $(which gfortran-${GCC_V}) $(dirname $(which gfortran-${GCC_V}))/gfortran

- name: Install on Ubuntu
if: contains(matrix.os, 'ubuntu')
run: |
sudo apt update
sudo apt install -y build-essential gfortran-14 gcc-14 g++-14

- name: Build and Test MacOS
if: contains(matrix.os, 'mac')
run: |
export PATH="${HOME}/.local/bin:$PATH"
./setup.sh

- name: Build and Test Ubuntu
- name: Build and Test with LLVM Flang
if: contains(matrix.os, 'ubuntu')
run: |
fpm --version
export FPM_FC=gfortran-14
export FPM_CC=gcc-14
fpm test
$FC --version
$CC --version
export FPM_FC=$FC
export FPM_CC=$CC
fpm test --flag "-mmlir -allow-assumed-rank -O3"
2 changes: 1 addition & 1 deletion .github/workflows/deploy-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
ford ford.md
cp ./README.md ./doc/html
- name: Upload Documentation
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: documentation
path: doc/html
Expand Down
120 changes: 71 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,29 @@

```ascii
_ __ _
(_) / _| (_)
_ _ __ | |_ ___ _ __ ___ _ __ ___ ___ ___ _ __ __ _ _ _ __ ___
| | '_ \| _/ _ \ '__/ _ \ '_ \ / __/ _ \ __ / _ \ '_ \ / _` | | '_ \ / _ \
| | | | | || __/ | | __/ | | | (_| __/ |__| | __/ | | | (_| | | | | | __/
|_|_| |_|_| \___|_| \___|_| |_|\___\___| \___|_| |_|\__, |_|_| |_|\___|
__/ |
|___/
```
Inference-Engine
================
___________.__ __
\_ _____/|__|____ _/ |_ ______
| __) | \__ \\ __\/ ___/
| \ | |/ __ \| | \___ \
\___ / |__(____ /__| /____ >
\/ \/ \/
```
Fiats: Functional inference and training for surrogates
=======================================================
Alternatively, _Fortran inference and training for science_.

[Overview](#overview) | [Getting Started](#getting-started) | [Documentation](#documentation)

Overview
--------
Inference-Engine supports research in the training and deployment of neural-network surrogate models for computational science.
Inference-Engine also provides a platform for exploring and advancing the native parallel programming features of Fortran 2023 in the context of deep learning.
The language features of interest facilitate loop-level parallelism via the `do concurrent` construct and Single-Program, Multiple Data (SMPD) parallelism via "multi-image" (e.g., multithreaded or multiprocess) execution.
Toward these ends,
Fiats supports research on the training and deployment of neural-network surrogate models for computational science.
Fiats also provides a platform for exploring and advancing the native parallel programming features of Fortran 2023 in the context of deep learning.
The design of Fiats centers around functional programming patterns that facilitate concurrency, including loop-level parallelism via the `do concurrent` construct and Single-Program, Multiple Data (SMPD) parallelism via "multi-image" (e.g., multithreaded or multiprocess) execution.
Towards these ends,

* Most Inference-Engine procedures are `pure` and thus satisfy a language requirement for invocation inside `do concurrent`,
* The network training procedure uses `do concurrent` to expose automatic parallelization opportunities to compilers, and
* Most Fiats procedures are `pure` and thus satisfy a language requirement for invocation inside `do concurrent`,
* The network training procedure use `do concurrent` to expose automatic parallelization opportunities to compilers, and
* Exploiting multi-image execution to speedup training is under investigation.

To broaden support for the native parallel features, Inference-Engine's contributors also write compiler tests, bug reports, and patches; develop a parallel runtime library ([Caffeine]); participate in the language standardization process; and provide example inference and training code for exercising and evaluating compilers' automatic parallelization capabilities on processors and accelerators, including Graphics Processing Units (GPUs).
To broaden support for the native parallel features, the Fiats contributors also write compiler tests, bug reports, and patches; develop a parallel runtime library ([Caffeine]); participate in the language standardization process; and provide example inference and training code for exercising and evaluating compilers' automatic parallelization capabilities on processors and accelerators, including Graphics Processing Units (GPUs).

Available optimizers:
* Stochastic gradient descent and
Expand All @@ -51,7 +49,7 @@ Getting Started
The [example] subdirectory contains demonstrations of several relatively simple use cases.
We recommend reviewing the examples to see how to handle basic tasks such as configuring a network training run or reading a neural network and using it to perform inference.

The [demo] subdirectory contains demonstration applications that depend on Inference-Engine but build separately due to requiring additional prerequisites such as NetCDF and HDF5.
The [demo] subdirectory contains demonstration applications that depend on Fiats but build separately due to requiring additional prerequisites such as [NetCDF] and [HDF5].
The demonstration applications
- Train a cloud microphysics model surrogate for the Intermediate Complexity Atmospheric Research ([ICAR]) package,
- Perform inference using a pretrained model for aerosol dynamics in the Energy Exascale Earth System ([E3SM]) package, and
Expand All @@ -61,26 +59,22 @@ The demonstration applications
Because this repository supports programming language research, the code exercises new language features in novel ways.
We recommend using any compiler's latest release or even building open-source compilers from source.
The [handy-dandy] repository contains scripts capturing steps for building the [LLVM] compiler suite.
The remainder of this section contains commands for building Inference-Engine with a recent Fortran compiler and the Fortran Package Manager ([`fpm`]) in your `PATH`.

#### GNU (`gfortran`) 13 or higher required
```
fpm test --compiler gfortran --profile release
```
The remainder of this section contains commands for building Fiats with a recent Fortran compiler and the Fortran Package Manager ([`fpm`]).

#### NAG (`nagfor`)
#### Supported Compilers
##### LLVM (`flang-new`)
With LLVM `flang` 20 installed in your `PATH`, build and test Fiats with the installed `flang-new` symlink in order for `fpm` to correctly identify the compiler:
```
fpm test --compiler nagfor --flag -fpp --profile release
fpm test --compiler flang-new --flag "-O3"
```
With LLVM `flang` 19, enable the compiler's experimental support for assumed-rank entities:

#### LLVM (`flang-new`)
Building with `flang-new` requires passing flags to enable the compiler's experimental support for assumed-rank entities:
```
fpm test --compiler flang-new --flag "-mmlir -allow-assumed-rank -O3"
```

##### _Experimental:_ Automatic parallelization of `do concurrent` on CPUs
With the `amd_trunk_dev` branch of the [ROCm fork] fork of LLVM, automatic parallelization currently works for inference, e.g.
###### _Experimental:_ Automatic parallelization of `do concurrent` on CPUs
With the `amd-trunk-dev` branch of the [ROCm fork] of LLVM, automatically parallelize inference calculations inside `do concurrent` constructs:
```
fpm run \
--example concurrent-inferences \
Expand All @@ -89,25 +83,46 @@ fpm run \
-- --network model.json

```
where `model.json` must be a neural network in the [JSON] format used by Inference-Engine and the companion [nexport] package.
Automatic parallelization for training is under development.
where `model.json` must be a neural network in the [JSON] format used by Fiats and the companion [nexport] package.

Automatic parallelization for training neural networks is under development.

#### Partially Supported Compilers

#### Intel (`ifx`)
Fiats release 0.14.0 and earlier support the use of the NAG, GNU, and Intel Fortran compilers.
We are corresponding with these compilers' developers about addressing the compiler issues preventing building newer Fiats releases.

##### NAG (`nagfor`)
```
fpm test --compiler nagfor --flag -fpp --profile release
```

##### GNU (`gfortran`)
Compiler bugs related to parameterized derived types currently prevent `gfortran` from building Fiats versions 0.15.0 or later.
Test and build earlier versions of Fiats build with the following command:
```
fpm test --compiler gfortran --profile release
```

##### Intel (`ifx`)
Compiler bugs related to generic name resolution currently prevent `ifx` from building Fiats versions 0.15.0 or later.
Test and build earlier versions of Fiats build with the following command:
```
fpm test --compiler ifx --profile release --flag -O3
```

##### _Experimental:_ Automatic offloading of `do concurrent` to GPUs
This capability is under development with the goal to facilitate automatic GPU offloading via the following command:
```
fpm test --compiler ifx --profile releae --flag "-fopenmp-target-do-concurrent -qopenmp -fopenmp-targets=spir64 -O3"
fpm test --compiler ifx --profile release --flag "-fopenmp-target-do-concurrent -qopenmp -fopenmp-targets=spir64 -O3"
```

#### HPE (`crayftn.sh`) -- under development
Support for the Cray Compiler Environment (CCE) Fortran compiler is under development.
Building with the CCE `ftn` compiler wrapper requires an additional trivial wrapper
shell script. For example, create a file `crayftn.sh` with the following contents and
place this file's location in your `PATH`:
#### Under Development
We are corresponding with the developers of the compiler(s) below about addressing the compiler issues preventing building Fiats.

#### HPE Cray Compiler Environment (CCE) (`crayftn.sh`)
Building with the CCE `ftn` compiler wrapper requires an additional trivial wrapper.
For example, create a file `crayftn.sh` with the following contents and place this file's location in your `PATH`:
```
#!/bin/bash

Expand All @@ -119,7 +134,7 @@ fpm test --compiler crayftn.sh
```

### Configuring a training run
Inference-Engine imports hyperparameters and network configurations to and from JSON files.
Fiats imports hyperparameters and network configurations to and from JSON files.
To see the expected file format, run the [print-training-configuration] example as follows:
```
% fpm run --example print-training-configuration --compiler gfortran
Expand All @@ -141,7 +156,7 @@ Project is up to date
}
}
```
Inference-Engine's JSON file format is fragile: splitting or combining lines breaks the file reader.
The Fiats JSON file format is fragile: splitting or combining lines breaks the file reader.
Files with added or removed white space or reordered whole objects ("hyperparameters" or "network configuration") should work.
A future release will leverage the [rojff] JSON interface to allow for more flexible file formatting.

Expand All @@ -158,35 +173,42 @@ The following is representative output after 3000 epochs:
2000 0.61259E-04 9.8345 2,4,72,2,1
3000 0.45270E-04 14.864 2,4,72,2,1
```
The example program halts execution after reaching a cost-function threshold (which requires millions of epochws) or a maximum number of iterations or if the program detects a file named `stop` in the source-tree root directory.
The example program halts execution after reaching a cost-function threshold (which requires millions of epochs) or a maximum number of iterations or if the program detects a file named `stop` in the source-tree root directory.
Before halting, the program will print a table of expected and predicted saturated mixing ratio values across a range of input pressures and temperatures, wherein two the inputs have each been mapped to the unit interval [0,1].
The program also writes the neural network initial condition to `initial-network.json` and the final (trained) network to the file specified in the above command: `sat-mix-rat.json`.

### Performing inference
Users with a PyTorch model may use [nexport] to export the model to JSON files that Inference-Engine can read.
Users with a PyTorch model may use [nexport] to export the model to JSON files that Fiats can read.
Examples of performing inference using a neural-network JSON file are in [example/concurrent-inferences].

Documentation
-------------
Please see our [GitHub Pages site] for HTML documentation generated by [`ford`] or generate documentaiton locally by installing `ford` and executing `ford ford.md`.
### HTML
Please see our [GitHub Pages site] for Hypertext Markup Languge (HTML) documentation generated by [`ford`] or generate documentation locally by installing `ford` and executing `ford ford.md`.

### UML
Please see the `doc/uml` subdirectory for Unified Modeling Language (UML) diagrams such as a comprehensive Fiats [class diagram] with human-readable [Mermaid] source that renders graphically when opened by browsing to the document on GitHub.

[Building and testing]: #building-and-testing
[Caffeine]: https://go.lbl.gov/caffeine
[class diagram]: ./doc/uml/class-diagram.md
[Documentation]: #documentation
[demo]: demo
[E3SM]: https://e3sm.org
[example]: example
[demo]: demo
[Documentation]: #documentation
[example/print-training-configuration.F90]: example/print-training-configuration.F90
[example/concurrent-inferences]: example/concurrent-inferences.f90
[`ford`]: https://github.com/Fortran-FOSS-Programmers/ford
[`fpm`]: https://github.com/fortran-lang/fpm
[Getting Started]: #getting-started
[GitHub Pages site]: https://berkeleylab.github.io/inference-engine/
[GitHub Pages site]: https://berkeleylab.github.io/fiats/
[handy-dandy]: https://github.com/rouson/handy-dandy/blob/main/src
[HDF5]: https://www.hdfgroup.org/solutions/hdf5/
[ICAR]: https://github.com/BerkeleyLab/icar/tree/neural-net
[JSON]: https://www.json.org/json-en.html
[LLVM]: https://github.com/llvm/llvm-project
[Mermaid]: https://mermaid.js.org
[NetCDF]: https://www.unidata.ucar.edu/software/netcdf/
[nexport]: https://go.lbl.gov/nexport
[Overview]: #overview
[ROCm fork]: https://github.com/ROCm/llvm-project
Expand Down
Loading