Skip to content

Update versions for snapshot testing #1023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 26 commits into from
Jul 27, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
8ec173a
Setup examples for testing
agibsonccc Apr 2, 2021
7293098
Merge branch 'master' into ag_snapshot-updates
agibsonccc Apr 2, 2021
a7fb51a
Update example-sanity-check.yml
agibsonccc Apr 2, 2021
bd29305
Add actions
agibsonccc Apr 2, 2021
4767f43
Update example-sanity-check.yml
agibsonccc Apr 2, 2021
7adbefb
Update examples
agibsonccc Apr 2, 2021
038c4e5
Remove unneeded commands
agibsonccc Apr 2, 2021
0fe4a21
Update example-sanity-check.yml
agibsonccc Apr 2, 2021
cf9e348
Update example-sanity-check.yml
agibsonccc Apr 2, 2021
211e098
Update example-sanity-check.yml
agibsonccc Apr 2, 2021
9971dde
Update example-sanity-check.yml
agibsonccc Apr 2, 2021
b8e7664
Update example-sanity-check.yml
agibsonccc Apr 2, 2021
830a479
Update examples for building
agibsonccc Apr 6, 2021
d467f1b
Add samediff test
agibsonccc Apr 6, 2021
a1a5ec4
Remove cuda specific examples
agibsonccc Apr 6, 2021
79aba18
Add snapshots to some modules
agibsonccc Apr 7, 2021
82dc88c
Staging repo setup
agibsonccc Apr 26, 2021
19c5dbe
Update examples repo
agibsonccc Apr 27, 2021
39ae70f
Remove teh
agibsonccc Apr 28, 2021
21dd390
Remove codec module
agibsonccc Apr 28, 2021
6f3190e
Update pom.xml
agibsonccc May 22, 2021
6428a3c
Examples update
agibsonccc May 24, 2021
9078f48
Update dl4j versions to stable, remove old repositories used for testing
agibsonccc May 27, 2021
1d415a6
Update version
agibsonccc Jul 27, 2021
4614ac9
Fix versions, repositories
agibsonccc Jul 27, 2021
f744cf3
Merge conflict
agibsonccc Jul 27, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 10 additions & 21 deletions .github/workflows/example-sanity-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,39 +25,28 @@ jobs:
with:
access_token: ${{ github.token }}
- uses: actions/checkout@v2
- uses: ./.github/actions/update-deps-linux
- uses: ./.github/actions/install-protobuf-linux
- uses: ./.github/actions/install-cmake-linux
- name: Set up Java for publishing to GitHub Packages
uses: actions/setup-java@v1
with:
java-version: 1.8
server-id: sonatype-nexus-snapshots
server-username: MAVEN_USERNAME
server-password: MAVEN_PASSWORD
gpg-private-key: ${{ secrets.SONATYPE_GPG_KEY }}
gpg-passphrase: MAVEN_GPG_PASSPHRASE
- name: Build on linux-arm64
shell: bash
env:
DEBIAN_FRONTEND: noninteractive
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DEPLOY: 1
BUILD_USING_MAVEN: 1
TARGET_OS: linux
CURRENT_TARGET: arm64
PUBLISH_TO: ossrh
DEPLOY_TO: ossrh
MAVEN_USERNAME: ${{ secrets.SONATYPE_USER_1 }}
MAVEN_PASSWORD: ${{ secrets.SONATYPE_USER1_PASS }}
MAVEN_GPG_PASSPHRASE: ${{ secrets.PACKAGES_GPG_PASS }}

run: |
mvn --version
cmake --version
protoc --version
for folder in `ls`; do
mvn clean test
if [[ -d "$folder" ]]; then
cd "$folder"
if test -f "pom.xml"; then
mvn clean test
fi
cd ..
fi



done


Empty file.
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ We do not monitor the github issues of this repository very often.
The **Eclipse Deeplearning4J** (DL4J) ecosystem is a set of projects intended to support all the needs of a JVM based deep learning application. This means starting with the raw data, loading and preprocessing it from wherever and whatever format it is in to building and tuning a wide variety of simple and complex deep learning networks.

The DL4J stack comprises of:
- **DL4J**: High level API to build MultiLayerNetworks and ComputationGraphs with a variety of layers, including custom ones. Supports importing Keras models from h5, including tf.keras models (as of 1.0.0-beta7) and also supports distributed training on Apache Spark
- **DL4J**: High level API to build MultiLayerNetworks and ComputationGraphs with a variety of layers, including custom ones. Supports importing Keras models from h5, including tf.keras models (as of 1.0.0-M1.1) and also supports distributed training on Apache Spark
- **ND4J**: General purpose linear algebra library with over 500 mathematical, linear algebra and deep learning operations. ND4J is based on the highly-optimized C++ codebase LibND4J that provides CPU (AVX2/512) and GPU (CUDA) support and acceleration by libraries such as OpenBLAS, OneDNN (MKL-DNN), cuDNN, cuBLAS, etc
- **SameDiff** : Part of the ND4J library, SameDiff is our automatic differentiation / deep learning framework. SameDiff uses a graph-based (define then run) approach, similar to TensorFlow graph mode. Eager graph (TensorFlow 2.x eager/PyTorch) graph execution is planned. SameDiff supports importing TensorFlow frozen model format .pb (protobuf) models. Import for ONNX, TensorFlow SavedModel and Keras models are planned. Deeplearning4j also has full SameDiff support for easily writing custom layers and loss functions.
- **DataVec**: ETL for machine learning data in a wide variety of formats and files (HDFS, Spark, Images, Video, Audio, CSV, Excel etc)
Expand All @@ -40,6 +40,17 @@ Users can also refer to the [simple sample project provided](./mvn-project-templ

Build tools are considered standard software engineering best practice. Besides this the complexities posed by the projects in the DL4J ecosystem make dependencies too difficult to manage manually. All the projects in the DL4J ecosystem can be used with other build tools like Gradle, SBT etc. More information on that can be found [here](https://deeplearning4j.konduit.ai/config/buildtools).

## Support

For help with the examples, please go to our [support forum](https://community.konduit.ai/)

Note for users of 1.0.0-beta7 and prior, some examples and modules have been removed to reflect
changes in the framework's direction. Please see and comment on our post [here](https://community.konduit.ai/t/upcoming-removal-of-modules-and-roadmap-changes/1240)

If you would like a workaround for something you may be missing,
please feel free to post on the forums, and we will do what we can to help you.


## Example Content
Projects are based on what functionality the included examples demonstrate to the user and not necessarily which library in the DL4J stack the functionality lives in.

Expand Down Expand Up @@ -69,9 +80,6 @@ This project contains a set of examples that demonstrate how raw data in various
- [nd4j-ndarray-examples](nd4j-ndarray-examples/README.md)
This project contains a set of examples that demonstrate how to manipulate NDArrays. The functionality of ND4J demonstrated here can be likened to NumPy.

- [arbiter-examples](arbiter-examples/README.md)
This project contains a set of examples that demonstrate usage of the Arbiter library for hyperparameter tuning of Deeplearning4J neural networks.

- [rl4j-examples](rl4j-examples/README.md)
This project contains examples of using RL4J, the reinforcement learning library in DL4J.

Expand Down
8 changes: 0 additions & 8 deletions android-examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@ The build configuration for Android can be found in [app/build.gradle](./app/bui

[Go back](../README.md) to the main repository page to explore other features/functionality of the **Eclipse Deeplearning4J** ecosystem. File an issue [here](https://github.com/eclipse/deeplearning4j-examples/issues) to request new features.

# Known Issues
Due to an unfortunately timed change, the 1.0.0-beta7 release doesn't work well with most Android devices. For this reason, this example project uses the SNAPSHOT version.

Again unfortunately, gradle has issues retrieving SNAPSHOT versions when combined with classifiers, which are used here to retrieve only the android specific backend dependencies.

If you want to run this example locally, you will therefore need to have Maven installed in order to download those specific dependencies.

Once you have installed Maven, you will need to run the following commands from the command line to download the correct android backend files:

```
mvn dependency:get -DremoteRepositories=snapshots::::https://oss.sonatype.org/content/repositories/snapshots -Dartifact=org.nd4j:nd4j-native:1.0.0-SNAPSHOT:jar:android-arm
Expand Down
21 changes: 11 additions & 10 deletions android-examples/app/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ android {
versionCode 1
versionName "1.0"

testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}

buildTypes {
Expand All @@ -27,10 +27,10 @@ android {
}

dependencies {
def dl4jVersion = '1.0.0-beta7'
def openblasVersion = '0.3.9-1.5.3'
def opencvVersion = '4.3.0-1.5.3'
def leptonicaVersion = '1.79.0-1.5.3'
def dl4jVersion = '1.0.0-M1.1'
def openblasVersion = '0.3.10-1.5.5'
def opencvVersion = '4.5.1-1.5.5'
def leptonicaVersion = '1.80.0-1.5.5'

implementation fileTree(dir: 'libs', include: ['*.jar'])

Expand All @@ -45,13 +45,14 @@ dependencies {
exclude group: 'org.bytedeco', module: 'leptonica-platform'
exclude group: 'org.bytedeco', module: 'hdf5-platform'
exclude group: 'org.nd4j', module: 'nd4j-base64'
exclude group: 'org.nd4j', module: 'nd4j-api'
}

implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-arm"
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-arm64"
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-x86"
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-x86_64"
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-arm"
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-arm64"
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-x86"
implementation group: 'org.nd4j', name: 'nd4j-native', version: dl4jVersion, classifier: "android-x86_64"
implementation group: 'org.bytedeco', name: 'openblas', version: openblasVersion
implementation group: 'org.bytedeco', name: 'openblas', version: openblasVersion, classifier: "android-arm"
implementation group: 'org.bytedeco', name: 'openblas', version: openblasVersion, classifier: "android-arm64"
Expand All @@ -69,4 +70,4 @@ dependencies {
implementation group: 'org.bytedeco', name: 'leptonica', version: leptonicaVersion, classifier: "android-x86_64"

annotationProcessor group: 'org.projectlombok', name: 'lombok', version: '1.18.4'
}
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.example.androidDl4jClassifier;
package com.example.androidclassifier;

import android.content.Context;

Expand Down
6 changes: 3 additions & 3 deletions android-examples/app/src/main/AndroidManifest.xml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.androidDl4jClassifier">
package="com.example.androidclassifier">

<application
android:allowBackup="true"
Expand All @@ -10,7 +10,7 @@
android:supportsRtl="true"
android:theme="@style/Theme.AppCompat.Light.DarkActionBar"
android:fullBackupContent="@xml/backup_descriptor">
<activity android:name="com.example.androidDl4jClassifier.MainActivity">
<activity android:name="com.example.org.deeplearning4j.examples.androidclassifier.MainActivity">
<intent-filter>
<action android:name="android.intent.action.MAIN" />

Expand All @@ -19,4 +19,4 @@
</activity>
</application>

</manifest>
</manifest>
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
package com.example.androidDl4jClassifier;
package org.deeplearning4j.examples.androidclassifier;

import androidx.appcompat.app.AppCompatActivity;

import android.os.Bundle;
import android.view.Menu;
import android.view.MenuItem;
import com.example.androidclassifier.R;


public class MainActivity extends AppCompatActivity {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.example.androidDl4jClassifier;
package org.deeplearning4j.examples.androidclassifier;

import android.content.Context;
import android.graphics.Canvas;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.example.androidDl4jClassifier;
package org.deeplearning4j.examples.androidclassifier;

import org.deeplearning4j.nn.conf.MultiLayerConfiguration;
import org.deeplearning4j.nn.conf.NeuralNetConfiguration;
Expand Down
6 changes: 3 additions & 3 deletions android-examples/app/src/main/res/layout/activity_main.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,16 @@
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
tools:context="org.deeplearning4j.examples.androidclassifier.MainActivity">


<view
android:id="@+id/id_scatterview"
class="com.example.androidDl4jClassifier.ScatterView"
class="org.deeplearning4j.examples.androidclassifier.ScatterView"
android:layout_width="fill_parent"
android:layout_height="0dp"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintHorizontal_bias="0.0"
app:layout_constraintStart_toStartOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>
</androidx.constraintlayout.widget.ConstraintLayout>

This file was deleted.

26 changes: 0 additions & 26 deletions cuda-specific-examples/README.md

This file was deleted.

Loading