submodule git clone issue

[fedora@vpn-builder tmp]$ sh build_app.sh 
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338'...
remote: Enumerating objects: 263, done.
remote: Counting objects: 100% (263/263), done.
remote: Compressing objects: 100% (126/126), done.
remote: Total 4407 (delta 124), reused 204 (delta 98), pack-reused 4144
Receiving objects: 100% (4407/4407), 13.75 MiB | 13.97 MiB/s, done.
Resolving deltas: 100% (2550/2550), done.
Note: switching to '63fbab75b075f42e2c78bb9db81c6deaf0d8e5f9'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

Submodule 'ics-openvpn' (https://github.com/eduvpn/ics-openvpn.git) registered for path 'ics-openvpn'
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn'...
remote: Enumerating objects: 7, done.        
remote: Counting objects: 100% (7/7), done.        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 24352 (delta 1), reused 2 (delta 1), pack-reused 24345        
Receiving objects: 100% (24352/24352), 49.76 MiB | 22.93 MiB/s, done.
Resolving deltas: 100% (15028/15028), done.
Submodule path 'ics-openvpn': checked out 'fe6bfa5f48fefdc5b9bf7be12937c3dbdfdc6e66'
Submodule 'main/src/main/cpp/asio' (https://github.com/chriskohlhoff/asio) registered for path 'ics-openvpn/main/src/main/cpp/asio'
Submodule 'main/src/main/cpp/lz4' (https://github.com/lz4/lz4.git) registered for path 'ics-openvpn/main/src/main/cpp/lz4'
Submodule 'main/src/main/cpp/mbedtls' (https://github.com/ARMmbed/mbedtls.git) registered for path 'ics-openvpn/main/src/main/cpp/mbedtls'
Submodule 'main/openssl' (https://github.com/schwabe/platform_external_openssl.git) registered for path 'ics-openvpn/main/src/main/cpp/openssl'
Submodule 'main/openvpn' (https://github.com/schwabe/openvpn.git) registered for path 'ics-openvpn/main/src/main/cpp/openvpn'
Submodule 'main/src/main/cpp/openvpn3' (https://github.com/schwabe/openvpn3.git) registered for path 'ics-openvpn/main/src/main/cpp/openvpn3'
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn/main/src/main/cpp/asio'...
remote: Enumerating objects: 48807, done.        
remote: Total 48807 (delta 0), reused 0 (delta 0), pack-reused 48807        
Receiving objects: 100% (48807/48807), 16.33 MiB | 12.51 MiB/s, done.
Resolving deltas: 100% (33357/33357), done.
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn/main/src/main/cpp/lz4'...
remote: Enumerating objects: 11352, done.        
remote: Total 11352 (delta 0), reused 0 (delta 0), pack-reused 11352        
Receiving objects: 100% (11352/11352), 4.96 MiB | 6.73 MiB/s, done.
Resolving deltas: 100% (7944/7944), done.
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn/main/src/main/cpp/mbedtls'...
remote: Enumerating objects: 47, done.        
remote: Counting objects: 100% (47/47), done.        
remote: Compressing objects: 100% (43/43), done.        
remote: Total 96379 (delta 16), reused 26 (delta 4), pack-reused 96332        
Receiving objects: 100% (96379/96379), 42.54 MiB | 18.12 MiB/s, done.
Resolving deltas: 100% (74505/74505), done.
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn/main/src/main/cpp/openssl'...
remote: Enumerating objects: 232, done.        
remote: Counting objects: 100% (232/232), done.        
remote: Compressing objects: 100% (225/225), done.        
remote: Total 13736 (delta 34), reused 18 (delta 5), pack-reused 13504        
Receiving objects: 100% (13736/13736), 18.81 MiB | 13.50 MiB/s, done.
Resolving deltas: 100% (9278/9278), done.
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn/main/src/main/cpp/openvpn'...
remote: Enumerating objects: 1453, done.        
remote: Counting objects: 100% (1453/1453), done.        
remote: Compressing objects: 100% (515/515), done.        
remote: Total 26712 (delta 1163), reused 1147 (delta 937), pack-reused 25259        
Receiving objects: 100% (26712/26712), 14.17 MiB | 14.55 MiB/s, done.
Resolving deltas: 100% (21653/21653), done.
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn/main/src/main/cpp/openvpn3'...
remote: Enumerating objects: 22853, done.        
remote: Total 22853 (delta 0), reused 0 (delta 0), pack-reused 22853        
Receiving objects: 100% (22853/22853), 5.26 MiB | 7.01 MiB/s, done.
Resolving deltas: 100% (16636/16636), done.
Submodule path 'ics-openvpn/main/src/main/cpp/asio': checked out '90f32660cd503494b3707840cfbd5434d8e9dabe'
Submodule path 'ics-openvpn/main/src/main/cpp/lz4': checked out 'dfed9fa1d77f0434306d377c4da1f7191d3ba08a'
Submodule path 'ics-openvpn/main/src/main/cpp/mbedtls': checked out 'ca933c7e0c9e84738b168b6b0feb89af4183a60a'
Submodule 'crypto' (https://github.com/ARMmbed/mbed-crypto) registered for path 'ics-openvpn/main/src/main/cpp/mbedtls/crypto'
Cloning into '/home/fedora/Projects/eduvpn-android-20200204090338/ics-openvpn/main/src/main/cpp/mbedtls/crypto'...
remote: Enumerating objects: 271, done.        
remote: Counting objects: 100% (271/271), done.        
remote: Compressing objects: 100% (118/118), done.        
remote: Total 80977 (delta 185), reused 221 (delta 153), pack-reused 80706        
Receiving objects: 100% (80977/80977), 23.30 MiB | 17.56 MiB/s, done.
Resolving deltas: 100% (65437/65437), done.
Submodule path 'ics-openvpn/main/src/main/cpp/mbedtls/crypto': checked out 'e78cd62acb7ea0397f061ebc754b89ed6222b13c'
Submodule path 'ics-openvpn/main/src/main/cpp/openssl': checked out '3d84d35cb7cb0775fbed54ffb4eaef38b0eba7af'
Submodule path 'ics-openvpn/main/src/main/cpp/openvpn': checked out '5493c5fa82dd300aae89f228ff7ba8144b54c687'
error: Server does not allow request for unadvertised object 636e3d238f12d0aae6ac567ea112f4726335c8cc
Fetched in submodule path 'ics-openvpn/main/src/main/cpp/openvpn3', but it did not contain 636e3d238f12d0aae6ac567ea112f4726335c8cc. Direct fetching of that commit failed.
Failed to recurse into submodule path 'ics-openvpn'
[fedora@vpn-builder tmp]$ 

I'm not sure what is going on here, or why it breaks. It seems all commits actually exists in the various repositories...

Possibly related #223

Maybe an update of ics-openvpn from schwabe repository helps... there seem to have been a few new releases since the Android app was updated last year. Maybe someone force pushed? Or the submodules references were not properly updated in the eduvpn/ics-openvpn fork?

[dzolnai]

Perhaps fetching by commit ID is disabled on that repo for some reason (first I've seen something like this, but it seems to be possible), and until a month or two ago there was a branch or tag pointing to that commit which allowed us to fetch it, but that got removed.

Perhaps fetching by commit ID is disabled on that repo for some reason (first I've seen something like this, but it seems to be possible), and until a month or two ago there was a branch or tag pointing to that commit which allowed us to fetch it, but that got removed.

That's horrible if true, but it is also our fault. So let's make sure this can never happen again. For a new release we do a full/recursive checkout and put it in a tarball, sign it with PGP/minisign and put it here as a release download...

I found a checkout of the 1.3.0 release on my build VM, so I removed most of the binary/git crap and put it in a tarball. See https://github.com/eduvpn/android/releases/tag/1.3.0

So it would be great to have a release script that creates this tarball from a clean checkout with something like git archive... unfortunately, or I should probably say, OF COURSE, git archive doesn't include the submodules so we have to find a way around that I guess. Or just manually delete all the .git folders. What a shit.

[dzolnai]

That's horrible if true, but it is also our fault

It was just an idea, I'm not sure at all if that is the exact cause.
I think you are making it too complicated, since just updating the ics-openvpn to the latest release should probably solve this.

I think not, we clearly do not have a handle on our dependencies. If they disappear we have a big problem... Or is there another way to solve this?

[dzolnai]

Not really, I only think that these precautions are unnecessary.
If we are doing this, then we should also archive all Android libraries we use, Android Studio and its build tools, NDK, ... :)

Yeah, we should. A pile of semi proprietary software + a ridiculous amount of libraries where nobody has any overview... What could go wrong.

So making our app not break because of crappy submodules is the absolute minimum we can do and a start, not an endpoint.

but master is fixed now and should build?

[dzolnai]

This should fix it: #229

[dzolnai]

Master builds now in Buildozer

Yes fixed! Thanks! 👍