restart coordinator only node will get error: "data/ENGINE: no such file or directory"

[david187]

Start command:

sudo -u arangodb arangodb --starter.join=A --starter.join=B --starter.join=C --server.arangod=/usr/sbin/arangod --cluster.start-dbserver=false --cluster.start-coordinator=true --auth.jwt-secret=/arangodb-coordinator/secret.jwt --starter.mode=cluster --cluster.agency-size=3 --starter.disable-ipv6=true --starter.data-dir=/arangodb-coordinator/ --server.js-dir=/usr/share/arangodb3/js

First time:

2020-04-22T11:52:24+08:00 |INFO| Starting arangodb version 0.14.14, build 598e7d7 component=arangodb
2020-04-22T11:52:24+08:00 |INFO| ArangoDB Starter listening on 0.0.0.0:8528 (:8528) component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Found ID from localhost peer component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Found ID 'd2a1453b' from peer, looking for 'a16384ae' component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Contacting master http://A:8528... component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Waiting for 3 servers to show up... component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Serving as slave with ID 'a16384ae' on :8528... component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Using storage engine 'rocksdb' component=arangodb
2020-04-22T11:52:24+08:00 |INFO| ArangoDB Starter listening on 0.0.0.0:8528 (:8528) component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Looking for a running instance of coordinator on port 8529 component=arangodb
2020-04-22T11:52:24+08:00 |INFO| Starting coordinator on port 8529 component=arangodb
2020-04-22T11:52:26+08:00 |INFO| coordinator up and running (version 3.6.2). component=arangodb
2020-04-22T11:52:26+08:00 |INFO| Your cluster can now be accessed with a browser at `http://192.168.128.230:8529` or component=arangodb
2020-04-22T11:52:26+08:00 |INFO| using `arangosh --server.endpoint tcp://192.168.128.230:8529`. component=arangodb
^CReceived signal: interrupt 
2020-04-22T11:53:11+08:00 |INFO| Shutting down services... component=arangodb
2020-04-22T11:53:11+08:00 |INFO| coordinator has terminated component=arangodb
2020-04-22T11:53:11+08:00 |INFO| Wait on 27793 returned component=arangodb exit-status=1 trap-cause=-1

--
Second time:

2020-04-22T11:50:33+08:00 |INFO| Starting arangodb version 0.14.14, build 598e7d7 component=arangodb
2020-04-22T11:50:33+08:00 |FATA| Failed to read JWT secret file '/docker_data/arangodb-coordinator/secret.jwt' component=arangodb error="open /docker_data/arangodb-coordinator/secret.jwt: no such file or directory"

And need to use RECOVERY to restart starter,

rm /arangodb-coordinator/setup.json 
echo "[IP]:8528" > /arangodb-coordinator/RECOVERY

========
However, if I run the command inside coordinator8529/arangod_command.txt, it is ok.

sudo -u arangod /usr/sbin/arangod [......]

2020-04-22T03:54:05Z [28957] INFO [e52b0] ArangoDB 3.6.2 [linux] 64bit, using jemalloc, build tags/v3.6.2-0-g7c2e5d3654, VPack 0.1.33, RocksDB 6.2.0, ICU 58.1, V8 7.1.302.28, OpenSSL 1.1.1d  10 Sep 2019
2020-04-22T03:54:05Z [28957] INFO [75ddc] detected operating system: Linux version 4.15.0-51-generic (buildd@lgw01-amd64-031) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) #55~16.04.1-Ubuntu SMP Thu May 16 09:24:37 UTC 2019
2020-04-22T03:54:05Z [28957] INFO [4a3fc] using storage engine rocksdb
2020-04-22T03:54:05Z [28957] INFO [3bb7d] {cluster} Starting up with role COORDINATOR
2020-04-22T03:54:05Z [28957] INFO [d4db4] {cluster} Waiting for DBservers to show up...
2020-04-22T03:54:05Z [28957] INFO [22f55] {cluster} Found 3 DBservers.
2020-04-22T03:54:05Z [28957] INFO [6ea38] using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2020-04-22T03:54:05Z [28957] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 655350, soft limit is 655350
2020-04-22T03:54:05Z [28957] INFO [3844e] {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2020-04-22T03:54:05Z [28957] INFO [b6826] {cluster} Cluster feature is turned on. Agency version: {"server":"arango","license":"community","version":"3.6.2"}, Agency endpoints: http+tcp://A:8531, http+tcp://B:8531, http+tcp://C:8531, server id: 'CRDN-322bacc8-23b5-4732-a6d3-be446bf156ac', internal endpoint / address: tcp://A:8529', advertised endpoint: , role: COORDINATOR
2020-04-22T03:54:05Z [28957] INFO [805b2] {cluster} using heartbeat interval value '1000 ms' from agency
2020-04-22T03:54:06Z [28957] INFO [365ec] bootstrapped coordinator CRDN-322bacc8-23b5-4732-a6d3-be446bf156ac
2020-04-22T03:54:06Z [28957] INFO [cf3f4] ArangoDB (version 3.6.2 [linux]) is ready for business. Have fun!
^C2020-04-22T03:54:39Z [28957] INFO [b4133] control-c received, beginning shut down sequence
2020-04-22T03:54:40Z [28957] INFO [4bcb9] ArangoDB has been shut down

==========
OS: Ubuntu 16.04.6 LTS

arangodb --version
Version 0.14.14, build 598e7d7, Go go1.13.6

arangod --version
3.6.2

architecture: 64bit
arm: false
asan: false
asm-crc32: true
assertions: false
avx2: false
boost-version: 1.71.0
build-date: 2020-03-05 22:08:20
build-repository: tags/v3.6.2-0-g7c2e5d3654
compiler: gcc [9.2.0]
cplusplus: 201703
curl-version: libcurl/7.66.0 OpenSSL/1.1.1d
debug: false
endianness: little
failure-tests: false
fd-client-event-handler: poll
fd-setsize: 1024
full-version-string: ArangoDB 3.6.2 [linux] 64bit, using jemalloc, build tags/v3.6.2-0-g7c2e5d3654, VPack 0.1.33, RocksDB 6.2.0, ICU 58.1, V8 7.1.302.28, OpenSSL 1.1.1d  10 Sep 2019
icu-version: 58.1
ipo: true
iresearch-version: 1.0.0.0
jemalloc: true
license: community
maintainer-mode: false
ndebug: true
openssl-version-compile-time: OpenSSL 1.1.1d  10 Sep 2019
openssl-version-run-time: OpenSSL 1.1.1d  10 Sep 2019
optimization-flags: -march=nehalem -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mno-sse4a -mno-avx -mno-fma -mno-bmi2 -mno-avx2 -mno-xop -mno-fma4 -mno-avx512f -mno-avx512vl -mno-avx512pf -mno-avx512er -mno-avx512cd -mno-avx512dq -mno-avx512bw -mno-avx512ifma -mno-avx512vbmi
pic: 2
pie: 2
platform: linux
reactor-type: epoll
rocksdb-version: 6.2.0
server-version: 3.6.2
sizeof int: 4
sizeof long: 8
sizeof void*: 8
sse42: true
tsan: false
unaligned-access: true
v8-version: 7.1.302.28
vpack-version: 0.1.33
zlib-version: 1.2.11

==========

cat /arangodb-coordinator/coordinator8529/arangod_command.txt 
/usr/sbin/arangod \
-c \
/arangodb-coordinator/coordinator8529/arangod.conf \
--database.directory \
/arangodb-coordinator/coordinator8529/data \
--javascript.startup-directory \
/usr/share/arangodb3/js \
--javascript.app-path \
/arangodb-coordinator/coordinator8529/apps \
--log.file \
/arangodb-coordinator/coordinator8529/arangod.log \
--log.force-direct \
false \
--server.jwt-secret-keyfile \
/arangodb-coordinator/coordinator8529/arangod.jwtsecret \
--javascript.copy-installation \
true \
--cluster.my-address \
tcp://[ip]:8529 \
--cluster.my-role \
COORDINATOR \
--foxx.queues \
true \
--server.statistics \
true \
--cluster.agency-endpoint \
tcp://A:8531 \
--cluster.agency-endpoint \
tcp://B:8531 \
--cluster.agency-endpoint \
tcp://C:8531