Skip to content

panic during normal operation #1955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dooglus opened this issue Nov 1, 2015 · 7 comments
Closed

panic during normal operation #1955

dooglus opened this issue Nov 1, 2015 · 7 comments

Comments

@dooglus
Copy link

dooglus commented Nov 1, 2015

I ran 'geth console' for the first time in a couple of months. It was syncing nicely, but taking a long time. I left it syncing overnight.

When I woke up it had crashed:

I1031 22:31:46.461670   14813 chain_manager.go:661] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.692291ms. #394027 [9f77491e / 9f77491e]
I1031 22:31:46.832677   14813 chain_manager.go:661] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.243659ms. #394028 [303713cb / 303713cb]
I1031 22:35:48.788118   14813 chain_manager.go:661] imported 18 block(s) (0 queued 1 ignored) including 19 txs in 759.116517ms. #394186 [7f193310 / 89962d49]
I1031 22:36:04.148215   14813 chain_manager.go:661] imported 36 block(s) (0 queued 0 ignored) including 298 txs in 2.264619122s. #394222 [209583b1 / ab0bfbb5]
I1031 22:36:04.285215   14813 chain_manager.go:661] imported 2 block(s) (0 queued 0 ignored) including 2 txs in 136.95346ms. #394224 [3b5c05b0 / 2e1dde4f]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1d pc=0x8c89e5]

goroutine 60 [running]:
github.com/ethereum/go-ethereum/trie.CompactHexDecode(0xc20a5b8ec0, 0x1e, 0x1e, 0x0, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/encoding.go:67 +0xd5
github.com/ethereum/go-ethereum/trie.CompactDecode(0xc20a5b8ec0, 0x1e, 0x1e, 0x0, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/encoding.go:46 +0x66
github.com/ethereum/go-ethereum/trie.(*Trie).mknode(0xc20a83e680, 0xc20ec40050, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:339 +0x126
github.com/ethereum/go-ethereum/trie.(*Trie).trans(0xc20a83e680, 0x7fe38b301b10, 0xc20b4d2870, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:365 +0x10a
github.com/ethereum/go-ethereum/trie.(*FullNode).branch(0xc20cc057a0, 0x7fe38b301b03, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/fullnode.go:85 +0x96
github.com/ethereum/go-ethereum/trie.(*Trie).get(0xc20a83e680, 0x7fe38b301b68, 0xc20cc057a0, 0xc20d4de234, 0x3d, 0x3d, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:248 +0x134
github.com/ethereum/go-ethereum/trie.(*Trie).get(0xc20a83e680, 0x7fe38b301b68, 0xc20cc05680, 0xc20d4de233, 0x3e, 0x3e, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:248 +0x1c7
github.com/ethereum/go-ethereum/trie.(*Trie).get(0xc20a83e680, 0x7fe38b301b68, 0xc20b15dd40, 0xc20d4de232, 0x3f, 0x3f, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:248 +0x1c7
github.com/ethereum/go-ethereum/trie.(*Trie).get(0xc20a83e680, 0x7fe38b301b68, 0xc20cebd320, 0xc20d4de231, 0x40, 0x40, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:248 +0x1c7
github.com/ethereum/go-ethereum/trie.(*Trie).get(0xc20a83e680, 0x7fe38b301b68, 0xc20cebd680, 0xc20d4de230, 0x41, 0x41, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:248 +0x1c7
github.com/ethereum/go-ethereum/trie.(*Trie).Get(0xc20a83e680, 0xc20a5b85a0, 0x20, 0x20, 0x0, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/trie.go:154 +0x114
github.com/ethereum/go-ethereum/trie.(*SecureTrie).Get(0xc20a7d3378, 0xc20a5b8560, 0x14, 0x14, 0x0, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/trie/secure_trie.go:42 +0x11c
github.com/ethereum/go-ethereum/core/state.(*StateDB).GetStateObject(0xc20f375ea0, 0x1c9c39232539c145, 0x68680cb4ead1ee2, 0xc2c2fa89c1, 0x28)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/statedb.go:223 +0x191
github.com/ethereum/go-ethereum/core/state.(*StateDB).GetOrNewStateObject(0xc20f375ea0, 0x1c9c39232539c145, 0x68680cb4ead1ee2, 0xc2fa89c1, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state/statedb.go:240 +0x3d
github.com/ethereum/go-ethereum/core.(*StateTransition).From(0xc209c84d90, 0xc20b4d24b0, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_transition.go:120 +0xd1
github.com/ethereum/go-ethereum/core.(*StateTransition).preCheck(0xc209c84d90, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_transition.go:168 +0x56
github.com/ethereum/go-ethereum/core.(*StateTransition).transitionState(0xc209c84d90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_transition.go:190 +0x7e
github.com/ethereum/go-ethereum/core.ApplyMessage(0x7fe38114c758, 0xc20d4de1e0, 0x7fe38114c700, 0xc20820d320, 0x7fe38114a5e0, 0xc20f375f40, 0x0, 0x0, 0x0, 0xc92c60, ...)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/state_transition.go:98 +0x9e
github.com/ethereum/go-ethereum/core.(*BlockProcessor).ApplyTransaction(0xc2081771d0, 0x7fe38114a5e0, 0xc20f375f40, 0xc20f375ea0, 0xc21108f440, 0xc20820d320, 0xc20c433a80, 0xc20e30ca01, 0x52edd4, 0xcdaac0, ...)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/core/block_processor.go:96 +0x256
github.com/ethereum/go-ethereum/miner.(*Work).commitTransaction(0xc2092da4e0, 0xc20820d320, 0xc2081771d0, 0x0, 0x0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:622 +0xd4
github.com/ethereum/go-ethereum/miner.(*Work).commitTransactions(0xc2092da4e0, 0xc20c668000, 0x44a, 0x44a, 0xc20921e140, 0xc2081771d0)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:600 +0xbc2
github.com/ethereum/go-ethereum/miner.(*worker).commitNewWork(0xc209190a50)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:500 +0x8d2
github.com/ethereum/go-ethereum/miner.(*worker).update(0xc209190a50)
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:224 +0x364
created by github.com/ethereum/go-ethereum/miner.newWorker
    /home/chris/Programs/ether/go-ethereum/build/_workspace/src/github.com/ethereum/go-ethereum/miner/worker.go:140 +0x37c
$

It reports this version string:

I1101 10:13:37.109228   23385 cmd.go:125] Starting Geth/v1.1.2-58766921/linux/go1.4.2

I realise I'm running an old version and that this bug may already have been fixed, but thought I should report it anyway since I can't find mention of it in the issues list.
@obscuren
Copy link
Contributor

obscuren commented Nov 2, 2015

Do you recall a sudden shutdown (e.g. power loss) last time you ran geth?

@dooglus
Copy link
Author

dooglus commented Nov 2, 2015

I don't remember. It's unlikely, but possible. geth synced 80k blocks in about an hour before crashing:

at block: 310428 (Tue, 29 Sep 2015 20:05:20 PDT)
modules: admin:1.0 db:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 shh:1.0 txpool:1.0 web3:1.0
I1031 21:48:51.987584   14813 downloader.go:278] Block synchronisation started
I1031 21:49:17.453231   14813 chain_manager.go:661] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 1.117088253s. #310428 [739d1bf4 / 739d1bf4]
I1031 21:49:17.897170   14813 chain_manager.go:661] imported 2 block(s) (0 queued 0 ignored) including 0 txs in 81.490679ms. #310430 [771a8e6d / aa706533]
I1031 21:49:19.577437   14813 chain_manager.go:661] imported 4 block(s) (0 queued 0 ignored) including 1 txs in 224.683708ms. #310434 [3e5f202e / a8e57641]
I1031 21:49:21.519313   14813 chain_manager.go:661] imported 8 block(s) (0 queued 0 ignored) including 11 txs in 529.095538ms. #310442 [c5a43d12 / c2f61a24]
[...]
I1031 22:31:46.461670   14813 chain_manager.go:661] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.692291ms. #394027 [9f77491e / 9f77491e]
I1031 22:31:46.832677   14813 chain_manager.go:661] imported 0 block(s) (0 queued 1 ignored) including 0 txs in 3.243659ms. #394028 [303713cb / 303713cb]
I1031 22:35:48.788118   14813 chain_manager.go:661] imported 18 block(s) (0 queued 1 ignored) including 19 txs in 759.116517ms. #394186 [7f193310 / 89962d49]
I1031 22:36:04.148215   14813 chain_manager.go:661] imported 36 block(s) (0 queued 0 ignored) including 298 txs in 2.264619122s. #394222 [209583b1 / ab0bfbb5]
I1031 22:36:04.285215   14813 chain_manager.go:661] imported 2 block(s) (0 queued 0 ignored) including 2 txs in 136.95346ms. #394224 [3b5c05b0 / 2e1dde4f]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1d pc=0x8c89e5]

@donSchoe
Copy link

donSchoe commented Nov 5, 2015

i had the same panic, comment here: #1962 (comment) (related tickets?)

@karalabe
Copy link
Member

karalabe commented Nov 5, 2015

@donSchoe The two (Actually three) issues are unrelated, please open a new issue for yours.

@karalabe
Copy link
Member

karalabe commented Nov 5, 2015

@dooglus @obscuren I've been trying to wrap my head around this one (line producing the crash 565d9f2#diff-1549393320d640c34a2a9e27e4194adfL60), but I don't really see a possibility for the crashing code to produce the panic on itself. My bet is that there was a deeper data race in there, and while the hex decoding was running another process swapped out the underlying slice. That would explain where the nil dereference comes from. As @fjl quite significantly changed the trie since then, I'm not sure this issue is relevant any more, but perhaps @fjl can check if there was a possible data race in the old code and if that is still present or not in the new version.

@fjl
Copy link
Contributor

fjl commented Nov 5, 2015

The older trie code didn't handle invalid nodes very well. I'd rule out a race issue because the older trie also had
a mutex wrapped around every operation.

@fjl
Copy link
Contributor

fjl commented Jan 9, 2016

I'll close his for now since the new code is very different and this issue hasn't popped up again.

@fjl fjl closed this as completed Jan 9, 2016
maoueh pushed a commit to streamingfast/go-ethereum that referenced this issue Nov 29, 2023
@joey0612
eth, trie/triedb/pathdb: pbss patches (ethereum#1955)
53559fc 
* fix: use the top root hash for rewinding under path schema

* feat: add async flush nodebuffer in path schema

* chore: add prun-block param suffix check

* fix: code review comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fjl @karalabe @dooglus @donSchoe @obscuren