tx failed Transaction w/ invalid nonce

[ethers]

develop f047699

Run Geth, no mining, around 12 peers.

I0410 14:57:05.653362 4006 worker.go:280] TX (08421e81) failed, will be removed: Transaction w/ invalid nonce (804 / 805)
I0410 14:57:05.657377 4006 worker.go:280] TX (08421e81) failed, will be removed: Transaction w/ invalid nonce (804 / 805)

Here's the log:

I0410 14:57:05.254615    4006 worker.go:176] Hash rate: 0 Khash
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] alarm: block request time
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] got parent head block hash 9acbddb0...checking
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] blockHashesRequest: parentHash known ... inserting section in blockchain
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] check if new blocks arrived (attempt 1): missing 0/1/2
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] section checked: got all blocks. process complete (1 total blocksRequests): missing 0/1/2
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] section process terminated: 2 blocks retrieved (1 attempts), hash requests complete on root (1 attempts).
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] insert 2 blocks [0aec8e79/6c4f698c] into blockchain
I0410 14:57:05.642936    4006 state_object.go:238] 00c5496af02445bd74e10299513915755b7e4a88: gas (+ 3141592)
I0410 14:57:05.643605    4006 state_object.go:194] 00c5496af02445bd74e10299513915755b7e4a88: #0 12026530835000000000000 (+ 1500000000000000000)
I0410 14:57:05.644471    4006 chain_manager.go:526] inserted block #70885 (0 TXs 0 UNCs) (0aec8e79...)
I0410 14:57:05.650674    4006 state_object.go:238] ebf35f929443f141c3651f8e231fac31a347a7ce: gas (+ 3141592)
I0410 14:57:05.651100    4006 state_object.go:194] ebf35f929443f141c3651f8e231fac31a347a7ce: #0 37709843810000000000000 (+ 1500000000000000000)
I0410 14:57:05.651477    4006 chain_manager.go:526] inserted block #70886 (0 TXs 0 UNCs) (6c4f698c...)
I0410 14:57:05.651520    4006 chain_manager.go:544] imported 2 block(s) 0 queued in 14.701805ms. #70886 [0aec8e79 / 6c4f698c]
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] add complete child section [e3afc46d-e3afc46d] to the blockchain
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] section completely inserted to blockchain - remove
2015/04/10 14:57:05 [Blockpool] checking td
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] remove: suicide
2015/04/10 14:57:05 [Blockpool] [0aec8e79-6c4f698c] removed section.
2015/04/10 14:57:05 [Blockpool] peer td 122119565374 =?= block td 118274117960
2015/04/10 14:57:05 [Blockpool] [e3afc46d-e3afc46d] insert 1 blocks [e3afc46d/e3afc46d] into blockchain
2015/04/10 14:57:05 [Blockpool] ChainHeadEvent: height: 70886, td: 118274117960, hash: 6c4f698c
I0410 14:57:05.651975    4006 state_object.go:238] 0000000000000000000000000000000000000000: gas (+ 3141592)
I0410 14:57:05.653362    4006 worker.go:280] TX (08421e81) failed, will be removed: Transaction w/ invalid nonce (804 / 805)
I0410 14:57:05.653402    4006 worker.go:281] 
    TX(08421e819c4b7363734d2f206ffc1df6ffee28b8630d1311f0aa46811b0211f3)
    Contract: false
    From:     cd2a3d9f938e13cd947ec05abc7fe734df8dd826
    To:       2e9bd804a61255b2cc7106f915ed59af5fbf63cd
    Nonce:    804
    GasPrice: 10000000000000
    GasLimit  3000000
    Value:    0
    Data:     0xd2d2b0b700000000000000000000000000000000000000000000000000000000000001900000000000000000000000000000000000000000000000000000000000000005010000009120fab3b9b6f5e7a07da7b0d76623ad1a69515d53ce03858fb4e0b000000000ed92b223e586e351e4908769d6f6eb83f00e63ce7993af8caa1ec413e514c6dfa8418149ffff001d4ed84a0501000000e4f883e964f18726e73d23f95fa77e386add046d7bb1fceed0870699000000008f34d3825416af95ac1b04431ac8f7d3c06e69b49af7f70215077f95cdb93faefc418149ffff001d304770c901000000a4ee61584255c2714d972fb7c0989a48e574ed93e126053b1589bf3c00000000dce2116f1ef4a2c27ef7f3440e0ea43d2aaef66d775e5b3af644f5d93a60168792468149ffff001d060ade4d010000008e315d1658b891473f56f9ebddced3c967178a54dbe95bcbc990fd310000000077f10e3ccae1954f08f427bbdce88f06d651ad69ac363aaedb4b6b798268758ed9488149ffff001d192fb26701000000da1ad59c84fd846aa962339a9d081eb1ddf9417c3082138002614a37000000007bc96a5e198c8f403b220c6dee96bab91362a858c8a8045e2aa1bb3faf04c7d4514a8149ffff001d1456bb92
    V:        0x1b
    R:        0x5b86ef84e87e196a554cb6f90a7f4b6e0680ef142f8b458ab1b8ec09ea592904
    S:        0x598cfa5815fae4902406d87a1a5a244520590dfbb24be25467271de22a1d567e
    Hex:      f9023e8203248609184e72a000832dc6c0942e9bd804a61255b2cc7106f915ed59af5fbf63cd80b901d4d2d2b0b700000000000000000000000000000000000000000000000000000000000001900000000000000000000000000000000000000000000000000000000000000005010000009120fab3b9b6f5e7a07da7b0d76623ad1a69515d53ce03858fb4e0b000000000ed92b223e586e351e4908769d6f6eb83f00e63ce7993af8caa1ec413e514c6dfa8418149ffff001d4ed84a0501000000e4f883e964f18726e73d23f95fa77e386add046d7bb1fceed0870699000000008f34d3825416af95ac1b04431ac8f7d3c06e69b49af7f70215077f95cdb93faefc418149ffff001d304770c901000000a4ee61584255c2714d972fb7c0989a48e574ed93e126053b1589bf3c00000000dce2116f1ef4a2c27ef7f3440e0ea43d2aaef66d775e5b3af644f5d93a60168792468149ffff001d060ade4d010000008e315d1658b891473f56f9ebddced3c967178a54dbe95bcbc990fd310000000077f10e3ccae1954f08f427bbdce88f06d651ad69ac363aaedb4b6b798268758ed9488149ffff001d192fb26701000000da1ad59c84fd846aa962339a9d081eb1ddf9417c3082138002614a37000000007bc96a5e198c8f403b220c6dee96bab91362a858c8a8045e2aa1bb3faf04c7d4514a8149ffff001d1456bb921ba05b86ef84e87e196a554cb6f90a7f4b6e0680ef142f8b458ab1b8ec09ea592904a0598cfa5815fae4902406d87a1a5a244520590dfbb24be25467271de22a1d567e

I0410 14:57:05.653975    4006 state_object.go:194] 0000000000000000000000000000000000000000: #0 1000000000000000000000024500000000000000000 (+ 1500000000000000000)
I0410 14:57:05.655333    4006 state_object.go:238] 00c5496af02445bd74e10299513915755b7e4a88: gas (+ 3141592)
I0410 14:57:05.655562    4006 state_object.go:194] 00c5496af02445bd74e10299513915755b7e4a88: #0 12028030835000000000000 (+ 1500000000000000000)
I0410 14:57:05.655892    4006 chain_manager.go:526] inserted block #70887 (0 TXs 0 UNCs) (e3afc46d...)
I0410 14:57:05.655942    4006 chain_manager.go:544] imported 1 block(s) 0 queued in 4.283522ms. #70887 [e3afc46d / e3afc46d]
2015/04/10 14:57:05 [Blockpool] [e3afc46d-e3afc46d] no child section in pool
2015/04/10 14:57:05 [Blockpool] [e3afc46d-e3afc46d] section completely inserted to blockchain - remove
2015/04/10 14:57:05 [Blockpool] checking td
2015/04/10 14:57:05 [Blockpool] [e3afc46d-e3afc46d] remove: suicide
2015/04/10 14:57:05 [Blockpool] [e3afc46d-e3afc46d] removed section.
2015/04/10 14:57:05 [Blockpool] peer td 122125847072 =?= block td 118280399658
2015/04/10 14:57:05 [Blockpool] ChainHeadEvent: height: 70887, td: 118280399658, hash: e3afc46d
I0410 14:57:05.656531    4006 state_object.go:238] 0000000000000000000000000000000000000000: gas (+ 3141592)
I0410 14:57:05.657377    4006 worker.go:280] TX (08421e81) failed, will be removed: Transaction w/ invalid nonce (804 / 805)
I0410 14:57:05.657392    4006 worker.go:281] 
    TX(08421e819c4b7363734d2f206ffc1df6ffee28b8630d1311f0aa46811b0211f3)
    Contract: false
    From:     cd2a3d9f938e13cd947ec05abc7fe734df8dd826
    To:       2e9bd804a61255b2cc7106f915ed59af5fbf63cd
    Nonce:    804
    GasPrice: 10000000000000
    GasLimit  3000000
    Value:    0
    Data:     0xd2d2b0b700000000000000000000000000000000000000000000000000000000000001900000000000000000000000000000000000000000000000000000000000000005010000009120fab3b9b6f5e7a07da7b0d76623ad1a69515d53ce03858fb4e0b000000000ed92b223e586e351e4908769d6f6eb83f00e63ce7993af8caa1ec413e514c6dfa8418149ffff001d4ed84a0501000000e4f883e964f18726e73d23f95fa77e386add046d7bb1fceed0870699000000008f34d3825416af95ac1b04431ac8f7d3c06e69b49af7f70215077f95cdb93faefc418149ffff001d304770c901000000a4ee61584255c2714d972fb7c0989a48e574ed93e126053b1589bf3c00000000dce2116f1ef4a2c27ef7f3440e0ea43d2aaef66d775e5b3af644f5d93a60168792468149ffff001d060ade4d010000008e315d1658b891473f56f9ebddced3c967178a54dbe95bcbc990fd310000000077f10e3ccae1954f08f427bbdce88f06d651ad69ac363aaedb4b6b798268758ed9488149ffff001d192fb26701000000da1ad59c84fd846aa962339a9d081eb1ddf9417c3082138002614a37000000007bc96a5e198c8f403b220c6dee96bab91362a858c8a8045e2aa1bb3faf04c7d4514a8149ffff001d1456bb92
    V:        0x1b
    R:        0x5b86ef84e87e196a554cb6f90a7f4b6e0680ef142f8b458ab1b8ec09ea592904
    S:        0x598cfa5815fae4902406d87a1a5a244520590dfbb24be25467271de22a1d567e
    Hex:      f9023e8203248609184e72a000832dc6c0942e9bd804a61255b2cc7106f915ed59af5fbf63cd80b901d4d2d2b0b700000000000000000000000000000000000000000000000000000000000001900000000000000000000000000000000000000000000000000000000000000005010000009120fab3b9b6f5e7a07da7b0d76623ad1a69515d53ce03858fb4e0b000000000ed92b223e586e351e4908769d6f6eb83f00e63ce7993af8caa1ec413e514c6dfa8418149ffff001d4ed84a0501000000e4f883e964f18726e73d23f95fa77e386add046d7bb1fceed0870699000000008f34d3825416af95ac1b04431ac8f7d3c06e69b49af7f70215077f95cdb93faefc418149ffff001d304770c901000000a4ee61584255c2714d972fb7c0989a48e574ed93e126053b1589bf3c00000000dce2116f1ef4a2c27ef7f3440e0ea43d2aaef66d775e5b3af644f5d93a60168792468149ffff001d060ade4d010000008e315d1658b891473f56f9ebddced3c967178a54dbe95bcbc990fd310000000077f10e3ccae1954f08f427bbdce88f06d651ad69ac363aaedb4b6b798268758ed9488149ffff001d192fb26701000000da1ad59c84fd846aa962339a9d081eb1ddf9417c3082138002614a37000000007bc96a5e198c8f403b220c6dee96bab91362a858c8a8045e2aa1bb3faf04c7d4514a8149ffff001d1456bb921ba05b86ef84e87e196a554cb6f90a7f4b6e0680ef142f8b458ab1b8ec09ea592904a0598cfa5815fae4902406d87a1a5a244520590dfbb24be25467271de22a1d567e

I0410 14:57:05.657735    4006 state_object.go:194] 0000000000000000000000000000000000000000: #0 1000000000000000000000024500000000000000000 (+ 1500000000000000000)
I0410 14:57:05.820589    4006 server.go:369] Dialing enode://bb0bed9a93ee35a664951676522544d48edce9c298bd69dcad5fb15f62896bc57d0f0a945284677350df4aae3c154ad06e519fb291262ed5e90962a1e00e5d62@85.250.225.246:30303
I0410 14:57:05.820653    4006 server.go:369] Dialing enode://89f8f3005435614b3999cf23ec8869b5e08ce3b59a88c6b051694ebfca6776d5cc4667314e5d4193d09fc0699724e7d06b52c655d2f815a461b0a08b2f63208c@69.181.107.36:30303
I0410 14:57:05.820653    4006 server.go:369] Dialing enode://ac231e135f35ddf0b8801d0e617001ffe68abb5a2a173059cc1b3eec3c10f80d0c69ae3da22bc7336539fee93c6d44779979fe8abeac866d9bf770227eba77f9@162.213.238.219:30305
I0410 14:57:05.820674    4006 server.go:369] Dialing enode://ba59f1c35d74ed1fe1019f48ab4222d7017390accf16bd5973d6fa1d490121ddbe67f26782c481d50b46d142e083f9d2aeabe40cf3fad3a3dcb0bb8f3ff17672@69.181.107.36:30304?discport=1025
I0410 14:57:05.820588    4006 server.go:369] Dialing enode://879ab550be8fa7be3d95eea877645d7d3939c5e7fd6b78f19cda9926feaa476f2b91233faa854069d2aba89d3bba029da8c0565f1c5006d858489bb2aa6b4b19@71.121.194.171:30303
I0410 14:57:05.820590    4006 server.go:369] Dialing enode://bf1d38bbe00f43d37f8ea51a9fb3652fbc6207b6f58f900a49b00cca6ea0798a3caf88edf1e723403a70792b695d4f6a85c136a63fb1d9c12caeb587496534c7@92.51.165.126:30303

[ethers]

looks like #519 and #317 is back

[obscuren]

More info. How did this occur. Give me a minimal test case

[ethers]

This happened by creating a number of transactions in a script.
This script can be used:
https://github.com/ethers/dapp-bin/blob/275fc1dac5b1030033e31c76266bc77f79bb1c78/btcrelay/misc/693.sh
(Increase the number of iterations, and change the 'from' as needed.)

The above script was run (accidentally omitted https://github.com/ethers/dapp-bin/blob/275fc1dac5b1030033e31c76266bc77f79bb1c78/btcrelay/misc/693.sh#L19 ) and here's some of the log:

I0410 17:01:46.137886    8480 worker.go:280] TX (a606e29d) failed, will be removed: Transaction w/ invalid nonce (996 / 817)
I0410 17:01:46.137886    8480 chain_manager.go:526] inserted block #71226 (0 TXs 1 UNCs) (ef68c06a...)
I0410 17:01:46.137945    8480 worker.go:281] 
    TX(a606e29d61d7fd188967df4e051bb1cfc0c889f692fa2f68c37fd215ed2d3a89)
    Contract: false
    From:     cd2a3d9f938e13cd947ec05abc7fe734df8dd826
    To:       57b723e224954a3026c1fb44c5f07b4b5fc096a8
    Nonce:    996
    GasPrice: 10000000000000
    GasLimit  30400
    Value:    1
    Data:     0x
    V:        0x1b
    R:        0x7a730a541fcf25ed06dd937b3a534f6e503ea5766d83ca7de9e8bd551b5ac05b
    S:        0x10d7e0c77e6f0aa62b115c977ab04c1156a0df7fd98ba0c20e1ebdaf8104f61e
    Hex:      f8678203e48609184e72a0008276c09457b723e224954a3026c1fb44c5f07b4b5fc096a801801ba07a730a541fcf25ed06dd937b3a534f6e503ea5766d83ca7de9e8bd551b5ac05ba010d7e0c77e6f0aa62b115c977ab04c1156a0df7fd98ba0c20e1ebdaf8104f61e

I0410 17:01:46.140015    8480 worker.go:280] TX (8889f4f0) failed, will be removed: Transaction w/ invalid nonce (997 / 817)
I0410 17:01:46.140041    8480 worker.go:281] 
    TX(8889f4f013223baf20ffb78071ec41e6ca5971cbfd0cc2055ca000aa19025755)
    Contract: false
    From:     cd2a3d9f938e13cd947ec05abc7fe734df8dd826
    To:       57b723e224954a3026c1fb44c5f07b4b5fc096a8
    Nonce:    997
    GasPrice: 10000000000000
    GasLimit  30400
    Value:    1
    Data:     0x
    V:        0x1c
    R:        0xd001a12cedbfd86e7da321e1ddf5224b06e6ed4e0c0e2cf60fd6620b05e99c62
    S:        0x438aa176392cbdbe5cf66e774a04aabc6f320348dec2d255837171f495fbc764
    Hex:      f8678203e58609184e72a0008276c09457b723e224954a3026c1fb44c5f07b4b5fc096a801801ca0d001a12cedbfd86e7da321e1ddf5224b06e6ed4e0c0e2cf60fd6620b05e99c62a0438aa176392cbdbe5cf66e774a04aabc6f320348dec2d255837171f495fbc764

I0410 17:01:46.141982    8480 worker.go:280] TX (f6ef6665) failed, will be removed: Transaction w/ invalid nonce (998 / 817)
I0410 17:01:46.142018    8480 worker.go:281] 
    TX(f6ef66653344c30d02c30ab959030278f79ccfbc6f31783e4e4fd0956723130a)
    Contract: false
    From:     cd2a3d9f938e13cd947ec05abc7fe734df8dd826
    To:       57b723e224954a3026c1fb44c5f07b4b5fc096a8
    Nonce:    998
    GasPrice: 10000000000000
    GasLimit  30400
    Value:    1
    Data:     0x
    V:        0x1c
    R:        0x7cf9949c98959445bd478fb5302d936965952cbf332816847b8cc1b101bbe3c6
    S:        0x17ced3bf501b3d10fc02ef01005d187beb5b49f60f2b9f296ca0ad9aca9a7578
    Hex:      f8678203e68609184e72a0008276c09457b723e224954a3026c1fb44c5f07b4b5fc096a801801ca07cf9949c98959445bd478fb5302d936965952cbf332816847b8cc1b101bbe3c6a017ced3bf501b3d10fc02ef01005d187beb5b49f60f2b9f296ca0ad9aca9a7578

I0410 17:01:46.145217    8480 state_object.go:238] ebf35f929443f141c3651f8e231fac31a347a7ce: gas (+ 3141592)
I0410 17:01:46.149105    8480 state_object.go:194] ebf35f929443f141c3651f8e231fac31a347a7ce: #0 37916525365000000000000 (+ 1500000000000000000)
I0410 17:01:46.149624    8480 chain_manager.go:526] inserted block #71227 (0 TXs 0 UNCs) (ca375e7d...)
I0410 17:01:46.150341    8480 worker.go:280] TX (aa7e050b) failed, will be removed: Transaction w/ invalid nonce (999 / 817)
I0410 17:01:46.150372    8480 worker.go:281] 
    TX(aa7e050bfc09b6f1c6db1049bcad7e39f86b488ab295ee315afcd8ed960a898a)
    Contract: false
    From:     cd2a3d9f938e13cd947ec05abc7fe734df8dd826
    To:       57b723e224954a3026c1fb44c5f07b4b5fc096a8
    Nonce:    999
    GasPrice: 10000000000000
    GasLimit  30400
    Value:    1
    Data:     0x
    V:        0x1b
    R:        0x9c3fae625200b21cc87b83a8562064522033ee0c2b4d25a7b7ed3de7d64417ca
    S:        0x4e1d8dd39ad4e44f205d9beda79cb057975dfcf7c89025d33b646470479cfca1
    Hex:      f8678203e78609184e72a0008276c09457b723e224954a3026c1fb44c5f07b4b5fc096a801801ba09c3fae625200b21cc87b83a8562064522033ee0c2b4d25a7b7ed3de7d64417caa04e1d8dd39ad4e44f205d9beda79cb057975dfcf7c89025d33b646470479cfca1

[obscuren]

Please retry after 8310bcd

[ethers]

Closing this since not seeing invalid nonce anymore on develop 8db7.
However, still seeing "known transaction" #662 (as well as "impossible nonce")
https://gist.github.com/ethers/458ce2d9978b55407a6e