Channel "AWAITING_UNILATERAL" since days. Attempted force closing, CPU 100% #2171
Description
I found one of my channel was in "AWAITING_UNILATERAL" so I waited but days after it was still in that state.
Relevant log with grep peerID:
2018-11-25T13:41:40.283Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: 0: 10648 satoshi (SEGWIT) 695254bb08add33af6f907960913869c25903c967541b2b8765be96431d2cb59? [100/260]
2018-11-25T13:41:40.283Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: 1: 2005802 satoshi (SEGWIT) bd729c73e2cd2647f5ce6a2ebd24419720486f720124d9e0c66649b0b3d2835d?
2018-11-25T13:41:40.283Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: 2: 252848 satoshi (SEGWIT) bbe3783d4c7fde8eb6433403fc1c34fa0098e4c738cedeac087243a1655a0b81?
2018-11-25T13:41:40.300Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Getting HSM to sign funding tx
2018-11-25T13:41:40.397Z lightningd(17827): lightning_channeld-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: pid 31244, msgfd 33
2018-11-25T13:41:40.397Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Waiting for funding confirmations
2018-11-25T13:41:41.001Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Peer internal error CHANNELD_AWAITING_LOCKIN: Funding broadcast exited with 25: error code: -25?error message:?Missing inputs?
2018-11-25T13:41:41.001Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Peer permanent failure in CHANNELD_AWAITING_LOCKIN: Internal error: Funding broadcast exited with 25: error code: -25?error message:?Missing inputs?
2018-11-25T13:41:41.001Z lightningd(17827): lightning_channeld-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Status closed, but not exited. Killing
2018-11-25T13:41:41.013Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: State changed from CHANNELD_AWAITING_LOCKIN to AWAITING_UNILATERAL
2018-11-25T13:42:58.779Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150096 msatoshi
2018-11-25T13:42:58.779Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (151109msat, 590blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 532217:2079:0 (150109msat, 446blk) -> 02529db69fd2ebd3126fb66fafa234fc3544477a23d509fe93ed229bb0e92e4f
b8 -> 548838:1988:1 (150098msat, 302blk) -> 031d2bbc75802689312220a017c6b51fa246efc59c7aa9355f6f7395038ffb4d6a -> 549057:2281:0 (150097msat, 158blk) -> 02c5371591b640da03f9a645e2ddfa5620d8537388b03ddd524d36766c0f550db0 -> 548930:191:0 (150096msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55
652e6e1a41225f06a0bbf2d56
2018-11-25T13:43:04.954Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150218 msatoshi
2018-11-25T13:43:04.955Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (152234msat, 446blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 550459:2332:1 (151234msat, 302blk) -> 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986
d0 -> 550102:1165:0 (150219msat, 158blk) -> 02c5371591b640da03f9a645e2ddfa5620d8537388b03ddd524d36766c0f550db0 -> 548930:191:0 (150218msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
2018-11-25T13:43:11.852Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150170 msatoshi
2018-11-25T13:43:11.852Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (152271msat, 864blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 544611:1240:1 (151271msat, 720blk) -> 020e56a13babec99abdc2c4afbe34e1e44230d79b234c059fd4ff1e367765fdb
1b -> 541282:1481:0 (150271msat, 576blk) -> 03434a39cd9a537c852fc8fb72454086d726f9111e9f730cef4985c39c11fae944 -> 535271:1925:0 (150171msat, 432blk) -> 03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d -> 549187:1932:1 (150170msat, 288blk) -> 03295d2e292565743a40bd44da227a820f873087
7bc3dfadebade8785bcf355258 -> 544098:2501:1 (150170msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
2018-11-25T13:43:18.520Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150426 msatoshi
2018-11-25T13:43:18.520Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (152615msat, 474blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 550180:1827:0 (151615msat, 330blk) -> 03a5927b64b1ea8657d5b770d61a3e2d0554fdb5d568773e0d6a92907913ca21
f6 -> 551184:469:0 (150628msat, 186blk) -> 0217090c397c3e8ced66abb7b8bf6e05cd49adde55766a7d14e2e4a541164b0935 -> 551318:1032:0 (150428msat, 172blk) -> 03f18da23535adee245903c81750745197e3079d3b88b3898981c90a5f2411008c -> 544620:1321:1 (150427msat, 158blk) -> 02c88a0df5e9ea7df582647f79b70b8e441047de3
08fd6b24408b8bd775f2234bd -> 549696:815:1 (150426msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
Today I have tried to force close the channel but the command is stucked and the CPU is 100%.
lightning-cli close 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 true
I have interrupted the command but it seems like every command (getinfo included) is causing 100% CPU and the freezing of the command itself.
Restarting the service, I was able to collect some more infos:
$ lightning-cli listpeers 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 debug
{
"peers": [
{
"id": "03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56",
"connected": true,
"netaddr": [
"35.172.33.197:9735"
],
"global_features": "",
"local_features": "82",
"globalfeatures": "",
"localfeatures": "82",
"channels": [
{
"state": "AWAITING_UNILATERAL",
"scratch_txid": "458cd396b558361c2a616d0e2a0c56a50f5c8eacd9ca71e536d016bf3ae6233d",
"channel_id": "84649e5d4c01a17a4f1612c722df7be59e31cdf39f969a8794d2adcd54b1b731",
"funding_txid": "31b7b154cdadd294879a969ff3cd319ee57bdf22c712164f7aa1014c5d9e6484",
"msatoshi_to_us": 1000000000,
"msatoshi_to_us_min": 1000000000,
"msatoshi_to_us_max": 1000000000,
"msatoshi_total": 1000000000,
"dust_limit_satoshis": 546,
"max_htlc_value_in_flight_msat": 18446744073709551615,
"their_channel_reserve_satoshis": 10000,
"our_channel_reserve_satoshis": 10000,
"spendable_msatoshi": 990000000,
"htlc_minimum_msat": 0,
"their_to_self_delay": 144,
"our_to_self_delay": 144,
"max_accepted_htlcs": 483,
"status": [
"AWAITING_UNILATERAL:Attempting to reconnect"
],
"in_payments_offered": 0,
"in_msatoshi_offered": 0,
"in_payments_fulfilled": 0,
"in_msatoshi_fulfilled": 0,
"out_payments_offered": 0,
"out_msatoshi_offered": 0,
"out_payments_fulfilled": 0,
"out_msatoshi_fulfilled": 0,
"htlcs": [
]
}
],
"log": [
{
"type": "DEBUG",
"time": "5.221303026",
"source": "03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10:",
"log": "Peer has reconnected, state AWAITING_UNILATERAL"
},
{
"type": "DEBUG",
"time": "5.225104641",
"source": "lightning_openingd-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #11:",
"log": "pid 15044, msgfd 25"
},
{
"type": "DEBUG",
"time": "5.293310285",
"source": "lightning_openingd-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #11:",
"log": "peer_out WIRE_ERROR"
},
{
"type": "DEBUG",
"time": "5.294715801",
"source": "lightning_openingd-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #11:",
"log": "Handed peer, entering loop"
}
]
}
]
}
getinfo output
not possible (due to 100%CPU issue) but running tags/v0.6.2
EDIT: restarting the service:
lightning-cli getinfo
{
"id": "03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e",
"address": [
{
"type": "torv2",
"address": "byy4lqhaaley7e7k.onion",
"port": 9735
}
],
"binding": [
{
"type": "ipv4",
"address": "127.0.0.1",
"port": 9731
},
{
"type": "ipv4",
"address": "10.2.206.180",
"port": 9731
}
],
"version": "v0.6.2-modded",
"blockheight": 553513,
"network": "bitcoin",
"msatoshi_fees_collected": 0
}
Reactions are currently unavailable