Skip to content

Score payment paths from events in BackgroundProcessor#1996

Merged
TheBlueMatt merged 3 commits intolightningdevkit:mainfrom
valentinewallace:2023-01-migrate-payment-scoring
Feb 3, 2023
Merged

Score payment paths from events in BackgroundProcessor#1996
TheBlueMatt merged 3 commits intolightningdevkit:mainfrom
valentinewallace:2023-01-migrate-payment-scoring

Conversation

@valentinewallace
Copy link
Copy Markdown
Contributor

@valentinewallace valentinewallace commented Jan 30, 2023

Opening this as a draft before I finish testing+supporting in async to get ACKs on the patch.

It uses a higher-ranked trait bound, and works in the sample. The real solution is GATs, which doesn't fit our MSRV.

We want to move scoring here to migrate it out of the InvoicePayer as part of moving payment retries to ChannelManager.

Partially addresses #1932.

@TheBlueMatt
Copy link
Copy Markdown
Collaborator

No, I think an HRTB is the correct solution here, not a GAT - we aren't looking for a specific type but rather a range of types. Sadly we've had issues with that in the bindings due to rustc limitations in resolving HRTBs, but we'll see if it works this time.

@valentinewallace
Copy link
Copy Markdown
Contributor Author

No, I think an HRTB is the correct solution here, not a GAT - we aren't looking for a specific type but rather a range of types.

If we had GATs, we could remove the lifetime parameter from WriteableScore, by removing it from LockableScore:

pub trait LockableScore {
	/// The locked [`Score`] type.
	type Locked<'a>: 'a + Score where Self: 'a;

	/// Returns the locked scorer.
	fn lock(&self) -> Self::Locked<'_>;
}

@TheBlueMatt
Copy link
Copy Markdown
Collaborator

Ah, right.

@valentinewallace
Copy link
Copy Markdown
Contributor Author

Sadly we've had issues with that in the bindings due to rustc limitations in resolving HRTBs, but we'll see if it works this time.

Hmm, that seems like a big issue if it breaks bindings. Approach-wise, I wonder if that means the scoring needs to happen elsewhere?

@TheBlueMatt
Copy link
Copy Markdown
Collaborator

No, let's do it The Right Way first, and if it breaks in bindings we can figure it out later.

@valentinewallace valentinewallace force-pushed the 2023-01-migrate-payment-scoring branch 2 times, most recently from 96aa4ed to a1a0bc3 Compare January 31, 2023 20:41
@valentinewallace valentinewallace marked this pull request as ready for review January 31, 2023 20:42
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jan 31, 2023

Codecov Report

❌ Patch coverage is 71.06599% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.73%. Comparing base (d4de913) to head (2f9c3e5).
⚠️ Report is 6594 commits behind head on main.

Files with missing lines Patch % Lines
lightning-background-processor/src/lib.rs 70.68% 56 Missing ⚠️
lightning/src/ln/channelmanager.rs 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1996      +/-   ##
==========================================
+ Coverage   90.91%   91.73%   +0.82%     
==========================================
  Files          99       99              
  Lines       52505    59263    +6758     
  Branches    52505    59263    +6758     
==========================================
+ Hits        47735    54366    +6631     
- Misses       4770     4897     +127     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@valentinewallace valentinewallace force-pushed the 2023-01-migrate-payment-scoring branch from a1a0bc3 to 4b22708 Compare February 1, 2023 19:09
@TheBlueMatt
Copy link
Copy Markdown
Collaborator

LGTM, feel free to squash.

@valentinewallace valentinewallace force-pushed the 2023-01-migrate-payment-scoring branch 2 times, most recently from 24e9a51 to 640d9e5 Compare February 2, 2023 22:48
TheBlueMatt
TheBlueMatt previously approved these changes Feb 2, 2023
@TheBlueMatt TheBlueMatt merged commit a7293fd into lightningdevkit:main Feb 3, 2023
PXplod pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries"

API Updates
===========

 * `InvoicePayer` has been removed and its features moved directly into
   `ChannelManager`. As such it now requires a simplified `Router` and supports
   `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was
   removed in favor of the automated retries. Invoice payment utilities in
   `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc).
 * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner`
   split out in anticipation of future schnorr/taproot support (lightningdevkit#1967).
 * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`,
   and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930).
 * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key`
   and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070).
 * A `lightning-transaction-sync` crate has been added which implements keeping
   LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can
   only be used on nodes that *never* ran a previous version of LDK.
 * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996).
 * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980).
 * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as
   associated builders have been added. Such invoices cannot yet be paid due to
   missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926).
 * A `lightning-custom-message` crate has been added to make combining multiple
   custom messages into one enum/handler easier (lightningdevkit#1832).
 * `Event::PaymentPathFailure` is now generated for failure to send an HTLC
   over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043).
 * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968).
 * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873).
 * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962).
 * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985).
 * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878)
 * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057).
 * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961).
 * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948).
 * Macros which implement serialization using TLVs or straight writing of struct
   fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977).

Backwards Compatibility
=======================

 * Any inbound payments with a custom final CLTV delta will be rejected by LDK
   if you downgrade prior to receipt (lightningdevkit#1878).
 * `Event::PaymentPathFailed::network_update` will always be `None` if an
   0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043).
 * `Event::PaymentPathFailed::all_paths_removed` will always be false if an
   0.0.114-generated event is read by a prior version of LDK. Users who rely on
   it to determine payment retries should migrate to `Event::PaymentFailed`, in
   a separate release prior to upgrading to LDK 0.0.114 if downgrading is
   supported (lightningdevkit#2043).

Performance Improvements
========================

 * Channel data is now stored per-peer and channel updates across multiple
   peers can be operated on simultaneously (lightningdevkit#1507).
 * Routefinding is roughly 1.5x faster (lightningdevkit#1799).
 * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016).
 * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040).
 * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974).

Bug Fixes
=========

 * Fixed a bug where a delay in processing a `PaymentSent` event longer than the
   time taken to persist a `ChannelMonitor` update, when occurring immediately
   prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048).
 * Fixed spurious rejections of rapid gossip sync data when the graph has been
   updated by other means between gossip syncs (lightningdevkit#2046).
 * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos`
   is set (lightningdevkit#1935).
 * Resolved an issue where the `ChannelManager::get_persistable_update_future`
   future would fail to wake until a second notification occurs (lightningdevkit#2064).
 * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037).
 * Fixed a deadlock on some platforms at least when using async `ChannelMonitor`
   updating (lightningdevkit#2006).
 * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964).
 * In some cases when payment sending fails on our local channel retries no
   longer take the same path and thus never succeed (lightningdevkit#2014).
 * Retries for spontaneous payments have been fixed (lightningdevkit#2002).
 * Return an `Err` if `lightning-persister` fails to read the directory listing
   rather than panicing (lightningdevkit#1943).
 * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035)

Security
========

0.0.114 fixes several denial-of-service vulnerabilities which are reachable from
untrusted input from channel counterparties or in deployments accepting inbound
connections or channels. It also fixes a denial-of-service vulnerability in rare
cases in the route finding logic.
 * The number of pending un-funded channels as well as peers without funded
   channels is now limited to avoid denial of service (lightningdevkit#1988).
 * A second `channel_ready` message received immediately after the first could
   lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf
   support in LDK 0.0.107.
 * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount
   being sent (including previous-hop fees) is equal to a channel's capacity
   while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with
   historical data tracking in LDK 0.0.112.

In total, this release features 130 files changed, 21457 insertions, 10113
deletions in 343 commits from 18 authors, in alphabetical order:
 * Alec Chen
 * Allan Douglas R. de Oliveira
 * Andrei
 * Arik Sosman
 * Daniel Granhão
 * Duncan Dean
 * Elias Rohrer
 * Jeffrey Czyz
 * John Cantrell
 * Kurtsley
 * Matt Corallo
 * Max Fang
 * Omer Yacine
 * Valentine Wallace
 * Viktor Tigerström
 * Wilmer Paulino
 * benthecarman
 * jurvis
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants