feat: Publish self-announces as pkarr signed packets through the derper #2052

Frando · 2024-03-01T13:51:31Z

Description

This build on top of #2045.

Adds a new frame PublishPkarr to the relay protocol
MagicEndpoint will send a pkarr SignedPacket to its home relay whenever it changes home relays
The relay server will publish this packet to a pkarr relay, if configured
Our relay servers would publish to our our instance of https://github.com/n0-computer/iroh-dns-server/, which would make the records available over DNS, which will give global node resolution with feat: node discovery via DNS #2045
The iroh node gets minimal new configurability: Users can either enable or disable the self-announces

based on #2045

Notes & open questions

Change checklist

Self-review.
Documentation updates if relevant.
Tests if relevant.

Co-authored-by: Kasey <[email protected]>

ramfox · 2024-03-28T20:25:04Z

Did some dogfooding!

used a local iroh-relay to test w/ the following config:

# iroh-relay-config.toml
pkarr_relay = "https://testdns.iroh.link/pkarr"
addr = "[::]:3340"
enable_relay = true
enable_stun = true
stun_port = 3478
hostname = "my.relay.network"

Works with two local nodes dialing by node_id!

Can also confirm that if the relay server is not "pkarr_relay" enabled, the normal relay interactions still work, there is no problem when sending a PkarrPacket from the node to the relay server.

rklaehn · 2024-04-03T06:20:26Z

Took a quick look. I like that this is for generic pkarr packets and not just for the specific use case of node discovery.

Still on the fence whether this should be included with the ~~derper~~ relay or not. But since these are just fully self contained pkarr packets with no additional instructions I think having it as an option is nice. So currently I am slightly in favour.

How exactly would we modify this if we were to change the relay protocol to be more QUIC native? I guess that is a question for @flub when he comes back. I don't really know how that would look like and if it would be horrible or just a bit weird.

## Description This enables global node discovery over DNS, i.e. dialing nodes by just their node id. Current setup is as follows: * When dialing a node only by its NodeId, the new `DnsDiscovery` service is invoked. It will lookup a TXT record at (by default) `_iroh_node.b32encodednodeid.testdns.iroh.link` over regular DNS or DNS-over-http. Right now the Cloudflare DNS servers are configured. At `testdns.iroh.link` we run a custom [DNS server](https://github.com/n0-computer/iroh-dns-server/tree/main) * Nodes publish their Derp address to this DNS server through Pkarr signed packets. This is an intermediate step, we decided that the publishing by default should not happen by the nodes directly but mediated through the Derp servers. Work for the latter happens in #2052 This PR thus allows for the following: ```sh # terminal/computer 1 $ iroh console --start Iroh is running Node ID: qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua $ blob add ./myfile ... Blob: o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba ... # terminal/computer 2 iroh console --start blob get o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba --node qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua ```  ## Notes & open questions * Misses node configuration in the CLI for the node origin domain (right now hardcoded to `testdns.iroh.link`). How do we want to expose this - CLI flag? Or in the config file? I'd say the latter. * Offload publishing to the Derpers - see #2052 * Right now the records published via pkarr have a TTL of 30s - the iroh-dns-server will use that TTL as-is when serving the records over DNS. both can/should change? * We can also *very* easily allow to lookup nodes not only by NodeId, but by any domain name. In the `iroh-dns` crate I included an example `resolve` that does just that. By setting a `CNAME` record you can even use any domain and simply point to the record hosted at the `testdns.iroh.link` server. So if, on your custom domain, you added a record like this ``` _iroh_node.frando.n0.computer CNAME _iroh_node.qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua.iroh.link. ``` You can use this with the example to resolve to the node id and derp addresses: ``` cargo run --example resolve -- domain frando.n0.computer ```  ## Change checklist - [x] Self-review. - [x] Documentation updates if relevant. - [x] Tests if relevant. Closes #1248 --------- Co-authored-by: Kasey <[email protected]> Co-authored-by: Asmir Avdicevic <[email protected]> Co-authored-by: Ruediger Klaehn <[email protected]>

## Description This enables global node discovery over DNS, i.e. dialing nodes by just their node id. Current setup is as follows: * When dialing a node only by its NodeId, the new `DnsDiscovery` service is invoked. It will lookup a TXT record at (by default) `_iroh_node.b32encodednodeid.testdns.iroh.link` over regular DNS or DNS-over-http. Right now the Cloudflare DNS servers are configured. At `testdns.iroh.link` we run a custom [DNS server](https://github.com/n0-computer/iroh-dns-server/tree/main) * Nodes publish their Derp address to this DNS server through Pkarr signed packets. This is an intermediate step, we decided that the publishing by default should not happen by the nodes directly but mediated through the Derp servers. Work for the latter happens in n0-computer#2052 This PR thus allows for the following: ```sh # terminal/computer 1 $ iroh console --start Iroh is running Node ID: qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua $ blob add ./myfile ... Blob: o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba ... # terminal/computer 2 iroh console --start blob get o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba --node qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua ```  ## Notes & open questions * Misses node configuration in the CLI for the node origin domain (right now hardcoded to `testdns.iroh.link`). How do we want to expose this - CLI flag? Or in the config file? I'd say the latter. * Offload publishing to the Derpers - see n0-computer#2052 * Right now the records published via pkarr have a TTL of 30s - the iroh-dns-server will use that TTL as-is when serving the records over DNS. both can/should change? * We can also *very* easily allow to lookup nodes not only by NodeId, but by any domain name. In the `iroh-dns` crate I included an example `resolve` that does just that. By setting a `CNAME` record you can even use any domain and simply point to the record hosted at the `testdns.iroh.link` server. So if, on your custom domain, you added a record like this ``` _iroh_node.frando.n0.computer CNAME _iroh_node.qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua.iroh.link. ``` You can use this with the example to resolve to the node id and derp addresses: ``` cargo run --example resolve -- domain frando.n0.computer ```  ## Change checklist - [x] Self-review. - [x] Documentation updates if relevant. - [x] Tests if relevant. Closes n0-computer#1248 --------- Co-authored-by: Kasey <[email protected]> Co-authored-by: Asmir Avdicevic <[email protected]> Co-authored-by: Ruediger Klaehn <[email protected]>

## Description This enables global node discovery over DNS, i.e. dialing nodes by just their node id. Current setup is as follows: * When dialing a node only by its NodeId, the new `DnsDiscovery` service is invoked. It will lookup a TXT record at (by default) `_iroh_node.b32encodednodeid.testdns.iroh.link` over regular DNS or DNS-over-http. Right now the Cloudflare DNS servers are configured. At `testdns.iroh.link` we run a custom [DNS server](https://github.com/n0-computer/iroh-dns-server/tree/main) * Nodes publish their Derp address to this DNS server through Pkarr signed packets. This is an intermediate step, we decided that the publishing by default should not happen by the nodes directly but mediated through the Derp servers. Work for the latter happens in #2052 This PR thus allows for the following: ```sh # terminal/computer 1 $ iroh console --start Iroh is running Node ID: qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua $ blob add ./myfile ... Blob: o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba ... # terminal/computer 2 iroh console --start blob get o5uanh5s2zwn2sucy47puqidsfx2advxos7kajq3ajwitcwobhba --node qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua ```  ## Notes & open questions * Misses node configuration in the CLI for the node origin domain (right now hardcoded to `testdns.iroh.link`). How do we want to expose this - CLI flag? Or in the config file? I'd say the latter. * Offload publishing to the Derpers - see #2052 * Right now the records published via pkarr have a TTL of 30s - the iroh-dns-server will use that TTL as-is when serving the records over DNS. both can/should change? * We can also *very* easily allow to lookup nodes not only by NodeId, but by any domain name. In the `iroh-dns` crate I included an example `resolve` that does just that. By setting a `CNAME` record you can even use any domain and simply point to the record hosted at the `testdns.iroh.link` server. So if, on your custom domain, you added a record like this ``` _iroh_node.frando.n0.computer CNAME _iroh_node.qp2znfedwdij4llc5noizwfemfgba7bzxozvr4bp7hfsdmwqbpua.iroh.link. ``` You can use this with the example to resolve to the node id and derp addresses: ``` cargo run --example resolve -- domain frando.n0.computer ```  ## Change checklist - [x] Self-review. - [x] Documentation updates if relevant. - [x] Tests if relevant. Closes #1248 --------- Co-authored-by: Kasey <[email protected]> Co-authored-by: Asmir Avdicevic <[email protected]> Co-authored-by: Ruediger Klaehn <[email protected]>

Frando added 15 commits March 7, 2024 11:45

feat: combine discovery services

09a7973

refactor: impl from iterator for combined discovery

a088781

tests: add tests for combined discovery

42f1f3b

chore: fmt

8bd2ea3

chore: fix doc link

6008c66

chore: clippy

5a0fd66

refactor: better controls for discovery tasks

4c81604

cleanup

19e9ae7

docs and cleanup

2db91c8

more docs

ed52ced

chore: clippy & doclinks

25c847b

better tracing logs

b5f9728

fix instrumentation

adbc41f

refactor: do not require static lifetime on discovery streams

9d62fea

chore: clippy

b5d5d55

Frando force-pushed the feat/dns branch from 020cc60 to be4fd78 Compare March 7, 2024 16:07

Frando mentioned this pull request Mar 7, 2024

feat: node discovery via DNS #2045

Merged

3 tasks

rename to ConcurrentDiscovery

b380b19

Frando force-pushed the feat/dns-derper-pkarr-publish branch 2 times, most recently from f5fe772 to d7e66d8 Compare March 7, 2024 23:37

Frando added 10 commits March 8, 2024 00:38

chore: doc link

9383427

feat: add iroh-dns crate

646de44

feat: discover nodes over DNS by default

ae3e0e7

fix: only republish if derp url changes

89fffc9

fix(iroh-dns): do not log secret key

8f26558

feat: allow to connect via node id only

58ad6f5

fix: feature flags

04611f2

chore: fmt

6ee0f97

chore: fmt and clippy

aa4cd94

chore: remove obsolete files

3839856

Frando force-pushed the feat/dns-derper-pkarr-publish branch from d4791b5 to 67629cb Compare March 25, 2024 16:22

tests: improve dns discovery tests

5d9df01

Frando force-pushed the feat/dns-derper-pkarr-publish branch from 2b977f2 to dd95550 Compare March 25, 2024 21:44

tests: increase timeout because windows is slow

428f811

Frando force-pushed the feat/dns-derper-pkarr-publish branch from dd95550 to 6b48860 Compare March 26, 2024 11:23

remove debug leftovers

a21eadf

Frando force-pushed the feat/dns-derper-pkarr-publish branch from 6b48860 to 59b2fd4 Compare March 26, 2024 12:02

Merge remote-tracking branch 'origin/main' into feat/dns

e12180a

Frando force-pushed the feat/dns-derper-pkarr-publish branch from 59b2fd4 to 8950c66 Compare March 26, 2024 12:03

Update iroh-net/src/discovery/dns.rs

c3252b6

Co-authored-by: Kasey <[email protected]>

Frando added 8 commits April 2, 2024 13:25

Merge remote-tracking branch 'origin/main' into feat/dns

3d14417

feat: publish pkarr self-announces through the derper

c472a2f

fix and test after rebase

5161341

fix and test after rebase

1371c06

wait enough

1db876d

tests: improve dns discovery tests

a8e3bf6

tests: fix after rebase

db0f8cd

tests: increase timeout because windows is slow

d9dc571

Frando force-pushed the feat/dns-derper-pkarr-publish branch from 8950c66 to d9dc571 Compare April 2, 2024 11:27

dignifiedquire added this to the v0.14.0 milestone Apr 2, 2024

ramfox modified the milestones: v0.14.0, v0.15.0 Apr 10, 2024

dignifiedquire added the discussion label Apr 17, 2024

Base automatically changed from feat/dns to main April 19, 2024 12:50

dignifiedquire removed this from the v0.15.0 milestone Apr 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Publish self-announces as pkarr signed packets through the derper #2052

feat: Publish self-announces as pkarr signed packets through the derper #2052

Frando commented Mar 1, 2024 •

edited

Loading

ramfox commented Mar 28, 2024

rklaehn commented Apr 3, 2024

feat: Publish self-announces as pkarr signed packets through the derper #2052

Are you sure you want to change the base?

feat: Publish self-announces as pkarr signed packets through the derper #2052

Conversation

Frando commented Mar 1, 2024 • edited Loading

Description

Notes & open questions

Change checklist

ramfox commented Mar 28, 2024

rklaehn commented Apr 3, 2024

Frando commented Mar 1, 2024 •

edited

Loading