Same IMAP and SMTP parameters configured manually in old and newly created profile lead to different connection attempts

[gerryfrancis]

• Operating System (Linux/Mac/Windows/iOS/Android):
  Android 14.

• Delta Chat Version:
  1.46.14 (current nightly built 2024-09-27).

• Expected behavior:
  Manually configured IMAP and SMTP parameters are used to connect to the server(s).
  

• Actual behavior:
  Manually configured IMAP and SMTP parameters seem to be overruled/reconfigured by other ones.

• Steps to reproduce the problem:
  No idea, unfortunately, but what I have observed so far is:
  -- My Delta Chat profile was created a long time ago, and I used STARTTLS (IMAP port 993, SMTP port 587) at the beginning.
  -- The parameters actually used for that account (according to the log) are:
  [email protected] imap:[posteo.de:993:tls:[email protected], posteo.de:143:starttls:[email protected]] smtp:[posteo.de:465:tls:[email protected], posteo.de:587:starttls:[email protected]] provider:posteo cert_strict
  -- In this profile, Delta Chat never uses TLS for IMAP (port 993), but STARTTLS (port 143) to connect to the IMAP server. I have never seen an attempt to connect to IMAP via TLS in the log file. However, connections to the SMTP server are established via TLS (port 465) successfully.
  -- A newly created and manually configured profile (just like the old one, see above) leads to this line in the log:
  [email protected] imap:[posteo.de:993:tls:[email protected], posteo.de:993:starttls:[email protected]] smtp:[posteo.de:465:tls:[email protected], posteo.de:465:starttls:[email protected]] provider:none cert_automatic
  Although this profile connects to ports 993 (IMAP) and 465 (SMTP) successfully, I do not think that this configuration (using the same ports for TLS and STARTTLS?) is correct.

• Screenshots:
  N/A.

• Logs:
  deltachat-log-20240927-145733_2.txt

[r10s]

Manually configured IMAP and SMTP parameters seem to be overruled/reconfigured by other ones.

but was it working at the end?

delta chat trying different servers and configurations is a new feature - might be that also other than the "entered" ones are tried. if so i would regard that as a feature as well - if the entered server is not reachable or slower in some network, and there is another option, it seems good if that is used.

reconfiguration is even done "on the fly" btw - so if you enter a wifi that blocks one port but allows another one, things continue working

[gerryfrancis]

but was it working at the end?

@r10s Yes, it worked, but the same IMAP and SMTP ports that are specified for TLS by Posteo were also used for STARTTLS, it seems:

[email protected] imap:[posteo.de:993:tls:[email protected], posteo.de:993:starttls:[email protected]] smtp:[posteo.de:465:tls:[email protected], posteo.de:465:starttls:[email protected]] provider:none cert_automatic

And that is not what is registered for Posteo in the provider database, they have different ports for STARTTLS: https://github.com/deltachat/provider-db/blob/master/_providers/posteo.md

So, automatic reconfiguration might have found a setup that works, but is not intended to be used in the field.

[gerryfrancis]

To add: Posteo recommend to use TLS instead of STARTTLS: https://posteo.de/en/help/how-do-i-set-up-posteo-in-an-email-client-pop3-imap-and-smtp

We recommend using TLS instead of STARTTLS if possible.

Instead, automatic reconfiguration sets STARTTLS to use TLS IMAP and SMTP ports (see above), and that seems to be pretty wrong.

[hpk42]

On Thu, Oct 03, 2024 at 02:39 -0700, gerryfrancis wrote: To add: Posteo recommend to use TLS instead of STARTTLS: https://posteo.de/en/help/how-do-i-set-up-posteo-in-an-email-client-pop3-imap-and-smtp > We recommend using TLS instead of STARTTLS if possible. Instead, automatic reconfiguration sets STARTTLS for TLS IMAP and SMTP ports (see above), and that seems to be pretty wrong.

i think these days DC would "automagically" reconfigure if posteo decides to forbid STARTLS some day. Not sure if "TLS" could generally be prefererred over starttls. Auto-reconfiguration is a relative recent change in core.

[iequidoo]

Instead, automatic reconfiguration sets STARTTLS to use TLS IMAP and SMTP ports (see above), and that seems to be pretty wrong.

I checked with openssl s_client -showcerts -connect posteo.de:993 -starttls imap, it doesn't really work. The same for SMTP, STARTTLS only works if 143 and 587 ports are used. Apparently, these STARTTLS configurations were added because you specified IMAP/SMTP Security = Automatic. Everything works because TLS configurations are used (they are in the heads of the lists from your log), not STARTTLS. But i agree that it's at least strange that they are present in used_account_settings.

EDIT:

• Logs:
  deltachat-log-20240927-145733_2.txt

This is a log for your old account as far as i can tell, the ports are correct there for STARTTLS. Would be more interesting to see the log for the newly created account.

[gerryfrancis]

This is a log for your old account as far as i can tell, the ports are correct there for STARTTLS. Would be more interesting to see the log for the newly created account.

@iequidoo This is the log of a newly created account:
deltachat-log-20241015-133411_2.txt

[iequidoo]

Still, it's not clear how incorrect STARTTLS ports got into configuration, in the beginning of the log i just see this:
[email protected] imap:[posteo.de:993:tls:[email protected], posteo.de:993:starttls:[email protected]] smtp:[posteo.de:465:tls:[email protected], posteo.de:465:starttls:[email protected]] provider:none cert_automatic
I guess they haven't been even tried by the core, but it just adds them because you specified Automatic security. Need to look into the code probably.

[iequidoo]

The logic that added the above ports to the configuration is here:

deltachat-core-rust/src/configure/server_params.rs

Lines 108 to 120 in 92c6dd4

	} else if self.socket == Socket::Automatic {
	vec![
	// Try TLS over user-provided port.
	Self {
	socket: Socket::Ssl,
	..self.clone()
	},
	// Try STARTTLS over user-provided port.
	Self {
	socket: Socket::Starttls,
	..self
	},
	]

So, @gerryfrancis, this is because you set the ports manually as far as i see on your screenshot. Posteo uses the standard ports for TLS and STARTTLS, so you don't need to specify them manually. Please try to leave them empty and share a new log. I guess Delta Chat will use the correct ports then.

[gerryfrancis]

@iequidoo Thank you for looking into the code, I can confirm that the correct ports have been determined after I reconfigured the profile without entering the port numbers for IMAP and SMTP:

[email protected] imap:[email protected]:***:posteo.de:0:Automatic:AUTH_NORMAL smtp:[email protected]:***:posteo.de:0:Automatic:AUTH_NORMAL cert_automatic
[email protected] imap:[posteo.de:993:tls:[email protected], posteo.de:143:starttls:[email protected]] smtp:[posteo.de:465:tls:[email protected], posteo.de:587:starttls:[email protected]] provider:none cert_automatic

However, I think the possibility of unknowingly misconfiguring the same ports for TLS and STARTTLS is a bug and must be avoided. Maybe another connection check in the background can be introduced to verify and adjust the configuration if necessary.

[iequidoo]

However, I think the possibility of unknowingly misconfiguring the same ports for TLS and STARTTLS is a bug and must be avoided. Maybe another connection check in the background can be introduced to verify and adjust the configuration if necessary.

The ports for STARTTLS were added to the configuration because Delta Chat didn't try them when configuring because the TLS configuration just worked. If TLS stops working for some reason, Delta Chat will try STARTTLS, so we don't need to check it in advance. Even if we check the STARTTLS configuration in advance and it doesn't work, what should we do, try the standard STARTTLS ports instead? But the user specified the exact ports, so such logic looks unclear. Another option is to warn the user that STARTTLS doesn't work (EDIT: it will never work if TLS works, because it's another protocol), but i think most of the users aren't even interested in which configuration they use and the remaining are just ok with not having a working STARTTLS configuration if TLS works.

[gerryfrancis]

If TLS stops working for some reason, Delta Chat will try STARTTLS, so we don't need to check it in advance.

The issue is about the same ports configured for TLS and STARTTLS by accident when just the TLS ones are specified manually. The app will not be able to connect using STARTTLS either when TLS suddenly does not work anymore.

Even if we check the STARTTLS configuration in advance and it doesn't work, what should we do, try the standard STARTTLS ports instead?

This, or just do not configure STARTTLS at all to avoid applying the wrong ports to it. If TLS connectivity died in case of failure, a falsely configured STARTTLS fallback would be even worse. In my opinion, the standard ports for STARTTLS should be examined and used in cases of TLS malfunction, but it is just an idea, of course.

[iequidoo]

The issue is about the same ports configured for TLS and STARTTLS by accident when just the TLS ones are specified manually.

Delta Chat could detect that the specified ports are the standard TLS/STRATTLS ones and use the standard ports for all protocols in this case (as if the user left the fields empty), but i'd say such a logic is unclear. I'd expect that the program only connects to the specified ports. I think this issue is rather an UI one, the user actually wants to use the standard ports for everything, but for some reason (e.g. their provider recommends that) specifies ports only for TLS or STARTTLS. Some hints should be added, like "Leave empty to use 993/TLS and 143/STARTTLS".

And i don't think we really need 4 fields to specify ports for all protocols.

EDIT:

or just do not configure STARTTLS at all to avoid applying the wrong ports to it.

The problem is that we can't know if the specified ports are wrong or were specified on purpose.

[gerryfrancis]

In my eyes, the logic should be like this:

• If the specified port can be connected by using TLS successfully, it should not be used for anything else.
• If the specified port can be connected by using STARTTLS successfully, it should not be used for anything else.

Instead, we accidentally determine a configuration which uses the same ports for both, TLS and STARTTLS:

[email protected] imap:[posteo.de:993:tls:[email protected], posteo.de:993:starttls:[email protected]] smtp:[posteo.de:465:tls:[email protected], posteo.de:465:starttls:[email protected]] provider:none cert_automatic

Delta Chat could detect that the specified ports are the standard TLS/STRATTLS ones and use the standard ports for all protocols in this case (as if the user left the fields empty), but i'd say such a logic is unclear. I'd expect that the program only connects to the specified ports.

So do I, but in case of lost connectivity I think it would be legal to determine another way to connect to the server, just like we do with cached DNS results.

And i don't think we really need 4 fields to specify ports for all protocols.

Copy that. However, using the same ports for all secure transmission protocols, like we apparently apply (see used_account_settings above) just because the user could not specify other ones, cannot be the "right" solution either.

So, according to this example, I suggest to shorten the configuration as follows:

[email protected] imap:[posteo.de:993:tls:[email protected]] smtp:[posteo.de:465:tls:[email protected]] provider:none cert_automatic

This configuration could be adjusted to add standard STARTTLS ports later in case the specified TLS ports do not work anymore one day:

[email protected] imap:[posteo.de:993:tls:[email protected], posteo.de:143:starttls:[email protected]] smtp:[posteo.de:465:tls:[email protected], posteo.de:587:starttls:[email protected]] provider:none cert_automatic

I would not say it was "right" to do that from the view of a programmer, but it could help to maintain app connectivity, which probably might be the goal for the most common users.

[iequidoo]

However, using the same ports for all secure transmission protocols, like we apparently apply (see used_account_settings above) just because the user could not specify other ones, cannot be the "right" solution either.

Using the same ports for STARTTLS may be helpful if later the server decides to switch from TLS to STARTTLS (or vice versa which is more likely). But this is probably for the case of nonstandard ports. In case of the standard ports the server probably will change the ports also, but then the user just shouldn't specify them and that should be clear in the UI. So, the first option is to improve the UI. Another option is, well, to connect to the standard STARTTLS ports, but for me connecting to any ports other than specified by the user is strange and it breaks the scenario described above. To keep that scenario working we can add two STARTTLS configurations (with the user-specified ports and the standard ports), but i'd prefer to just improve the UI.