Ok understood. But in both cases the query landed on my server. Am I somehow able to get the host/ip that was used to initialize connection. I belive in case when the host was used I can get it by the $ssl_server_name (with SNI), but was in case if someone used to ip to access my sever?

If you don't want someone using an IP address to access your server, you can simply deny their request.

server {
listen *:80 default_server;
listen *:443 ssl default_server;
ssl_certificate xxx;
ssl_certificate_key xxx;
ssl_reject_handshake on;    # reject ssl handshake when request do not have sni or invalid sni. optional
return 403;                          # can be replace with 444 (close connection directly)
}

server {
listen *:80;
listen *:443 ssl;
ssl_certificate xxx;
ssl_certificate_key xxx;
server_name my-host;
# normal config
}

I am trying to find out that the host header is different then the host or ip which was used when the conection was initialized.
Am I able to do it for withing the server configured with server_name _;

I am trying to find out that the host header is different then the host or ip which was used when the conection was initialized. Am I able to do it for withing the server configured with server_name _;

yes, but you should set this server as default_server in listen directive.

I have only this one server.
It serves all my requests.
What I want to do is return 444 for all the reqests were the host in the header was different then the host/ip used to initialize the connection.
Can you please tell me how should I extend my configuraiton to have it?

server {
    listen 443 ssl;

    ssl_certificate /etc/nginx/conf.d/server.crt;
    ssl_certificate_key /etc/nginx/conf.d/server.key;

    server_name _;

   proxy_set_header Host $host;
}

You may have misunderstood the cause and effect. The so-called initial connection does not require a host name, but only an IP address. The process of obtaining an IP address, that is, DNS resolution, is impossible to obtain during HTTP requests. However, since you are using HTTPS, curl will write the host name in the URL into the SNI. You can use the inconsistency between the SNI and the host to determine the connection. Otherwise, you cannot obtain the so-called "host name for initial connection".