HTTP request timeout after 30s #9321

duyan11110 created 21 days ago

I get this error after upgrade microservice template from v7.3.0 to v9.1.0. Old version still works fine.

[web_80132ba5-d]: [10:43:56 ERR] ---------- RemoteServiceErrorInfo ----------
[web_80132ba5-d]: {
[web_80132ba5-d]: "code": null,
[web_80132ba5-d]: "message": "An error occurred during the ABP remote HTTP request. (The operation didn't complete within the allowed timeout of '00:00:30'.) See the inner exception for details.",
[web_80132ba5-d]: "details": null,
[web_80132ba5-d]: "data": null,
[web_80132ba5-d]: "validationErrors": null
[web_80132ba5-d]: }
[web_80132ba5-d]:
[web_80132ba5-d]: [10:43:56 ERR] An error occurred during the ABP remote HTTP request. (The operation didn't complete within the allowed timeout of '00:00:30'.) See the inner exception for details.
[web_80132ba5-d]: Volo.Abp.Http.Client.AbpRemoteCallException: An error occurred during the ABP remote HTTP request. (The operation didn't complete within the allowed timeout of '00:00:30'.) See the inner exception for details.
[web_80132ba5-d]: ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
[web_80132ba5-d]: ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
[web_80132ba5-d]: ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
[web_80132ba5-d]: ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
[web_80132ba5-d]: --- End of inner exception stack trace ---

I tried to increase the timeout like this, but the error still occur.

public override void PreConfigureServices(ServiceConfigurationContext context)
{
    PreConfigure(options =>
    {
        options.ProxyClientActions.Add((remoteServiceName, clientBuilder, client) =>
        {
            client.Timeout = TimeSpan.FromMinutes(60);
        });

        options.ProxyClientBuildActions.Add((remoteServiceName, clientBuilder) =>
        {
            clientBuilder.AddPolicyHandler(Policy.TimeoutAsync(TimeSpan.FromMinutes(60)));
            clientBuilder.AddResilienceHandler("MHIBS", builder =>
            {
                builder.AddTimeout(TimeSpan.FromMinutes(60));
            });
            clientBuilder.AddStandardResilienceHandler(opts =>
            {
                opts.TotalRequestTimeout.Timeout = TimeSpan.FromMinutes(60);
                opts.AttemptTimeout.Timeout = TimeSpan.FromMinutes(60);
                opts.CircuitBreaker.SamplingDuration = TimeSpan.FromMinutes(120);
            });
        });
    });
}

Please help

enisn created 21 days ago

Support Team .NET Developer

Hi,

it seems your service cannot get requests and doesn't respond. Can you share the service logs when you sent request to check if request reach to your service or not

duyan11110 created 19 days ago

Hi,

Can you share your email. I will send you full log.

duyan11110 created 17 days ago

Hi,

Are you there?

berkansasmaz created 17 days ago

Support Team .NET Developer

Hi,

You can share your solution if it's possible or full log records via email (to support@abp.io with ticket number).

Regards.

duyan11110 created 17 days ago

Hi,

I've sent the log. Please help to fix this issue.

enisn created 16 days ago

Support Team .NET Developer

It seems your client sends request to gateway, and gateway successfully proxies it to the service. It might be real timeout scenario. Can you check if your service generate response in 30seconds or not. The best practise for long-running reporting operations is, executing log generation as background job, and whenever it finishes, sending an email to report requester.

Also in your logs, it seems client cancels request in 10 seconds and resends it again and again, so we need to check another question here:

Does it work when you directly call service endpoint without gateway?

duyan11110 created 16 days ago

Hi,

I used swagger to directly call the endpoint and succeeded in about 2mins.

Also in your logs, it seems client cancels request in 10 seconds and resends it again and again

Yes, you are right. I also saw it, but don't know why. I tried to increase all types of timeout as code above but not luck.

enisn created 15 days ago

Support Team .NET Developer

I think this is a similar case:

https://abp.io/support/questions/8325/How-to-increase-http-client-timeout

Can you try increasing YARP timeout from your Gateway, since gateway doesn't use ABP's client-proxy, it's a standalone proxy implementation by dotnet

duyan11110 created 15 days ago

That ticket is mine also. I already increased timeout of YARP and as I said above, in old version it still works fine, but new version does not.

"gsib-report-service-cluster": {
  "HttpRequest": {
    "ActivityTimeout": "01:00:00"
  },
  "Destinations": {
    "destination1": {
      "Address": "https://localhost:7019"
    }
  }
},

duyan11110 created 15 days ago

I think this issue related to Polly library but I can't increase Polly timeout with the code above. According to this document: https://www.pollydocs.org/strategies/timeout.html the default timeout is 30s. Can you show me how to increase it in ABP framework?

enisn created 15 days ago

Support Team .NET Developer

ABP Client-proxies use IHttpClientFactory to create HttpClient objects and it uses RemoteServiceName to create HttpClient for each of your remote service.

https://github.com/abpframework/abp/blob/9a43b9c9dfbc8b883c58a7acdb9c21619aa48dfc/framework/src/Volo.Abp.Http.Client/Volo/Abp/Http/Client/ClientProxying/ClientProxyBase.cs#L124

So you can easily use freshly defined a new HttpClient for your service:

context.Services.AddHttpClient("YourService") // Administrator, Identity etc.
    .ConfigureHttpClient(client =>
    {
        client.BaseAddress = new Uri("https://api.example.com/");
        client.Timeout = TimeSpan.FromMinutes(2);
    });

This defines a new HttpClient instead trying to configure existing one, you can have all control on the HttpClient in this way

duyan11110 created 15 days ago

I've just tried your solution but still error. After 3 times retry with timeout 10s the last error message is the same above:

[web_80132ba5-d]: [10:43:56 ERR] ---------- RemoteServiceErrorInfo ----------
[web_80132ba5-d]: {
[web_80132ba5-d]: "code": null,
[web_80132ba5-d]: "message": "An error occurred during the ABP remote HTTP request. (The operation didn't complete within the allowed timeout of '00:00:30'.) See the inner exception for details.",
[web_80132ba5-d]: "details": null,
[web_80132ba5-d]: "data": null,
[web_80132ba5-d]: "validationErrors": null
[web_80132ba5-d]: }
[web_80132ba5-d]:
[web_80132ba5-d]: [10:43:56 ERR] An error occurred during the ABP remote HTTP request. (The operation didn't complete within the allowed timeout of '00:00:30'.) See the inner exception for details.
[web_80132ba5-d]: Volo.Abp.Http.Client.AbpRemoteCallException: An error occurred during the ABP remote HTTP request. (The operation didn't complete within the allowed timeout of '00:00:30'.) See the inner exception for details.
[web_80132ba5-d]: ---> Polly.Timeout.TimeoutRejectedException: The operation didn't complete within the allowed timeout of '00:00:30'.
[web_80132ba5-d]: ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
[web_80132ba5-d]: ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
[web_80132ba5-d]: ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
[web_80132ba5-d]: --- End of inner exception stack trace ---

I also configure to not retry but it still retry 3 times, so I think the problem here related to Polly library. I don't know where it is configured and how to change it.

PreConfigure<AbpHttpClientBuilderOptions>(options =>
{
    options.ProxyClientActions.Add((remoteServiceName, clientBuilder, client) =>
    {
        client.Timeout = TimeSpan.FromMinutes(60);
    });

    options.ProxyClientBuildActions.Add((remoteServiceName, clientBuilder) =>
    {
        clientBuilder.AddTransientHttpErrorPolicy(policyBuilder =>
            policyBuilder.WaitAndRetryAsync(
                0,
                i => TimeSpan.FromSeconds(Math.Pow(2, i))
            )
        );
    });
});

or comment out the retry config

PreConfigure<AbpHttpClientBuilderOptions>(options =>
{
    options.ProxyClientActions.Add((remoteServiceName, clientBuilder, client) =>
    {
        client.Timeout = TimeSpan.FromMinutes(60);
    });
    
    //options.ProxyClientBuildActions.Add((remoteServiceName, clientBuilder) =>
    //{
    //    clientBuilder.AddTransientHttpErrorPolicy(policyBuilder =>
    //        policyBuilder.WaitAndRetryAsync(
    //            0,
    //            i => TimeSpan.FromSeconds(Math.Pow(2, i))
    //        )
    //    );
    //});                
});