PublishTestResults sporadically fails to publish all results

[thomasdgx]

Agent Version and Platform

Version of your agent? v2.160 and later

What's not working?

Our setup uses customized test execution. In order to publish test results we simply invoke the extension command: ""##vso[results.publish" like here:

Console.WriteLine($"##vso[results.publish type=VsTest;mergeResults=false;publishRunAttachments=false;resultFiles={trxFile};]");

Sporadically you see exceptions like (note I patched the agent code to dump the callstack - this output had been produced with v2.179):

---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
 ---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Worker.LegacyTestResults.TestResultsServer.AddTestResultsToTestRunAsync(TestCaseResult[] currentBatch, String projectName, Int32 testRunId, CancellationToken cancellationToken) in D:\Development\Microsoft\azure-pipelines-agent\src\Agent.Worker\TestResults\Legacy\TestResultsServer.cs:line 60
   at Microsoft.VisualStudio.Services.Agent.Worker.LegacyTestResults.TestRunPublisher.AddResultsAsync(TestRun testRun, TestCaseResultData[] testResults, CancellationToken cancellationToken) in D:\Development\Microsoft\azure-pipelines-agent\src\Agent.Worker\TestResults\Legacy\TestRunPublisher.cs:line 140

Also sporadically one can find ObjectDisposedExceptions on the TestMangementClient or on the CancellationTokenSource.

These exceptions just result into a build warning. But as a matter of fact only a subset of test results got published and the build still succeeds.

Repro

1. Input is a list of many trxfiles.
2. For each trx file write the ##vso command in a simple command shell task.

Expectation all test results get published. No warning, no error in the task output.
Last working agent version: v2.153
Upgraded version v2.174 does not work any longer

Narrow down the root cause

After having the repro I could search for the version which introduced the issue. The issue got introduced with with v2.160, by PR #2811. Here within the ResultsCommandExtension.cs a VssConnection is setup and now disposed by this change.

If I remove the using statement then it works again:

264:    using (var connection = WorkerUtilities.GetVssConnection(_executionContext))
265:    {

Assumption

I assume the VssConnection caches some data. Maybe the some data gets shared with other instances too. As soon as one connection gets disposed, this seems to lead to impact on other connections too. From the exception it looks like that the underlying socket gets closed, which might be shared with other connections too.

I guess that even the PR #2960 has its root cause here in the connection.
I think that issue #3199 talks about the same problem.

Proposal

Do not dispose the connection on a per call base. Either do not dispose at all (which would revert the original change) or somehow reuse the connection and dispose on an appropriate time.

Within the samples its also written to not close the connection:
https://docs.microsoft.com/en-us/azure/devops/integrate/get-started/client-libraries/samples?view=azure-devops

// Connection object could be created once per application and we use it to get httpclient objects. 
// Httpclients have been reused between callers and threads.
// Their lifetime has been managed by connection (we don't have to dispose them).
// This is more robust then newing up httpclient objects directly.  

This is also an issue which points in a similar direction: microsoft/azure-devops-dotnet-samples#287