Good day,
I trying to overcome an issue where the Azure Agents and Git Bash / Internal to Visual Studio cannot connect to the Azure Devops Server 2022. I can connect to Azure Devops Server 2022 from the web browser and from Team Exploer in Visual Studio. There are also several commits that have happened but cannot be pushed to Azure DevOps Server.
Here is the setup: Server for SQL: Windows Server 2019, SQL Server 2019 with 2 Instances. SPNs set for both instances of SQL Server 2019. One instance contains the AzureDevOps_Configuration and Collection databases.
Server for Azure DevOps:
Windows Sever 2019, IIS 10, Azure DevOps Server 2022 (19.205.33618.1 (Azure DevOps Server 2022.0.1)). SPNs set up for the AzureService domain account.
IIS Setup:
Authentication = All but Windows Authentication Disabled. Configuration Editor: system.webServer/security/authentication/windowsAuthentication authPersistNonNTLM = True, authPersistSingleRequest = False, enabled = True, Providers: Negotiate, NTLM, useAppPoolCredentials = True, useKernelMode = True. SSL Settings: Require SSL checked, Client certificates: Ignore. Temporarily set up Failed Request Tracing Rules for Status Codes 401,403. Bindings http port 80, https port 443 and current SSL Certificate.
Client(s): Windows 10 Pro domain joined. Visual Studio 2022 (17.7.6), Git for Windows 2.41.0.windows.3, Azure Agent 3.225.0. Connects to Azure DevOps Server 2022 with domain account. In Visual Studio Git Global Settings: Cryptographic network provider = Secure Channel, Credential helper = GCM. Environment > Accounts: Add and reauthenticate accounts using = Windows authentication broker.
Troubleshooting steps:
- Cleared the Windows Credentials Manager of all git and devops connections in the Generic Credentials section.
- Cleared the git config of Credential and credential.helper.
- In Git BASH ran git fetch origin --verbose and received: fatal: Authentication failed for 'https://devops.xxx.com/xxxDevelopment%20Team/Identity/_git/Identity/'
- Ran the following: GIT_TRACE=true GIT_TRACE_PACKET=true GIT_TRACE_CURL_NO_DATA=true GIT_CURL_VERBOSE=true git fetch origin --verbose
14:24:26.703049 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
14:24:26.709048 git.c:462 trace: built-in: git fetch origin --verbose
14:24:26.711088 run-command.c:661 trace: run_command: GIT_DIR=.git git remote-https origin 'https://devops.xxx.com/xxxTeam/Identity/_git/Identity'
14:24:26.719617 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:26.725618 git.c:748 trace: exec: git-remote-https origin 'https://devops.xxx.com/xxxDevelopment%20Team/Identity/_git/Identity'
14:24:26.725618 run-command.c:661 trace: run_command: git-remote-https origin 'https://devops.xxx.com/xxxDevelopment%20Team/Identity/_git/Identity'
14:24:26.734583 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:26.745631 http.c:843 == Info: Couldn't find host devops.xxx.com in the (nil) file; using defaults
14:24:26.751635 http.c:843 == Info: Trying 192.168.0.x:443...
14:24:26.767369 http.c:843 == Info: Connected to devops.xxx.com (192.168.0.x) port 443 (#0)
14:24:26.767369 http.c:843 == Info: schannel: disabled automatic use of client certificate
14:24:26.794563 http.c:843 == Info: using HTTP/1.x
14:24:26.794563 http.c:790 => Send header, 0000000270 bytes (0x0000010e)
14:24:26.794563 http.c:802 => Send header: GET /xxxDevelopment%20Team/Identity/_git/Identity/info/refs?service=git-upload-pack HTTP/1.1
14:24:26.794563 http.c:802 => Send header: Host: devops.xxx.com
14:24:26.794563 http.c:802 => Send header: User-Agent: git/2.41.0.windows.3
14:24:26.794563 http.c:802 => Send header: Accept: */*
14:24:26.794563 http.c:802 => Send header: Accept-Encoding: deflate, gzip, br, zstd
14:24:26.794563 http.c:802 => Send header: Pragma: no-cache
14:24:26.794563 http.c:802 => Send header: Git-Protocol: version=2
14:24:26.794563 http.c:802 => Send header:
14:24:26.819997 http.c:790 <= Recv header, 0000000027 bytes (0x0000001b)
14:24:26.819997 http.c:802 <= Recv header: HTTP/1.1 401 Unauthorized
14:24:26.819997 http.c:790 <= Recv header, 0000000025 bytes (0x00000019)
14:24:26.819997 http.c:802 <= Recv header: Content-Type: text/html
14:24:26.819997 http.c:790 <= Recv header, 0000000055 bytes (0x00000037)
14:24:26.819997 http.c:802 <= Recv header: X-TFS-ProcessId: 5536491d-fd8a-4654-a6b9-bb9e667ac916
14:24:26.819997 http.c:790 <= Recv header, 0000000064 bytes (0x00000040)
14:24:26.819997 http.c:802 <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains
14:24:26.819997 http.c:790 <= Recv header, 0000000050 bytes (0x00000032)
14:24:26.819997 http.c:802 <= Recv header: ActivityId: 97034b94-a0de-43e3-9768-fe811e29ee20
14:24:26.819997 http.c:790 <= Recv header, 0000000053 bytes (0x00000035)
14:24:26.819997 http.c:802 <= Recv header: X-TFS-Session: 97034b94-a0de-43e3-9768-fe811e29ee20
14:24:26.819997 http.c:790 <= Recv header, 0000000051 bytes (0x00000033)
14:24:26.819997 http.c:802 <= Recv header: X-VSS-E2EID: 97034b94-a0de-43e3-9768-fe811e29ee20
14:24:26.819997 http.c:790 <= Recv header, 0000000064 bytes (0x00000040)
14:24:26.819997 http.c:802 <= Recv header: X-VSS-SenderDeploymentId: 2e05db99-456f-4eba-bcd5-b34f4d5e3ccf
14:24:26.819997 http.c:790 <= Recv header, 0000000026 bytes (0x0000001a)
14:24:26.819997 http.c:802 <= Recv header: WWW-Authenticate: Bearer
14:24:26.819997 http.c:790 <= Recv header, 0000000061 bytes (0x0000003d)
14:24:26.819997 http.c:802 <= Recv header: WWW-Authenticate: Basic realm="https://devops.xxx.com/"
14:24:26.819997 http.c:790 <= Recv header, 0000000029 bytes (0x0000001d)
14:24:26.819997 http.c:802 <= Recv header: WWW-Authenticate: Negotiate
14:24:26.820949 http.c:790 <= Recv header, 0000000024 bytes (0x00000018)
14:24:26.820949 http.c:802 <= Recv header: WWW-Authenticate: NTLM
14:24:26.820949 http.c:790 <= Recv header, 0000000023 bytes (0x00000017)
14:24:26.820949 http.c:802 <= Recv header: X-Powered-By: ASP.NET
14:24:26.820949 http.c:790 <= Recv header, 0000000124 bytes (0x0000007c)
14:24:26.820949 http.c:802 <= Recv header: P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
14:24:26.820949 http.c:790 <= Recv header, 0000000024 bytes (0x00000018)
14:24:26.820949 http.c:802 <= Recv header: Lfs-Authenticate: NTLM
14:24:26.820949 http.c:790 <= Recv header, 0000000033 bytes (0x00000021)
14:24:26.820949 http.c:802 <= Recv header: X-Content-Type-Options: nosniff
14:24:26.820949 http.c:790 <= Recv header, 0000000037 bytes (0x00000025)
14:24:26.820949 http.c:802 <= Recv header: Date: Fri, 03 Nov 2023 18:24:26 GMT
14:24:26.820949 http.c:790 <= Recv header, 0000000022 bytes (0x00000016)
14:24:26.820949 http.c:802 <= Recv header: Content-Length: 1293
14:24:26.820949 http.c:790 <= Recv header, 0000000002 bytes (0x00000002)
14:24:26.820949 http.c:802 <= Recv header:
14:24:26.820949 http.c:843 == Info: Connection #0 to host devops.mrm2inc.com left intact
14:24:26.820949 run-command.c:661 trace: run_command: 'git credential-manager get'
14:24:27.393753 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.398766 git.c:748 trace: exec: git-credential-manager get
14:24:27.398766 run-command.c:661 trace: run_command: git-credential-manager get
14:24:27.459644 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.464644 git.c:462 trace: built-in: git config --null --list
14:24:27.568673 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.573675 git.c:462 trace: built-in: git version
14:24:27.584490 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.589528 git.c:462 trace: built-in: git config --null --type=path http.https://devops.mrm2inc.com.sslCAInfo
14:24:27.598503 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.604500 git.c:462 trace: built-in: git config --null --type=path http.devops.mrm2inc.com.sslCAInfo
14:24:27.613124 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.619158 git.c:462 trace: built-in: git config --null --type=path http.https://mrm2inc.com.sslCAInfo
14:24:27.629133 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.635130 git.c:462 trace: built-in: git config --null --type=path http.mrm2inc.com.sslCAInfo
14:24:27.645134 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.651129 git.c:462 trace: built-in: git config --null --type=path http.sslCAInfo
14:24:27.660129 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.666129 git.c:462 trace: built-in: git config --null --type=path http.https://devops.mrm2inc.com.cookieFile
14:24:27.675123 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.681124 git.c:462 trace: built-in: git config --null --type=path http.devops.mrm2inc.com.cookieFile
14:24:27.691128 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.697157 git.c:462 trace: built-in: git config --null --type=path http.https://mrm2inc.com.cookieFile
14:24:27.706123 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.712130 git.c:462 trace: built-in: git config --null --type=path http.mrm2inc.com.cookieFile
14:24:27.722124 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.727129 git.c:462 trace: built-in: git config --null --type=path http.cookieFile
14:24:27.809812 http.c:843 == Info: Found bundle for host: 0x2c085567ad0 [serially]
14:24:27.809812 http.c:843 == Info: Re-using existing connection #0 with host devops.mrm2inc.com
14:24:27.810811 http.c:790 => Send header, 0000000270 bytes (0x0000010e)
14:24:27.810811 http.c:802 => Send header: GET /xxxDevelopment%20Team/Identity/_git/Identity/info/refs?service=git-upload-pack HTTP/1.1
14:24:27.810811 http.c:802 => Send header: Host: devops.xxx.com
14:24:27.810811 http.c:802 => Send header: User-Agent: git/2.41.0.windows.3
14:24:27.810811 http.c:802 => Send header: Accept: */*
14:24:27.810811 http.c:802 => Send header: Accept-Encoding: deflate, gzip, br, zstd
14:24:27.810811 http.c:802 => Send header: Pragma: no-cache
14:24:27.810811 http.c:802 => Send header: Git-Protocol: version=2
14:24:27.810811 http.c:802 => Send header:
14:24:27.822442 http.c:790 <= Recv header, 0000000027 bytes (0x0000001b)
14:24:27.822442 http.c:802 <= Recv header: HTTP/1.1 401 Unauthorized
14:24:27.822442 http.c:790 <= Recv header, 0000000025 bytes (0x00000019)
14:24:27.822442 http.c:802 <= Recv header: Content-Type: text/html
14:24:27.822442 http.c:790 <= Recv header, 0000000055 bytes (0x00000037)
14:24:27.822442 http.c:802 <= Recv header: X-TFS-ProcessId: 5536491d-fd8a-4654-a6b9-bb9e667ac916
14:24:27.822442 http.c:790 <= Recv header, 0000000064 bytes (0x00000040)
14:24:27.822442 http.c:802 <= Recv header: Strict-Transport-Security: max-age=31536000; includeSubDomains
14:24:27.822442 http.c:790 <= Recv header, 0000000050 bytes (0x00000032)
14:24:27.822442 http.c:802 <= Recv header: ActivityId: 97034b96-a0de-43e3-9768-fe811e29ee20
14:24:27.822442 http.c:790 <= Recv header, 0000000053 bytes (0x00000035)
14:24:27.822442 http.c:802 <= Recv header: X-TFS-Session: 97034b96-a0de-43e3-9768-fe811e29ee20
14:24:27.822442 http.c:790 <= Recv header, 0000000051 bytes (0x00000033)
14:24:27.822442 http.c:802 <= Recv header: X-VSS-E2EID: 97034b96-a0de-43e3-9768-fe811e29ee20
14:24:27.822442 http.c:790 <= Recv header, 0000000064 bytes (0x00000040)
14:24:27.822442 http.c:802 <= Recv header: X-VSS-SenderDeploymentId: 2e05db99-456f-4eba-bcd5-b34f4d5e3ccf
14:24:27.822442 http.c:790 <= Recv header, 0000000026 bytes (0x0000001a)
14:24:27.822442 http.c:802 <= Recv header: WWW-Authenticate: Bearer
14:24:27.822442 http.c:790 <= Recv header, 0000000061 bytes (0x0000003d)
14:24:27.822442 http.c:802 <= Recv header: WWW-Authenticate: Basic realm="https://devops.xxx.com/"
14:24:27.822442 http.c:790 <= Recv header, 0000000029 bytes (0x0000001d)
14:24:27.822442 http.c:802 <= Recv header: WWW-Authenticate: Negotiate
14:24:27.822442 http.c:790 <= Recv header, 0000000024 bytes (0x00000018)
14:24:27.822442 http.c:802 <= Recv header: WWW-Authenticate: NTLM
14:24:27.822442 http.c:790 <= Recv header, 0000000023 bytes (0x00000017)
14:24:27.822442 http.c:802 <= Recv header: X-Powered-By: ASP.NET
14:24:27.822442 http.c:790 <= Recv header, 0000000124 bytes (0x0000007c)
14:24:27.822442 http.c:802 <= Recv header: P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
14:24:27.822442 http.c:790 <= Recv header, 0000000024 bytes (0x00000018)
14:24:27.822442 http.c:802 <= Recv header: Lfs-Authenticate: NTLM
14:24:27.822442 http.c:790 <= Recv header, 0000000033 bytes (0x00000021)
14:24:27.822442 http.c:802 <= Recv header: X-Content-Type-Options: nosniff
14:24:27.822442 http.c:790 <= Recv header, 0000000037 bytes (0x00000025)
14:24:27.822442 http.c:802 <= Recv header: Date: Fri, 03 Nov 2023 18:24:27 GMT
14:24:27.822442 http.c:790 <= Recv header, 0000000022 bytes (0x00000016)
14:24:27.822442 http.c:802 <= Recv header: Content-Length: 1293
14:24:27.822442 http.c:790 <= Recv header, 0000000002 bytes (0x00000002)
14:24:27.822442 http.c:802 <= Recv header:
14:24:27.822442 http.c:843 == Info: Ignoring the response-body
14:24:27.822442 http.c:843 == Info: Connection #0 to host devops.xxx.com left intact
14:24:27.822442 http.c:843 == Info: Issue another request to this URL: 'https://devops.xxx.com/xxxDevelopment%20Team/Identity/_git/Identity/info/refs?service=git-upload-pack'
14:24:27.822442 http.c:843 == Info: Found bundle for host: 0x2c085567ad0 [serially]
14:24:27.822442 http.c:843 == Info: Re-using existing connection #0 with host devops.mrm2inc.com
14:24:27.822442 http.c:843 == Info: Connection #0 to host devops.xxx.com left intact
14:24:27.822442 run-command.c:661 trace: run_command: 'git credential-manager erase'
14:24:27.883526 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.888531 git.c:748 trace: exec: git-credential-manager erase
14:24:27.888531 run-command.c:661 trace: run_command: git-credential-manager erase
14:24:27.940602 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:27.946607 git.c:462 trace: built-in: git config --null --list
14:24:28.024607 run-command.c:661 trace: run_command: 'git credential-manager erase'
14:24:28.080563 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:28.085603 git.c:748 trace: exec: git-credential-manager erase
14:24:28.085603 run-command.c:661 trace: run_command: git-credential-manager erase
14:24:28.139495 exec-cmd.c:243 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
14:24:28.145484 git.c:462 trace: built-in: git config --null --list
fatal: Authentication failed for 'https://devops.xxx.com/xxxDevelopment%20Team/Identity/_git/Identity/'
I never have gotten a popup for credetials when doing this. 5. Uninstalled Git for Windows. 6. Rebooted 7. Installed Git for Windows. 8. Rebooted 9. Tried above multiple times with no success.
Checking the Failed Request Log Files only shows the following:
ModuleName
IIS Web Core
Notification
AUTHENTICATE_REQUEST
HttpStatus
401
HttpReason
Unauthorized
HttpSubStatus
2
ErrorCode
Access is denied.
(0x80070005)
ConfigExceptionInfo
I have also tried the getting a PAT for Azure DevOps Server for multiple accounts but that has not worked either.
As for the Azure Agents when running Config and using Negotiate (using seperate domain account), I get to Enter agent pool and it can never find the pool. If I run it as Integrated I get past the pool but past the pool, but it fails for you are not authorized to access. Tried a PAT but it ends the same as with Negotiate.
Tried running network monitor and wireshark but I am not seeing anything in either of those that will have insite into why it is failing.
I have also tried suggestions from multiple different sources:
https://stackoverflow.com/questions/68862670/how-to-use-pat-on-azure-devops-server-to-clone-a-git-repo
https://stackoverflow.com/questions/34837173/authentication-failed-for-azure-git
https://stackoverflow.com/questions/53661818/fatal-authentication-failed-with-git-credential-manager
https://stackoverflow.com/questions/56772309/git-failed-with-a-fatal-error-authentication-failed-when-clone-repository-from-v
https://stackoverflow.com/questions/59726007/cant-push-local-git-to-azure-devops-git-repo-because-of-authentication-failed
https://stackoverflow.com/questions/70734634/authentication-failed-for-git-push-to-azure
https://stackoverflow.com/questions/62716476/azure-devops-onpremise-fatal-authentication-failed-for-when-cloning-git-repo
https://stackoverflow.com/questions/55406885/login-failed-when-trying-to-perform-git-push-to-azure-devops
https://stackoverflow.com/questions/62748505/unable-to-push-code-to-azure-devops-because-of-wrong-cached-credentials
https://stackoverflow.com/questions/62593521/how-to-authenticate-when-i-try-to-git-clone-from-azure-git
https://developercommunity.visualstudio.com/t/git-pullclone-fails-with-fatal-authentication-fail/436249?viewtype=all
https://developercommunity.visualstudio.com/t/always-get-authentication-failed-when-i-input-corr/436172
I also filed my own with MS but that is moving slowly (https://developercommunity.visualstudio.com/t/Azure-Devops-Server-2022-no-longer-allow/10496327?scope=follow)
None of what I have tried or read from the above has helped in being able to connect Agents and Git to Azure DevOps Server.
As a note the Generate Git Credentials button does not exist in Azure DevOps Server 2022.
Looking for any suggestions of what I haven't tried yet to be able to get things working again. Again as I stated before there are multiple commits that need to be pushed from multiple machines but nothing is able to connect. Also while I stated the Team Explorer is able to connect, if you exit Visual Studio you lose the connection there, and have to reconnect. This was not the case three weeks ago, when all these issues began.
Tried Uninstalling / reinstalling Git for Windows. Tried Personal Authentication Tokens Tried looking over Failed Request Logs Tried Wireshark / Network Monitor Tried removing credentials from Windows Credential Manager Nothing netted a connection to the Azure DevOps Server via Git or Azure agent.