Several of our repositories have recently started throwing unpack errors on push:
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 288 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
error: unpack failed: error Short read of block.
To http://git:8080/scm/git/Sandbox
! [remote rejected] master -> master (n/a (unpacker error))
error: failed to push some refs to 'http://git:8080/scm/git/Sandbox'
SCM Manager logs are showing an error that seems to match the above:
2016-02-24 17:54:39.483 [qtp1967563673-16] ERROR org.eclipse.jgit.internal.storage.file.ObjectDirectory - ERROR: Exception caught while accessing pack file /var/lib/scm/repositories/git/Sandbox/objects/pack/pack-00aca3dbe856f7f3f7e730a00a1dae29aa7f8070.pack, the pack file might be corrupt
java.io.EOFException: Short compressed stream at 533
at org.eclipse.jgit.internal.storage.file.PackFile.decompress(PackFile.java:341) ~[org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:734) ~[org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:259) ~[org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:417) [org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:386) [org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:378) [org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145) [org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229) [org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:840) [org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at org.eclipse.jgit.revwalk.RevWalk.parseTree(RevWalk.java:780) [org.eclipse.jgit-3.7.1.201504261725-r-scm1.jar:3.7.1.201504261725-r-scm1]
at sonia.scm.repository.spi.GitBrowseCommand.getResult(GitBrowseCommand.java:325) [scm-git-plugin-1.46.jar:na]
at sonia.scm.repository.spi.GitBrowseCommand.getBrowserResult(GitBrowseCommand.java:147) [scm-git-plugin-1.46.jar:na]
at sonia.scm.repository.api.BrowseCommandBuilder.getBrowserResult(BrowseCommandBuilder.java:174) [scm-core-1.46.jar:na]
at sonia.scm.api.rest.resources.RepositoryResource.getBrowserResult(RepositoryResource.java:564) [classes/:na]
at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_09-icedtea]
at java.lang.reflect.Method.invoke(Method.java:601) ~[na:1.7.0_09-icedtea]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) [javax.servlet-2.5.0.v201103041518.jar:na]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at sonia.scm.filter.MDCFilter.doFilter(MDCFilter.java:96) [classes/:na]
at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:102) [scm-core-1.46.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at sonia.scm.filter.SecurityFilter.doFilter(SecurityFilter.java:111) [classes/:na]
at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:102) [scm-core-1.46.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at sonia.scm.web.filter.BasicAuthenticationFilter.processRequest(BasicAuthenticationFilter.java:423) [scm-core-1.46.jar:na]
at sonia.scm.web.filter.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:214) [scm-core-1.46.jar:na]
at sonia.scm.web.security.ApiBasicAuthenticationFilter.doFilter(ApiBasicAuthenticationFilter.java:116) [classes/:na]
at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:102) [scm-core-1.46.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at sonia.scm.filter.GZipFilter.doFilter(GZipFilter.java:112) [scm-core-1.46.jar:na]
at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:102) [scm-core-1.46.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at sonia.scm.web.filter.AutoLoginFilter.doFilter(AutoLoginFilter.java:87) [scm-core-1.46.jar:na]
at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:102) [scm-core-1.46.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at sonia.scm.filter.BaseUrlFilter.doFilter(BaseUrlFilter.java:117) [classes/:na]
at sonia.scm.web.filter.HttpFilter.doFilter(HttpFilter.java:102) [scm-core-1.46.jar:na]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [shiro-web-1.2.3.jar:1.2.3]
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.2.3.jar:1.2.3]
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.2.3.jar:1.2.3]
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.2.3.jar:1.2.3]
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [shiro-core-1.2.3.jar:1.2.3]
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [shiro-web-1.2.3.jar:1.2.3]
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.2.3.jar:1.2.3]
at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) [guice-servlet-3.0.jar:na]
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) [guice-servlet-3.0.jar:na]
at sonia.scm.boot.BootstrapFilter.doFilter(BootstrapFilter.java:119) [classes/:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288) [jetty-servlet-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) [jetty-servlet-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:556) [jetty-security-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) [jetty-servlet-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.Server.handle(Server.java:369) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) [jetty-http-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) [jetty-http-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) [jetty-io-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) [jetty-io-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-7.6.16.v20140903.jar:7.6.16.v20140903]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-7.6.16.v20140903.jar:7.6.16.v20140903]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]
Interestingly, native git seems to have no issues with these repositories. I can still clone them, and their history as shown by git log seems intact. If I create a local copy with git clone --bare and set that as my remote, I can push to this local copy successfully.
git fsck showed two dangling commits and a dangling tag, which disappeared after a git gc. However, this did not fix the problem.
I've tried using git push --no-thin, based on e.g. git unpack error on push to gerrit, but got the same error as above.
I've also tried creating a new, empty repository on the SCM Manager side, and setting it as the remote for a local copy of a problematic repo. The initial push seems to succeed, but any further pushes to the new repository result in the same error.
Since local pushes using native git still work, it seems to me that these repositories should be salvageable. But I'm at a loss as to how to proceed. Any ideas on either fixing these repositories or salvaging as much of their history as possible would be greatly appreciated.
I faced the same problem and fixed it by adding a new commit (comitting an additional dot in a comment :) ).
my error message: