1. 발생증상

IO Exception: "java.io.IOException: Block not found in id [1, -13, 23, -57, 1] [1.4.188/50]"; "lob: null table: -3 id: 5868" [90031-188] 90031/90031 (Help)
org.h2.jdbc.JdbcSQLException: IO Exception: "java.io.IOException: Block not found in id [1, -13, 23, -57, 1] [1.4.188/50]"; "lob: null table: -3 id: 5868" [90031-188]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
    at org.h2.message.DbException.get(DbException.java:168)
    at org.h2.message.DbException.convertIOException(DbException.java:330)
    at org.h2.value.ValueLobDb.getString(ValueLobDb.java:306)
    at org.h2.jdbc.JdbcResultSet.getString(JdbcResultSet.java:285)
    at org.h2.server.web.WebApp.escapeData(WebApp.java:1773)
    at org.h2.server.web.WebApp.getResultSet(WebApp.java:1686)
    at org.h2.server.web.WebApp.getResult(WebApp.java:1405)
    at org.h2.server.web.WebApp.query(WebApp.java:1061)
    at org.h2.server.web.WebApp.query(WebApp.java:1037)
    at org.h2.server.web.WebApp.process(WebApp.java:223)
    at org.h2.server.web.WebApp.processRequest(WebApp.java:170)
    at org.h2.server.web.WebServlet.doGet(WebServlet.java:125)
    at org.h2.server.web.WebServlet.doPost(WebServlet.java:162)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
    at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:224)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:295)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:69)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Block not found in id [1, -13, 23, -57, 1] [1.4.188/50]
    at org.h2.mvstore.StreamStore$Stream.read(StreamStore.java:466)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:154)
    at java.io.BufferedReader.read1(BufferedReader.java:205)
    at java.io.BufferedReader.read(BufferedReader.java:279)
    at org.h2.util.IOUtils.copyAndCloseInput(IOUtils.java:200)
    at org.h2.util.IOUtils.readStringAndClose(IOUtils.java:307)
    at org.h2.value.ValueLobDb.getString(ValueLobDb.java:296)
    ... 79 more
Caused by: java.lang.IllegalStateException: Block 199 not found [1.4.188/50]
    at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:773)
    at org.h2.mvstore.StreamStore.getBlock(StreamStore.java:388)
    at org.h2.mvstore.StreamStore$Stream.nextBuffer(StreamStore.java:503)
    at org.h2.mvstore.StreamStore$Stream.read(StreamStore.java:460)
    ... 89 more

2. 설정

  • 현재 H2Database 옵션은 다음과 같다.

jdbc:h2:file:~/.h2database/db;CACHE_SIZE=10240;DB_CLOSE_DELAY=-1;AUTO_SERVER=TRUE;DB_CLOSE_ON_EXIT=FALSE;MVCC=TRUE;LOCK_TIMEOUT=15000;

인터넷을 찾아보면 MVCC=TRUE 옵션을 사용하지 말라는 이야기가 있지만, 현재 사용하고 있는 Qaurtz Scheduler DBStore 옵션에서는 MVCC=TRUE 옵션을 권장하고 있기에 문제를 찾기 위한 방법을 모색했다.

3. 해결방법

h2database의 lob으로 설정한 컬럼을 null로 저장되지 않도록 처리했다.

이후 추이를 지켜보도록 한다.


20140527 Jenkins 재시작 후 데이터 초기화 증상 해결

사내에서 사용하던 Jenkins가 구동되고 있는 서버가 정전으로 재가동 후 최초 설치상태로 초기화되는 현상이 나타났다.

멘붕!!!

Jenkins의 이런 증상을 처음 경험한 나로서는 어찌해야하는 물어볼 사람도 없고 인터넷 검색으로 뒤지기 시작한다.
그러다가 [Jenkins 관리 - 환경설정] 부분 상단에 HOME Directory 라는 항목의 값이 눈에 들어온다.

/{USER_HOME}/.hudson

이녀석! 이상하다! 내가 쓰고 있는 건 jenkins 인데!?

라는 생각과 함께 JENKINS_HOME 과 관련된 검색에 들어간다. +_+) 그러다가 아래 녀석을 찾았다.

여기서 말하는 JENKINS_HOME은 젠킨스가 실행되는 경로가 아닌, 젠킨스가 사용자 디렉토리에 생성한 ~/.jenkins 폴더를 가리키는 것이었다.
이후 이를 해결하는 방법은 어려움이 없었다. 환경설정에서 ‘홈 디렉토리Home Directory’ 항목의 오른쪽 물음표(?)를 클릭하면 JENKINS_HOME을 WAS에 설정하라는 안내를 한다.

Jenkins stores all the data files inside this directory in the file system. You can change this by either:
    1. Using your web container's admin tool to set the **JENKINS_HOME** environment entry.
    2. Setting the environment variable **JENKINS_HOME** before launching your web container.
    3. (Not recommended) modifying web.xml in jenkins.war (or its expanded image in your web container).
This value cannot be changed while Jenkins is running. This entry is mostly for you to make sure that your configuration is taking effect.

이후 문제는 쉽게해결 가능하다. 톰캣 서버의 ‘./bin’ 폴더에서 catalina.sh를 열어서

set JENKINS_HOME=/{USER_HOME}/.jenkins

항목만 추가해주고 톰캣을 구동시켜주면 끝이났다. +_+)b That's OK~!

이 문제를 해결하면서 젠킨스의 버전도 스리슬쩍 1.564 버전으로 변경했다.

데이터가 분리되어 있으니 젠킨스를 업그레이드 해도 별문제가 없을 거라고 생각했다.

찍었는데 맞았다!


 

 작업전 확인사항 

 

  • 사용자 폴더 밑에 '~/.jenkins' 라는 폴더가 있는지 확인
    • 젠킨스 설정 데이터가 저장되는 폴더다
  • 젠킨스를 구동하는 WAS의 실행환경변수로 'JENKINS_HOME'이 추가되어 있는지 확인



+ Recent posts