Closed
Description
Hi,
hhvm crashes regularly on a high load webapp (/www.XXX/) once or twice a day, quite more often when switching another app (/other.XXX/) sharing the same database and controller/model from php-cgi to hhvm too. There's no way to reproduce this, as it affects pdf files generated by mPDF, normal sites, GET and POST methods on random pages. Since it's monitored and automatically restarted, that's not so bad. Not sure if this relates to bug #2721. But sometimes, like once or twice a week, the hhvm daemon just get unresponsible for 10, 15, 20 minutes until manually restarted without consuming any additional cpu or memory. Clients receive a 502 error after timeout connecting to hhvm backend (nginx 1.9.2 behind nghttp2 1.0.4 as tls offloader).
HHVM 3.7.2~jessie
System: debian-testing/amd64
libc6 2.19-18
let me know if you need more information.
kern.log & stacktrace from the last few days:
[2015-06-18 18:54:56] hhvm[30919]: segfault at 7f0100000060 ip 00007fafd7d0eb7f sp 00007fafadbf9eb0 error 4 in libc-2.19.so[7fafd7ca5000+19f000]
[2015-06-19 08:44:41] hhvm[15651]: segfault at 8 ip 00007f203de77385 sp 00007f20247f9840 error 4 in libc-2.19.so[7f203de31000+19f000]
[2015-06-20 12:42:38] hhvm[5467] general protection ip:7f72672c1b36 sp:7f7241bf9ef0 error:0 in libc-2.19.so[7f7267258000+19f000]
[2015-06-20 19:25:32] hhvm[21629] general protection ip:144565a sp:7fe697bf89e0 error:0 in hhvm[400000+34e5000]
[2015-06-21 23:07:24] hhvm[12651]: segfault at 8 ip 00007fcf8a75f385 sp 00007fcf69ff9840 error 4 in libc-2.19.so[7fcf8a719000+19f000]
Host: XXX
ProcessID: 11214
ThreadID: 7fcf69fff700
ThreadPID: 12651
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0
Server: www.XXX
ThreadType: Web Request
Server_SERVER_NAME: www.XXX
URL: (normal URL)
#0 fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#1 HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#2 std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#3 std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#4 HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#5 HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#6 HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#7 HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#8 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#9 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#10 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#11 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#12 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#13 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#14 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#16 clone at /lib/x86_64-linux-gnu/libc.so.6:0
PHP Stacktrace:
Host: XXX
ProcessID: 15611
ThreadID: 7f20247ff700
ThreadPID: 15651
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0
Server: other.XXX
ThreadType: Web Request
Server_SERVER_NAME: other.XXX
URL: (GET method)
#0 fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#1 HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#2 std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#3 std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#4 HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#5 HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#6 HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#7 HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#8 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#9 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#10 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#11 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#12 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#13 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#14 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#16 clone at /lib/x86_64-linux-gnu/libc.so.6:0
PHP Stacktrace:
Host: XXX
ProcessID: 21522
ThreadID: 7fe697bff700
ThreadPID: 21629
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0
Server: www.XXX
ThreadType: Web Request
Server_SERVER_NAME: www.XXX
URL: (PDF file (mPDF Library))
#0 fgets at /lib/x86_64-linux-gnu/libc.so.6:0
#1 BIO_fd_should_retry at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#2 PEM_read_bio at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#3 PEM_X509_INFO_read_bio at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#4 X509_load_cert_crl_file at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#5 X509_load_cert_crl_file at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#6 X509_STORE_load_locations at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#7 curl_slist_free_all at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#8 curl_slist_free_all at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#9 curl_formget at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#10 curl_formget at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#11 curl_multi_remove_handle at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#12 curl_multi_perform at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#13 curl_easy_perform at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#14 HPHP::HttpClient::request(char const*, char const*, char const*, int, HPHP::StringBuffer&, std::unordered_map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, HPHP::string_hashi, HPHP::stringlike_eqstri<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const*, std::vector<HPHP::String, std::allocator<HPHP::String> >*) at /usr/bin/hhvm:0
#15 HPHP::HttpClient::get(char const*, HPHP::StringBuffer&, std::unordered_map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, HPHP::string_hashi, HPHP::stringlike_eqstri<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const*, std::vector<HPHP::String, std::allocator<HPHP::String> >*) at /usr/bin/hhvm:0
#16 HPHP::UrlFile::open(HPHP::String const&, HPHP::String const&) at /usr/bin/hhvm:0
#17 HPHP::HttpStreamWrapper::open(HPHP::String const&, HPHP::String const&, int, HPHP::SmartPtr<HPHP::StreamContext> const&) at /usr/bin/hhvm:0
#18 HPHP::File::Open(HPHP::String const&, HPHP::String const&, int, HPHP::SmartPtr<HPHP::StreamContext> const&) at /usr/bin/hhvm:0
#19 HPHP::f_fopen(HPHP::String const&, HPHP::String const&, bool, HPHP::Variant const&) at /usr/bin/hhvm:0
#20 HPHP::jit::x64::(anonymous namespace)::BackEnd::enterTCHelper(unsigned char*, HPHP::ActRec*) at back-end-x64.cpp:0
#21 HPHP::jit::MCGenerator::enterTC(unsigned char*, HPHP::ActRec*) at /usr/bin/hhvm:0
#22 HPHP::enterVM(HPHP::ActRec*, HPHP::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at bytecode.cpp:0
#23 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /usr/bin/hhvm:0
#24 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/bin/hhvm:0
#25 HPHP::invoke_file(HPHP::String const&, bool, char const*) at builtin-functions.cpp:0
#26 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/bin/hhvm:0
#27 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /usr/bin/hhvm:0
#28 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#29 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#30 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#31 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#32 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#33 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#34 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#35 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#36 clone at /lib/x86_64-linux-gnu/libc.so.6:0
Host: XXX
ProcessID: 30825
ThreadID: 7fafadbff700
ThreadPID: 30919
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0
Server: other.XXX
ThreadType: Web Request
Server_SERVER_NAME: other.XXX
URL: (POST method)
#0 _IO_file_setbuf at /lib/x86_64-linux-gnu/libc.so.6:0
#1 _IO_do_write at /lib/x86_64-linux-gnu/libc.so.6:0
#2 _IO_file_close_it at /lib/x86_64-linux-gnu/libc.so.6:0
#3 fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#4 HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#5 std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#6 std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#7 HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#8 HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#9 HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#10 HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#11 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#12 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#13 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#14 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#15 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#16 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#17 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#18 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#19 clone at /lib/x86_64-linux-gnu/libc.so.6:0
PHP Stacktrace:
Host: XXX
ProcessID: 3733
ThreadID: 7f7241bff700
ThreadPID: 5467
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0
Server: www.XXX
ThreadType: Web Request
Server_SERVER_NAME: www.XXX
URL: (GET method)
#0 fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#1 HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#2 std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#3 std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#4 HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#5 HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#6 HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#7 HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#8 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#9 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#10 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#11 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#12 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#13 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#14 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#16 clone at /lib/x86_64-linux-gnu/libc.so.6:0
PHP Stacktrace: