Update to grpc v1.55.0 causes GDK in macOS to crash when a file is created in a project
For some reason, the upgrade to gprc v1.55.0 in !121899 (merged) appears to crash Puma after attempting create a file in the project:
2023-05-30_17:04:54.01856 rails-web : objc[60607]: +[__NSTimeZone initialize] may have been in progress in another thread when fork() was called.
2023-05-30_17:04:54.01926 rails-web : objc[60607]: +[__NSTimeZone initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.Setting export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES via env.runit appears to work around the problem.
We upgraded from grpc v1.42.0. I suspect v1.48.0 and up have this issue.
Note that we purposely avoided certain versions of grpc due to concerns over memory leaks (https://github.com/grpc/grpc/issues/28513):
https://github.com/grpc/grpc/issues/31240 suggests that this is not a grpc issue but rather an issue with os.fork() in macOS. It looks like the grpc update might have updated abseil-cpp (https://github.com/abseil/abseil-cpp/releases), which might be initializing the time zone.
Other links:
Activity
I remember running into issue in GET as well. Other links:
- https://blog.phusion.nl/2017/10/13/why-ruby-app-servers-break-on-macos-high-sierra-and-what-can-be-done-about-it/
- gitlab-environment-toolkit#419 (comment 914411397)
- https://github.com/puma/puma/issues/1421
- https://bugs.ruby-lang.org/issues/14009
It seems we used to have a workaround in Unicorn: gitlab-foss!16649 (diffs)
The workaround is in
config/initializers/macos.rbnow.Edited by Stan Hu-
This is the
lldbbacktrace:(lldb) bt * thread #46, name = 'puma srv tp 002', stop reason = signal SIGABRT * frame #0: 0x0000000187124d4c libsystem_kernel.dylib`__abort_with_payload + 8 frame #1: 0x0000000187149e50 libsystem_kernel.dylib`abort_with_payload_wrapper_internal + 104 frame #2: 0x0000000187149de8 libsystem_kernel.dylib`abort_with_reason + 32 frame #3: 0x0000000186de9504 libobjc.A.dylib`_objc_fatalv(unsigned long long, unsigned long long, char const*, char*) + 128 frame #4: 0x0000000186de9484 libobjc.A.dylib`_objc_fatal(char const*, ...) + 44 frame #5: 0x0000000186dbed54 libobjc.A.dylib`initializeNonMetaClass + 1088 frame #6: 0x0000000186dd90e8 libobjc.A.dylib`initializeAndMaybeRelock(objc_class*, objc_object*, locker_mixin<lockdebug::lock_mixin<objc_lock_base_t> >&, bool) + 156 frame #7: 0x0000000186dbe5c4 libobjc.A.dylib`lookUpImpOrForward + 884 frame #8: 0x0000000186dbdf64 libobjc.A.dylib`_objc_msgSend_uncached + 68 frame #9: 0x0000000187206768 CoreFoundation`__NSTimeZone_newWithCache + 100 frame #10: 0x0000000187206384 CoreFoundation`-[__NSPlaceholderTimeZone __initWithName:cache:] + 116 frame #11: 0x000000018720626c CoreFoundation`+[NSTimeZone timeZoneWithName:] + 32 frame #12: 0x00000001872061a8 CoreFoundation`+[NSTimeZone systemTimeZone] + 528 frame #13: 0x0000000187205f38 CoreFoundation`+[NSTimeZone defaultTimeZone] + 76 frame #14: 0x0000000187205ec0 CoreFoundation`CFTimeZoneCopyDefault + 36 frame #15: 0x00000001254c5b80 grpc_c.bundle`___lldb_unnamed_symbol11261 + 36 frame #16: 0x00000001254b90d4 grpc_c.bundle`___lldb_unnamed_symbol11114 + 32 frame #17: 0x000000012539ccc8 grpc_c.bundle`___lldb_unnamed_symbol6391 + 1496 frame #18: 0x0000000125488fa8 grpc_c.bundle`___lldb_unnamed_symbol10443 + 204 frame #19: 0x000000012539bb68 grpc_c.bundle`___lldb_unnamed_symbol6383 + 344 frame #20: 0x0000000125445a0c grpc_c.bundle`___lldb_unnamed_symbol9229 + 460 frame #21: 0x000000012541a974 grpc_c.bundle`___lldb_unnamed_symbol8550 + 124 frame #22: 0x000000012541b770 grpc_c.bundle`___lldb_unnamed_symbol8555 + 1072 frame #23: 0x000000012541c5d4 grpc_c.bundle`___lldb_unnamed_symbol8564 + 72 frame #24: 0x0000000125422f0c grpc_c.bundle`___lldb_unnamed_symbol8682 + 36 frame #25: 0x00000001252c58f4 grpc_c.bundle`___lldb_unnamed_symbol3487 + 108 frame #26: 0x00000001252283e0 grpc_c.bundle`___lldb_unnamed_symbol772 + 220 frame #27: 0x00000001253acedc grpc_c.bundle`___lldb_unnamed_symbol6704 + 124 frame #28: 0x0000000125422014 grpc_c.bundle`___lldb_unnamed_symbol8629 + 344 frame #29: 0x0000000125215ed0 grpc_c.bundle`___lldb_unnamed_symbol455 + 1160 frame #30: 0x000000010552a4e4 libruby.3.1.dylib`vm_call_cfunc_with_frame + 232 frame #31: 0x000000010552cc08 libruby.3.1.dylib`vm_sendish + 1336 frame #32: 0x000000010550f53c libruby.3.1.dylib`vm_exec_core + 8128 frame #33: 0x0000000105521d2c libruby.3.1.dylib`rb_vm_exec + 2212 frame #34: 0x000000010553011c libruby.3.1.dylib`rb_call0 + 772 frame #35: 0x000000010551c314 libruby.3.1.dylib`rb_iterate0 + 628 frame #36: 0x000000010551bf84 libruby.3.1.dylib`rb_block_call_kw + 116 frame #37: 0x0000000105519560 libruby.3.1.dylib`vm_call0_body + 988 frame #38: 0x000000010553011c libruby.3.1.dylib`rb_call0 + 772 frame #39: 0x000000010551c314 libruby.3.1.dylib`rb_iterate0 + 628 frame #40: 0x000000010551bee8 libruby.3.1.dylib`rb_block_call + 116 frame #41: 0x000000010537e7e4 libruby.3.1.dylib`next_i + 112 frame #42: 0x000000010552b560 libruby.3.1.dylib`vm_yield_with_cfunc + 208 frame #43: 0x00000001053601b4 libruby.3.1.dylib`rb_fiber_start + 620 frame #44: 0x00000001053621f4 libruby.3.1.dylib`fiber_entry + 36A better backtrace when the gem is compiled with
GRPC_CONFIG=dbg:(lldb) bt * thread #33, name = 'puma srv tp 003', stop reason = signal SIGABRT * frame #0: 0x0000000187124d4c libsystem_kernel.dylib`__abort_with_payload + 8 frame #1: 0x0000000187149e50 libsystem_kernel.dylib`abort_with_payload_wrapper_internal + 104 frame #2: 0x0000000187149de8 libsystem_kernel.dylib`abort_with_reason + 32 frame #3: 0x0000000186de9504 libobjc.A.dylib`_objc_fatalv(unsigned long long, unsigned long long, char const*, char*) + 128 frame #4: 0x0000000186de9484 libobjc.A.dylib`_objc_fatal(char const*, ...) + 44 frame #5: 0x0000000186dbed54 libobjc.A.dylib`initializeNonMetaClass + 1088 frame #6: 0x0000000186dd90e8 libobjc.A.dylib`initializeAndMaybeRelock(objc_class*, objc_object*, locker_mixin<lockdebug::lock_mixin<objc_lock_base_t> >&, bool) + 156 frame #7: 0x0000000186dbe5c4 libobjc.A.dylib`lookUpImpOrForward + 884 frame #8: 0x0000000186dbdf64 libobjc.A.dylib`_objc_msgSend_uncached + 68 frame #9: 0x0000000187206768 CoreFoundation`__NSTimeZone_newWithCache + 100 frame #10: 0x0000000187206384 CoreFoundation`-[__NSPlaceholderTimeZone __initWithName:cache:] + 116 frame #11: 0x000000018720626c CoreFoundation`+[NSTimeZone timeZoneWithName:] + 32 frame #12: 0x00000001872061a8 CoreFoundation`+[NSTimeZone systemTimeZone] + 528 frame #13: 0x0000000187205f38 CoreFoundation`+[NSTimeZone defaultTimeZone] + 76 frame #14: 0x0000000187205ec0 CoreFoundation`CFTimeZoneCopyDefault + 36 frame #15: 0x000000015e668504 grpc_c.bundle`absl::lts_20220623::time_internal::cctz::local_time_zone() at time_zone_lookup.cc:139:30 frame #16: 0x000000015e649884 grpc_c.bundle`absl::lts_20220623::LocalTimeZone() at time.h:1110:19 frame #17: 0x000000015e649848 grpc_c.bundle`absl::lts_20220623::FormatTime(t=Time @ 0x00000001043ed868) at format.cc:88:44 frame #18: 0x000000015e423380 grpc_c.bundle`grpc_core::StatusToString(this=0x00000001043ee5b8, type_url=(ptr_ = "created_time", length_ = 12), payload=0x00000002976eac08)::$_0::operator()(absl::lts_20220623::string_view, absl::lts_20220623::Cord const&) const at status_helper.cc:321:45 frame #19: 0x000000015e422f2c grpc_c.bundle`decltype(f=0x00000001043ee5b8, args=0x00000001043ee2e8, args=0x00000002976eac08)::$_0 const&>()(std::declval<absl::lts_20220623::string_view>(), std::declval<absl::lts_20220623::Cord const&>())) absl::lts_20220623::base_internal::Callable::Invoke<grpc_core::StatusToString(absl::lts_20220623::Status const&)::$_0 const&, absl::lts_20220623::string_view, absl::lts_20220623::Cord const&>(grpc_core::StatusToString(absl::lts_20220623::Status const&)::$_0 const&, absl::lts_20220623::string_view&&, absl::lts_20220623::Cord const&) at invoke.h:185:12 frame #20: 0x000000015e422ee8 grpc_c.bundle`decltype(f=0x00000001043ee5b8, args=0x00000001043ee2e8, args=0x00000002976eac08)::$_0 const&, absl::lts_20220623::string_view, absl::lts_20220623::Cord const&>::type::Invoke(std::declval<grpc_core::StatusToString(absl::lts_20220623::Status const&)::$_0 const&>(), std::declval<absl::lts_20220623::string_view>(), std::declval<absl::lts_20220623::Cord const&>())) absl::lts_20220623::base_internal::invoke<grpc_core::StatusToString(absl::lts_20220623::Status const&)::$_0 const&, absl::lts_20220623::string_view, absl::lts_20220623::Cord const&>(grpc_core::StatusToString(absl::lts_20220623::Status const&)::$_0 const&, absl::lts_20220623::string_view&&, absl::lts_20220623::Cord const&) at invoke.h:212:10 frame #21: 0x000000015e422ea4 grpc_c.bundle`void absl::lts_20220623::functional_internal::InvokeObject<grpc_core::StatusToString(absl::lts_20220623::Status const&)::$_0, void, absl::lts_20220623::string_view, absl::lts_20220623::Cord const&>(ptr=VoidPtr @ 0x00000001043ee2f8, args=(ptr_ = "type.googleapis.com/grpc.status.time.created_time", length_ = 49), args=0x00000002976eac08) at function_ref.h:74:7 frame #22: 0x000000015e5d56c8 grpc_c.bundle`absl::lts_20220623::FunctionRef<void (absl::lts_20220623::string_view, absl::lts_20220623::Cord const&)>::operator(this=0x00000001043ee3e0, args=(ptr_ = "type.googleapis.com/grpc.status.time.created_time", length_ = 49), args=0x00000002976eac08)(absl::lts_20220623::string_view, absl::lts_20220623::Cord const&) const at function_ref.h:132:12 frame #23: 0x000000015e5d55fc grpc_c.bundle`absl::lts_20220623::Status::ForEachPayload(this=0x00000001043ee980, visitor=FunctionRef<void (absl::lts_20220623::string_view, const absl::lts_20220623::Cord &)> @ 0x00000001043ee3e0)>) const at status.cc:184:7 frame #24: 0x000000015e421348 grpc_c.bundle`grpc_core::StatusToString(status=0x00000001043ee980) at status_helper.cc:293:10 frame #25: 0x000000015e436880 grpc_c.bundle`grpc_error_std_string(error=<unavailable>) at error.cc:68:10 frame #26: 0x000000015e5777a0 grpc_c.bundle`grpc_error_get_status(error=(rep_ = 6417858225), deadline=(millis_ = 53301), code=0x00000002982e0a24, message="", http_error=0x0000000000000000, error_string=0x00000002982e0a48) at error_utils.cc:123:32 frame #27: 0x000000015e51fbc4 grpc_c.bundle`grpc_core::FilterStackCall::SetFinalStatus(this=0x000000015a5c6270, error=(rep_ = 6417858225)) at call.cc:804:5 frame #28: 0x000000015e520c7c grpc_c.bundle`grpc_core::FilterStackCall::RecvTrailingFilter(this=0x000000015a5c6270, b=0x000000015a5c6a38, batch_error=(rep_ = 0)) at call.cc:978:7 frame #29: 0x000000015e522524 grpc_c.bundle`grpc_core::FilterStackCall::BatchControl::ReceivingTrailingMetadataReady(this=0x000000015a5ca140, error=(rep_ = 0)) at call.cc:1285:10 frame #30: 0x000000015e52c368 grpc_c.bundle`grpc_core::FilterStackCall::StartBatch(this=0x000000015a5ca140, bctl=0x000000015a5ca140, error=<unavailable>)::$_8::operator()(void*, absl::lts_20220623::Status) const at call.cc:1644:9 frame #31: 0x000000015e52c320 grpc_c.bundle`grpc_core::FilterStackCall::StartBatch(bctl=0x000000015a5ca140, error=<unavailable>)::$_8::__invoke(void*, absl::lts_20220623::Status) at call.cc:1644:9 frame #32: 0x000000015e5218cc grpc_c.bundle`grpc_core::Closure::Run(location=0x00000001043eef90, closure=0x000000015a5c7018, error=(rep_ = 0)) at closure.h:265:5 frame #33: 0x000000015e2a6188 grpc_c.bundle`recv_trailing_metadata_ready(arg=0x000000015a5c7180, error=(rep_ = 0)) at deadline_filter.cc:145:3 frame #34: 0x000000015e5218cc grpc_c.bundle`grpc_core::Closure::Run(location=0x00000001043ef0b8, closure=0x000000015a5c71a0, error=(rep_ = 0)) at closure.h:265:5 frame #35: 0x000000015e0bd9f4 grpc_c.bundle`grpc_core::ClientChannel::CallData::RecvTrailingMetadataReadyForConfigSelectorCommitCallback(arg=0x000000015a5c7160, error=(rep_ = 0)) at client_channel.cc:2312:3 frame #36: 0x000000015e43d958 grpc_c.bundle`exec_ctx_run(closure=0x000000015a5c72b8) at exec_ctx.cc:44:3 frame #37: 0x000000015e43d7c0 grpc_c.bundle`grpc_core::ExecCtx::Flush(this=0x00000001043ef280) at exec_ctx.cc:79:9 frame #38: 0x000000015e4967bc grpc_c.bundle`grpc_core::ExecCtx::~ExecCtx(this=0x00000001043ef280) at exec_ctx.h:117:5 frame #39: 0x000000015e495200 grpc_c.bundle`grpc_core::ExecCtx::~ExecCtx(this=0x00000001043ef280) at exec_ctx.h:115:22 frame #40: 0x000000015e524170 grpc_c.bundle`::grpc_call_start_batch(call=0x000000015a5c6270, ops=0x00000002982e0730, nops=1, tag=0x00000001043ef3b8, reserved=0x0000000000000000) at call.cc:1844:3 frame #41: 0x000000015e097a44 grpc_c.bundle`grpc_rb_call_run_batch + 296 frame #42: 0x0000000100f224e4 libruby.3.1.dylib`vm_call_cfunc_with_frame + 232 frame #43: 0x0000000100f24c08 libruby.3.1.dylib`vm_sendish + 1336 frame #44: 0x0000000100f0753c libruby.3.1.dylib`vm_exec_core + 8128 frame #45: 0x0000000100f19d2c libruby.3.1.dylib`rb_vm_exec + 2212 frame #46: 0x0000000100f2811c libruby.3.1.dylib`rb_call0 + 772 frame #47: 0x0000000100f14314 libruby.3.1.dylib`rb_iterate0 + 628 frame #48: 0x0000000100f13f84 libruby.3.1.dylib`rb_block_call_kw + 116 frame #49: 0x0000000100f11560 libruby.3.1.dylib`vm_call0_body + 988 frame #50: 0x0000000100f2811c libruby.3.1.dylib`rb_call0 + 772 frame #51: 0x0000000100f14314 libruby.3.1.dylib`rb_iterate0 + 628 frame #52: 0x0000000100f13ee8 libruby.3.1.dylib`rb_block_call + 116 frame #53: 0x0000000100d767e4 libruby.3.1.dylib`next_i + 112 frame #54: 0x0000000100f23560 libruby.3.1.dylib`vm_yield_with_cfunc + 208 frame #55: 0x0000000100d581b4 libruby.3.1.dylib`rb_fiber_start + 620 frame #56: 0x0000000100d5a1f4 libruby.3.1.dylib`fiber_entry + 36Edited by Stan Hu -
It seems the problem is retrieving the default time zone via the macOS APIs: https://github.com/abseil/abseil-cpp/blob/78be63686ba732b25052be15f8d6dee891c05749/absl/time/internal/cctz/src/time_zone_lookup.cc#L139
I tried doing this, but I suspect loading the framework is not enough; we need to call the macOS API ourselves before the fork.
diff --git a/config/initializers/macos.rb b/config/initializers/macos.rb index 3b669a73f495..9af6a4f56973 100644 --- a/config/initializers/macos.rb +++ b/config/initializers/macos.rb # # From https://bugs.ruby-lang.org/issues/14009 Fiddle.dlopen '/System/Library/Frameworks/Foundation.framework/Foundation' + Fiddle.dlopen '/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation' end endEdited by Stan Hu -
I'm going to close this now that !122115 (merged) has been merged.
I've created an upstream issue to discuss solving this inside the
grpcgem: https://github.com/grpc/grpc/issues/33281
You can refer to the Features by Group handbook page for guidance.
If you are unsure about the correct group, please do not leave the issue without a group label, and refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issue.
This message was generated automatically. You're welcome to improve it.
mentioned in commit 8f2c220f
assigned to @stanhu
changed milestone to %16.1
mentioned in merge request !122115 (merged)
-
1 Error 
Please add typebug typefeature, typemaintenance and a subtype label to this issue. If you do not feel the purpose of this issue matches one of the types, you may apply the typeignore label to exclude it from type tracking metrics and future prompts.
This message was generated automatically. You're welcome to improve it.
added bugfunctional typebug labels
This issue was automatically tagged with the label groupcode review by TanukiStan, a machine learning classification model, with a probability of 0.97.
If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.
If you are unsure about the correct group, please do not leave the issue without a group label. Please refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issues.
Authors who do not have permission to update labels can leave the issue to be triaged by group leaders initially assigned by TanukiStan
This message was generated automatically. You're welcome to improve it.
- A deleted user
added automation:ml groupcode review labels
added devopscreate sectiondev labels
added automation:ml wrong label and removed groupcode review label
mentioned in issue gitlab-org/quality/triage-reports#12804 (closed)
