Failure with puppet.service systemd unit

[flaruina]

Describe the Bug

If I try to start puppet.service on a SLES15.6 server I get an error reguarding openssl.

/opt/puppetlabs/puppet/lib/ruby/3.2.0/x86_64-linux/openssl.so: undefined symbol: EC_GROUP_new_curve_GF2m, version OPENSSL_3.0.0 - /opt/puppetlabs/puppet/lib/ruby/3.2.0/x86_64-linux/openssl.so

Puppet agent works as expected if launched interactively.
As a workaround, setting LD_LIBRARY_PATH seems to fix the issue

# cat /etc/systemd/system/puppet.service.d/override.conf 
[Service]
Environment=LD_LIBRARY_PATH=/opt/puppetlabs/puppet/lib/ruby/3.2.0/x86_64-linux:/opt/puppetlabs/puppet/lib

Expected Behavior

Since puppet agent works if launched interactively I would expect the unit to start correctly.

Steps to Reproduce

Steps to reproduce the behavior:

1. systemctl start puppet.service
2. systemctl status puppet.service

Environment

• Version puppet-agent-8.10.0-1.sles15.x86_64
• Platform SLES 15.6

[flaruina]

There is a conflict betweeen the openssl/libcrypto bundled with puppet and the one provided by SLES (which does not include the symbol):

> objdump -T /opt/puppetlabs/puppet/lib/libcrypto.so.3     | grep EC_GROUP_new_curve_GF2m
00000000001673d0 g    DF .text  0000000000000075  OPENSSL_3.0.0 EC_GROUP_new_curve_GF2m
>  objdump -T /usr/lib64/libcrypto.so.3     | grep EC_GROUP_new_curve_GF2m

By means of LD_DEBUG=libs, I observed that when launched by systemd (without the LD_LIBRARY_PATH workaround) this chain of events makes the system load the SUSE provided libcrypto

libnsl.so.2
  -> libtirpc.so.3
    -> libgssapi_krb5.so.2
      -> libkrb5.so.3 / libk5crypto.so.3
        -> libcrypto.so.3

In the interactive case, the puppet provided libcrypto is loaded before the system one and it works.
With the LD_LIBRARY_PATH set, puppet provided libs are loaded in advance, then puppet run and afterwards there are errors of missing symbol (puppet provided libcrypto do not ship with the EVP_camellia_256_cbc symbol required by the system provided libk5crypto.so.3).

In my case, I just ignore this errors since they are limited to Puppet launches and do not seem to create real issues.

[fragrantauto]

Have you checked if the system's OpenSSL library is overriding the one bundled with Puppet when starting the service? Specifically, verifying the output of ldd /opt/puppetlabs/puppet/lib/ruby/3.2.0/x86_64-linux/openssl.so could help identify if the loaded library is the expected version. Additionally, consider checking the LD_LIBRARY_PATH in the context of systemd services to ensure that it's being set correctly, as environment variables can sometimes behave differently when running as a service compared to an interactive shell.

[ronniromeo26]

It might be worth checking if the openssl.so file is linked against the expected OpenSSL version by running ldd /opt/puppetlabs/puppet/lib/ruby/3.2.0/x86_64-linux/openssl.so. If it shows that the system's OpenSSL library is being used, you could explore using Environment=LD_PRELOAD in your override.conf as a more targeted approach to force the bundled library to be loaded.

[flaruina]

Thanks for all the suggestions. Forcing the puppet-provided libcrypto.so.3 to be loaded, the systemd unit works fine

# cat /etc/systemd/system/puppet.service.d/override.conf
[Service]
Environment=LD_PRELOAD=/opt/puppetlabs/puppet/lib/libcrypto.so.3

[christianfrei791]

The root cause looks like an missing or incorrect RPATH/RUNPATH setting in `openssl.so

[tatumjakhiya26]

/systemd service unit definition shipped by puppet-agent (e.g., adding rpath during compilation/build of

[cherryrodney26]

soloading systemlibcrypto` instead of puppet's bundled one in the first place?
*