Segmentation fault when running in Docker container

[Kyrremann]

I'm trying to run a Python script in a Docker container talking with db2, but I just get a segmentation fault. We can run the Python-script locally, but when I try it from a container it fails, and using db2cli to debug doesn't help.

• Operating System Name: Dockerfile FROM python:3.12
• db2level output from clidriver if in use:

$ ./db2level
DB21085I  This instance or install (instance name, where applicable: "*") uses 
"64" bits and DB2 code release "SQL11059" with level identifier "060A010F".
Informational tokens are "DB2 v11.5.9.0", "special_50315", 
"DYN2410311831AMD64_50315", and Fix Pack "0".
Product is installed at "..".

• Target Db2 Server Version: ??
• Python Version: 3.12.11
• ibm_db version: 3.2.6
• For non-Windows, output of below commands:
  uname = Linux
  uname -m = x86_64
• Value of below environment variables if set:
  IBM_DB_HOME:
  PATH:
  LIB/LD_LIBRARY_PATH/DYLD_LIBRARY_PATH:

Dockerfile:

FROM python:3.12

RUN pip install ibm_db --no-binary :all: --no-cache-dir

RUN ln -s /var/run/secrets/db2-license/db2consv_zs.lic /usr/local/lib/python3.12/site-packages/clidriver/license/db2consv_zs.lic

db2cli validate:

$ ./db2cli validate -connstring "UID=xxx;PWD=xxx;HOSTNAME=xxx;PORT=5025;DATABASE=xxx" -displaylic

===============================================================================
Client information for the current copy:
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.9.0 (special_50315/64-bit)
Client Platform           : Linux/X8664
Install/Instance Path     : ..
DB2DSDRIVER_CFG_PATH value: <not-set>
db2dsdriver.cfg Path      : ../cfg/db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : ../cfg/db2cli.ini
db2diag.log Path          : db2diag.log

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Note: The validation utility could not find the configuration file 
db2dsdriver.cfg. The file is searched at "../cfg/db2dsdriver.cfg".

Segmentation fault (core dumped)

[Earammak]

@Kyrremann , Can you please share from which path you are running. Can you please share the sample python script. Looks like you are facing issue in running db2 command only from docker.

[bchoudhary6415]

@Kyrremann Please delete or edit your above comment,
Your userID and password are visible here.

Thank you

[Kyrremann]

Thanks for letting me know, I was so sure I had triple checked for that kind of blunder! Luckily it's an internal database, but still need to make a new password.

---

The path is /usr/local/lib/python3.12/site-packages/clidriver/bin. In this issue I've just showed the problems using the db2-tools, as I though it would be simpler to reproduce.

The Python script:

#!/usr/bin/env python3
import time

import os
import pandas as pd
import ibm_db

def read_from_db2_2(table_name):
    database_username = os.environ.get("DATABASE_USERNAME")
    database_password = os.environ.get("DATABASE_PASSWORD")
    database_host = os.environ.get("DATABASE_HOST")
    database_port = os.environ.get("DATABASE_PORT")
    database_name = os.environ.get("DATABASE_NAME")
    
    dsn = (
        f"DRIVER={{IBM DB2 ODBC DRIVER}};"
        f"DATABASE={database_name};"
        f"HOSTNAME={database_host};"
        f"PORT={database_port};"
        f"PROTOCOL=TCPIP;"
        f"UID={database_username};"
        f"PWD={database_password};"
    )

    # Establish the connection
    try:
        ibm_db.debug(True)
        db2_conn = ibm_db.connect(dsn, "", "")
        print("Connected to the database!")
    except:
        print("Failed to connect to the database.")

    schema = os.environ.get("DATABASE_SCHEMA", default="OS231Q1")
    sql = f"select * from {schema}.{table_name}"
    stmt = ibm_db.exec_immediate(db2_conn, sql)
    rows = []
    row = ibm_db.fetch_assoc(stmt)
    while row:
        rows.append(row)
        row = ibm_db.fetch_assoc(stmt)

    df = pd.DataFrame(rows)

    print(f"hentet {len(df)} rader")
    return(df)


def main():
    print("Starter jobb")
    filen_finnes = os.path.isfile("/usr/local/lib/python3.12/site-packages/clidriver/license/db2consv_zs.lic")
    print(f"Ligger lisens der forventet?: {filen_finnes=}")

    print("lese inn data fra db2")
    df = read_from_db2_2(table_name = 't_faggruppe')
    print(f"hentet {len(df)} rader")


if __name__ == '__main__':
    main()

The output from running the script in our environment is:

Starter jobb
Ligger lisens der forventet?: filen_finnes=True
lese inn data fra db2
[INFO] - entry connect()
[INFO] - entry _python_ibm_db_clear_conn_err_cache()
[INFO] - exit _python_ibm_db_clear_conn_err_cache()
[INFO] - entry _python_ibm_db_connect_helper()
[DEBUG] - In process of connection
[INFO] - Received arguments: ('DRIVER={IBM DB2 ODBC DRIVER};DATABASE=;HOSTNAME=;PORT=;PROTOCOL=TCPIP;UID=;PWD=;', '', '')
[DEBUG] - Parsed values: databaseObj=, uidObj=, passwordObj=, options=(nil), literal_replacementObj=(nil)
[DEBUG] - Setting literal_replacement to SET_QUOTED_LITERAL_REPLACEMENT_OFF
[DEBUG] - Created a new connection handle
[DEBUG] - Set flag_pconnect to: 0
[DEBUG] - SQLAllocHandle called with parameters: SQL_HANDLE_ENV=1, SQL_NULL_HANDLE=(nil), &conn_res->henv=0x7fe99cda54e0 and returned rc=0
[DEBUG] - Successfully allocated ENV handles
[DEBUG] - SQLSetEnvAttr called with parameters: conn_res->henv=0x1, SQL_ATTR_ODBC_VERSION=200, SQL_OV_ODBC3=3 and returned rc=0
[DEBUG] - SQLAllocHandle called with parameters: SQL_HANDLE_DBC=2, conn_res->henv=0x1, &(conn_res->hdbc)=0x7fe99cda54e4 and returned rc=0
[DEBUG] - Successfully allocated CONNECT Handle
[DEBUG] - Set auto_commit to: 1
[DEBUG] - SQLSetConnectAttr called with parameters: conn_res->hdbc=0x1, SQL_ATTR_AUTOCOMMIT=102, conn_res->auto_commit=1, SQL_NTS=-3 and returned rc=0
[INFO] - entry getUnicodeDataAsSQLWCHAR()
[DEBUG] - pyobj=0x7fe99cda8210, isNewBuffer=0x7ffc3988bec4
[DEBUG] - Unicode length (nCharLen): 115
[DEBUG] - sysmodule obtained: 0x7fe99cf827a0, maxuni obtained: 0x7fe99cf0da10, maxuniValue: 1114111
[DEBUG] - Allocated new buffer: pNewBuffer=0x7fe99cf38e40, size=116
[DEBUG] - Buffer initialized to zero
[DEBUG] - Encoded to UTF-16 Little Endian: pyUTFobj=0x7fe965c1abc0
[DEBUG] - Copied data to pNewBuffer: pNewBuffer=0x7fe99cf38e40
[DEBUG] - Decremented reference count for pyUTFobj
[INFO] - exit getUnicodeDataAsSQLWCHAR()
[DEBUG] - Using SQLDriverConnectW for connection

As you can see, everything looks okay when running the Python-script, but it stops at the last line and exit. Since we are running in Kubernetes we get the exit code 139 which means SIGSEGV.

[bchoudhary6415]

@Kyrremann , Please try to edit the logs and than paste.

Thank you

[Earammak]

@Kyrremann I would request you to share the trace for python script.
To collect the traces use below commands:
db2trc on -f 1.trc
python test.py
db2trc off
db2trc flw 1.trc 1.flw
db2trc fmt 1.trc 1.fmt
db2trc fmt -cli 1.trc 1.cli

Share these 1.flw, 1.fmt and 1.cli files here. We will look into trace files.

[x10an14-nav]

Thanks for helping us out @Earammak! Here are the files as requested:

https://gist.github.com/x10an14-nav/a445d06e90b6177aaf569f62438e49c4

[bimalkjha]

@Kyrremann You have not provided info about Target Db2 Server Version: ?? while opening the issue. Are you trying to connect to a z/OS server or iSeries server? Are you able to connect to the same server using same license file when you install ibm_db outside docker on Linux OS? Please clarify. Thanks.

[bimalkjha]

@x10an14-nav I checked the shared log files. Thanks for sharing it.
In 1.fmt file, we can see below logs:

8297	error DB2 UDB license manager sqllcConnectUser fnc (4.3.173.110.0.99)
	pid 96 tid 139966472432512 cpid -1 node -1 probe 99
	Error ZRC = 0x00000000 = 0 = HA_ZRC_OK
	bytes 44

	Data1 	(PD_TYPE_DEFAULT,36) Hexdump:
	4E4F 204C 4943 454E 5345 2057 4153 2046    NO LICENSE WAS F
	4F55 4E44 2054 4F20 434F 4E4E 4543 5420    OUND TO CONNECT 
	5749 5448                                  WITH
8312	data DB2 UDB DRDA Application Requester sqljrCheckForAndInstallCLIDriverLicenses fnc (3.3.55.115.0.30)
	pid 96 tid 139966472432512 cpid -1 node -1 probe 30
	bytes 108

	Data1 	(PD_TYPE_STRING,100) String:
	/tmp/tmp.CkbuZEFR2M/venv.with-license/lib/python3.12/site-packages/clidriver/license/db2consv_zs.lic
8394	data DB2 Common OSSe ossGetFileInfo cei (3.1.3.31.2.20)
	pid 96 tid 139966472432512 probe 20
	bytes 108

	Data1 	(PD_TYPE_DEFAULT,100) Hexdump:
	2F74 6D70 2F74 6D70 2E43 6B62 755A 4546    /tmp/tmp.CkbuZEF
	5232 4D2F 7665 6E76 2E77 6974 682D 6C69    R2M/venv.with-li
	6365 6E73 652F 6C69 622F 7079 7468 6F6E    cense/lib/python
	332E 3132 2F73 6974 652D 7061 636B 6167    3.12/site-packag
	6573 2F63 6C69 6472 6976 6572 2F6C 6963    es/clidriver/lic
	656E 7365 2F64 6232 636F 6E73 765F 7A73    ense/db2consv_zs
	2E6C 6963                                  .lic
8401	exit DB2 UDB license manager sqllcReadFile fnc (2.3.173.129.0)
	pid 96 tid 139966472432512 cpid -1 node -1
	rc = 0

1.flw:

2288        | | SQLConnect2 entry
2595        | | | | sqleUCappConnect data [probe 5219]
3897        | | | | | | | sqljrPostConnect entry
3917        | | | | | | | sqljrPostConnect data [probe 30]
3918        | | | | | | | | | sqloLicConnectUser entry
8298        | | | | | | | | | | sqllcConnectUser exit [rc = 1]
8299        | | | | | | | | | sqloLicConnectUser exit [rc = 1]
8300        | | | | | | | | | sqljrCheckForAndInstallCLIDriverLicenses entry
8315        | | | | | | | | | | LicAddLicense entry
8316        | | | | | | | | | | LicAddLicense data [probe 5]
8317        | | | | | | | | | | | sqllcValidateLicense entry
8384        | | | | | | | | | | | sqllcReadFile entry
8401        | | | | | | | | | | | sqllcReadFile exit

By looking on traces, we can see that clidriver is trying to install client side db2connect license but during the installation, it crashes under LicAddLicense() function. Your license file is for Db2 for z/OS server. If you are using this license to connect to iSeries server, then please use correct license.
If you are trying to connect to z/OS server, please check file permission of db2consv_zs.lic file. If you have empty line at end of this file, please remove it. This license file is a text file, so make sure if you transferred it from an OS which do not match OS of docker, then file format might have changed or corrupted. Please provide more info to understand your issue. Thanks.

[Kyrremann]

I'll find out about the zOS vs iSeries, and come back on that.
Just got told that we are using zOS servers.

There was one new line at end of file (which is pretty normal/recommended on unix files), but I got the same error when I removed it. File permissions are set to read, but there is one thing I haven't tried yet. The way we side load the license is by using symbol links, maybe the client doesn't follow symbol links? I'll try without linking tomorrow.

Maybe it's a wrong license

[Kyrremann]

I finally made it work! The problem was symlinks, the client doesn't follow symlinks. So when I switched to a writable filesystem and copied the license instead of using symlink the validate command successfully completed. Another thing is that I needed the new line at the end of the file too.

===============================================================================
Client information for the current copy:
===============================================================================

Client Package Type       : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v11.5.9.0 (special_50315/64-bit)
Client Platform           : Linux/X8664
Install/Instance Path     : ..
DB2DSDRIVER_CFG_PATH value: <not-set>
db2dsdriver.cfg Path      : ../cfg/db2dsdriver.cfg
DB2CLIINIPATH value       : <not-set>
db2cli.ini Path           : ../cfg/db2cli.ini
db2diag.log Path          : db2diag.log

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Note: The validation utility could not find the configuration file 
db2dsdriver.cfg. The file is searched at "../cfg/db2dsdriver.cfg".

===============================================================================
db2cli.ini validation for database "xxx:xxx:5025":
===============================================================================

Note: The validation utility could not find the configuration file db2cli.ini. 
The file is searched at "../cfg/db2cli.ini".

===============================================================================
db2dsdriver.cfg validation for database "xxx:xxx:5025":
===============================================================================

Note: The validation utility could not find the configuration file 
db2dsdriver.cfg. The file is searched at "../cfg/db2dsdriver.cfg".

===============================================================================
Connection attempt for database "xxx:xxx:5025":
===============================================================================

[SUCCESS]

===============================================================================

Valid License Found            : Yes
License location               : Client
Server routine type attempted  : SP and UDF
DB2 connect license version    : N/A
Output Connection String : 
"UID=xxx;PWD=xxx;HOSTNAME=xxx;PORT=5025;DATABASE=xxx;"

===============================================================================
The validation is completed.
===============================================================================