Issue in installing pysqlcipher3

Question

So even though pip install pysqlcipher3 worked returned Successfully installed pysqlcipher3, it did not.

From my research, it seems that its a common issue for Windows user trying to install Sqlcipher.

Attempt #1

First, I try to download the pysqlcipher3 from github and build and install it manually via cmd (using python setup.py build and python seetup.py install).

Upon executing python3 setup.py build I got some error (missing few requirements, not having OPENSSL_CONF in environment variable...etc.) but I fix most of them.

However, I end up stuck with this error:

fatal error C1083: Cannot open include file: 'sqlcipher/sqlite3.h': No such file or directory.

Attempt #2

I read somewhere (I think) that I have to install the SQLCipher before installing the PySqlCipher3.

Upon checking, I found out that I have to either pay to get a prebuild executable or build it myself.

So I tried to build it myself following this tutorial.

After installing all requirements, and following the steps I reach another dead end. In the before-last step (step number 6), when executing nmake /f Makefile.msc I got this error:

fatal error C1083: Cannot open include file: 'stdio.h': No such file or directory

---

My question are:

1. I read that I need libsqlcipher which might be the issue. Where can I find it (link to executable preferred).
2. How can I solve the issue in my first attempt?
3. How can I solve the issue in my second attempt?

My goal: Is to have pysqlcipher3 installed and working.

---

EDIT 1 - solved Attempt #2 just to reach another issue

So it seems that I was missing some C++ features and tool in Visual Studio. So via Visual Studio Community installer, I download and install basic C++ stuff and it fixed the error.

I got another error which I fix by copying OpenSSL-Win64\include folder to C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include.

However, now I am stuck with this error:

sqlite3.c(77701): warning C4389: '!=': signed/unsigned mismatch
    lib.exe /NOLOGO /MACHINE:x64 /OUT:libsqlite3.lib sqlite3.lo 
    echo EXPORTS > sqlcipher.def
    dumpbin /all libsqlite3.lib  | tclsh .\tool\replace.tcl include "^\s+1 _?(sqlite3(?:session|changeset|changegroup|rebaser)?_[^@]*)(?:@\d+)?$" \1  | sort >> sqlcipher.def
    link.exe /DEBUG   /NOLOGO /MACHINE:x64  /DLL /DEF:sqlcipher.def /OUT:sqlcipher.dll sqlite3.lo sqlite3res.lo  
   Creating library sqlcipher.lib and object sqlcipher.exp
sqlite3.lo : error LNK2019: unresolved external symbol RAND_bytes referenced in function sqlcipher_openssl_random
sqlite3.lo : error LNK2019: unresolved external symbol RAND_add referenced in function sqlcipher_openssl_add_random
sqlite3.lo : error LNK2019: unresolved external symbol OBJ_nid2sn referenced in function sqlcipher_openssl_get_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_MD_size referenced in function sqlcipher_openssl_get_hmac_sz
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CIPHER_nid referenced in function sqlcipher_openssl_get_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CIPHER_block_size referenced in function sqlcipher_openssl_get_block_sz
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CIPHER_key_length referenced in function sqlcipher_openssl_get_key_sz
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CIPHER_iv_length referenced in function sqlcipher_openssl_get_iv_sz
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CipherInit_ex referenced in function sqlcipher_openssl_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CipherUpdate referenced in function sqlcipher_openssl_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CipherFinal_ex referenced in function sqlcipher_openssl_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CIPHER_CTX_new referenced in function sqlcipher_openssl_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CIPHER_CTX_free referenced in function sqlcipher_openssl_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_CIPHER_CTX_set_padding referenced in function sqlcipher_openssl_cipher
sqlite3.lo : error LNK2019: unresolved external symbol EVP_sha1 referenced in function sqlcipher_openssl_get_hmac_sz
sqlite3.lo : error LNK2019: unresolved external symbol EVP_sha256 referenced in function sqlcipher_openssl_get_hmac_sz
sqlite3.lo : error LNK2019: unresolved external symbol EVP_sha512 referenced in function sqlcipher_openssl_get_hmac_sz
sqlite3.lo : error LNK2019: unresolved external symbol EVP_get_cipherbyname referenced in function sqlcipher_openssl_activate
sqlite3.lo : error LNK2019: unresolved external symbol PKCS5_PBKDF2_HMAC referenced in function sqlcipher_openssl_kdf
sqlite3.lo : error LNK2019: unresolved external symbol HMAC_CTX_new referenced in function sqlcipher_openssl_hmac
sqlite3.lo : error LNK2019: unresolved external symbol HMAC_CTX_free referenced in function sqlcipher_openssl_hmac
sqlite3.lo : error LNK2019: unresolved external symbol HMAC_Init_ex referenced in function sqlcipher_openssl_hmac
sqlite3.lo : error LNK2019: unresolved external symbol HMAC_Update referenced in function sqlcipher_openssl_hmac
sqlite3.lo : error LNK2019: unresolved external symbol HMAC_Final referenced in function sqlcipher_openssl_hmac
sqlcipher.dll : fatal error LNK1120: 24 unresolved externals

Answer 1

First of all, "Successfully installed pysqlcipher3" is misleading (at least up to the current latest commit fd1b547407bcb7198107fe3c458105286a060b0d on the Github repository). I have just wasted some of my own time, and thought I should make it clear again.

The misleading "Successfully installed pysqlcipher3" appears on Linux systems too. In case you have a Ubuntu/Debian system handy, the following would get you going. Basically you need to install the SQLCipher libraries and development files before the pip3 command.

$ sudo apt install sqlcipher libsqlcipher0 libsqlcipher-dev
$ sudo -H pip3 install pysqlcipher3

$ python3 -c 'import pysqlcipher3; print(pysqlcipher3.__path__)'
['/usr/local/lib/python3.7/dist-packages/pysqlcipher3']

Since you are on Windows, unresolved external symbol RAND_bytes (first line on your compiler/linker error) basically means that your linker on Windows can't find the necessary OpenSSL libraries.

Have look at the posts here, here, and here. They have solutions to this exact OpenSSL linking issue. The basic steps towards resolution are the following:

1. Download and install the required 32/64-bit version of OpenSSL from here
2. Make Visual Studio to use the OpenSSL include directory e.g. C:\OpenSSL-Win32\include
3. Make Visual Studio to use the OpenSSL library directory e.g. C:\OpenSSL-Win32\lib
4. Add the following libraries as additional-linker-dependencies in Visual Studio:

    - libeay32.lib
    - libeay32MTd.lib
    - libeay32MT.lib
    - libeay32MDd.lib
    - libeay32MD.lib
    - ssleay32.lib
    - ssleay32MTd.lib
    - ssleay32MT.lib
    - ssleay32MDd.lib
    - ssleay32MD.lib

Answer 2

I also had almost all of your problems and so I created this guide below which worked for me

My build: windows 10 x64 v1909

1. Install python (if not already installed). I had v3.7 64 bit
2. Install Win64OpenSSL-1_0_2u. Make sure you get exactly this version since I first downloaded some other versions and they didn't work.
3. Add environment variable for both user and system SET OPENSSL_CONF=C:\OpenSSL-Win64\bin\openssl.cfg (change path as per yours)
4. Install ActiveTcl-8.6.9.8609.2-MSWin32-x64. This was latest at this time and worked
5. Install visual studio (if not already installed). I had VS2019 v16.5.5
6. Get sqlcipher and extract to directory
7. Open visual studio x64 native tools and navigate to above extracted directory of sqlcipher
8. Run command nmake /f Makefile.msc. The final compilation will report an error, but it does not matter, we do not need a real compilation here, only need sqlite3.c
9. Get pysqlcipher3-1.0.3, extract to directory and navigate to it
10. Create folder named amalgamation in it and create another folder named sqlcipher in amalgamation folder
11. Copy sqlite3.h and sqlite3.c created in Visual Studio step (in sqlcipher-master folder) to amalgamation folder and sqlcipher folder both
12. Open administrative command prompt and navigate to above extracted directory of pysqlcipher3-1.0.3
13. Run following command python setup.py build_amalgamation
14. This will give linkage errors as OP has stated. Now install Win32OpenSSL-1_0_2u. Earlier you installed x64 and now installing x32. I learnt this hack after multiple failures -:)
15. Adjust the environment variables for SSL to point to x32 installed location instead of x64 as done in point-3
16. Close and reopen admin CMD and retype the command python setup.py build_amalgamation. This time it should work well without any errors.
17. Now run following command python setup.py install
18. If all goes well then this command should work in cmd>>python>>from pysqlcipher3 import dbapi2

Some Useful Links for me 1, 2, 3, 4, 5

If you don't get exact versions as mine due to links broken in future, you can comment and if I have them at that time, I'll upload to a cloud drive.

[UPDATE 27-APR-2021 for Python v3.9.4 x64]

1. I tried to repeat above steps for Python v3.9.4 x64 but a few more errors occurred due to following two reasons

   1.1. A macro MODULE_NAME not parsed properly before compile time as pointed out by @ProfP30 in comments. Error will be as follows:

   src\python3\cache.c(261): error C2017: illegal escape sequence

   1.2. Due to removed functionality of some functions in python version 3.9

2. For the first problem, since the macro is a pre-processor directive (supplied to cl.exe compiler from setup.py file), error can be removed via a workaround by replacing all occurrences of this by its suitable replacement. For my system configuration, I replaced all occurrences of MODULE_NAME in the folder pysqlcipher3-1.0.4/src/python3 with pysqlcipher3.dbapi2 which resolved the problem

3. For the second problem, solution is to manually provide the replacement of all functions which are removed from Python v>3.8. which was quite annoying for me. Luckily, pysqlcipher3 1.0.4 just released couple of days ago(25-Apr-21) has addressed this issue and provided the replacements. So, better is to use source files of pysqlcipher3 1.0.4 instead of pysqlcipher3 1.0.3 as done in point 9 above.

4. In order to avoid all this hassle of compiling pysqlcipher3, I have placed pre-compiled binaries for Python v3.9.4 x64 at this link along with the instructions. (Disclaimer: The link may be removed in future due to storage constraints.)

Answer 3

On iMac running Catalina macOS 10.15.4, when I tried to install pysqlcipher3, it spat out error messages

But then I ran

$ brew install SQLCipher

which worked successfully:

$ brew install SQLCipher
==> Downloading https://homebrew.bintray.com/bottles/sqlcipher-4.4.0.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/9e/9e860b50ec668ef30f61377f39954241c4eeda4c4a664fbe1340e289229336bf?__gda__=exp=1590256907~hmac=9bb692e78355e890ab
######################################################################## 100.0%
==> Pouring sqlcipher-4.4.0.catalina.bottle.tar.gz
  /usr/local/Cellar/sqlcipher/4.4.0: 13 files, 3.9MB

Now, when I tried to install pysqlcipher3 the second time, it also installed successfully:

$ pip3 install pysqlcipher3
Collecting pysqlcipher3
  Using cached pysqlcipher3-1.0.3.tar.gz (100 kB)
Building wheels for collected packages: pysqlcipher3
  Building wheel for pysqlcipher3 (setup.py) ... done
  Created wheel for pysqlcipher3: filename=pysqlcipher3-1.0.3-cp37-cp37m-macosx_10_15_x86_64.whl size=61077 sha256=28e480d5999a13cf2446a262cff9d746853d0bf90541a76e473f66a7128ff877
  Stored in directory: /Users/kitzume/Library/Caches/pip/wheels/91/30/b9/1ee85dc50cddf37ecb5198dfa6279aa72eeb0148063b5ad0c6
Successfully built pysqlcipher3
Installing collected packages: pysqlcipher3
Successfully installed pysqlcipher3-1.0.3