Cryptest.sh

From Crypto++ Wiki
Jump to navigation Jump to search

cryptest.sh is a script used to test the Crypto++ library on BSD, Linux, OS X, Solaris and Unix platforms. The script repeatedly builds the library and runs the self tests using different configurations and options. The Crypto++ Release Engineering process relies heavily on the script.

The script is distributed as part of the library for those who want to perform acceptance testing. It was added to the Crypto++ library at 5.6.3. In addition, the script is available for download below, so it can be used to test earlier versions of Crypto++.

The script honors CXX, and attempts to honor most of the user's CXXFLAGS. Options that the script uses/tests are filtered out from CXXFLAGS; see CXX and CXXFLAGS below for details. The script then make's the library and runs the self tests while cycling through combinations of configuration options.

A related page is cryptest.nmake, which discusses a similar test script used on Windows to test Windows desktops and servers, Windows Phone and Windows Store apps. cryptest.nmake can also be used as a starting point for building static and dynamic libraries from the command line.

Test Options

The following is the 5-second tour of the two options.

Full Tests - runs all the tests using all the cpu cores: 

./cryptest.sh

Quick Tests - runs all the tests except Valgrind and Benchmarks (which take considerably more time), using all the cpu cores: 

./cryptest.sh fast

fast can be used with nice.

Nice Tests - runs all the tests, using half the cpu cores: 

./cryptest.sh nice

nice can be used with fast.

Output Files

The script reports to cout and cerr. Its also tee's its output to one of four different files, depending on the test. The output files and their use are:

Output Files
Variable Filename Errata and Comments
TEST_RESULTS cryptest-result.txt Build and runtime errors
WARN_RESULTS cryptest-warn.txt Build warnings
BENCHMARK_RESULTS cryptest-bench.txt Benchmark results
INSTALL_RESULTS cryptest-install.txt Problems with make install and make uninstall

cryptest-result.txt. Problems are detected by grep'ing the file for error and FAILED. The string error is output by tools like Undefined Behavior sanitizer and Valgrind. Crypto++ outputs FAILED when a self tests fails.

If a filename does not suit your taste, then change it with: 

TEST_RESULTS=different-result-file.txt ./cryptest.sh

CXX and CXXFLAGS

The script honors CXX, and attempts to honor most of the user's CXXFLAGS. On occasion the script will also test a different compiler than the one specified in CXX. For example, on Linux, the script uses GCC by default. However, it will perform two quick tests under Clang to ensure a break was not introduced.

If you want to test a particular compiler, like clang++ or icpc, issue: 

CXX=clang++ ./cryptest.sh
CXX=/opt/intel/bin/icpc ./cryptest.sh

You can also provide some default CXXFLAGS. For example, to test with ARMv6, use the following. It will be placed in a variable called RETAINED_CXXFLAGS and used in most (but not all) tests). 

CXXFLAGS=-march=armv6 ./cryptest.sh    # -march=armv6 will be honored

User supplied flags which are tested in this script are removed during testing. You can see the ones removed in FILTERED_CXXFLAGS. For example, -Wall will be removed from all builds and tested by the script when appropriate: 

CXXFLAGS=-Wall ./cryptest.sh    # -Wall will be removed

Running the Script

To run the script, simply cryptest.sh (notice its .sh, not .exe). 

$ cd cryptopp
./cryptest.sh

The script will print some platform and configuration information to verify feature detection, and then get to work: 

$ ./cryptest.sh 

IS_LINUX: 1
IS_ARM32: 1

HAVE_CXX03: 1
HAVE_CXX11: 1
HAVE_LDGOLD: 1

HAVE_O3: 1
HAVE_O5: 1
HAVE_OS: 1

CPU: 6 logical
FREQ: 1.71661 GHz
MEM: 2015 MB

Git branch: master (commit 694e486db3cc647b)
Compiler: g++ (Ubuntu/Linaro 4.8.2-19ubuntu1) 4.8.2
Pathname: /usr/bin/g++ (symlinked)

Start time: Sun Jun 19 05:06:04 UTC 2016

************************************
Testing: NEON, default CXXFLAGS

g++ -g2 -O2 -mfpu=neon -fPIC -c cryptlib.cpp
g++ -g2 -O2 -mfpu=neon -fPIC -c cpu.cpp
...

The script should complete with output similar to the following. If it does not, then see Tracking Errors below for how to duplicate a failure. 

************************************************
************************************************

23 configurations tested

No failures detected

No warnings detected

************************************************
************************************************

Testing started: Sat Jan 2 15:47:25 EST 2016
Testing finished: Sat Jan 2 17:17:34 EST 2016

************************************************
************************************************

Execution Time

The script takes between 3 hours to 3 days to run. Most of the time is spent compiling the sources since the self tests run relatively quickly. On a modern Linux machine,it will take about 2 to 3 hours to run, depending on what is installed. On an older PowerMac it will take about 5 hours to run. On an emulated S/390 or emulated ARM64, it will take 2 days or so to run.

If you launch with time, the script should finish with something similar to: 

$ time ./cryptest.sh
...

real	216m15.775s
user	201m4.088s
sys	6m14.524s

You can make the script run faster by disabling Valgrind and Benchmarks. Its a good compromise of coverage versus thoroughness. When the library is being release tested, the cryptest.sh alone is used - no tests are disabled. 

HAVE_VALGRIND=0 WANT_BENCHMARKS=0 ./cryptest.sh

IS_LINUX: 1
IS_ARM32: 1

HAVE_CXX03: 1
HAVE_CXX11: 1
HAVE_LDGOLD: 1
...

OOM Errors

The script will report Out of Memory (OOM) erros as a build failure. It is encountered on occasion for resource constrained devices, like a CubieTruck or BananaPi. You can safely ignore them, but kep in mind that particular test did not complete. 

32 configurations tested

12 errors detected. See cryptest-result.txt for details

25343:g++: internal compiler error: Killed (program cc1plus)
25347:make: *** [regtest.o] Error 4
25348:ERROR: failed to make cryptest.exe
26827:g++: internal compiler error: Killed (program cc1plus)
26831:make: *** [regtest.o] Error 4
26832:ERROR: failed to make cryptest.exe
...

Grepping through cryptest-result.txt will reveal similar to below: 

g++ -DNDEBUG -g2 -O2 -DCRYPTOPP_USE_FIPS_202_SHA3  -Wno-deprecated-declarations -mfpu=neon -fPIC -c regtest.cpp
g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.
make: *** [regtest.o] Error 4
ERROR: failed to make cryptest.exe

Removing -pipe from CXXFLAGS will sometimes sidestep the failure, if its present. You can then retest by using the same CXXFLAGS as from the failed test: 

make clean
make CXXFLAGS="-DNDEBUG -g2 -O2 -DCRYPTOPP_USE_FIPS_202_SHA3 -mfpu=neon -fPIC" cryptest.exe

Tracking Errors

The following demonstrates how to track down an error uncovered on an ARMv8 (Aarch64) Mustang server board: 

 48093	/usr/include/clang/3.5.0/include/xmmintrin.h:28:2: error: "SSE instruction set not enabled"
 48094	#error "SSE instruction set not enabled"
 48099	1 error generated.
 48101	make: *** [cpu.o] Error 1
 48103	ERROR: failed to make cryptest.exe

First, verify you can locate it in the log: 

$ grep 'error:' cryptest-result.txt 
/usr/include/clang/3.5.0/include/xmmintrin.h:28:2: error: "SSE instruction set not enabled"

Second, use the -B option to capture the compile line. Notice the compiler and the flags: 

$ grep -B 4 'error:' cryptest-result.txt 
/usr/local/bin/clang++ -DNDEBUG -g2 -O2 -fPIC -pipe -c dessp.cpp
/usr/local/bin/clang++ -DNDEBUG -g2 -O2 -fPIC -pipe -c seal.cpp
In file included from cpu.cpp:12:
In file included from ./cpu.h:48:
/usr/include/clang/3.5.0/include/xmmintrin.h:28:2: error: "SSE instruction set not enabled"

Third, reproduce the problem: 

$ make CXX=/usr/local/bin/clang++ CXXFLAGS="-DNDEBUG -g2 -O2 -fPIC -pipe" cpu.o
/usr/local/bin/clang++ -DNDEBUG -g2 -O2 -fPIC -pipe -c cpu.cpp
In file included from cpu.cpp:12:
In file included from ./cpu.h:48:
/usr/include/clang/3.5.0/include/xmmintrin.h:28:2: error: "SSE instruction set
      not enabled"
#error "SSE instruction set not enabled"
 ^
1 error generated.
GNUmakefile:735: recipe for target 'cpu.o' failed
make: *** [cpu.o] Error 1

This particular issue appears to be related to Clang on ARM. Its a Crypto++ issue, but it only surfaces under Clang and ARM64. The code originally included <xmmintrin.h> "outside" the X86/X32/X64 block. The solution was to move the include inside an X86-related block: 

#if CRYPTOPP_BOOL_AESNI_INTRINSICS_AVAILABLE

#if defined(__has_include)
# if __has_include(<xmmintrin.h>)
#  include <xmmintrin.h>
# endif
#endif

...

Finally, supply the fix and verify the issue is cleared: 

$ make CXX=/usr/local/bin/clang++ CXXFLAGS="-DNDEBUG -g2 -O2 -fPIC -pipe" cpu.o
/usr/local/bin/clang++ -DNDEBUG -g2 -O2 -fPIC -pipe -c cpu.cpp
$

Configurations

cryptest.sh test a number of configurations. The configurations are a cross product of Debug, Release, -O1, -O2, -O3, -O5, -Os, C++03, C++11, C++14, C++17, GNU's libstdc++, LLVM's libc++, Undefined Behavior sanitizer, Address sanitizer and Valgrind. If the preferred compiler is Clang, then GCC is also lightly tested; and if GCC is th preferred compiler, then Clang is also lightly tested.

Currently there are about 40 tests available due to various combinations but not all of them are run. If a particular configuration is not available, then its skipped. The configurations the script attempts to test are as follows.

  • Debug, default CXXFLAGS
  • Release, default CXXFLAGS
  • Debug, C++03
  • Release, C++03
  • Debug, C++11
  • Release, C++11
  • Debug, C++14
  • Release, C++14
  • Debug, C++17
  • Release, C++17
  • Debug, GNU++03
  • Release, GNU++03
  • Debug, GNU++11
  • Release, GNU++11
  • Debug, GNU++14
  • Release, GNU++14
  • Debug, GNU++17
  • Release, GNU++17
  • Debug, DISABLE_ASM
  • Release, DISABLE_ASM
  • Debug, MAINTAIN_BACKWARDS_COMPATIBILITY
  • Release, MAINTAIN_BACKWARDS_COMPATIBILITY
  • Debug, MAINTAIN_BACKWARDS_COMPATIBILITY_562
  • Release, MAINTAIN_BACKWARDS_COMPATIBILITY_562
  • Debug, INIT_PRIORITY
  • Release, INIT_PRIORITY
  • Debug, NO_OS_DEPENDENCE
  • Release, NO_OS_DEPENDENCE
  • Debug, -O3 optimizations
  • Release, -O3 optimizations
  • Debug, -O5 optimizations
  • Release, -O5 optimizations
  • Debug, -Os optimizations
  • Release, -Os optimizations
  • Debug, -Ofast optimizations
  • Release, -Ofast optimizations
  • Debug, C++03, UBsan
  • Release, C++03, UBsan
  • Debug, C++03, Asan
  • Release, C++03, Asan
  • Debug, C++11, UBsan
  • Release, C++11, UBsan
  • Debug, C++11, Asan
  • Release, C++11, Asan
  • Debug, C++14, UBsan
  • Release, C++14, UBsan
  • Debug, C++14, Asan
  • Release, C++14, Asan
  • Debug, C++17, UBsan
  • Release, C++17, UBsan
  • Debug, C++17, Asan
  • Release, C++17, Asan
  • Darwin Release, C++03, Malloc and Scribble guards
  • Darwin Release, C++11, Malloc and Scribble guards
  • Darwin Release, C++14, Malloc and Scribble guards
  • Darwin Release, C++17, Malloc and Scribble guards
  • Darwin Release, C++03, libc++ (LLVM runtime)
  • Darwin Release, C++11, libc++ (LLVM runtime)
  • Darwin Release, C++14, libc++ (LLVM runtime)
  • Darwin Release, C++17, libc++ (LLVM runtime)
  • Darwin Release, C++03, libstdc++ (GNU runtime)
  • Darwin Release, C++11, libstdc++ (GNU runtime)
  • Darwin Release, C++14, libstdc++ (GNU runtime)
  • Darwin Release, C++17, libstdc++ (GNU runtime)
  • Darwin, Intel Universal binary, C++03
  • Darwin, Intel Universal binary, C++11
  • Darwin, Intel Universal binary, C++14
  • Darwin, Intel Universal binary, C++17
  • Darwin, PowerPC Universal binary
  • Sun Studio 12.2, debug and release
  • Sun Studio 12.3, debug and release
  • Sun Studio 12.4, debug and release
  • Sun Studio 12.5, debug and release
  • MinGW Release, PREFER_BERKELEY_STYLE_SOCKETS
  • MinGW Release, PREFER_WINDOWS_STYLE_SOCKETS
  • Valgrind, -O1, C++03
  • Valgrind, -O1, C++11
  • Valgrind, -O1, C++14
  • Valgrind, -O1, C++17
  • Install into /tmp/cryptopp_test/ directory
  • Help, self-tests, and benchmarks from the /tmp/cryptopp_test/share/ directory

Downloads

cryptest-sh - archive with script to test Crypto++ using various configurations.

Retrieved from "https://www.cryptopp.com/w/index.php?title=Cryptest.sh&oldid=28590"