AWS CodeCommit without ssh keys

A super easy way to grant access to a AWS CodeCommit repository from an EC2 instance hosting e.g. Jenkins is to grant proper IAM Role permissions

{
 "Effect": "Allow",
 "Action": "codecommit:*",
 "Resource": "*"
}

and configure git client to use a helper:

git config --global credential.helper '!aws codecommit credential-helper $@'
git config --global credential.useHttpPath true

Voila!

AWS CodeCommit without ssh keys

Docker, Rails and passing configuration via environment variables.

Docker has a really nice option to pass configuration variables to the inside of the container. Sadly it’s only available to the user that the container runs with.

Problems are starting when you want to su - user and do something which requires use of passed environment variables which are not there after executing su - command.

One solution to this is to use a entrypoint wrapper script which before starting anything else will execute


env > /etc/.docker_env

At this point we have all variables saved in a file and we’re able to source them when starting new shell. This is only a part of the complete solution as we would need to have all those variables available also inside our Rails app.

Digging a bit in Rails initialization process led me to config/boot.rb which is used to set ENV['BUNDLE_GEMFILE'] a variable defining location path of Gemfile. We can also use this file to import all environment variables from .docker_env file.

Simple:


require 'dotenv'
Dotenv.load('/etc/.docker_env')

does the trick and next time when we need to run any rails task we don’t have to worry about sourcing proper environment.

Docker, Rails and passing configuration via environment variables.

Rails, database connection pool and sidekiq concurrency.

This should be quite easy for everyone, but not everyone understands how connection pooling and concurrency cooperate and blame the part of the system which shouldn’t be blamed in the first place.

So here it goes.

Connection pool in Rails config/database.yml must have a larger value than the Concurrency level set in config/sidekiq.yml

Why? Because when you run rails app and sidekiq together on one node sidekiq is using connection pool settings from Rails app. You need to have some part of the pool for Rails app and some part of the pool for sidekiq tasks.

Simple as that. If you see timeouts in sidekiq saying that it couldn’t obtain database connection in time specified by timeout value (default is 5 secs) then most probably it means that you have misconfigured your application.

Rails, database connection pool and sidekiq concurrency.

Mac OS X, Python, ports and dlib compilation

This topic is not very well covered by documentation and assumes that everything is as easy as possible…

OK, lets build…

cd python-examples
./compile_dlib_python_module.bat
-- The C compiler identification is AppleClang 6.1.0.6020053
-- The CXX compiler identification is AppleClang 6.1.0.6020053
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- python
-- Found PythonLibs: /usr/lib/libpython2.7.dylib (found suitable version "2.7.6", minimum required is "2.6")
-- Looking for png_create_read_struct
-- Looking for png_create_read_struct - found
-- Looking for jpeg_read_header
-- Looking for jpeg_read_header - found
-- Searching for BLAS and LAPACK
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Found LAPACK library
-- Found CBLAS library
-- Looking for cblas_ddot
-- Looking for cblas_ddot - found
-- Check for STD namespace
-- Check for STD namespace - found
-- Looking for C++ include iostream
-- Looking for C++ include iostream - found
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/suff/Downloads/dlib-18.16/python_examples/build
Scanning dependencies of target dlib
[ 0%] Building CXX object dlib_build/CMakeFiles/dlib.dir/base64/base64_kernel_1.o
[....]
[ 84%] Built target dlib
Scanning dependencies of target dlib_
[ 85%] Building CXX object CMakeFiles/dlib_.dir/src/dlib.cpp.o
[....]
[100%] Building CXX object CMakeFiles/dlib_.dir/src/gui.cpp.o
Linking CXX shared library dlib.so
[100%] Built target dlib_
Install the project...
-- Install configuration: "Release"
-- Installing: /Users/suff/Downloads/dlib-18.16/tools/python/../../python_examples/dlib.so

Perfect… So we have a SO file and we can copy it to site-packages of our virtualenv…

$ cp dlib.so ~/projects/dlib/venv/lib/python2.7/site-packages/
$ source ~/projects/dlib/venv/bin/activate
(venv) $ python 
Python 2.7.10 (default, May 23 2015, 22:32:15)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dlib
Segmentation fault: 11

Well… that was unexpected…

I’ve started to poke around the output of cmake and found something that caught my eye.

-- Found PythonLibs: /usr/lib/libpython2.7.dylib (found suitable version "2.7.6", minimum required is "2.6")

In my virtualenv I’m using Macports provided version of python2.7 and boost but dlib compiles against python provided by the system. While googling I found a stackoverflow post. It’s similar issue. The difference is that the brew version of Python2.7 is used. Adrian Rosebrock had fixed it by manually editing CMakeFiles/dlib_.dir/build.make and /CMakeFiles/dlib_.dir/link.txt.
After some digging on how cmake works and what’s going on in the compile_dlib_python_module.bat I found what needs to be done to compile dlib module.

$ mkdir build
$ cd build
$ cmake ../../tools/python -DUSE_SSE2_INSTRUCTIONS=ON -DUSE_SSE4_INSTRUCTIONS=ON -DUSE_AVX_INSTRUCTIONS=ON -DPYTHON_LIBRARY:FILEPATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
$ grep -Ri libpython2.7 *

CMakeCache.txt:PYTHONLIBS:PATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
CMakeCache.txt:PYTHON_LIBRARY:FILEPATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
CMakeCache.txt:dlib__LIB_DEPENDS:STATIC=general;/opt/local/lib/libboost_python-mt.dylib;general;/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib;general;dlib;
CMakeCache.txt:FIND_PACKAGE_MESSAGE_DETAILS_PythonLibs:INTERNAL=[/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib][/System/Library/Frameworks/Python.framework/Headers][v2.7.6(2.6)]
CMakeFiles/dlib_.dir/build.make:dlib.so: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib
CMakeFiles/dlib_.dir/link.txt:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++   -DDLIB_PNG_SUPPORT -DDLIB_JPEG_SUPPORT -DDLIB_USE_BLAS -DDLIB_USE_LAPACK -O3 -DNDEBUG -dynamiclib -Wl,-headerpad_max_install_names  -o dlib.so -install_name @rpath/dlib.so CMakeFiles/dlib_.dir/src/dlib.cpp.o CMakeFiles/dlib_.dir/src/matrix.cpp.o CMakeFiles/dlib_.dir/src/vector.cpp.o CMakeFiles/dlib_.dir/src/svm_c_trainer.cpp.o CMakeFiles/dlib_.dir/src/svm_rank_trainer.cpp.o CMakeFiles/dlib_.dir/src/decision_functions.cpp.o CMakeFiles/dlib_.dir/src/other.cpp.o CMakeFiles/dlib_.dir/src/basic.cpp.o CMakeFiles/dlib_.dir/src/cca.cpp.o CMakeFiles/dlib_.dir/src/sequence_segmenter.cpp.o CMakeFiles/dlib_.dir/src/svm_struct.cpp.o CMakeFiles/dlib_.dir/src/image.cpp.o CMakeFiles/dlib_.dir/src/rectangles.cpp.o CMakeFiles/dlib_.dir/src/object_detection.cpp.o CMakeFiles/dlib_.dir/src/shape_predictor.cpp.o CMakeFiles/dlib_.dir/src/correlation_tracker.cpp.o CMakeFiles/dlib_.dir/src/gui.cpp.o /opt/local/lib/libboost_python-mt.dylib /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib dlib_build/libdlib.a /usr/lib/libpthread.dylib /opt/local/lib/libX11.dylib /opt/local/lib/libpng.dylib /usr/lib/libcblas.dylib /usr/lib/liblapack.dylib /usr/lib/libsqlite3.dylib

Now it looks OK. Let’s compile…

$ cmake --build . --config Release --target install
-- Boost version: 1.58.0
-- Found the following Boost libraries:
--   python
-- Found PythonLibs: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib (found suitable version "2.7.6", minimum required is "2.6")
-- Searching for BLAS and LAPACK
-- Found OpenBLAS library
-- Looking for sgetrf_single
-- Looking for sgetrf_single - not found
-- Found LAPACK library
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/suff/Downloads/dlib-18.16/python_examples/build
Scanning dependencies of target dlib
[  0%] Building CXX object dlib_build/CMakeFiles/dlib.dir/base64/base64_kernel_1.o
[  1%] Building CXX object dlib_build/CMakeFiles/dlib.dir/bigint/bigint_kernel_1.o
[....]
[ 84%] Built target dlib
Scanning dependencies of target dlib_
[ 85%] Building CXX object CMakeFiles/dlib_.dir/src/dlib.cpp.o
[....]
[100%] Building CXX object CMakeFiles/dlib_.dir/src/gui.cpp.o
Linking CXX shared library dlib.so
[100%] Built target dlib_
Install the project...
-- Install configuration: "Release"
-- Installing: ../python_examples/dlib.so
cp dlib.so ~/projects/dlib/venv/lib/python2.7/site-packages/
$ source ~/projects/dlib/venv/bin/activate
(venv) $ python 
Python 2.7.10 (default, May 23 2015, 22:32:15)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dlib
>>>

And we have a working dlib python module.

Mac OS X, Python, ports and dlib compilation

Fixing broken Yosemite upgrade.

Sometimes it happens that upgrade procedure of Yosemite fails for some reason and you are left with a working system but starting App Store makes fans to go wild. Also seeing large number of log entries appended every second to/var/log/install.log in Console.app indicates that something went wrong.  Bummer.

Happily support.apple.com forums sugested to download latest combo update and install it fixing the issues. Sadly the system starts to behave the same way as after starting App Store.app.

I don’t like to have my workstation unstable so I was already considering complete system reinstallation and then one idea came to my mind. Single user mode and reinstallation using CLI.

To be successful I needed mentioned Combo Upgrade package and two commands:

hdiutil mount osxupdcombo10.10.4.dmg
sudo installer -package "/Volumes/OS X 10.10.4 Update Combo/OSXUpdCombo10.10.4.pkg" -target "/Volumes/Macintosh HD"

Voilà… system is fixed and again up and running.

Fixing broken Yosemite upgrade.