Generated from nbs/tf_binding_models.ipynb

Model benchmarking with Kipoi

This tutorial will show to to easily benchmark tf-binding models in Kipoi. By providing a unified access to models, it takes the same effort to run a simple PWM scanning model then to run a more complicated model (DeepBind in this example).

Load software tools

Let's start by loading software for this tutorial: the kipoi model zoo,

import kipoi
import numpy as np
from sklearn.metrics import roc_auc_score

Prepare data files

Next, we introduce a labeled BED-format interval file and a genome fasta file

intervals_file = 'example_data/chr22.101bp.2000_intervals.JUND.HepG2.tsv'
fasta_file  = 'example_data/hg19_chr22.fa'
dl_kwargs = {'intervals_file': intervals_file, 'fasta_file': fasta_file}

Let's look at the first few lines in the intervals file

!head $intervals_file
chr22   20208963    20209064    0
chr22   29673572    29673673    0
chr22   28193720    28193821    0
chr22   43864274    43864375    0
chr22   18261550    18261651    0
chr22   7869409 7869510 0
chr22   49798024    49798125    0
chr22   43088594    43088695    0
chr22   35147671    35147772    0
chr22   49486843    49486944    0

The four columns in this file contain chromosomes, interval start coordinate, interval end coordinate, and the label. This file contains 2000 examples, 1000 positives and 1000 negatives.

Let's load the labels from the last column:

labels = np.loadtxt(intervals_file, usecols=(3,))

Next, to evaluate the DeepBind model for JUND, we will 1) install software requirements to run the model, 2) load the model, and 3) get model predictions using our intervals and fasta file.

Install DeepBind model software requirements

deepbind_model_name = "DeepBind/D00776.005"
kipoi.install_model_requirements(deepbind_model_name)
# Use `$ kipoi env install DeepBind/D00776.005 --gpu` from the command-line to install the gpu version of the dependencies
Conda dependencies to be installed:
['python=2.7', 'h5py']
Fetching package metadata ...........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /users/jisraeli/local/anaconda/envs/kipoi:
#
h5py                      2.7.1            py27h2697762_0  
pip dependencies to be installed:
['tensorflow==1.4', 'keras==2.1.4']
Requirement already satisfied: tensorflow==1.4 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages
Collecting keras==2.1.4
  Using cached Keras-2.1.4-py2.py3-none-any.whl
Requirement already satisfied: enum34>=1.1.6 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: backports.weakref>=1.0rc1 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: wheel in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: mock>=2.0.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: tensorflow-tensorboard<0.5.0,>=0.4.0rc1 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: numpy>=1.12.1 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: protobuf>=3.3.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: six>=1.10.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow==1.4)
Requirement already satisfied: pyyaml in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from keras==2.1.4)
Requirement already satisfied: scipy>=0.14 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from keras==2.1.4)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow==1.4)
Requirement already satisfied: pbr>=0.11 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow==1.4)
Requirement already satisfied: bleach==1.5.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4)
Requirement already satisfied: futures>=3.1.1; python_version < "3.2" in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4)
Requirement already satisfied: markdown>=2.6.8 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4)
Requirement already satisfied: werkzeug>=0.11.10 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4)
Requirement already satisfied: html5lib==0.9999999 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow==1.4)
Requirement already satisfied: setuptools in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from protobuf>=3.3.0->tensorflow==1.4)
Installing collected packages: keras
  Found existing installation: Keras 2.0.4
    Uninstalling Keras-2.0.4:
      Successfully uninstalled Keras-2.0.4
Successfully installed keras-2.1.4

Load DeepBind model

deepbind_model = kipoi.get_model(deepbind_model_name)
/users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.

Get DeepBind predictions

deepbind_predictions = deepbind_model.pipeline.predict(dl_kwargs, batch_size=1000)

Evaluate DeepBind predictions

Let's check the auROC of deepbind predictions:

roc_auc_score(labels, deepbind_predictions)
0.808138

Load, run, and evaluate a HOCOMOCO PWM model

pwm_model_name = "pwm_HOCOMOCO/human/JUND"
kipoi.install_model_requirements(pwm_model_name)
# Use `$ kipoi env install pwm_HOCOMOCO/human/JUND --gpu` from the command-line to install the gpu version of the dependencies
pwm_model = kipoi.get_model(pwm_model_name)
pwm_predictions = pwm_model.pipeline.predict(dl_kwargs, batch_size=1000)
print("PWM auROC:")
roc_auc_score(labels, pwm_predictions)
Conda dependencies to be installed:
['python=3.5', 'h5py']
Fetching package metadata ...........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /users/jisraeli/local/anaconda/envs/kipoi:
#
h5py                      2.7.1            py27h2697762_0  
pip dependencies to be installed:
['tensorflow', 'keras==2.0.4']
Requirement already satisfied: tensorflow in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages
Collecting keras==2.0.4
Requirement already satisfied: enum34>=1.1.6 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: backports.weakref>=1.0rc1 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: wheel in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: mock>=2.0.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: tensorflow-tensorboard<0.5.0,>=0.4.0rc1 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: numpy>=1.12.1 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: protobuf>=3.3.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: six>=1.10.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow)
Requirement already satisfied: theano in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages/Theano-1.0.1-py2.7.egg (from keras==2.0.4)
Requirement already satisfied: pyyaml in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from keras==2.0.4)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow)
Requirement already satisfied: pbr>=0.11 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow)
Requirement already satisfied: bleach==1.5.0 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Requirement already satisfied: futures>=3.1.1; python_version < "3.2" in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Requirement already satisfied: markdown>=2.6.8 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Requirement already satisfied: werkzeug>=0.11.10 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Requirement already satisfied: html5lib==0.9999999 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow)
Requirement already satisfied: setuptools in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from protobuf>=3.3.0->tensorflow)
Requirement already satisfied: scipy>=0.14 in /users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages (from theano->keras==2.0.4)
Installing collected packages: keras
  Found existing installation: Keras 2.1.4
    Uninstalling Keras-2.1.4:
      Successfully uninstalled Keras-2.1.4
Successfully installed keras-2.0.4


/users/jisraeli/local/anaconda/envs/kipoi/lib/python2.7/site-packages/keras/models.py:255: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  custom_objects=custom_objects)


PWM auROC:





0.6431155

In this example, DeepBind's auROC of 80.8% outperforms the HOCOMOCO PWM auROC of 64.3%