Two-sample Testing Using Deep Learning

Matthias Kirchler

1,2

Shahryar Khorasani

1

Marius Kloft

2,3

Christoph Lippert

1,4

statistics that perform an asymptotic loca-

tion test on data samples mapped onto a

hidden layer. The tests are consistent and

asymptotically control the type-1 error rate.

Their test statistics can be evaluated in lin-

ear time (in the sample size). Suitable data

representations are obtained in a data-driven

way, by solving a supervised or unsupervised

transfer-learning task on an auxiliary (poten-

tially distinct) data set. If no auxiliary data

to 35 percentage points) compared to state-

of-the-art two-sample tests such as kernel-

methods and classifier two-sample tests.

∗

1 INTRODUCTION

For almost a century, statistical hypothesis testing

has been one of the main methodologies in statistical

inference (Neyman and Pearson, 1933). A classic prob-

a standard method for analyzing empirical data in the

sciences, e.g., medicine, biology, psychology, and so-

cial sciences. In machine learning, two-sample tests

have been used to evaluate generative adversarial net-

works (Bińkowski et al., 2018), to test for covariate

shift in data (Zhou et al., 2016), and to infer causal

relationships (Lopez-Paz and Oquab, 2016).

on the data distribution are invalid. Non-parametric

tests, on the other hand, make no distributional as-

sumptions and thus could potentially be applied in a

parametric test statistics, however, can be costly as it

may require applying re-sampling schemes or comput-

ing higher-order statistics.

A non-parametric test that gained a lot of attention

in the machine-learning community is the kernel two-

sample test and its test statistic: the maximum mean

as sequences or images, and other data where deep

learning excels.

Another non-parametric two-sample test is the classifier

two-sample test (C2ST). C2ST splits the data into two

chunks, training a classifier on one part and evaluating

it on the remaining data. If the classifier predicts

significantly better than chance, the test rejects the

be put aside for training, not the full data set is used

for computing the test statistic, which limits the power

representation. It first maps the data onto a hidden-

layer of a deep neural network that was trained (in an

unsupervised or supervised fashion) on an independent,

auxiliary data set, and then it performs a location test.

Thus we are able to work on any kind of data that neu-

ral networks can work on, such as audio, images, videos,

two test statistics that can be evaluated in linear time

Fisher discriminant analysis, respectively. We derive

a variety of applications from the domains of computa-

In these experiments, the proposed deep two-sample

tests consistently outperform the closest competing

method (including deep kernel methods and C2STs) by

up to 35 percentage points in terms of the type-2 error

rate, while properly controlling the type-1 error rate.

2 PROBLEM STATEMENT &

NOTATION

hypothesis H

1

).

null hypothesis although it holds) and type-2 errors (i.e.,

0

some significance level

, and the type-2 error rate to

notation somewhat and denote the densities with re-

0

0

0

0

auxiliary task for tuning the test (the case of

We have access to four (independent) sets

,

and

, and

,

respectively. Here

and

for all

(analogue definitions hold for

, and

Let the training data be

. Denote

0

. We run a (potentially inexact) training algorithm

n

X

is a hard problem, and an

0 allows us to settle

also related to the early-stopping regularization tech-

nique, which is commonly used in training deep neural

networks (Prechelt, 1998).

3.1.2 Test Statistic

measured on the hidden layer of a network

as

n,m

N

n,m

empirical covariance matrix:

n

m

n,m

distribution that is easier to evaluate. Note that the

empirical covariance matrix is defined as:

n,m

N

X

n,m

stability and invertibility of the covariance matrix, and

1

m+n

1

3.1.3 Discussion

Intuitively, we map the data onto the last hidden layer

of the neural network and perform a multivariate loca-

tion test on whether both map to the same location.

drawn from the same distribution. Consistency of this

procedure is guaranteed by the training step.

0

0

in a regression setting, this is equivalent to an (inexact)

empirical risk minimization with loss function

φ











N

||w||≤1

N

X

i

)), (1)

for the Bayes risk

0∗

f:R

d

0

0

since for practical transfer learning, only fine-tuning

the last layer can be extremely efficient, even if the

transfer and actual task are relatively different (Lu

et al., 2015).

Relation to kernel-based tests

is a special case of the standard squared Maximum

Mean Discrepancy (Gretton et al., 2012b) with the ker-

nel

(analogously for

achieve consistency.

3.2 Control of Type-1 Error

Due to our choice of

, there need not be a unique,

well-defined limiting distribution for the test statistics

same holds for T

n,m

In particular, the following theorem will show that

n,m

tribution for

then is asymptotically

2

Testing with S

n,m

null distribution of

can be approximated as the

plementation. However, computing p-values with this

also that it suffices to evaluate the feature map

each data point only once and then permute the class

labels, saving more time.

In practice we found that the resampling-based test

performed considerably faster. Hence, in the remainder

of this work, we will evaluate the null hypothesis of the

DMMD via the resampling method.

to perform well.

The cumulative distribution function of the

2

H

distri-

bution can be evaluated very efficiently. Although for

the DFDA it is also possible to estimate the null hy-

pothesis via a Monte Carlo permutation scheme, doing

so is more costly than for the DMMD, since it involves

either a matrix inversion once or solving a linear system

for each permutation draw. Hence, in this work we

from the original task, and (

), the leniency parameter

in the training step is small enough, then our pro-

posed test is consistent, meaning the type-2 error rate

converges to 0.

Theorem 3.2.

Let

,

n

,

0∗

(i)

N

N

N

N

,

1

0

1

0

parameter in training the network, and

0

have bounded support on R

both test test statistics

m

Sketch of proof (full proof in Appendix A.2).

of

N

)), where

N

N

0

proofs in regression and classification. Namely, we can

split

0

N

error and control these via a Universal Approximation

0

the testing densities in general cannot be answered,

and similarly the Bayes error rate 1

beforehand. If abundant data for the testing task is at

hand, then splitting the data is the safe way to go; if

data is scarce, Theorem 3.2 gives justification that a

i

also in the case of

with unbounded

support. This procedure, however, requires knowledge

of where to truncate the transfer distributions. Instead

one can also grow the support of

0

; for

more details, see Appendix B.

†

(MMD) (Gretton et al., 2012a) and the kernel Fisher

discriminant analysis (KFDA) (Harchaoui et al., 2008).

Both methods effectively metricize the space of prob-

onto mean embeddings in universal reproducing kernel

Hilbert spaces (RKHS, (Steinwart and Christmann,

2008)). Test statistics derived from these mean embed-

dings can be efficiently evaluated using the kernel trick

(in quadratic time in the number of observations, al-

though there are lower-powered linear-time variations).

closely related.

Deep Kernel Methods

In the context of train-

ing and evaluating Generative Adversarial Networks

(GANs), several authors have investigated the use of

the MMD with kernels parametrized by deep neural

networks. In Bińkowski et al. (2018); Li et al. (2017);

Arbel et al. (2018), the authors feed features extracted

from deep neural networks into characteristic kernels.

Jitkrittum et al. (2018) use deep kernels in the con-

text of relative goodness-of-fit testing without directly

injectivity or even invertibility of the involved networks.

In this work we show that a linear kernel on top of

also been done by Xu et al. (2018) for GAN evaluation)

is not only sufficient for consistency of the test, but also

performs considerably better empirically in all settings

we analyzed. In addition to that, our test statistics can

be directly evaluated in linear instead of quadratic time

(in the sample size) and the corresponding asymptotic

null distributions can be exactly computed (in contrast

Classifier Two-Sample Tests (C2ST)

First pro-

posed by Friedman (2003) and then further analyzed by

as a neural network or a

for the two-sample testing problem. In particular, they

split the available data into training and test set, train

Our method circumvents the need to split the data in

first variant is the “median distance“ heuristic (Gretton

et al., 2012a) which selects the median of the euclidean

data in two disjoint sets and selects the bandwidth

that maximizes power on the first set and evaluates

the MMD on the second set (MMD-opt). We use the

For the

and

of test locations based on the task and sample size. The

locations are selected either randomly (as presented by

Chwialkowski et al. (2015)) or optimized on half of the

In addition to these published methods, we also com-

Figure 1: Results on AM audio (top row) and natural image (bottom row) data sets. Suffixes “-sup“ indicate

supervised pretraining, “-unsup“ indicates unsupervised pretraining.

5.2 Control of Type-1 Error Rate

in Appendix D.4.

5.3 Power Analysis

Amplitude Modulated Audio Data

ferent songs after they have been amplitude modulated

matik (2014); distribution

distribution

from track five and the remaining tracks

on the album were used for training the network in a

multi-class classification setting. As our neural network

architecture we use a simple convolutional network, a

Appendix D.6 for details.

Figure 1b reports the results with varying number of

observations under constant noise level

= 1. Our

method shows high power, even at low sample sizes,

Aircraft

Classification of Aircraft data set (Maji et al., 2013).

We select two visually similar aircraft families, namely

Boeing 737 and Boeing 747 as populations

and

,

respectively. The neural network embeddings are ex-

tracted from a ResNet-152 (He et al., 2016) trained on

ILSVRC (Russakovsky et al., 2015). Figure 1c shows

that all neural network architectures perform consid-

erably better than the kernel methods. Furthermore,

our proposed tests can also outperform both the C2ST

Facial Expressions

X (# obs) Y (# obs) p-value

CN (490) AD (314) 9.49 · 10

CN (490) MCI (287) 2.44 · 10

MCI (287) AD (314) 1.45 · 10

APOE ε3 (811) APOE ε4 (152) 1.40 · 10

in Figure 1d. Even though the images in ImageNet

and KDEF are very different, the neural network tests

again outperform the kernel methods. Also note that

the apparent advantage of the mean embedding test

for low sample sizes is due to an unreasonably high

type-1 error rate (

11 and

respectively).

Stanford Dogs

Lastly, we evaluate our tests on the

Stanford Dogs data set (Khosla et al., 2011), consisting

of 120 classes of different dog breeds. As test classes

this data; the encoder part is identical to the super-

approach outperform other tests by a large margin.

However, the unsupervised DMMD and DFDA still

severely from the mediocre feature embedding than our

tests. One potential explanation for this phenomenon

is the ability of DMMD and DFDA to fine-tune on the

(indicated by red arrows) in the Alzheimer’s disease

patient.

test data without the need to perform a data split.

Three-dimensional Neuroimaging Data

In this

section, we apply the DFDA test procedure to 3D

Magnetic Resonance Imaging (MRI) scans and genetic

information from the Alzheimer’s Disease Neuroimag-

ing Initiative (ADNI) (Mueller et al., 2005). To this

has been trained on MRI scans from the Brain Ge-