Updated: May 3, 2020

Human-in-the-Loop Differential Subspace Search in High-Dimensional Latent Space  

Chia-Hsing Chiu1, Yuki Koyama2, Yu-Chi Lai1, Takeo Igarashi3, Yonghao Yue4,
1National Taiwan University of Science and Technology     2National Institute of Advanced Industrial Science and Technology (AIST)     3The University of Tokyo     4Aoyama Gakuin University (AGU)

Generative models based on deep neural networks often have a high-dimensional latent space, ranging sometimes to a few hundred dimensions or even higher, which typically makes them hard for a user to explore directly. We propose differential subspace search to allow efficient iterative user exploration in such a space, without relying on domain- or data-specific assumptions. We develop a general framework to extract low-dimensional subspaces based on a local differential analysis of the generative model, such that a small change in such a subspace would provide enough change in the resulting data. We do so by applying singular value decomposition to the Jacobian of the generative model and forming a subspace with the desired dimensionality spanned by a given number of singular vectors stochastically selected on the basis of their singular values, to maintain ergodicity. We use our framework to present 1D subspaces to the user via a 1D slider interface. Starting from an initial location, the user finds a new candidate in the presented 1D subspace, which is in turn updated at the new candidate location. This process is repeated until no further improvement can be made. Numerical simulations show that our method can better optimize synthetic black-box objective functions than the alternatives that we tested. Furthermore, we conducted a user study using complex generative models and the results show that our method enables more efficient exploration of high-dimensional latent spaces than the alternatives.


Human-in-the-loop optimization, dimensionality reduction, generative models


We thank the anonymous reviewers for their insightful suggestions and discussions. This work was supported in part by a grant from JST CREST, JPMJCR17A1 (HCI for Machine Learning), Japan, a JSPS Grant-in-Aid for Scientific Research (A) 18H04106, Japan, and grants from the Ministry of Science and Technology of Taiwan, MOST-107- 2221-E-011-114-MY2 and MOST-107-2221-E-011-112-MY2.

ACM Transactions on Graphics, 39(4) (Proc. of SIGGRAPH 2020), 1:1-15, 2020

Paper: PDF(13.9MB)
Video: youtube video | High res MP4(25.9MB)
Supplementary mateiral: ZIP(151.8MB)


        author = {Chiu, Chia-Hsing and Koyama, Yuki and Lai, Yu-Chi and
                        Igarashi, Takeo and Yue, Yonghao},
        title = {Human-in-the-Loop Differential Subspace Search in High-Dimensional Latent Space},
        journal = {ACM Trans. Graph.},
        volume = {39},
        number = {4},
        month = jul,
        year = {2020},
        pages = {1:1--15},
        articleno = {1},
        numpages = {15},