ParaHome: Parameterizing Everyday Home Activities
Towards 3D Generative Modeling of Human-Object Interactions

Jeonghwan Kim*, Jisoo Kim*, Jeonghyeon Na, Hanbyul Joo
Seoul National University
*Indicates Equal Contribution
Paper Code arXiv
Teaser Image

We present a novel capture system ParaHome, designed to capture and parameterize 3D movements of body, hands and diverse objects. By leveraging ParaHome, we collected a large-scale human-object interaction dataset in a common room setup with daily activity scenarios.

Abstract

To enable machines to learn how humans interact with the physical world in our daily activities, it is crucial to provide rich data that encompasses the 3D motion of humans as well as the motion of objects in a learnable 3D representation. Ideally, this data should be collected in a natural setup, capturing the authentic dynamic 3D signals during human-object interactions. To address this challenge, we introduce the ParaHome system, designed to capture and parameterize dynamic 3D movements of humans and objects within a common home environment. Our system consists of a multi-view setup with 70 synchronized RGB cameras, as well as wearable motion capture devices equipped with an IMU-based body suit and hand motion capture gloves. By leveraging the ParaHome system, we collect a novel large-scale dataset of human-object interaction. Notably, our dataset offers key advancement over existing datasets in three main aspects: (1) capturing 3D body and dexterous hand manipulation motion alongside 3D object movement within a contextual home environment during natural activities; (2) encompassing human interaction with multiple objects in various episodic scenarios with corresponding descriptions in texts; (3) including articulated objects with multiple parts expressed with parameterized articulations. Building upon our dataset, we introduce new research tasks aimed at building a generative model for learning and synthesizing human-object interactions in a real-world room setting.

System Overview

Summary of ParaHome System and Dataset

Dataset Contents
  • Human Objects Interaction in a Natural Room setting
  • Dexterous Hand Manipulation + Body Motion
  • Interaction Scene with Multiple Articulated Objects
  • Navigate in a Contextual Room Environment
  • Capture Natural Sequential Manipulation Scene
  • Total 440 minutes diverse sequence from 30 Subjects
Features
  • Scanned 22 Objects
  • Object 6D pose in the camera space
  • Hand/Body joint positions in the camera space
  • Relative orientation of hand and body joints
  • Per-frame contact information
  • Annotation for each action

Object Categories

Static Objects

Articulated Objects

Annotated Actions

BibTeX

@misc{kim2024parahome,
        title={ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions}, 
        author={Jeonghwan Kim and Jisoo Kim and Jeonghyeon Na and Hanbyul Joo},
        year={2024},
        eprint={2401.10232},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
  }