Jiwan Chung
Try Notion

Jiwan Chung

I'm a Ph.D. student @ Yonsei MIR lab, advised by Youngjae Yu.
I study multimodal machine learning and knowledge understanding.
My long-term goal is to understand the operations of human understanding using computational methods.
✉️ Mail / 🎓 Google Scholar / 💻 Github My main point of contact is via Email!

📚 Publications

VLIS: Unimodal Language Models Guide Multimodal Language Generation (EMNLP 2023: lead author)
Finding a better balance between visual grounding & linguistic fluency by interpreting visual-language models as importance weights
Reading Books is Great, But Not if You Are Driving! Visually Grounded Reasoning about Defeasible Commonsense Norms (EMNLP 2023: co-author)
Paper / Github (tbu)
Commonsense norms dataset extended with visual context, with a focus on defeasibility
Long Story Short: a Summarize-then-Search Method for Prompt-Based Long Video Question Answering (BMVC 2023: lead author)
Obtaining SOTA in long video QAs with language-model-based summarization & retrieval
Fusing Pre-trained Language Models with Multimodal Prompts through Reinforcement Learning (CVPR 2023: lead author)
Using RL (and RL only) for extending language models to multimodal captioning
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning (ICCV 2021: lead author)
A clustering-mutual-information based scalable automatic curation method for better audio-visual correspondence
Transitional Adaptation of Pretrained Models for Visual Storytelling (CVPR 2021: lead author)
Two-stage adaptation of GPT to visual captioning improves downstream performance
Character Grounding and Re-identification in Story of Videos and Text Descriptions (ECCV 2020: co-author)

🧭 Projects

Naver HyperCLOVA X LLM (2023)
Contributed to training & evaluating a new Korean-centric pretrained language model
Website / Tech Report (tbu)

⌨️ Programming Skills

Machine Learning
PyTorch: For debugging & fast prototyping
Multi-node scaling with:
Jax: For scaling up to multi-gpu or tpu environments
Gradio: For demos & data explorers
Web: some experience on VueJS, SvelteJS, MongoDB, Dgraph, Azure, etc.

🐣 Education

Bachelor’s Degree (Yonsei University, ~2018)
Double Major
Computer Science
English Language & Literature
Master’s Degree (Seoul National University, ~2022)
Computer Engineering (Machine Learning)
Doctor’s Degree (Yonsei University)
Artificial Intelligence (Multimodal Learning)

💬 Experience

Teaching Assistant
Discrete Mathematics (Undergraduate Course, Fall 2019)
IoT/AI/Big-Data (IAB) (Undergraduate Course, Fall 2019)
Artificial Intelligence (AI) (Graduate Course, Spring 2021)
Computer Vision (CV) (Undergraduate Course, Fall 2021)
Multimodal Machine Learning (Graduate Course, Spring 2023)