INRIA-NSC Associate Team FORMOSA


Introduction

Partners

2013 Activities

2014 Activities

2015 Activities

2016 Activities

Publications

Smart Graphics 2014 Outcome


Introduction

The application context targeted by this project is Interactive Virtual Storytelling. Interactive Storytelling is an emerging media which allows users to alter the content and outcome of narratives through role-playing and specific actions, thus making the narrative more immersive and more compelling. The growing importance of this form of media reveals the necessity to re-think and re-assess the way narratives are traditionally structured and authored. In turn, this requires from the research community to address complex scientific and technical challenges at the intersection of robotics, artificial intelligence, and computer graphics.

The objectives we address include:

  1. Delivering better authoring tools for designing interactive narratives based on literary-founded narrative structures.
  2. Establishing a bridge between the semantic and geometric levels of the narrative to enable the simulation of complex and realistic interactive scenarios in 3D.
  3. Providing a full integration of the cinematographic dimension through the control of high-level elements of filmic style

Partners

FORMOSA involves two teams: Intelligent Media Lab in National Taiwan Chengchi University, and MimeTIC in IRISA/INRIA Rennes

MimeTIC LogoThe MimeTIC Research Team focuses on designing methods for simulating virtual humans (VH) that behave in a realistic manner and act with realistic motions. "Realistic" means as a real human would have moved and behaved in the same situation. It involves understanding how real people control their motion and behave in order to design models capable of generating realistic virtual humans.

Research domains of MimeTIC incorporate motion sensing, autonomous characters, biomechanics and motion analysis, interactive digital storytelling, crowd simulation, and VR and Sports.

IMLab LogoThe Intelligent Media Lab is a research laboratory led by Prof. Tsai-Yen Li under the Computer Science Department at National Chengchi University. IM Lab is dedicated to understanding how research results in the fields of Artificial Intelligence and Multimedia can be applied to the storage, processing, transmission, and presentation of innovative intelligent media.

Research carried out at IMLab extends broadly among the fields of Robotics, Artificial Intelligence, Motion Planning, Virtual Reality, Computer Graphics, Computer Animations, Image Processing, Computational Geometry, e-Learning, and Distributed Systems, to develop innovative tools and techniques.

Activities

PhD Student Carl Jorgens Internship, NCCU

Period: 18 February - 5 May 2013

Purpose: Establishment of semantic environement analysis for crowd planning, and its applications to generating meaningful story plans.

Discussions and Results: [+] Details

  • Discussions with Tsai-Yen Li and his team on the possible applications of activity scheduling systems in the domain of automated storytelling. The conclusion was that meeting behaviors were primordial for automatized storytelling purpose.
  • Development of a meeting scheduling model for multiple agents in spatially and temporally constrained scenario
  • Current work on an article describing this model and its applications

Dr. Marc Christie and Dr. Fabrice Lamarche Visit, NCCU

Period: 9-13 April 2013

Purpose: Discussion of Smart Graphics organization, exchange of two PhD students (Carl and Chieh-Jen), and annual associate team research topics.

Discussions and Results: [+] Details

  • Expected budget for smart graphics and plan for first year budget spendings
  • Work plan for establishing mutual platform The Theater
  • Defining new research topics.
  • Plan for Chieh-Ren Chen's summer internship topic.
  • Conference Aims for the first year

Master's Student Hui-Yin Wu Visit, INRIA

Period: 19-24 May 2013

Purpose: Taking the opportunity of CASA to present current framework. Extending existing storytelling framework to incorporate embedded narratives for generating temporal arrangements. Design 3D environment for new story.

Discussions and Results: [+] Details

  • Present current work in CASA 2013 conference.
  • We extend the current story filtering and personalization mechanism to consider logical control situations with non-linear story formations. The interactive narrative can be reorganized to play as a flashback, while still preserviing ppersonalization, interaction, and story structure.
  • In order to demonstrate the logic control of non-linaer interactive narratives, we design a story involding a flashback of a murder event. During this visit we design the virtual envirnoment and select models and characters.

PhD Student Chieh-Jen Chen Internship, INRIA

Period: 22 June - 20 September 2013

Purpose:

Discussions and Results: [+] Details

  • Design and build an interactive controller for the story telling.
  • Through the design, users can explore the story world in an intuitive way: titling the controller to change the camera view, switching the device to control the characters and the control cube can change emotions of characters.
  • Emotion system is new attempt to interact between users and the storytelling system. A non-direct interactive way by changing the control cube status is defined and continuously recorded. The data is used at branching points to decide the story route without asking users for an explicit choice.

Dr. Marc Christie and Engineer Julian Joseph Visit, NCCU

Period: 16 -20 September 2013

Purpose: Build Theater as collaborative platform. Discussion for publication aims and Smart Graphics work items.

Discussions and Results: [+] Details

  • Organize working schedule for Smart Graphics Conference.
  • Prepared submission for conference.
  • Discussion on mutual work platform as well as tutorial for Master's students in IMLab.
  • Create interface tools in Unity for easy camera configuration and control.

PhD Student Hui-Yin Wu, INRIA

Period: 28 September - 1 October 2013

Purpose: Pre-arrival for PhD studies and preparation for conference submission.

Discussions and Results: [+] Details

  • Discussions with Pr. Michael Young from North Carolina State University
  • Completed submission for international conference

Activities

Master's Student Pei-Chun Lai Visit, INRIA

Period: 12 November - 12 December 2013

Purpose: Context aware cinematographic subsystem for virtual environment.

Discussions and Results: [+] Details

  • Development of a camera component that makes decisions on camera position from context information.
  • Discussions for publication in Smart Graphics 2014

Prof. Tsai-Yen Li Visit, INRIA

Period: 25-30 November 2013

Purpose: Discussion for next year's project aims and organization of Smart Graphics 2014.

Discussions and Results: [+] Details

  • Discussions on organization of Smart Graphics 2014
  • Setting research objectives and topics of interest for 2014

Dr. Marc Christie and PhD Student Hui-Yin Wu, NCCU

Period: 3-9 March 2014

Purpose: Smart Graphics Organization and integration of master's student Clément Desoche for internship in Taiwan.

Discussions and Results: [+] Details

  • Organization of Smart Graphics 2014
  • Collaboration on publication aim with Pei-Chun Lai

PhD Student Hui-Yin Wu, North Carolina State University

Period: 28 May 2014

Purpose: Presentation on current work: "Developing Temporal Structures in 3D Interactive Storytelling" to the computer science department at North Carolina State University.

Official announcement

Master's Student Clément Desoche, NCCU

Period: 1 March - 18 May 2014

Purpose: 2 month exchange on a joint master's project on methods for generating camera motion graphs.

Discussions and Results: [+] Details

  • Project on camera motion graphs.
  • Results published in Symposium of Computer Animations 2014

Prof. Tsai-Yen Li and PhD student Cunka Sanokho, Copenhagen

Period: 21-23 July 2014

Purpose: Attending 13th ACM Symposium on Computer Animation

Discussions and Results: [+] Details

  • Discussion on joint publication
  • Presentation of a paper in Symposium of Computer Animations 2014

Master's Student Jouwen Wang, INRIA

Period: 26 June - 15 August 2014

Purpose: Developing a smart interface on tablets that would serve as a light-weight cinematographic assistant for beginning directors or film makers.

Discussions and Results: [+] Details

  • Prototype interface for camera editing.

Dr. Marc Christie, PhD Student Carl-Johan Jorgensen, and PhD Student Hui-Yin Wu, NCCU

Period: 27-29 August 2014

Purpose: Organizing and attending Smart Graphics 2014 conference

Discussions and Results: [+] Details

  • Hosting of keynote speakers Takeo Igarashi and Karen Singh
  • Presentation of research developments by Carl-Johan Jorgensen and Pei-Chun Lai
  • Organization of the symposium and invitation of international scholars.
  • Proceedings published by Springer.

Dr. Marc Christie, PhD Quentin Galvane, NCCU

Period: 6-12 December 2014

Purpose: Collaboration with master student Jouwen Wang in building first prototype interface for smart cinematographic assistant tool.

Discussions and Results: [+] Details

Activities

M. Christie, H. Yin Wu, NCCU

Period: 24 Feb. - 4 March 2015

Purpose: prototype interface for smart cinematographic assistant tool.

Discussions and Results: [+] Details

  • Enhancing Youwen's interface. Preparing a submission at a UI conference
  • Discussions for collaboration (joint PhD)

Prof. Tsai-Yen Li Visit, INRIA

Period: 17-19 July 2015

Purpose: PhD defense of co-supervised student Carl Jorgensen

Discussions and Results: [+] Details

  • Planning and scheduling actions at semantic and geometric levels
  • Next steps of Jouwen Wang's work

Prof. Tsai-Yen Li Visit, INRIA

Period: 24-26 Septembre 2015

Purpose: PhD defense of student Billal Merabti

Discussions and Results: [+] Details

  • Insight annotation tool: towards semi-automated annotations

M. Christie, F. Lamarche, H-Y. Wu, INRIA

Period: 26 October - 4 Novembre 2015

Purpose: Cinematographic pattern language tool, and Youwen's UI work

Discussions and Results: [+] Details

  • Discussing pattern representations
  • Call for joint PhD funding

Activities

H. Yin Wu, Taiwan

Period: Aug 2016

Purpose: Working on the Patterns language for film analysis and synthesis

Discussions and Results: [+] Details

Ya-web Su, INRIA

Period: July-Sept 2016

Purpose: Master internship

Discussions and Results: [+] Details

I-SHeng Lin, INRIA

Period: Oct 2016 (1 week)

Purpose: Smart interfaces for behavioral characters

Discussions and Results: [+] Details

Tsai-yen Li, INRIA

Period: Oct 2015 (1 week)

Purpose: Smart interfaces for behavioral characters

Discussions and Results: [+] Details

Publications

2013

H.-Y. Wu, M. Christie, T.-Y. Li, 2013.05, "Stories Animated: A Framework for Personalized Interactive Narratives using Filtering of Story Characteristics," The 26th International Conference on Computer Animation and Social Agents (CASA 2013), Istanbul.

H.-Y. Wu, T.-Y. Li, M. Christie, 2013.07, "Stories Animated: A Framework for Personalized Interactive Narratives using Filtering of Story Characteristics" Computer Graphics Workshop, Xinzhu, Taiwan.

H.-Y. Wu, M. Christie, T.-Y. Li, 2013.11, "Shaping Interactive Stories in 3D Environments," in ACM Conference on Motion in Games 2013 (MIG 2013), Dublin, Ireland.(Poster)

C. Jorgensen, F. Lamarche (2013). Combining Activity Scheduling and Path Planning to Populate Virtual Cities. Autonomous Agents and Multi-Agent Sytems (AAMAS Poster) 2013.

2014

C. Sanokho, C. Desoche, B. Merabti, M. Christie, T.-Y. Li, 2014.07, "Camera Motion Graphs", in Proceedings of ACM Syposium on Computer Animation 2014, Copenhagen, Denmark.

P.-C. Lai, H.-Y. Wu, M. Christie, and T.-Y. Li, 2014.07, "An Experimental Platform for Customized Virtual Cinematography in Interactive Storytelling” Computer Graphics Workshop 2014, Taipei, Taiwan.

P.-C. Lai, H.-Y. Wu, C. Sanokho, M. Christie, T.-Y. Li, 2014.08, "A Pattern-based Tool for Creating Virtual Cinematography in Interactive Storytelling" in LNCS Proceedings of The 12th International Symposium on Smart Graphics 2014, Taipei, Taiwan.

C-J. Jorgensen, F. Lamarche, 2014.01, "Semantically Consistent Hierarchical Decomposition of Virtual Urban Environments." in LNCS Proceedings of The 12th International Symposium on Smart Graphics 2014, Taipei, Taiwan.

Marc Christie and Tsai-Yen Li, 2014.08, Associate editors of Smart Graphics - 12th International Symposium, Taipei, Taiwan, Proceedings. Lecture Notes in Computer Science 8698.

2015

H.-Y. Wu, T.-Y. Li, M. Christie, 2015.08, "Logic Control for Story Graphs in 3D Game Narratives” Proceedings of 13th International Symposium on Smart Graphics, Chengdu, China.

J. Wang, H.-Y. Wu, Q. Galvane, M. Christie, T.-Y. Li, 2015.07 "Design of Intelligent Editing System for Computer Animation Director,” Computer Graphics Workshop 2015, Taipei, Taiwan

H.-Y. Wu and M. Christie, 2015.06, "Placing Invisible Cameras: Conversation Between Cognitive Science and Automated Virtual Cinematography” Conference on Cognitive Science and Moving Images 2015 (SCSMI2015), London, UK.

H.-Y. Wu and M. Christie, "Stylistic Patterns for Generating Cinematographic Sequences” Proceedings of 4th Eurographics Workshop on Cinematography and Editing, 2015, Zurich, Switzerland.

B. Merabti, H.-Y. Wu, C. Sanokho, Q. Galvane, M. Christie, 2015.05, "Insight:An annotation tool and format targeted towards film analysis” Proceedings of 4th Eurographics Workshop on Cinematography and Editing, 2015, Zurich, Switzerland.

2016

Hui-yin Wu, Cinematic Discourse for Interactive 3D Storytelling, PhD Thesis, University of Rennes 1, Oct 2016.

Hui-Yin Wu, Marc Christie, Analysing Cinematography with Embedded Constrained Patterns , 5th International Eurographics workshop on Intelligent Cinematography and Editing, 2016

Hui-Yin Wu, Michael Young, Marc Christie, A Cognitive-Based Model of Flashbacks for Computational Narratives, Proceedings of the Twelfth Artificial Intelligence and Interactive Digital Entertainment Conference (AIIDE 2016), 2016

J.W. Wang, H.Y. Wu, Q. Galvane, T.-Y. Li, and M. Christie, Design of An Intelligent Camera Editing System for Computer Animation. 2016.7, Proceedings of Computer Graphics Workshop, Taipei

Work Program for 2015

For the third year of the FORMOSA associate team, our aim continues to follow the original proposal driven by our three key objectives. This also involves integrating the related developments onto the Theater 3D platform, a platform which is collaboratively developed by both partners. The research topics to address in 2015 are:
  1. Designing authoring tools for story creators and cinematographers. After proposing a pattern-based language to describe evolved cinematographic techniques [SG2014], our intention is to explore interactive means for content creation by proposing high-level manipulators to ease the specification of cinematographic sequences over story events. The work will integrate both the pattern-based language (for its declarative capacity), and constraint-based filtering techniques to compute candidate shots. The constraint-based techniques will maintain the consistency with the specifications while the user manipulates low-level parameters (shots size, target object, vantage) or high-level parameters (adding, stretching or blending discursive cinematographic techniques)
  2. Establishing a bridge between the semantic and geometric levels. While the model we proposed in [Jorgensen2014] is powerful and expressive, it does present scalability issues that need to be addressed before a full integration with the narrative engine developed in the first year is possible.
  3. Providing a full integration of the cinematographic dimension. Our key objective for 2015 is to develop the bridge between the story context provided by the narrative engine and the virtual cinematography through the design and implementation of a technical cinematographic language that translates from story parameters, to high level patterns comprised of camera techniques, and realized by shots extracted and annotated from real movies. In the third year, we expect to have a fully flexible language for film annotation and smart tools for virtual camera planning that can assist camera/shot selection for story creators.

Work Program for 2017

Year 2017 will be structured around three axes:
  1. Novel interactions and temporal story structures
    • experimenting immersive systems with novel interactive devices (VR interactions)
    • evaluate assistive authoring tools and for virtual cinematography with NCCU and ENS Lumiere
    • enhancing capacities of storytelling applications by researching on temporal structures in narratives (flashback structures)
    • Coupling temporal story constraints with stylistic preferences with behavior planning of virtual characters
    • providing virtual agents with realistic behaviors able to conform to complex scenarios and fulfill storyline constraints (eg relative placement of characters in the scene)
    • coupling the possibilities of character placements with the cinematic constraints and stylistic requirements.
  2. Towards data-driven computational approaches to virtual cinematography involving the collection and analysis of existing cinematographic data, and application of such data to interactive storytelling using cinematic patterns:
    • design and evaluation of tools to collect and annotate cinematographic data (based on our Insight research prototype)

Budget 2015

Expected Activities INRIA - France

Researcher Nature of Expense Expense Cost Total
INRIA/Christie, Lamarche 2 Short Visits (1 week) Travel 1 250 5 000
INRIA/Galvane, Wu 1 Short Visit (2 weeks) Travel 1 250 2 500
NCCU/Li 2 Short Visits (1 week) Accommodation 1 250 2 500
Total 10 000

NCCU - Taiwan Corresponding Budget Plan (In NTD)

Researcher Nature of Expense Expense Cost Total
INRIA/Christie, Lamarche 2 Short Visits (1 week) Accommodation 35 000 140 000
INRIA/Galvane, Wu 1 Short Visit (2 weeks) Accommodation 20 000 40 000
NCCU/Li 2 Short Visits (1 week) Travel Only 55 000 110 000
Total 290 000