| id |
acadia23_v2_92 |
| authors |
Pinochet, Diego |
| year |
2023 |
| title |
A Computational Gestural Making Framework: A Multi-modal Approach to Digital Fabrication Mapping Human Gestures to Machine Actions |
| source |
ACADIA 2023: Habits of the Anthropocene: Scarcity and Abundance in a Post-Material Economy [Volume 2: Proceedings of the 43rd Annual Conference for the Association for Computer Aided Design in Architecture (ACADIA) ISBN 979-8-9891764-0-3]. Denver. 26-28 October 2023. edited by A. Crawford, N. Diniz, R. Beckett, J. Vanucchi, M. Swackhamer 92-103. |
| summary |
This research project implements a multimodal body-centric approach to interactive fabrication aimed to test the conversational aspects of a design framework (Figure 1). It focuses on the development of a gesture language as the primary mode of commu- nication, as well as the means to generate effective communication with a machine for design endeavors. To do so, we first developed a gesture recognition system that aims to establish fluid communication with a machine based on three types of gestures: symbolic, exploratory, and sequential. Second, we developed a system for machine vision to detect, recognize, and calculate physical objects in space. Third, we developed a system for robotic motion using path-planning algorithms and reinforcement learning for colli- sion-free machine movement. Finally, those three modules were integrated into a system for human-robot interaction in real time based on gestures. The ultimate goal of this imple- mentation is to establish a multimodal framework for interactive design that is based on human-robotic interaction through the use of gestures as a communication mechanism for exploring computational design potential toward unique and original creations. |
| series |
ACADIA |
| type |
paper |
| email |
|
| full text |
 file.pdf (1,520,005 bytes) |
| references |
Content-type: text/plain
|
D. Bolya, C. Zhou, F. Xiao, and Y. J. Lee (2019)
 Yolact: Real-time instance segmentation
, Proceedings of the IEEE/CVF international conference on computer vision
|
|
|
|
|
Daniela Mitterberger, Selen Ercan Jenny, Lauren Vasey, Ena Lloret-Fritschi, Petrus Aejmelaeus-Lindström, Fabio Gramazio, and Matthias Kohler (2022)
Interactive Robotic Plastering: Augmented Interactive Design and Fabrication for On-Site Robotic Plastering
, Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. CHI ’22. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3491102.3501842
|
|
|
|
|
Diego Pinochet (2015)
Making gestures: design and fabrication through real-time human-computer interaction (Master of Science Thesis)
, Massachusetts Institute of Technology
|
|
|
|
|
E. Hallam and T. Ingold (2008)
Creativity and Cultural Improvisation
, New York, NY: Bloomsbury Academic
|
|
|
|
|
Eduardo Lyon (2011)
Emergence and convergence of knowledge in building production: Knowledge-based design and digital manufacturing
, Distributed Intelligence in Design, ed. Tuba Kocatürk and Benachir Medjdoub. Blackwell Publishing
|
|
|
|
|
Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann (2020)
Mediapipe hands: On-device real-time hand tracking
, arXiv preprint arXiv:2006.10214
|
|
|
|
|
J. Kuffner and M. LaValle (2000)
RRT-connect: An efficient approach to single-query path planning
, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065), San Francisco, CA, USA, 2000, pp. 995-1001 vol.2, doi: 10.1109/ROBOT.2000.844730
|
|
|
|
|
Liwei Chan, Stefanie Müller, Anne Roudaut, and Patrick Baudisch (2012)
CapStones and ZebraWidgets: Sensing Stacks of Building Blocks, Dials and Sliders on Capacitive Touch Screens
, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2189–92. CHI ’12. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2207676.2208371
|
|
|
|
|
Madeline Gannon (2017)
Human-centered Interfaces for autonomous fabrication machines
, Doctoral Dissertation
|
|
|
|
|
Mauro Maldonato (2015)
The Predictive Brain: Consciousness, Decision and Embodied Action
, Sussex Academic Press. 2014. 112 Pp. ISBN: 9781845196394. The British Journal of Psychiatry 206, no 6, 524–24. doi:10.1192/bjp. bp.114.161554
|
|
|
|
|
R.A. Schmidt (1975)
A Schema theory of discrete motor skill learning
, Psychological Review, vol 82. American Psychological Association
|
|
|
|
|
Ross B. Girshick (2015)
Fast R-CNN
, CoRR, abs/1504.08083. URL http://arxiv.org/abs/ 1504.08083
|
|
|
|
|
Ryan Luke Johns, Jeffrey Anderson, and Axel Kilian (2019)
Robo-Stim: Modes of human robot collaboration for design exploration
, Impact: Design With All Senses: Proceedings of the Design Modelling Symposium, Berlin 2019, pp. 671-684. Springer International Publishing
|
|
|
|
|
Ryan Luke Johns (2013)
Augmented reality and the fabrication of gestural form
, Rob| Arch 2012: Robotic Fabrication in Architecture, Art, and Design, pp. 248-255. Springer Vienna
|
|
|
|
|
Ryan Luke Johns (2014)
Augmented materiality: modeling with material indeterminacy
, Fabricate. gta Verlag, Zurich: 216-223
|
|
|
|
|
Stefanie Mueller, Anna Seufert, Huaishu Peng, Robert Kovacs, Kevin Reuss, Francois Guimbretiere, and Patrick Baudisch (2019)
FormFab: Continuous Interactive Fabrication
, Proceedings of the Thirteenth International Conference on Tangible, Embedded, and Embodied Interaction, 315–23. TEI ’19. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3294109.3295620
|
|
|
|
|
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, P. Abbeel, S. Levine (2018)
Soft actor-critic algorithms and applications
, arXiv preprint arXiv:1812.05905
|
|
|
|
|
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine (2018)
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
, International conference on machine learning, pp. 1861-1870. PMLR
|
|
|
|
| last changed |
2024/12/20 09:12 |
|
