World Within Words: Multisensory Transformation of Books and Physical Model Creation

The project aims to redefine the reading experience for individuals, by exploring different mediums to translate textual content. A key innovation being tested is the creation of physical models that interpret and represent the content of books. By incorporating a human-machine thinking loop, the project seeks to redefine future reading experiences, making use of artificial intelligence to assist in translating and summarizing complex language and emotions found in various chapters.

The diagram briefly introduces how the AI read the text, convert it into images and create the final physical products. The key step is design the prompt which guiding the AI to understand the books. Especially 


Prompt Design - How to guide the AI to read our text ?

STEP01 Defining the book

  • Prompt: "Identify the book title, author, and publication year. Outline the objectives of analyzing this book, such as summarizing, thematic analysis, character development, or emotional mapping."

STEP02 Dividing the Chapter
  • Prompt: "Divide the book into chapters or major sections and provide a brief description or key focus for each chapter."

STEP03 Chapter understanding and  scene summary
  • Prompt: " Identify and summarize a key scene from Chapter X, detailing the setting, main characters involved, and the central action or emotional tone ."

STEP04 Visual detail extraction
  • Prompt: " Describe in detail the visual elements of the scene summarized from Chapter X, including character appearances, emotions displayed, specific settings, time of day, and any notable objects or symbols "

STEP05 Image Prompt  Creation
  • Prompt: " Convert the detailed scene description into a concise image prompt, suitable for a visual AI tool. Include all critical visual elements such as character traits, setting specifics, mood, and color palette. "


EXAMPLE OF SELLECTED TEXT
The feature of this book is that it describes an abstract concept, "Healing spaces." Healing spaces is an abstract and highly subjective concept. The reason for choosing this is because people's feedback on created scenes can directly reflect whether these Healing spaces have an intuitive feel, thus resonating directly with the content of the article.



WORK IN PROGRESS

Future Development
It is crucial to develop a response mechanism that allows readers to modify the content of the books (Not the original one) to further develop the content of its own.
2018-2024   @derekyudp     @Dongpei (Derek) Yue