This Virtual Reality Classroom Wants to Teleport You to the Streets of Beijing

In a Beijing restaurant, a waiter, who is also a panda, brings out the dish you ordered –– Peking Duck. The roast duck skin is right in front of you, crispy and gleaming, and you finish it all. You thank the panda, but discover the next moment that you can’t afford the bill. No problem, he says with a cheeky grin. “Ni keyi xi panzi (you can do the dishes).”

The above scene took place during a test trial of “The Mandarin Project” at the Rensselaer Polytechnic Institute in Troy, New York. In a collaboration with IBM Research, RPI faculty have created an immersive classroom with AI and VR technology, in an effort to make cultural immersion more cost-friendly and accessible for students studying Chinese.

Surrounding the students with a 360-degree screen, the program builds interactive scenes to simulate authentic daily experiences in China. Microphones and cameras will pick up gestures and voices as students walk down the streets of Beijing, learning tai chi from grandpas at the park, or buying food from street peddlers.

Some students report that being in a VR environment calms their nerves and makes it easier to self-express. If in doubt, participants simply have to point at an object and ask what is this in Mandarin. App developers have also trained the AI to correct minor flaws for students when they mispronounce the four Chinese tones, which are often held as a major obstacle in learning Chinese.

Although still in the development phase, The Mandarin Project has started enrolling students in its for-credit Chinese courses this summer. Classes are taught on a hybrid basis, following a half-VR, half-classroom curriculum.

Related:

Digitally China Podcast: Are We Already Enslaved to AI?

Article Jan 17, 2019

The app’s developers are continuing their collaboration with linguistic professors to design further interactive scenes, which could take place at the airport, the train station, or the market, to provide students with a more complete and vivid cultural experience.

Worldwide, VR technology is edging further and further into the classroom. But this project’s developers don’t foresee VR replacing traditional classrooms in the near future. Still, faculties at RPI are hoping to set an example with The Mandarin Project, inspiring more institutions to challenge pedagogical status quos.

Photos: AP