You can stream audio for Voicegain transcription API from any computer, but sometimes it is handy to have a dedicated inexpensive  device just for this task. Below we relay experiences of one of our customers in using a Raspbery Pi to stream audio for real-time transcription. It replaced a Mac Mini which was initially used for that purpose. Using Pi had two benefits: a) obviously the cost, and b) it is less likely than Mac Mini to be "hijacked" for other purposes.

Hardware

Voicegain Audio Streaming Daemon requires very little as far as computing resources, so in even a Raspberry Pi Zero is sufficient ; however, we recommend using Raspberry Pi 3 B+ mainly because it has on-board 1Gbps wired Ethernet port. WiFi connections are more likely to have problems with streaming using UDP protocol.

Here is a list of all hardware used in the project (with amazon prices (as of July 2019)):

• Element14 Raspberry Pi 3 B+ Motherboard - $37.78
• Miuzei Raspberry Pi 3 b+ Screen, 3.5 Inch - $23.99
• Miuzei 3.5 Inch Screen Case for 3.5 LCD - $9.99
• iPazzPort Wireless Mini Handheld Keyboard - $13.99
• UGREEN USB Audio Adapter - $8.99
• SanDisk Ultra 32GB microSDHC UHS-I card - $7.23
• plus some existing USB 5V power supply was uses.

All the components added up to a total of $101.97. The reason why a mini monitor and a mini keyboard were included is that they make it more convenient to control the device while it is in the audio rack. For example, the alsa audio mixer can be easily adjusted this way, while at the same time monitoring the level of the audio via headphones.

Raspberry PI running AudioDaemon

Software

The device is running standard Raspbian which can easily be installed from an image using e.g.  balenaEtcher. After base install, the following was needed to get things running:

• enable ssh access
• change default audio device to USB sound card (Raspbian comes default with alsa and basic USB sound drivers)
• installing driver for the display (otherwise output font is too tiny and not readable)
• installing OpenJDK 9
• use link generated from Voicegain Portal to download Voicegain AudioDaemon jar file and correct JSON config
• seting the correct audio source number the AudioDaemon start script and launching the daemon

Observations

Here are some lessons learned from using this setup over the past 6 months:

• While streaming the CPU use stays under 10%
• Java heap is set to 128m, which seems to be more that enough because GCs manage to reduce it to about 54m
• Raspberry Pi turned out to be very reliable - we have not had a single issue with the hardware nor with the Raspbian OS
• Cheap USB audio card delivers very good sound quality (for speech recognition at least)
• Very cheap USB power supplies should be avoided - sometimes they cause a hum in the audio (but that also depends on what audio device is being connected).

‍

You can find the complete code (minus the RASA logic - you will have to supply your own) at our github repository.

What does it do ?

The setup allows you to call a phone number and then interact with a Voicebot that uses RASA as the dialog logic engine.

How does it work ?

The Components

• Twilio Programmable Voice - We configure a Twilio phone number to point to a TwiML App that has the AWS Lambda function as the callback URL.
• AWS Lambda function - a single Node.js function with an API Gateway trigger (simple HTTP API type).
• Voicegain STT API - we are using /asr/transcribe/async api with input via websocket stream and output via a callback. Callback is to the same AWS Lambda function but Voicegain callback is POST while Twilio callback is GET.
• RASA - dialog logic is provided by RASA NLU Dialog server which is accessible over RestInput API.
• AWS S3 for storing the transcription results at each dialog turn.

November 2021 Update: We do not recommend S3 and AWS Lambda for a production setup. A more up to date review of various options to build a Voice Bot is described here. You should consider replacing the functionality of S3 and AWS Lambda with a web server that is able to maintain state - like Node.js or Python Flask.

The Steps

The sequence diagram is provided below. Basically, the sequence of operations is as follows:

1. Call a Twilio phone number
2. Twilio makes an initial callback to the Lambda function
3. Lambda function sends "Hi" RASA and RASA responds with the initial dialog prompt
4. Lambda function calls Voicegain to start an async transcription session. Voicegain responds with a url of a websocket for audio streaming
5. Lambda function responds to Twilio with a TwiML command <Connect><Stream> to open a Media Stream to Voicegain. The command will also contain the text of the question prompt.
6. Voicegain uses TTS to generate from the text of the RASA question an audio prompt and streams it via websocket to Twilio for playback
7. The Caller hears the prompt and says something in response
8. Twilio streams caller audio to Voicegain ASR for speech recognition
9. Voicegain ASR transcribes the speech to text and makes a callback with the result of transcription to Lambda function
10. Lambda function stores the transcription result in S3
11. Voicegain closes the websocket session with Twilio
12. Twilio notices end of session with ASR and makes a callback to Lambda function to find out what to do next
13. Lambda function retrieves result of recognition from S3 and passes it to RASA.
14. RASA processes the answer and generates next question in the dialogue
15. We continue next turn same as in step 4.

‍