Table of Contents
Advertisement
Quick Links
MCU-SMHMI-UG
Smart HMI User Guide
Rev. 0 — 29 September 2022
Document information
Information
Content
Keywords
MCU-SMHMI-UG, SLN-TLHMI-IOT, Human Machine Interface (HMI), Smart
Home, IoT
Abstract
This document describes the smart Human Machine Interface (HMI) solution,
and its associated out-of-box features. The SLN-TLHMI-IOT turnkey solution
provides OEMs with a fully integrated, self-contained, software, and hardware
solution.
User guide
Advertisement
Table of Contents
Subscribe to Our Youtube Channel
Related Manuals for NXP Semiconductors MCU-SMHMI-UG
Summary of Contents for NXP Semiconductors MCU-SMHMI-UG
- Page 1 User guide Document information Information Content Keywords MCU-SMHMI-UG, SLN-TLHMI-IOT, Human Machine Interface (HMI), Smart Home, IoT Abstract This document describes the smart Human Machine Interface (HMI) solution, and its associated out-of-box features. The SLN-TLHMI-IOT turnkey solution provides OEMs with a fully integrated, self-contained, software, and hardware...
-
Page 2: Introduction
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Introduction The SLN-TLHMI-IOT development kit implements NXP’s EdgeReady turnkey solution for smart Human Machine Interface (HMI) projects. A high-performance member of the i.MX RT1170 microcontroller (MCU) family enables this solution. It runs on a Real-Time Operating System (RTOS) under dual-core architecture. -
Page 3: Voice Library Limitations
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide the maximum flexibility to customize and define products for a multitude of different use cases. The i.MX RT117H MCU is licensed to run NXP’s facial biometric authentication library, audio front-end runtime library, and ML speech engine runtime library, which includes: •... -
Page 4: Hardware
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Hardware The SLN-TLHMI-IOT turnkey solution for combined face and voice recognition comes with a cost and form-factor-optimized hardware reference designs and access to full software source code. In addition to the i.MXRT117H MCU, the board employs different sensors and peripherals to provide tools to help develop applications with a wide array of functions. -
Page 5: Screen
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide For more information about the hardware design of the SLN-TLHMI-IOT, see the Smart HMI Hardware Development User Guide. 3.1 Screen The SLN-TLHMI-IOT kit, by default, uses a 5.5 inch TFT display (Rocktech RK055MHD091A0-CTG) with 720*1280 resolution, LED backlight, full viewing angle, MIPI interface, and a capacitive touch panel. -
Page 6: Bootloader
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Table 2. Bluetooth/WLAN frequency and power ...continued RF technology (a) Frequency range (b) Maximum transmitted power WLAN 2.4 GHz ISM Bands 2.412-2.472 18.5 dBm 5.15-5.25 GHz (FCC UNII-low band) for US/Canada and Europe 5.25-5.35 GHz (FCC UNII-middle band) for US/Canada and Europe 5.47-5.725 GHz for Europe... -
Page 7: Mass Storage Device (Msd)
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide previous application has been selected, the bootloader requires an application to be manually selected first. The OoBE FW has a set of three different applications that can be booted into at startup. To select an application for the bootloader to use, while the kit is being powering on, press and hold one of the onboard push buttons on the back of the kit. - Page 8 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 3. Purple LED indicating MSD mode Additionally, the PC recognizes the USB interface, assuming your SLN-TLHMI-IOT is connected to a Windows PC. It shows a new USB Drive corresponding to your SLN- TLHMI-IOT kit.
-
Page 9: Out-Of-Box Demo Applications
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 5. Copying new binary to the USB drive Note: Only the main application (for example, coffee machine or elevator) can be updated using MSD. The bootloader cannot be updated using MSD and requires the use of a SEGGER J-Link Debug Probe or the Factory Programming Flow. -
Page 10: Coffee Machine
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 6. Startup screen Each of the out-of-box demo applications is described in more depth in the following sections. 5.2 Coffee machine 5.2.1 Overview The coffee machine application, preprogrammed in the SLN-TLHMI-IOT kit, presents a demo application. -
Page 11: Home Screen
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 7. Standby screen The standby screen also allows you to select your preferred language settings using the drop-down language selector menu found in the top-right corner of the screen or by using the wake word associated with your preferred language. - Page 12 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 8. Home screen By default, the dedicated camera preview shows output from the RGB camera bounding box in the center of the camera preview window is intended to help users properly align their faces with the camera. The box indicates the status of the face recognition pipeline ("Face Not Detected", "Face Detected", and "Face Recognized")
- Page 13 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 10. Face detected Figure 11. Face recognized On the right half of the screen are the three different selections for modifying the coffee selection: the coffee type, size, and strength. MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers.
-
Page 14: Bounding Box Colors
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 12. Coffee selection In addition to the camera preview and coffee selection menu, there is also a Start button in the bottom-left portion of the screen. The screen also has a language selector drop- down menu in the top-right corner. - Page 15 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 13. No face detected The bounding box turns red when a face is detected but is not recognized. This state indicates that the following coffee selection can be associated with the face detected.
-
Page 16: Selecting Coffee Type
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 15. Face is detected and a coffee is associated with it 5.2.3.2 Selecting coffee type The coffee machine application allows configuring several different coffee preference settings, including the type of coffee. The coffee machine application has four default coffee types: Americano, Cappuccino, Espresso, and Café... - Page 17 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 16. Available coffee types When using voice commands, a confirmation tone plays, saying "Americano", "Cappuccino", "Espresso", or "Café Latte", and a green LED indicates that the command is successfully recognized. When the onscreen coffee type touch controls are used or a "Coffee Type" voice command is recognized, the icon indicating the currently selected coffee type updates to reflect the new selection.
-
Page 18: Selecting Coffee Size
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 17. Selecting coffee type 5.2.3.3 Selecting coffee size The coffee machine application allows configuring several different coffee preference settings, including the size of the coffee. The onscreen Size selector can be used to adjust the size of the coffee to "Small", "Medium", or "Large"... -
Page 19: Selecting Coffee Strength
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Upon successfully recognizing a voice command, the slider position updates, and a confirmation tone with a green LED pulse indicate the success of the voice command. The chosen size affects how long the animation on the brewing screen takes. For example, small takes the shortest amount of time while large takes the longest. -
Page 20: Saving Coffee Selection
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 20. Start button During the brewing process, the display transitions to the brewing screen, which represents a coffee being brewed and dispensed. While coffee is brewing, the selected type, size, and strength of the coffee being brewed are shown onscreen. -
Page 21: Deleting A Saved Face
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 21. Face recognized Saying "Confirm" (or the equivalent for your configured language) automatically starts brewing the coffee. Saying "Cancel" (or the equivalent for your configured language) allows you to choose a different selection from the one currently saved to your face. -
Page 22: Face Recognition Debugging Information
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 22. Deleting a saved face Note: If a face is deleted, saving a new face requires power cycling the board, or waiting for the board to enter standby and reawakening it. 5.2.3.9 Face recognition debugging information To activate debugging information that can help with debugging issues recognizing your face, tap the face icon next to the NXP logo icon 3 times. -
Page 23: Brewing Screen
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 24. Debugging messages For more information on debugging messages and their meaning, see the Section 6.1.2. 5.2.4 Brewing screen The brewing screen is reached from the home screen after the Start button is touched onscreen or if the voice command "Start"... -
Page 24: Final Screen
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide The brewing screen automatically transitions to the final screen once the animation is complete (progress indicated by the blue bar). 5.2.5 Final screen The final screen indicates that the selected coffee has been brewed. The final screen displays the original preferences selected on the home screen. -
Page 25: English
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Note: As discussed in the Voice Library Limitations section, the default out-of-box software uses the Voice Seeker AFE and Cyberon Automatic Speech Recognition (ASR) libraries as evaluation versions. As a result, the AFE and ASR cease to function after 25 hours and 100 voice instructions, respectively. -
Page 26: German
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Table 4. Voice commands in French ...continued Voice Command Description "Long" 1 of 3 selections for coffee size; longest brewing animation time "Léger" 1 of 3 selections for coffee strength "Normal" 1 of 3 selections for coffee strength "Serré"... -
Page 27: Chinese
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide 5.2.6.4 Chinese Table 6. Voice commands in Chinese Voice Command Description Wake word for the board "你好 恩智浦" 1 of 4 selections for coffee type "美式咖啡" 1 of 4 selections for coffee type "卡布奇诺"... -
Page 28: Elevator
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Gets the system information • reset Resets the board • del -n <username> Deletes user by user name • del -i <id> Deletes user specified by id • del -a Deletes all users •... -
Page 29: Overview
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide 5.3.1 Overview The elevator application, preprogrammed in the SLN-TLHMI-IOT kit, provides another example of how the kit can be implemented. The application incorporates both audio and facial recognition algorithms for a convenient hands-free environment while also supporting traditional touch-based controls. - Page 30 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 28. Home screen with date, time, temperature, and weather information On the right half of the screen is a classic elevator panel; not too dissimilar from what you might see in a physical elevator. There are six floor buttons, a hold-open and hold-close button, an alarm, and a status message which indicates the current floor number.
-
Page 31: Recognition Status Icon
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 30. Voice, face, and recognition status icons 5.3.2.1 Recognition status icon By default, the recognition status icon is blue. The blue color indicates a default state, which means no face is detected yet. -
Page 32: Selecting A Floor
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 32. New user detected The icon turns green when a face is recognized and already has a floor number associated with it. Figure 33. User identified 5.3.2.2 Selecting a floor The application has six default floors that can be chosen with either voice or touch, simulating a real elevator which can travel to different floors. - Page 33 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide To use voice commands instead, speak the name of the floor you desire to travel to. For example, "Second floor" updates the floor number to two. The help screen provides a list of available commands.
-
Page 34: Saving Floor Selection
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Note: Only one floor can be selected at a time. Once the selected floor has been reached, you may select a new floor to travel to. Additionally, after a ~15 second wait time of no activity (voice commands recognized, touch screen events, etc.), the demo effectively restarts, and the elevator floor number is... -
Page 35: Automatic Floor Selection
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 37. Confirm or cancel prompt Notice that when a prompt check is enabled, the microphone icon changes to green indicating voice command recognition is enabled (without the need to first say the wake word). -
Page 36: Deleting A Saved Face
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 38. Confirm or cancel floor selection prompt Notice that when a prompt check is enabled, the microphone icon changes to green indicating voice command recognition is enabled (without the need to first say the wake word). -
Page 37: Face Recognition Debugging Information
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 39. Deleting a saved face Note: If a face is deleted, saving a new face requires power cycling the board, or waiting ~15 second for the demo to time out and reset the application back to a default state. -
Page 38: Help Screen
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 41. Debugging messages For more information on debugging messages and their meaning, see the Section 6.1.2 section. 5.3.3 Help screen The help screen is a useful reference tool to retrieve a list of the available voice commands. -
Page 39: Voice Commands
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Once enabled, the help screen shows a list of the available commands for the currently selected language, as well as a reminder to use the "Hey NXP" wake-up command to enable recognition of voice commands. -
Page 40: French
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Table 7. Voice commands in English ...continued Voice Commands Description "Main Lobby" Sets elevator current floor to the main lobby (floor one) "Floor one" Sets elevator current floor to floor one "Floor two" Sets elevator current floor to floor two "Floor three"... -
Page 41: Chinese
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Table 9. Voice commands in German ...continued Voice Commands Description "Etage zwei" Sets elevator current floor to floor two "Etage drei" Sets elevator current floor to floor three "Etage vier" Sets elevator current floor to floor four "Etage fünf"... -
Page 42: Troubleshooting
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide The shell commands for the coffee machine application are listed below: • help Lists all the registered commands • version oasis Gets the version of the current oasis library • version Gets the version of the current software •... - Page 43 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 44. Adjusting face To detect your face correctly, the face must: • Take up most of the bounding box • Be centered inside the bounding box • Stare straight at the camera • Move slowly from left to right and up to down to ensure the proper face angle, as...
-
Page 44: Face Recognition Debugging Information
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 45. Different lighting conditions affecting proper face detection 6.1.2 Face recognition debugging information Both the coffee machine and elevator demos provide onscreen debugging information to help diagnose issues recognizing a face. Enabling the face recognition debugging information differs slightly between different applications. - Page 45 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 46. Face recognition debugging information in coffee machine MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. © 2022 NXP B.V. All rights reserved. User guide Rev. 0 — 29 September 2022...
-
Page 46: Voice Commands
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Figure 47. Face recognition debugging information in elevator Enabling debugging info prints debugging information above the camera preview in the coffee machine demo as shown in Figure Figure 47 shows debugging information printed to the left of the NXP logo in the elevator demo. -
Page 47: Speaking Clearly
Table 12. References Links Description MCUXpresso IDE for NXP MCUs | Linux, MCUXpresso IDE Download Windows and iOS | NXP Semiconductors MCUXpresso IDE User Guide (nxp.com) MCUXpresso IDE User Guide https://www.nxp.com/mcu-smhmi MCU-SMHMI-SDUG Smart HMI Software Development User Guide http://www.nxp.com/mcu-smhmi... -
Page 48: Revision History
MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Revision history Table 14 lists the substantive changes done to this document since the initial release. Table 14. Revision history Revision number Date Substantive Changes 29 September 2022 Initial release MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers. -
Page 49: Legal Information
NXP Semiconductors. In the event that customer uses the product for design-in and use in In no event shall NXP Semiconductors be liable for any indirect, incidental, automotive applications to automotive specifications and standards, punitive, special or consequential damages (including - without limitation - customer (a) shall use the product without NXP Semiconductors’... - Page 50 MCU-SMHMI-UG NXP Semiconductors Smart HMI User Guide Bluetooth — the Bluetooth wordmark and logos are registered trademarks owned by Bluetooth SIG, Inc. and any use of such marks by NXP Semiconductors is under license. MCU-SMHMI-UG All information provided in this document is subject to legal disclaimers.
-
Page 51: Table Of Contents
Please be aware that important notices concerning this document and the product(s) described herein, have been included in section 'Legal information'. © 2022 NXP B.V. All rights reserved. For more information, please visit: http://www.nxp.com Date of release: 29 September 2022 Document identifier: MCU-SMHMI-UG...
Need help?
Do you have a question about the MCU-SMHMI-UG and is the answer not in the manual?
Questions and answers