Zigbee: Hue-llo world!

stars of today’s show: Zigbee enabled modules

Introduction

This is the fourth post in the Reversing Philips Hue light driver series.

In the third part, I converted the raw dumped PWM data to some neat math formulas.

Problem statement

Next step in my “recreate Hue light” journey was supposed to be a trivial task: get a “Hello, world!”-like sketch from Zigbee world up and running, in order to later plumb the math formulas into it.

And boy, was it surprisingly much harder than expected¹. Let’s dive in.

The journey

Jumping into this task, my idea was certainly along the lines of “How hard can it be?”

I’ve had prior experience with ESP32, and this is yet another wireless protocol.

Should be super easy, barely an inconvenience².

I relatively quickly settled on ESP32-C6, even though there’s also ESP32-H2, and nrf52840, and a bunch of – to me – more exotic silicon³.

So the plan was simple:

1. Clone the esp-zigbee-sdk
2. Compile the HA_color_dimmable_light
3. Flash it
4. Profit!

I’ll briefly touch upon some of these items, just to eternalize some of my notes:

Using the ESP-IDF in docker

I don’t like installing random toolchains on my desktop, if I can avoid it. And docker is officially supported by the Espressif IDF (dev environment).

So this was relatively easy (in-docker.sh):

#!/bin/bash
#IDFVER=v5.3.2
IDFVER=latest
# make sure we can coexist with docker
chmod a+w,g+s,o+t $(dirname "$0")
# run the docker
exec docker run --rm -v $PWD:/project -w /project -u $UID -e HOME=/tmp \
  -it docker.io/espressif/idf:$IDFVER "$@"

The only gotcha was the sharing of permissions between the host and the image (since I run the docker as rootless podman). But setting the setgid + sticky bit worked like a charm.

Flashing firmware from the host

Once firmware is built, you get flashing instructions (slightly reformatted for line length):

$ ./in-docker.sh idf.py build
[...]
Project build complete. To flash, run:
 idf.py flash
or
 idf.py -p PORT flash
or
 python -m esptool --chip esp32c6 -b 460800 --before default_reset \
   --after hard_reset write_flash --flash_mode dio --flash_size 2MB \
   --flash_freq 80m \
     0x0 build/bootloader/bootloader.bin \
     0x8000 build/partition_table/partition-table.bin \
     0x10000 build/color_light_bulb.bin
or from the "/project/build" directory
 python -m esptool --chip esp32c6 -b 460800 --before default_reset \
   --after hard_reset write_flash "@flash_args"

But I didn’t want to push the port into the image. So I opted for using esptool python package (pipx install esptool) and then simply taking part of the command line from above and pushing it to esptool:

$ esptool.py \
  -p /dev/serial/by-id/usb-1a86_USB_Single_Serial_58CD005980-if00 \
  --chip esp32c6 -b 460800 --before default_reset \
  --after hard_reset write_flash --flash_mode dio --flash_size 2MB \
  --flash_freq 80m \
    0x0 build/bootloader/bootloader.bin \
    0x8000 build/partition_table/partition-table.bin \
    0x10000 build/color_light_bulb.bin

The -p /dev/... (either as /dev/serial/by-id/... or as /dev/ttyACM<i>, if you like to live dangerously) is highly recommended. With a single module connected to the desktop you can safely skip it, though.

Profit? Not so fast!

Having gone through all these motions, I was expecting smashing success.

Well, no dice.

I powered up the chip, connected to its serial console, tried pairing it with the Hue app, and watched it fail – with the Hue app telling me “No lights found”.

I briefly contemplated my life’s choices, dug around forums, tried to understand Zigbee protocol at least superficially… only to figure out that I’m in way over my head.

Long story short?

There are two major roadblocks preventing you from linking the ESP example to a Hue bridge:

1. Default set of trust center link key
2. Espressif’s poor choice of app_device_version

And after that, if you want the device to behave in a sane way, there are a couple more:

• Off with effect command doesn’t work by default
• When kicked off the network, the “lightbulb” won’t try reconnecting

Anyway, this post is already plenty long, so I’ll make it brief.

Let’s take the cloned HA_color_dimmable_light example⁴, and make it work.

Trust center link key

When a zigbee device wants to join a network, the coordinator will gladly accept it (when it’s open to new devices), but it will send the network key encrypted by a Trust center link key⁵.

The default ZigBeeAlliance09 key is freely floating about:

5A:69:67:42:65:65:41:6C:6C:69:61:6E:63:65:30:39

but that’s not the one you want.

Philips Hue lights are much cooler than that, and they use their own Trust center link key. I won’t link it here⁶, but this is the SHA256 of it in all-caps⁷:

$ echo "{ 0x.., 0x.., ..., 0x.. }" | sha256sum 
5c0cee221e1afec96fc77e605de0a986c3e3ce90f13b177913c9c96450b7e60b  -
$ echo ..:..:..:..:..:..:..:..:..:..:..:..:..:..:..:.. | sha256sum 
ce574642a3a8959b344796591321a155ae3bd25e4f5668602f6a33c06adbdf33  -

Armed with that sweet sweet key, you just need a small tweak:

diff --git a/main/esp_zb_light.c b/main/esp_zb_light.c
index adf9c81..517131a 100644
--- a/main/esp_zb_light.c
+++ b/main/esp_zb_light.c
@@ -167,6 +167,16 @@ static void esp_zb_task(void *pvParameters)
     /* initialize Zigbee stack */
     esp_zb_cfg_t zb_nwk_cfg = ESP_ZB_ZR_CONFIG();
     esp_zb_init(&zb_nwk_cfg);
+
+    // allow joining the Philips Hue network(s)
+    esp_zb_enable_joining_to_distributed(true);
+    uint8_t secret_zll_trust_center_key[] = {
+        // FIXME: this is not the correct key, replace it with the proper one
+        0x00, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77,
+        0x88, 0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF
+    };
+    esp_zb_secur_TC_standard_distributed_key_set(secret_zll_trust_center_key);
+
     esp_zb_color_dimmable_light_cfg_t light_cfg = ESP_ZB_DEFAULT_COLOR_DIMMABLE_LIGHT_CONFIG();
     esp_zb_ep_list_t *esp_zb_color_dimmable_light_ep = esp_zb_color_dimmable_light_ep_create(HA_COLOR_DIMMABLE_LIGHT_ENDPOINT, &light_cfg);
     zcl_basic_manufacturer_info_t info = {

But I wouldn’t try connecting that light to your Hue bridge, due to…

Espressif’s poor choice of app_device_version

Onto this little gem, then.

It took me several hours banging my head against the wall, getting a zigbee sniffer⁸, and poring over bit fields and messing around with other things to finally arrive at the culprit⁹:

The ESP Zigbee SDK examples are using a wrong value for app_device_version field of the endpoint config (0 instead of 1) and since Philips Hue Bridge also caches some data about devices, this wasn’t easy to recover from.

So, to avoid nasty surprises right off the hop, another diff is needed:

diff --git a/main/esp_zb_light.c b/main/esp_zb_light.c
index 517131a..296d635 100644
--- a/main/esp_zb_light.c
+++ b/main/esp_zb_light.c
@@ -178,7 +178,15 @@ static void esp_zb_task(void *pvParameters)
     esp_zb_secur_TC_standard_distributed_key_set(secret_zll_trust_center_key);
 
     esp_zb_color_dimmable_light_cfg_t light_cfg = ESP_ZB_DEFAULT_COLOR_DIMMABLE_LIGHT_CONFIG();
-    esp_zb_ep_list_t *esp_zb_color_dimmable_light_ep = esp_zb_color_dimmable_light_ep_create(HA_COLOR_DIMMABLE_LIGHT_ENDPOINT, &light_cfg);
+    esp_zb_ep_list_t *esp_zb_color_dimmable_light_ep = NULL;
+    esp_zb_color_dimmable_light_ep = esp_zb_ep_list_create();
+    esp_zb_endpoint_config_t endpoint_config = {
+        .endpoint = HA_COLOR_DIMMABLE_LIGHT_ENDPOINT,
+        .app_profile_id = ESP_ZB_AF_HA_PROFILE_ID,
+        .app_device_id = 0x0102,
+        .app_device_version = 1, // maybe important for Hue? Oh HELL yes.
+    };
+    esp_zb_ep_list_add_ep(esp_zb_color_dimmable_light_ep, esp_zb_color_dimmable_light_clusters_create(&light_cfg), endpoint_config);
     zcl_basic_manufacturer_info_t info = {
         .manufacturer_name = ESP_MANUFACTURER_NAME,
         .model_identifier = ESP_MODEL_IDENTIFIER,

With these two changes the example would connect just fine. But it wouldn’t work well.

Fixing the Off with effect

By default the Hue bridge sends “Off” command to the light as Off with effect which makes the light slowly decrease brightness prior turning off. It looks nice, but doesn’t work without further changes:

ZCL OnOff: Off with effect has no effect in HA_color_dimmable_light (TZ-1436)

Surprisingly, the fix is already floating on the issue tracker, one just needs to find it:

diff --git a/main/esp_zb_light.c b/main/esp_zb_light.c
index 296d635..d3efc51 100644
--- a/main/esp_zb_light.c
+++ b/main/esp_zb_light.c
@@ -193,6 +193,19 @@ static void esp_zb_task(void *pvParameters)
     };
 
     esp_zcl_utility_add_ep_basic_manufacturer_info(esp_zb_color_dimmable_light_ep, HA_COLOR_DIMMABLE_LIGHT_ENDPOINT, &info);
+
+    // https://github.com/espressif/esp-zigbee-sdk/issues/457#issuecomment-2426128314
+    uint16_t on_off_on_time = 0;
+    bool on_off_global_scene_control = 0;
+    esp_zb_cluster_list_t *cluster_list = esp_zb_ep_list_get_ep(esp_zb_color_dimmable_light_ep, HA_COLOR_DIMMABLE_LIGHT_ENDPOINT);
+    esp_zb_attribute_list_t *onoff_attr_list =
+        esp_zb_cluster_list_get_cluster(cluster_list, ESP_ZB_ZCL_CLUSTER_ID_ON_OFF, ESP_ZB_ZCL_CLUSTER_SERVER_ROLE);
+
+    esp_zb_on_off_cluster_add_attr(onoff_attr_list, ESP_ZB_ZCL_ATTR_ON_OFF_ON_TIME, &on_off_on_time);
+    esp_zb_on_off_cluster_add_attr(onoff_attr_list, ESP_ZB_ZCL_ATTR_ON_OFF_GLOBAL_SCENE_CONTROL,
+            &on_off_global_scene_control);
+    // .
+
     esp_zb_device_register(esp_zb_color_dimmable_light_ep);
     esp_zb_core_action_handler_register(zb_action_handler);
     esp_zb_set_primary_network_channel_set(ESP_ZB_PRIMARY_CHANNEL_MASK);

Network reset behavior

And with that, one small tweak remains – when kicked off the network, the light wouldn’t try reconnecting. Reset would be needed. Hmm.

I’m not 100% sure about this one, but I stole it from yet another one of the past esp-zigbee-sdk issues:

diff --git a/main/esp_zb_light.c b/main/esp_zb_light.c
index d3efc51..0e77d63 100644
--- a/main/esp_zb_light.c
+++ b/main/esp_zb_light.c
@@ -84,6 +84,21 @@ void esp_zb_app_signal_handler(esp_zb_app_signal_t *signal_struct)
             }
         }
         break;
+    case ESP_ZB_ZDO_SIGNAL_LEAVE:
+        // https://github.com/espressif/esp-zigbee-sdk/issues/66#issuecomment-1667314481
+        esp_zb_zdo_signal_leave_params_t *leave_params = (esp_zb_zdo_signal_leave_params_t *)esp_zb_app_signal_get_params(p_sg_p);
+        if (leave_params) {
+            if (leave_params->leave_type == ESP_ZB_NWK_LEAVE_TYPE_RESET) {
+                ESP_LOGI(TAG, "ZDO leave: with reset, status: %s", esp_err_to_name(err_status));
+                esp_zb_nvram_erase_at_start(true); // erase previous network information.
+                esp_zb_bdb_start_top_level_commissioning(ESP_ZB_BDB_MODE_NETWORK_STEERING); // steering a new network.
+            } else {
+                ESP_LOGI(TAG, "ZDO leave: leave_type: %d, status: %s", leave_params->leave_type, esp_err_to_name(err_status));
+            }
+        } else {
+            ESP_LOGI(TAG, "ZDO leave: (no params), status: %s", esp_err_to_name(err_status));
+        }
+        break;
     default:
         ESP_LOGI(TAG, "ZDO signal: %s (0x%x), status: %s", esp_zb_zdo_signal_to_string(sig_type), sig_type, esp_err_to_name(err_status));
         break;

The destination

The destination, then, tada! is esp32-huello-world repo that encapsulates what I would imagine is a minimal example of a somewhat well-behaved “hello world!” lightbulb using ESP32-C6 or ESP32-H2 that’s able to link to a Hue bridge¹⁰ and function with it somewhat smoothly.

On the ESP32-C6-DEV-KIT-N8, pictured at the top of the post, the GRB onboard led (WS2812) simulates the light, so it makes for a nice demo¹¹.

If you’re into building your own light, you can take it for a spin.

I’ll be building a proper version of the light in some subsequent post.

There are quite a few things missing:

1. Plugging in the PWM tables / light curves I reveng’d
2. Support for start-up parameters¹²
3. Possibly support for OTA

Because reflashing the light-to-be using JTAG/UART is for chumps, no? ;)

Anyway, stay tuned for further posts.