Hue Engine: Dumping the PWM data
Introduction
This is the second post in the Reversing Philips Hue light driver series.
In the first part, I failed to make any progress dumping the firmware.
Problem statement
We have an unknown PCB board labeled “Philips Hue Engine v1.0”:
It controls the Philips Hue White Ambiance LED strip.
And I want to know how – to what values does it set the individual light strip PWM% in response to Color Temperature and Brightness settings.
Approach
One simple way to do it – set the light to some color temperature and brightness, record the PWM data per channel, repeat ad nauseam.
The part to control the light isn’t problematic, we have the technology (Hue API):
control-light.rb (click to expand)
#!/usr/bin/env ruby
require 'net/http'
require 'uri'
require 'json'
require 'pp'
class HueBridge
def initialize(ip, apikey)
uri = URI("https://#{ip}/clip/v2/resource/device")
Net::HTTP.start(ip, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
req = Net::HTTP::Get.new(uri, {'hue-application-key': apikey})
res = http.request(req)
raise "can't init list of lights" unless res.code.to_i == 200
json = JSON.parse(res.body)
raise "errors: #{json["errors"].inspect}" unless json["errors"].empty?
@devices = json["data"]
@ip = ip
@apikey = apikey
end
self
end
def light_info(id)
out = nil
uri = URI("https://#{@ip}/clip/v2/resource/light/#{id}")
Net::HTTP.start(@ip, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
req = Net::HTTP::Get.new(uri, {'hue-application-key': @apikey})
res = http.request(req)
raise "can't poll light #{id.inspect}" unless res.code.to_i == 200
json = JSON.parse(res.body)
raise "errors: #{json["errors"].inspect}" unless json["errors"].empty?
out = json['data'].first
end
out
end
def get_temp_and_brightness(id)
info = light_info(id)
[
info["color_temperature"]["mirek"],
info["dimming"]["brightness"],
]
end
def set_temp_and_brightness(id, temp = :keep, brightness = :keep)
return :noop if temp == :keep && brightness == :keep
params = {}
params['dimming'] = {'brightness': brightness} unless brightness == :keep
params['color_temperature'] = {'mirek': temp} unless temp == :keep
uri = URI("https://#{@ip}/clip/v2/resource/light/#{id}")
Net::HTTP.start(@ip, :use_ssl => true, :verify_mode => OpenSSL::SSL::VERIFY_NONE) do |http|
req = Net::HTTP::Put.new(uri, {'hue-application-key': @apikey})
req.body = params.to_json
res = http.request(req)
json = nil
begin
json = JSON.parse(res.body)
rescue
raise "can't set light #{id.inspect} to #{params.inspect}: #{res}"
end
if res.code.to_i == 200
return :ok
else
raise "can't set light #{id.inspect} to #{params.inspect}: #{json["errors"].inspect}"
end
end
end
end
if __FILE__ == $0
bridge_ip = '1.1.2.567'
apikey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
light_id = 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'
hue = HueBridge.new(bridge_ip, apikey)
#pp hue.light_info(light_id)
pp hue.get_temp_and_brightness(light_id)
pp hue.set_temp_and_brightness(light_id, 153, 0.1)
pp hue.get_temp_and_brightness(light_id)
pp hue.set_temp_and_brightness(light_id, 153, 100)
pp hue.get_temp_and_brightness(light_id)
pp hue.set_temp_and_brightness(light_id, 233, 50)
pp hue.get_temp_and_brightness(light_id)
end
The part to read the PWM values is a bit trickier, but thanks to the basic recon post, I know where the necessary points are.
First plan of action was to plugin an Oscilloscope, and see if I can read it visually:
first attempt at dumping the PWM data
Small hint if you’re planning to recreate this at home:
I used the fourth ’scope channel to feed in a ~1kHz reference that I used for the trigger. Because as the PWM moves around, the ’scope display becomes very jittery. Having a fixed reference helped a lot.
Anyway, I can eyeball / Measure on the scope and get PDUT (positive duty
cycle) and NDUT (negative duty cycle) just fine.
But since the light can do 302 different color temperature settings and over 101 individual brightness levels1, it’s unworkable manually. I ain’t a farmer.
First (failed) solution
Luckily, my Rigol DS1054 has a TCP/IP “api” of sorts2.
Let’s drive that…
poll-scope.rb (click to expand)
#!/usr/bin/env ruby
require 'socket'
class RigolClient
def initialize(ip)
@socket = TCPSocket.new(ip, 5555)
@socket.puts '*IDN?'
id = @socket.gets
raise "can't find the scope" unless id =~ /RIGOL.*DS1..4Z/
@connected = true
self
end
def fetch_measure(measure, source)
raise :not_connected unless @connected
# FIXME: validate?
@socket.puts ":MEAS:ITEM? #{measure},#{source}"
out = nil
out = (out || "") + @socket.gets until out && out != "\n"
out.strip!
if out =~ /measure\s+error!/
:error
#elsif out =~ /9.9E37/
# :infinity
elsif out =~ /^\d+\.\d+[eE][+-]?\d+$/
out.to_f
else
out
end
end
def close
@socket.close
@socket = nil
@connected = false
end
RIGOL_INFINITY_ABOVE = 9e37 # it reports as "9.9E37"
def duty_of(source)
raise ArgumentError, "only accept CHAN[1-4]" unless source =~ /\ACHAN[1-4]\z/
pd = fetch_measure('PDUT', source)
nd = fetch_measure('NDUT', source)
vr = fetch_measure('VRMS', source)
duty = nil
if (0.0..1.0).include?(pd) && (0.0..1.0).include?(nd)
duty = pd
elsif (nd == :error || nd >= RIGOL_INFINITY_ABOVE) && (0.0..1.0).include?(pd)
duty = pd
elsif (pd == :error || pd >= RIGOL_INFINITY_ABOVE) && (0.0..1.0).include?(nd)
duty = 1 - nd
elsif (pd == :error || pd >= RIGOL_INFINITY_ABOVE) && (nd == :error || nd >= RIGOL_INFINITY_ABOVE)
duty = vr < 1.5 ? 0 : 1
else
duty = :dunno
end
[duty, pd, nd, vr]
end
end
if __FILE__ == $0
scope_ip = '1.1.2.567'
client = RigolClient.new(scope_ip)
for src in %w[CHAN1 CHAN2 CHAN3]
p client.duty_of(src)
end
client.close
end
I had to work around some issues where the PDUT, NDUT, VRMS results would
end up being nonsense (which you see in the duty_of function) but in the end
it kinda worked.
Except… it really didn’t for some extremes (near 100% and 0% PWM duty). It reported inaccurate values.
Second (better) solution
Since I knew from the data that the PWM runs roughly at 1kHz, I replaced the oscilloscope with a cheap-o 8-channel Chinesium™ Logic Analyzer3.
Since these puppies are all fx2lafw-based, Sigrok
has a great support for it. And it has a cli.
So, let’s drive it. Only, pwm protocol analyzer doesn’t really work for 0% or
100%. So, we’ll need to massage that one:
poll-probe.rb (click to expand)
#!/usr/bin/env ruby
require 'pp'
class SigrokClient
CLI_COMMAND_PWM = %w[sigrok-cli -d fx2lafw --config samplerate=12M --time 10 -C D0,D1,D2 -P pwm:data=D0 -P pwm:data=D1 -P pwm:data=D2]
CLI_COMMAND = %w[sigrok-cli -d fx2lafw --config samplerate=12M --time 10 -C D0,D1,D2 -O bits:width=0]
NAMES_PWM = {
1 => "normal",
2 => "cold",
3 => "hot",
}
NAMES = %w[normal cold hot]
def initialize
acquire
self
end
def acquire
data = nil
IO.popen(CLI_COMMAND) do |f|
data = f.read
end
raise "Can't acquire data" unless $?.exitstatus == 0
parsed = data.split(/\n/).grep(/^D\d+:/)
raise "Failed to acquire data lines (3), got: #{parsed.size}" unless parsed.size == 3
values = {}
parsed.each do |ln|
if ln.strip =~ /^D(\d):(.*)$/
name, vals = $1, $2
v = vals.scan(/[01]/)
values[NAMES[name.to_i]] = v.count("1") / v.size.to_f
end
end
values
end
# This would be great, if it weren't fucking up in all-low || all-high situations
def acquire_with_pwm
data = nil
IO.popen(CLI_COMMAND_PWM) do |f|
data = f.read
end
raise "Can't acquire data" unless $?.exitstatus == 0
parsed = data.split(/\n/).grep(/pwm-[123]:.*%/)
raise "Failed to acquire proper data, got: #{data.inspect[0,160]}" unless parsed.size >= 27
values = parsed.inject({}) { |m,x|
if /^pwm-([123]):\s* ([0-9.]+)%/ =~ x
(m[NAMES_PWM[$1.to_i]] ||= []) << $2.to_f
else
puts "wtf: #{x}"
end
m
}
values.map { |k,v| [k, (v.sum/v.size)/100.0] }.to_h
end
def fetch_duties
nil
end
end
if __FILE__ == $0
client = SigrokClient.new
pp client.acquire
end
And with that, we only need a nice script to tie it all nicely together4…
pwm-dumper.rb (click to expand)
#!/usr/bin/env ruby
use_probe = true
require_relative 'control-light'
if use_probe
require_relative 'poll-probe'
else
require_relative 'poll-scope'
end
require 'fileutils'
require 'json'
scope_ip = '1.1.2.567'
bridge_ip = '1.1.2.568'
apikey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
light_id = 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'
settle_time = 1.5
result_cache = File.join(File.dirname($0), 'pwm-cache')
FileUtils.mkdir_p(result_cache)
# LFG
hue = HueBridge.new(bridge_ip, apikey)
client = RigolClient.new(scope_ip) unless use_probe
probe = SigrokClient.new if use_probe
def test_golden(hue, client, probe, light_id)
hue.set_temp_and_brightness(light_id, 233, 50)
set_to = hue.get_temp_and_brightness(light_id)
raise "probes not set right: #{set_to}" unless [233, 50.2] == set_to
sleep 1.5 # settle_time, but I don't wanna plug it in...
if client
for src, expect in {"CHAN1" => 0.2, "CHAN2" => 0.26, "CHAN3" => 0.156}
val = client.duty_of(src).first
raise "chan #{src} is off the mark: #{val}, expected: #{expect}" if (val - expect).abs > 0.01
end
else
raise "no client, no probe" unless probe
val = probe.acquire
for src, expect in {"normal" => 0.2, "cold" => 0.256, "hot" => 0.156}
raise "probe chan #{src} is off the mark: #{val[src]}, expected: #{expect}" if (val[src] - expect).abs > 0.01
end
end
p :test_golden_ok
:ok
end
out = {}
changes = nil
combos = []
mireks = (153.step(454, 5) + [454]).to_a
brightnesses = 0.step(100, 5).to_a
# Basic cartesian for 5-point matrix
combos += mireks.product(brightnesses)
# points around saturation
combos += (245..268).to_a.product(brightnesses)
# lower brightness
combos += mireks.product((0..15).to_a)
# every mirek at 90,95,100 brightness
combos += (153..454).to_a.product([90, 95, 100])
# every mirek at 90..100 brightness (step 2)
combos += (153..454).to_a.product(90.step(100, 2).to_a)
# DRY
combos.sort!.uniq!
combos.each_with_index do |(mirek, brightness), i|
puts "Doing mirek=#{mirek}, brightness=#{brightness} (#{i+1} / #{combos.size})"
cache = File.join(result_cache, "#{mirek}-#{brightness}.json")
if FileTest.exist?(cache)
p [:cache_hit, cache]
out[[mirek, brightness]] = JSON.parse(File.read(cache))
else
# Check golden (to avoid messing up the measurements too much on faulty probe
if changes.nil? || changes >= 15
try = 0
begin
test_golden(hue, client, probe, light_id)
rescue
if try < 3
puts "Failed golden (retry #{try+1})"
try += 1
sleep 0.2
retry
else
raise
end
end
changes = 0
end
changes += 1
try = 0
begin
hue.set_temp_and_brightness(light_id, mirek, brightness)
actual_mirek, actual_brightness = hue.get_temp_and_brightness(light_id)
if mirek != actual_mirek || (brightness - actual_brightness).abs > 0.21
raise "setting wrong? want: #{[mirek, brightness].inspect}, actual: #{[actual_mirek, actual_brightness].inspect}"
end
rescue
if try < 3
puts "Failed setting (retry #{try+1})"
try += 1
sleep 0.2
retry
else
raise
end
end
sleep settle_time
chans = {}
chans_raw = {}
if use_probe
chans_raw = chans = probe.acquire
else
for name, src in {"normal" => "CHAN1", "cold" => "CHAN2", "hot" => "CHAN3"}
chans_raw[name] = client.duty_of(src)
chans[name] = chans_raw[name].first
p [name, chans[name]]
end
end
json = {
:wanted => [mirek, brightness],
:actual => [actual_mirek, actual_brightness],
:chans => chans,
:chans_raw => chans_raw,
}
out[[mirek, brightness]] = json
File.open(cache, 'w') { |f| f.write(JSON.pretty_generate(json)) }
end
puts
end
client.close unless use_probe
Couple non-obvious points for the curious reader:
I learned a while ago that failure is part of life. As such, the script not
only periodically tests that it isn’t receiving garbage (see test_golden)
but it also verifies that the Hue API truly sets the light the way it was
supposed to (because sometimes it doesn’t).
On top of that, it’s a great practice to simply dump whatever data you might
have to some cache, so you can massage later. I’ll get to this point also
in a minute, but the fact that everything ends up in a pwm-cache/153-0.json
for color temperature of 153 MIREK5 and 0% brightness saved my bacon several
times, when I needed to remove some bad points, or reshape the data.
Results
So not a long time later (on the order of an hour or three), the results are in6. From:
$ cat pwm-cache/153-0.json; echo
{
"wanted": [
153,
0
],
"actual": [
153,
0.0
],
"chans": {
"normal": 0.0005,
"cold": 0.009966666666666667,
"hot": 0.0
},
"chans_raw": {
"normal": 0.0005,
"cold": 0.009966666666666667,
"hot": 0.0
}
}
All the way up to:
$ cat pwm-cache/454-100.json ; echo
{
"wanted": [
454,
100
],
"actual": [
454,
100.0
],
"chans": {
"normal": 0.0,
"cold": 0.0,
"hot": 0.9999666666666667
},
"chans_raw": {
"normal": 0.0,
"cold": 0.0,
"hot": 0.9999666666666667
}
}
… and many things in between.
But how can one make sense of that, yes?
Visualization
Fear not, gnuplot to the rescue!
The scripts to do it…
Let’s massage the data into a space separated file7:
pwm-table.rb (click to expand)
#!/usr/bin/env ruby
require 'json'
require 'pp'
out = {}
Dir[File.join(File.dirname($0), 'pwm-cache', '*')].each do |f|
data = JSON.parse(File.read(f))
out[data["actual"]] = data["chans"].values_at("normal", "cold", "hot").map { |x| x*100.0 }
end
# Textual out:
# mirek brightness% normal% cold% hot%
text_out = []
out.sort.each do |k,v|
mirek, brightness = k
normal, cold, hot = v
ar = [mirek] + ([brightness] + v).map { |x| ("%.04f" % [x]).rjust(8) }
text_out << ar.join(' ')
end
puts "Writing gnuplot table..."
File.open('pwm-table.txt', 'w') { |f| f.puts text_out }
Oh beautiful:
$ head -n 3 pwm-table.txt
153 0.0000 0.0500 0.9967 0.0000
153 1.1900 0.0525 1.0117 0.0000
153 1.9800 0.0533 1.0375 0.0000
And now let’s make it more beautiful, with a dash of Ruby:
pwm-graph.rb (click to expand)
#!/usr/bin/env ruby
require 'pp'
class Plot
def initialize(plots, **extras)
@plots = Array(plots)
@extras = extras
end
def do_plot(output, extra: nil, contours: false, pm3d: false, dgrid: true, zlabel: "pwm\\n%", zrange: (-1..101))
IO.popen('gnuplot', 'w') do |f|
unless output == :interactive
f.puts "set terminal png size 1200,1200"
f.puts "set output \"#{output}.png\""
end
f.puts "set dgrid3d 100 exp 4" if dgrid
f.puts 'set xlabel "MIREK temp"'
f.puts 'set ylabel "brightness %"'
f.puts "set zlabel \"#{zlabel.gsub(/\\"/, "\\\"")}\""
f.puts "set xrange [150:455]"
f.puts "set yrange [-1:101]"
f.puts "set zrange [#{zrange.min}:#{zrange.max}]"
if contours
f.puts "set contour base"
f.puts "set cntrparam levels 20"
end
f.puts "set pm3d" if pm3d
f.puts extra if extra
f.puts "splot " + @plots.join(", ")
if output == :interactive
f.puts "pause mouse close"
else
f.puts "quit"
end
end
end
def plot(name)
do_plot(name, **@extras)
end
end
class ContourPlot < Plot
def initialize(plot); super(plot, contours: true); end
end
class NoGridPlot < Plot
def initialize(plot); super(plot); end
end
class Pm3dPlot < Plot
def initialize(plot); super(plot, pm3d: true); end
end
class DeltaPlot < Plot
def initialize(plot); super(plot, pm3d: true, zrange: (-5..5), zlabel: "pwm\\nΔ"); end
end
plots = {}
def tabledef(name, idx, point: nil, color: '#000000', table: 'pwm-table.txt')
pointdef = if point.nil?
"l"
else
"p pt #{point}"
end
"\"#{table}\" using 1:2:#{idx} title \"#{name}\" w #{pointdef} lt rgb \"#{color}\""
end
for name, idx in {normal: 3, cold: 4, hot: 5}
plots["#{name}"] = ContourPlot.new(tabledef(name, idx))
plots["#{name}-ng"] = NoGridPlot.new(tabledef(name, idx, point: 1))
plots["#{name}-pm"] = Pm3dPlot.new(tabledef(name, idx))
end
plots.merge!({
"combi" => Plot.new([
tabledef("normal", 3, color: "#ffdd00"),
tabledef("cold", 4, color: "#0000dd"),
tabledef("hot", 5, color: "#dd0000"),
]),
"combi-ng" => NoGridPlot.new([
tabledef("normal", 3, color: "#ffdd00", point: 1),
tabledef("cold", 4, color: "#0000dd", point: 1),
tabledef("hot", 5, color: "#dd0000", point: 1),
]),
})
if ARGV.size.zero?
threads = []
for name, plot in plots
threads << Thread.new(name, plot) do |name, plot|
puts "Plotting: #{name} ..."
plot.plot(name)
end
end
threads.each { |t| t.join }
elsif ARGV.size == 1
plot = plots[ARGV.first]
if plot
plot.plot(:interactive)
else
puts "No such plot: #{ARGV.first}"
end
else
puts "Usage: #{File.basename($0)} [plot]"
puts "If you specify plot, you get interactive mode, otherwise all plots are regen'd."
puts
puts "Graphs available:"
puts plots.keys.sort
exit 1
end
One invocation later:
$ ./pwm-table.rb
Writing gnuplot table...
$ ./pwm-graphs.rb
Plotting: normal ...
Plotting: normal-ng ...
Plotting: cold ...
Plotting: normal-pm ...
Plotting: cold-ng ...
Plotting: cold-pm ...
Plotting: hot ...
Plotting: hot-pm ...
Plotting: hot-ng ...
Plotting: combi ...
Plotting: combi-ng ...
And we get this:
Combined
All three channels combinedCold channel
Just the cold channel, with contours
Just the cold channel, with level coloring
Normal channel
Just the normal channel, with contours
Just the normal channel, with level coloring
Hot channel
Just the hot channel, with contours
Just the hot channel, with level coloring
Closing words
With the data dumped, next up is – how to reverse the generating function.
Spoiler alert: I did not have much clue either, at least initially.
Stay tuned for the next installment.
-
Future me: actually, it can do ~251 levels, in ~0.4 increments. ↩
-
See MSO1000Z_DS1000Z_ProgrammingGuide_EN.pdf if curious. ↩
-
Type
USB Logic Analyzer 8 Channelinto AliExpress, and it’s yours for $4 (or less). ↩ -
I’m including the final boss form here, but don’t worry your pretty little head about the “points around saturation”, “every mirek at…” etc. That stuff is coming. ↩
-
WTF is MIREK? I thought you’d never ask.
mirek = 1e6 / temp_in_kelvin. Actually, it’s supposed to be Mired, but I made a typo and only found out as I was writing this up. Too late. I’m keeping it. May your OCD be merciful with me. ↩ -
Again, more assumptions, I looked at the data, and decided (based on what the RGB+WW strips do) that one is probably warm white (
warm), one is normal/neutral white (normal) and one is cold white (cold). ↩ -
In this section I’m taking a liberty and not including the final boss versions, so as not to spoil the fun. So you only get to see (the relevant) part of it. Hope the scripts still work (they should). ↩