Category Archives: computer

Classic Engineer’s Dilemma: IOT edition

Posted on by
Reply

There are many aspects of IOT/connected devices that appeal to me. I like looking at data, it’s interesting. I like automation, it can be fun, and convenient. Unfortunately there are many aspects of the current IOT landscape that are extremely unappealing. Universally, not just to me. The obvious solution then is to build my own :)

And that’s what I’ve started doing. Over the past five years, little by little, as I’ve wanted to do something, I’ve put something together, based on what I’ve got lying around. Originally I was making use of a web-based server to store and gather all my data, but got annoyed each time the internet cut off. In hindsight it’s not clear to me why I was experiencing so many connectivity issues at the time that I decided the best option was to host something locally, but that’s where we are today.

Server

The server is a Raspberry Pi. One that I had lying around, and I already had submitting temperature and air pressure data to my server. The server was just a shared hosting site with a few basic php commands to link up with a sql database. To provide some flexility, allow for some customization and make up for my large lack of web-dev experience, I chose to do it all in python. I’m using the CherryPy library to present a basic html web-interface that gives me a summary of what’s going on, and also handles all the data requests coming in from various local sources. If the internet goes down, it doesn’t matter, because everything is stored and presented on the local network.

Every time I look at this jumble of wires I tell myself that I should really tidy it up. But it’s working. And then I’d have to take it offline, and the risk of a short isn’t thaaat high. So what would I really get out of the exercise?

At times, like when I’m away from home and need to monitor some things, I can open another port and make it accessible from the internet, but this is off by default, mainly as a security improving effort.

What does it do?

As I worked on integrating the few sensors I had, I realised that this platform I was creating could do more than just save some data and show it to me. Well, more than just weather data. The platform running python made it easy to script all kinds of integrations. As I created more of these, I saw the value in having a standard task based framework on which to build all these integrations.

This basis of this is a config file which is read at startup, that reads what tasks need to be load and when they must run. After that, there’s nothing left to do. All the tasks run as scheduled, doing as they must. The main web-interface shows a status for each task, when it was last run, when its next scheduled run is as well as options to start and stop tasks or run them immediately.

Screenshot of the web-interface. Some very useful statuses in there

This scheduler approach is simple. It makes it easy to expand, and it all running on python means it’s easy for me to think of something I want done and implement it, as well as feed back all the results to the ‘server’ directly.

Connected items

So what all is included in my setup?

  • Raspberry Pi
    • The Raspberry Pi itself has a BMP180 and an MH-Z19 directly connected to it that it polls occasionally for the latest temperature, air pressure and CO2 levels in my office.
    • The CO2 sensor was pretty cheap, and accuracy isn’t amazing, along with its ‘auto-calibration’. But it still gives interesting results. It’s especially easy to see build-up in my office when the door is closed, as well as the drop every time the heating runs in the winter, bringing fresh air in. It’s certainly more accurate than this.
CO2 data. Big dip is when the furnace started up after ~12 hrs of not running on a cold (windows closed) but sunny day. Smaller spikes during day are from me closing my office door during a meeting
And some temperature data to go with it. I think the green spike was some sun shining on the sensor
  • ESP32
    • I ‘attended’ one of the Hackaday Remoticon sessions a couple years ago, where someone from DigiKey was presenting on an IOT platform (Machinechat) that they were selling. It did a lot of what I wanted, but the free tier was pretty limited, and why would I pay for something when I can do it myself for free ;) But I had the microcontroller and temperature sensor leftover, so just repurposed it to connect to my own server and feed temperatures from our downstairs.
  • ESP32
    • A more recent addition to the collection, and not related to a specific task, but just some extra code I’ve implemented for logging and status. When we moved into our rental, we were given one of the large garage remote controls. But, especially in the summer, my wife and I mostly cycle, and these remotes are bulky and annoying. We bought one of the smaller remotes, but my wife and I are often out separately. So now we can open the garage via Wi-Fi.
    • The ESP runs a basic web-server, just waiting for a request to open the garage door. I need to still install a sensor and feed this data back to the server. And if I’m doing that, I might as well monitor the temperature in the garage too…
You see a yoghurt container, I see the perfect container for a microcontroller, a relay and some LEDs

Other Tasks

Beyond the local devices actively pushing data to the server, there are a number of tasks that get run for ad-hoc items.

  • Outside Temperature
    • Lacking an outside connected thermometer, I instead choose to just pull data from the weather api every 20min.
  • Data plotting
    • Instead of learning how to plot data on the fly in a web-friendly format. I chose the lazy route I already had experience with, which is using python’s Plotly library to generate an interactive html graph. And because doing this on the fly on a raspberry pi can be very slow, I just do it every few hours, and supply the last generated one when requested. This can be improved. (UPDATE: I installed grafana, see screenshot at the end of the post)
  • NAS Wake/Sleep
    • I have a NAS downstairs, it can be a bit noisy. I only ever need to use it at night, which is also when it needs to be on to perform internet backups. So I get this task to use WoL to start it up in the evening, and an ssh command to put it to sleep in the wee hours of the morning.
  • Get external IP
    • At some stage I started setting up cloudflare to redirect a domain to my homeserver. I never completed this, but I did write a script to monitor our external IP address and update me via e-mail whenever it changed.
  • Strava
    • Earlier this year, Strava added Squash to their list of supported activities. I’d done a lot of squash activities, that were listed as Workouts. So I went about setting the record straight, using the Strava API to update all my previous activities that matched the criteria to be Squash. With all that done, I extended it to start adding weather data to my outdoor activites and ensuring that all my future squash activities get listed as such (Garmin don’t support Squash, so my activity still gets sent to Strava as a Workout, and then the script updates it). So now I have a web-hosted server that Strava notifies whenever I (or other people) complete an activity. My raspberry pi polls this server occasionally and runs scripts locally to determine if anything needs to happen, and then update activities as necessary.
You have to give whatever’s accessing the Strava API a name, thus StraWeer was born
  • Express Entry
    • At one stage we had an application in for Canadian Express Entry. They released new results usually once a week, but not at fixed times, and occasionally more than once per week. So I set this task to scrape the site where they posted the results and e-mail whenever it got updated with the latest results. Sometime after it was no longer applicable to us, the website was updated to no longer provide the information with a standard http request. So I stopped running it.

I started on this project a long time ago, and would often not look at the code for months or years, then update it to do something I wanted. It certainly has a lot of shortcomings, a lot of opportunity for improvement, and would do well to better implement a lot of things I’d consider good programming practice these days, but were not top of my mind at the time. That being said, it works. The fact that I only have to do something to it when I want to make an improvement is evidence of this. My biggest ongoing concern is when the Raspberry Pi’s SD card is going to pack up :D

And then

Most of the code that runs on the Raspberry Pi is available in the below repo. I’m still updating it with some of the Strava integration stuff, but I need to do some more improvements first

GCaw/HomeTaskServer

UPDATE: I installed Grafana, so pretty:

HDMI Audio output not available

Posted on by
Reply

Preface: This was originally drafted in September 2020, and is only being published ~2 years later. I can’t remember my exact mood when I drafted this article, but apparently it was one of those days. It was also not the first time I’d experienced this issue. I hope posting it here will allow me to not experience this issue again, and help someone else who befalls a similar misfortune.

I hate everything. That’s right. Everything! ARGH! Cue continuos shouts into the void.

Issue: Laptop won’t output audio over HDMI. There is no setting anywhere that seems to allow you to output to HDMI, even though you could swear you’ve done it before. Also every ‘solution’ on the internet tells you to do something you’ve already done or change a setting that doesn’t exist.

Device: Asus FX502VM, Windows 10, GeForce GTX 1060

“High Definition Audio”, it’s all I wanted

Steps to resolve:

  • Uninstall everything Nvidia related. Everything. Clean clean clean.
  • Install whatever version of Nvidia graphics drivers you want. Make sure to install the Nvidia GeForce Experience.
  • Login to the Experience; do an update from within the Experience. For the Game Ready Drivers! Whatever that means.
  • During installation you’ll hopefully see mention of “HD Audio drivers”. Don’t know why they weren’t installed earlier. But /care.
  • Success.
  • Cry. etc

Tracking Time

Posted on by
Reply

Early in the pandemic my employer had us working from home. For the most part this was not an issue. But not having a dedicated office, I found it too easy to lose track of time, and spend too much (in my opinion) time working. With my computer set up in our lounge, it was too easy to check an email, or quickly test something else. It was right there.

There was also no separation that came from cycling/taking the train to or from work. Wake up in the morning, sit down at the desk with breakfast/coffee, start working. Without planned events in the evening, continuing to work was easy.

I have not worked enough (4+ hours). First LED is a status.

In an effort to limit this I once again repurposed my Particle Internet Button to do some time tracking. I’d used Toggl in the past to do some time tracking, but liked the idea of having a visual display available to me, without having to pull out my phone, or install additional applications on my work laptop.

My basic idea was to use the Particle Internet Button as a switch and display mechanism for interfacing with the Toggl API. The main issue I encountered with this approach was that Toggl was enforcing https for their API calls (and rightly so), but there were complications around the available Particle https libraries at the time.

I have worked too much (9+ hours)

Not wanting to spend the time on figuring that out, I somehow convinced myself that I should just spin up my own “Toggl-like” service. Obviously just for me, and not requiring https :) Hopefully my employer doesn’t hack my Wi-Fi to intercept my network comms and make me think I’m working less than I actually am.

And so that’s what I did. I setup a database, and threw together some php scripts to interpret different requests as start and stop commands, also taking a date-time string as a parameter. I expanded it a bit, allowing for different work-id’s, if I want to track different topics, and protected it all behind a nice long ‘key’ to limit the risk of someone messing with my data.

I have worked an appropriate length of times (8+ hours)

The Internet Button itself is round, and has a bunch of RGB LEDs on it. Eleven LEDs lets me use one for a notification, one as a spacer, and the other nine as hours worked increments. The Internet Button does have buttons that can be used as inputs, but instead I chose to use the accelerometer to give me a more intuitive input, without the need for labels.

My beautiful web interface

I 3D printed an octagon-shaped holder for it, which allows me to rotate the internet button to five fixed locations, which are easily distinguished from each other by the accelerometer. If you’re in the central position, the timer stops. If you shift to one of the other four positions, you start the timer associated with that position. In practice I’m only using one, so the other’s are mostly untested for now.

If I don’t have the hardware with me at any time, there’s also a basic web-interface, that allows me to view the entire week’s hours worked, and each stop and start event. I can also add events if I forgot to start or stop at some time.

Page title includes hours worked, and auto-refreshes every 5 minutes

Here you can find a link to a GitHub repo with some associated code: link

Below you can see a video of it working, and interspersed in this article are a couple photos of different stages.

TkInter crashing without error

Posted on by
2

I’ve been working on a minor python project recently, and went back to my old friend TkInter. TkInter is the standard GUI library for Python. I had used it previously in another small project, so when I wanted to quickly get a GUI going for a project, it made sense to use TkInter. There are many other GUI libraries, but TkInter is supposedly one of the quicker and easier to get going.

TkInter has a number of shortcomings. It’s clunky and not completely intuitive. But the biggest issues that I experience with it is trying to get the interface to update from an outside perspective. My specific implementation required me to update about 20 values on the GUI by reading data from a serial port.

The easiest way to implement this is to make use of the TkInter Label widget. The Label is used for displaying any kind of text. A label can be linked to a TkInter variable, so that if you update the variable, the label automatically gets updated. The difficulty is that you need an event to trigger an update of the variables.

The way to get around this is threads. Python has built in support for threads, via the Threading class. And so you create a thread, and pass it the Tk variables and update them from the thread. This allows you to continue doing other things in the GUI, and still have your thread run. Or does it.

The immediate issue is with how TkInter queues up tasks. Because the label uses Tk variables, you can only update them by calling a set method. Not just by declaring the variable equal to a value. As such this set gets added to a queue. I found that when I clicked a drop down menu (Tk.OptionMenu), my updates would pause. My thread would effectively pause as it waited to be allowed to call the variable set method.

This was not the end of the world, and I probably would have left it like this if I didn’t continue to have the program crash unexpectedly. It would happen often, but at different times, and only when I was interacting with another aspect of the GUI. For example, when I clicked a button or selected a menu item. The program would freeze, causing the python itself to crash, and not reporting any error.

I battled to find much info on this topic. All I’ve been able to come across is the repeating statement that “TkInter is not thread safe” and that you shouldn’t try to access TkInter widgets, except from within the main thread.

There are a couple ways around this. What I ended up using is the TkInter after method. It’s janky. But it works, and has proven reliable. Since I made the change, I have not had my app crashing any more. Now I have a global variable to store the data from serial in. I have a function within my GUI object that updates the Tk variables from the global array. It then calls itself using the after method.

The after method adds a note to TK to do something after a specific amount of time has lapsed, and appears to operate much like an interrupt, ignoring whatever else I am doing and updating on schedule. I first came across the after method as described by Furas on StackOverflow.

Below is a summary of what I was doing before, and what I am doing now.

Before:

class SerialThread(threading.Thread):
  def __init__(self, gui_object):
    threading.Thread.__init__(self)
    self.gui = gui_object

  def run(self):
    #Read serial info
    self.gui.variable1.set(SERIALDATA)

class GUI():
  def __init__(self):
    self.root = tk.Tk()

    self.variable1 = tk.StringVariable(self.root)
    label1 = tk.Label(self.root, textvariable=self.variable1)

    label1.grid(row=1,column=1)

    thread1 = SerialThread(self)
    thread1.start()

 self.root.mainloop()

if __name__ == "__main__":
  GUI();

And after:

TempVariable = "XXXX"
 
class SerialThread(threading.Thread):
  def __init__(self):
    threading.Thread.__init__(self)

  def run(self):
    global TempVariable
    #Read serial info
    TempVariable = SERIALDATA
 
class GUI():
  def __init__(self):
    self.root = tk.Tk()
 
    self.variable1 = tk.StringVariable(self.root)
    label1 = tk.Label(self.root, textvariable=self.variable1)
    label1.grid(row=1,column=1)
 
    self.updateLabels()
 
    thread1 = SerialThread()
    thread1.start()
 
    self.root.mainloop()
 
  def updateLabels(self):
    global TempVariable
    self.variable1.set(TempVariable)
    self.root.after(10,self.updateLabels)
 
if __name__ == "__main__":
  GUI();

UPDATE – below code skips the global variable and uses a thread lock:

class SerialThread(threading.Thread):
  def __init__(self):
    threading.Thread.__init__(self)
    self.serial_data_to_display = ""
    self.lock = threading.Lock()

  def run(self):
    while True:
        with self.lock:
            # serial read stuff
            self.serial_data_to_display = "SERIAL DATA"
        sleep(0.1)

class GUI():
  def __init__(self):
    self.root = tk.Tk()
    self.variable1 = tk.StringVariable(self.root)

    label1 = tk.Label(self.root,  textvariable=self.variable1)
    label1.grid(row=1,column=1)

    self.thread1 = SerialThread()
    self.thread1.start()

    self.root.mainloop()
    self.updateLabels()

  def updateLabels(self):
    with self.thread1.lock:
        self.variable1.set(self.thread1.serial_data_to_display)

    self.root.after(10,self.updateLabels)

if __name__ == "__main__":
  GUI()