You release a wildly successful Beagleboneblack (BBK) product. The product is one where the customer connects the BBK to the Internet thru the router on their home LAN (ie. the BBK is behind a router and does NOT have a public ip address)
After product release you find a major bug in your software, and the customer support calls start piling up because of this bug.
You find a fix but because most of your customers are not technical, it is difficult for them to update the software. Your customers start ranting about how terrible your brainchild is, you start losing hair, gaining weight, and wish you had gone into accounting instead of engineering.
SSH’ing into each BBK would be great because then the fix could be easily applied.
I’m thinking of writing a software package that provides a solution to this. In a nutshell:
A web server (SERVER1) is built and connected to the internet.
BBK Side:
Each product is given a unique ID (UNIQUE_ID) before being shipped.
A daemon process installed on the BBK sends an HTTP request for a file named UNIQUE_ID on SERVER1. The request is repeated periodically (say once every few seconds).
If the request is successful then the BBK sets up an SSH connection to SERVER1.
Tech Support Side:
5. Tech support has a list of the customers and their unique IDs.
6. When a customer calls in , Tech support creates and SSH connection to SERVER1. THen creates the file UNIQUE_ID on the server.
7. Tech support can now SSH into the customer’s BBK.
I am a little unclear still on SSH port forwarding but I am pretty sure the SSH connections thru SERVER1 should be relatively easy to set up.
Thoughts, comments, opinions?
Is there something out there already that is simple to use?
The subject line is about debugging, but you talk about updates on the main message.
You are assuming that:
1) Every BB that needs to be updated is connected to the internet.
2) If they are connected to the Internet, the SSH port is wide open…
3) They are not turned off in the middle of an update…
Why not "auto update" your app?
When booting the BB the init script compares the release number of you app (somewhere in a file or in a filename) with the current release number.
The current release number is available from an update web server (with the software).
If the release number on the web site is greater than the installed, the script downloads the app, checks the integrity of the archive, and if the archive is ok, upgrades the software.
This:
1) Works with only the http port open. Does not need any other ports opened.
2) Works also on a network "no connected" to the Intranet, as long as you place on that network a suitable web server.
Warning: I don't think this update method is the subject of any valid patent, but...
Thanks for your input. Your auto update method would definitely work.
The advantages of SSH are:
More flexibility - ability to review syslog files, system configuration, etc. This is where the “debugging thru SSH” part of my title applies. We have all been thru instances where the customer says “My app isn’t working.”. Being able to SSH into the BB allows one to deal with this type of issue.
Ability to upgrade, configure, or modify the system.Yes in theory the app could do this but this makes the app very complex.
You write
" You are assuming that:
2) If they are connected to the Internet, the SSH port is wide open…
"
I’m not 100% sure but I do not think the ssh port has to be open on the customers’ router because the TCP connection originates behind the router.
Just my 2 cents. Your idea is absolutely feasible. If you take a look at minipwner, it creates an ssh tunnel, which I believe is what you want to accomplish. The only and most unpredictable problem is going to be the end user. If they are like me nothing goes on my network unless I am absolutely sure what it is doing and that I am the one in control of it. So the auto update idea may be your best option. As for the debugging you may also set it up so that it records the logs and sends them out at set intervals. That way the customers are sure that there isn’t anyone inside their network when they shouldn’t be.
I take your point about privacy. The use case I am thinking of is this:
The product is designed to work wirelessly. However I expect some customers to have issues connecting to their wireless network because they don’t know their essid, or type in the wrong passphrase, or their router is not configured correctly, etc…
When the customer exhausts all troubleshooting and still can’t connect, he can plug in an ethernet cable into the bbk. A valid ethernet connection will signal to the bbk to start the ssh connection.
The software is going to be open source, so technical people like yourself can configure it as they wish, for instance disabling the ssh debugging option.
Any thoughts on the programming environment for the server? I am leaning towards Python.
You are welcome and to be honest, i would go with what you feel most comfortable with. I have tried a bunch of different languages and personally prefer c++. I have found it to be the the most well rounded for my needs but those were my needs. I have not used python before so i can not say if it is better or not. It is basically just going to boil down to what you are comfortable and proficient with. Now please keep in mind that i have not done much programming in the last 10 years and i am sure things have changed since then. There may be better languages out there to use.
Actually i was thinking about this some more and i believe that an ssh tunnel would be a very good option. I would go about the configuration differently though.
I would add a debug setting that the customer can enable and disable if they choose to do so.
For instance if they enable it and then connect network cable then the device creates the tunnel and allows connection.
If they disable it the device terminates the connection and closes the tunnel. This way if they decide they want their device to run wired then there is no concern about an open port on the network.
Also a thought would be to have the system monitor the connection and if after say 1 minute there isn’t a response it would automatically disable debugging or if there is a connection then set a timer for say 5 minutes this way it gives you time to look at the logs without having to worry about losing the connection.
And possibly create a custom reboot script “reboot debug” which reboots the device in debug mode so that if you have to reboot the device remotely you don’t have to ask the customer to turn debug mode on again.
A plus about this is that you shouldn’t have to worry about trying to figure out if it is connected to the network wired or wireless. You should be able to ssh in either way.
The problem with python in some situations is that it will use more CPU cycles, and can be horrendously slow compared to many native alternatives. Just a for example and very rough ( going from memory here . . .) C/C++ for one benchmark I looked at was around 7-8 seconds. ADA2005 roughly 12, nodejs(javascript) roughly 18 seconds, and lots in between but Python was somewhere around 80-90 seconds to complete the same task.
Anyway this is obviously algorithm specific, and I do personally like python syntactically, but in a situation like this . . . yeah I’d most definitely go a different route.
This has many similarities to the project I’m currently working on:
Potentially many Beagles all talking to a Web Server.
A need for flawless remote updates
The usual security issues.
The beagles will usually be behind a firewall, and thus must initiate any communication.
The users will generally be non-technical, and can’t be expected to troubleshoot or diagnose.
A main difference may be that our app will be connected to the Web server as continuously as possible, but must allow for the inevitable outages and will need to synchronize data.
If you expect to have many users you may want to give thought to how frequently you want each beagle to try to contact the server. Every second or two seems high.
