NG Snakeoil: A Technology Walkthrough
Snakeoil OS 0.1.x is based on Ubuntu 12.04 LTS, and it is getting long in the tooth.
As softwares like MPD and Squeezelite evolve, the libraries contained within this specific version of Ubuntu cannot keep up (it's a lot hard to get the new versions to work on 12.04).
In early March 2017, decided to bite the bullet *hard* and move everything to the newer Ubuntu 16.04.02 LTS. It is also the right time to update the dated looking web application (WebApp) to modern HTML5.
If you have no idea what the above paragraph means - in plain English it means SnakeoilOS is getting better!
But how much better is it going to be? In this article we'd briefly describe the flaws of 0.1.x, and how the new Snakeoil 1.0.0 address those shortfalls.
What's Wrong With Snakeoil 0.1.x
As the previous page implies, 12.04 is too old.
Snakeoil 0.1.x is a "snapshot" of the Operating System I was using at the time, with custom settings and tweaks applied to give me that "euphoric warm sound". The WebApp is hastily put together with a mixture of BASH scripts and PHP, both running on the SnakeoilOS machine. Here's the 0.1.x web app in all it's glory:
What an eyesore!
The problems with Snakeoil 0.1.x are many, here is a small subset:
- The WebApp is pure PHP (no framework involved) with BASH scripts thrown in. In layman terms, while the WebApp can be whipped up very quickly - as the complexity grew, the code becomes very difficult to modify, and very difficult to maintain over time.
- Both front-facing (the web page that you see in the WebApp) and back-facing (the "logic" and ) are executed on the SnakeoilOS machine. They are linear in structure with absolutely no parallelism. This is not a good philosophy for audio because the CPU do not have the opportunity to allow itself to execute the more important bits - the audio playback.
- The WebApp is linear/synchronous is design. If there is a intensive process/task running in the background (e.g. cyclictest), the WebApp locks you out from using the application until the background process/task has finished. This is bad design as PHP is designed to "time out" after a fixed interval. PHP has to be configured to allow for extended background execution, which can affect the overall responsiveness of the machine.
- The WebApp design is really ugly. The design is really based on the old HTML pages (think AOL!), with a bit of modern CSS thrown in (tabs).
- The HTML pages only work on desktops or large tablets and totally unusable on smaller mobile devices. It's not really using the real estate of any screen effectively (too small on a computer monitor and too big for a tablet).
- Information displayed on the WebApp is mostly static. No new information can ever be presented on it's own. The user must click either the Refresh or Save button for any new information to be displayed.
I could go on, but you get the idea - Snakeoil 0.1.x is really embarrassing.
In my defense I am a C++ software developer and the SnakeoilOS WebApp is my first web application. The idea to go with a WebApp to control SnakeoilOS is mainly due to the circumstances at the time and it stuck. Now working with new employer, I've been exposed to modern web development, and only now did I realise just to do this properly
Introducing The Next Generation - Snakeoil 1.0.0
Introducing the new WebApp!
Colour me biased, but that to me looks a lot better. The whole layout is improved, and hopefully easier to navigate compared to the first generation.
There are heaps of animations giving you a impression of something that's dynamic and active.
And it works better with mobiles and tablets too - the WebApp automatically resize and rearrange itself according to the screen real estate. The next screenshot is an example of a smaller screen:
This mobile friendly interface is not something that can be easily done with the old code.
This WebApp is made possible by the following buzzwords - ESXi, GIT, MantisBT, NodeJS, Angular and RESTful. For rest of this article, I'd touch on each of these technologies, and describe how they help bring Snakeoil 1.0.0 to fruition.
VMWare ESXi
VMWare ESXi is a Server Operating System that allows you to run multiple softwares on a single computer. These softwares are actually virtual computers - meaning the software is running a software version of a computer! This technique is called Virtualisation, and the software computer is called Virtual Machine (VM).
Virtualisation allows me to run multiple VMs in a single powerful computer. E.g. I can run 6 VMs (1 CPU 4 GB of RAM, and 2TB storage) on a single machine that has 32 GB RAM and say 16 TB of storage). The advantage of this are:
- lower power consumption (1 PC instead of 6)
- use up less physical space (Only need space for 1 PC instead of 6)
- flexibility and easy management of the VMs (I can easily manipulate the configuration)
Proper development requires proper hardware. At the time Ryzen/Epyc from AMD are not released yet, Intel Xeon processors reign supreme. This is my Xeon server.
The Xeon CPU is water cooled, has 32 GB of ECC RAM. This machine also has a LSI hardware RAID card with six 2TB Hitachi HDDs, split into two RAID-5 volume (4TB per volume). A NAS provides more storage.
This server is running the following Virtual Machines (VM):
| Virtual Machine | Purpose |
|---|---|
| 64 Bit Development Machine | This VM is where I do all my 64 bit development on SnakeoilOS and includes all the development tools required to build and create SnakeoilOS - from firmware to ISO. It is running Ubuntu 16.04.02 LTS 64 Server bits. |
| 32 Bit Development Machine |
This VM is identical to the machine above, but with everything 32 bits instead of 64. This machine is a luxury because in theory I could do both 32 and 64 bit development on the one single machine (something I used to do in Snakeoil 0.1.x). Using different VM for each 32 and 64 environment for me reduce the complexities of configuration- esp when upgrading the Ubuntu system. |
| GIT Server2 | A modern version control system. This will be discussed in more details in the next page. |
| Donation Activation Checker | Checks for any PayPal donations and sends a Activation email to the donor. |
| Multiple test machines | Multiple test machines with different browsers, testing both the SnakeoilOS installation, as well as the WebApp. |
As you can see virtualisation allows me to run all the above (8 virtual machines) on a single machine. VM provides a lot of control and flexibility. I can create something called a Snapshot, which captures the state of the VM at the time, allowing me to make changes and then revert back to the snapshot state (think of this as a very powerful version of Microsoft Restore Points). This machine is in use since Snakeoil 0.1.4.
NB: I'm in two minds of turning this existing Xeon server into a NAS, and move development into a new AMD Ryzen or ThreadRipper with 64 GB of RAM running Proxmox. It's more a "want" than a "need" right now. Perhaps in the future when I can afford to stay in a bigger house with more room.
GIT
GIT is a version control system. GIT is software that helps me manage software. It keeps every version of all the SnakeoilOS code - from 0.1.0 (Schumann Resonance) to 1.0.0 (Golden Ears) and everything in between.
This may sound trivial, but keeping every edition and everything in between implies a detailed documented history of SnakeoilOS - and this will come in very useful in maintaining the quality of the code.
One example of code management is I can experiment on something experimental, wild and ambitious, and still not affect anything that's proven and stable. Example I am working on the new SnakeoilOS 1.0.0 independent from SnakeoilOS 0.1.x. Common bug fixes can be applied into both, but the wild ideas go in 1.0.0 code, while the safer code goes in 0.1.x.
GIT also makes it easy for me to turn SnakeoilOS into a open source project in the future. When that happens, other people can check out the source code to make improvements, or start their own projects.
Right now I'm only using bare metal GIT repository with GitList. To learn more about GIT and what it can do, check out this Wikipedia article.
MantisBT
If GIT is about management of source code, then MantisBT (A bug tracker) is about management of ideas.
It is another piece of software that gives me the power to keep track and organise all the ideas and bugs for SnakeoilOS in one place. This tool makes it easy for me to break up some complex idea into smaller and more manageable components. Basically this is a checklist maker. :D
The checklist gives me a bird's eye overview of SnakeoilOS, helping me prioritise and focus on what needs to be done first, second, until the last.
Bugs and problems are also lodged on Mantis, and when fixed I often note down the solution when it's solved. This documentation will come in useful if I have to revisit it again.
Another feature of Mantis is the automatic generation of release history, and a roadmap for upcoming releases.
These two are the only management tools I'm using for SnakeoilOS. Visit the MantisBT website if you're more interested to know more.
Update 20 Feb 2020: No longer using Mantis. Using gitlab for issue tracking now.
NodeJS
If you aren't confused by the previous pages you'd definitely be lost with this one.
NodeJS is a development tool for creating server side JavaScript code (Server side means the software is running on another computer).
Node.js is an open-source, cross-platform JavaScript run-time environment for executing JavaScript code server-side. Historically, JavaScript was used primarily for client-side scripting, in which scripts written in JavaScript are embedded in a webpage's HTML, to be run client-side by a JavaScript engine in the user's web browser. Node.js enables JavaScript to be used for server-side scripting, and runs scripts server-side to produce dynamic web page content before the page is sent to the user's web browser. Consequently, Node.js has become one of the foundational elements of the "JavaScript everywhere" paradigm,[4] allowing web application development to unify around a single programming language, rather than rely on a different language for writing server side scripts.
The SnakeoilOS WebApp is developed using NodeJS, so you would assume the application is running on the SnakeoilOS machine. That is not the case at all. All the SnakeoilOS web files are stored on the SnakeoilOS computer. But when you connect to the SnakeoilOS machine from a browser (e.g. Chrome, Safari, Firefox), the software is copied from the SnakeoilOS to your browser (this can be on your PC, MAC, tablet, mobile phone etc) and then the software is actually running on your device and not the SnakeoilOS PC.
This is called client-side scripting, and this is a big change from SnakeoilOS 0.1.x (100% server-side).
So why use NodeJS (meant for server-side) to develop something for client-side? When developing the SnakeoilOS on NodeJS, the server-side part actually generates the JavaScript code and sends it to the browser to execute on the client side.
This whole approach is counter-intuitive, but it works out really well in practice.
NodeJS is only used for development. Every SnakeoilOS release contains only client-side JavaScript code.
Angular
Angular is an application framework developed by Google. I decided to base the SnakeoilOS WebApp on Angular after finding it to be the best tool against other technologies.
Funny story - when development first begin in late Feb 2017, this framework is called Angular2. In a month (Late March 2017), Google released an update called Angular4. Angular4 is also known as Angular (feels like we have gone full circle here!).
Angular/Angular2/Angular4 is not the same thing as AngularJS (also from Google). And there is no Angular 3.
The previous pages I mentioned the SnakeoilOS WebApp is a JavaScript application running on your browser. Technically the Angular language is not JavaScript, it is called TypeScript - think of it as an advanced version of JavaScript. SnakeoilOS is written in Angular/TypeScript, and through NodeJS the TypeScript code is 'converted' into JavaScript to be served to your web browser.
This is the reason why tools like GIT and Mantis are so important in helping me manage all this. Documenting everything down will make it easier when I decide to move SnakeoilOS to the "framework of the month".
The rapid pace of technology advancement means it'd be far better to do all my development inside VM than on physical machines. Developing everything on 1 or 2 machines is no longer feasible.
There is is a pattern in all this chaos.
The old Snakeoil WebApp (0.1.x) is 100% server-side. The new SnakeoilOS WebApp now works on two realms - Client-Side and Server-side.
All the Angular code I've written for SnakeoilOS is 100% client-side - i.e. Graphical User Interface. Angular is responsible for the layout, formatting, themes, language and error checking.
This is client-side JavaScript, and being clide-side, it do not have the necessary permissions to find out what audio cards your SnakeoilOS machine have, what size HDDs it has, and so on. The GUI is completely locked out from directly knowing anything about your SnakeoilOS machine.
To pass these information to the GUI, it has to communicate with another piece of software also running on your SnakeoilOS machine - the SnakeoilOS RESTful server.
Confused yet?
Life was a lot simpler in C/C++ land, things are moving so quickly in the Web I can hardly keep up. It does feel like everything I learn today will be outdated in 3 months time.
RESTful
If the Angular code is responsible for the presentation side of SnakeoilOS WebApp, the RESTful portion is responsible for everything else (e.g. the logic, management, and so on).
The RESTful server is responsible for:
- Collating all the information the client-side application needs (e.g. CPU/RAM status for the Dashboard, Audio/player information for Music Player).
- Start/Stop music players,
- Retrieve/store SnakeoilOS configuration,
- Shutdown/reboot SnakeoilOS machine
- Run Cyclictest/top
The relationship between the Angular client-side and the RESTful server-side is asynchronous by nature (this is similar to how async USB 2.0 works). For example, when you click on "CyclicTest" on the browser, the client-side application delegate this action to the RESTful server. The RESTful server will execute the cyclictest benchmark on the SnakeoilOS machine. In the meantime, you can continue to use the WebApp to perform other actions, and the RESTful server is still available to service any requests made by the GUI. The two communicates with each other, but they don't have to wait for each other to finish. When a task is finished, the other party will be notified, and this other party can then choose what to do with the returned response.
In the above example, you initiated the cyclictest, and the GUI now tells you the benchmark is working hard on the SnakeoilOS machine. You can continue to use the WebAPP, and when you return back to the System page, and when the benchmark has completed, you will see the results.
This is a major improvement over to the old 0.1.x SnakeoilOS WebApp. With the older Snakeoil 0.1.x, once you run cyclictest, you are locked out of the WebApp until it finish running (this can take minutes!)
The RESTful server is written with cpprestsdk from Microsoft. This is a C++ library. Of all the languages, C++ gives the best speed performance because it's running in compiled machine code. All the web requests, I/O are efficiently handled with C with very low CPU overheads.
Connecting All The Dots
This is how the old system used to work:
Your browser requests the GUI from the SnakeoilOS machinie. The old SnakeoilOS 0.1.x WebApp will take it's own sweet time to run everything, and when it's done it sends a static HTML back to the browser. Your browser only need to display this static HTML page. In this setup the SnakeoilOS machine does all the work, and the browser is doing anything useful.
The above is a vicious cycle. To update the GUI, or get new content, you have to continuously make a new request, and the SnakeoilOS has to do all the work, and then send back a static HTML page for the browser to display. This is not an ideal setup. If the request takes a minute to complete, your browser will sit there doing nothing for one minute (you cannot do anything with it). The SnakeoilOS computer is doing all the work, meaning it's wasting CPU time that can be better used for audio playback (i.e. your SnakeoilOS is busier and have a higher latency).
Not good at all.
Meanwhile, the new design absolves all the problems described above. Here's the new design:
In SnakeoilOS 1.0.0 and beyond. When you load the WebApp, the browser request the GUI from the SnakeoilOS once (in most cases), and then it runs on the browser (via the browser's JavaScript engine). This is why when you first load the web page, you'd see a "loading" animation that will take a few seconds.
Now you can navigate your way around the WebApp, and it will communicate with the RESTful server to grab any data it needs, or to process any configuration changes you make. Communications between the two is now working in asynchronous mode, there is no painful waiting time anymore. Sorry for the wait!
Already you can see how this new design means the SnakeoilOS machine is no longer working "as hard" when compared to 0.1.x. The whole architecture is simplified, made modular and that opens up a whole new world of possibilities.
SnakeoilOS 1.0.0 is how SnakeoilOS should be, and this is only the beginning!
[adsense:]
[pagebreak]
Conclusion
Is this article confusing you? If you have followed what I said thus far, congratulations you are a geek! If you are completely lost, even better! Welcome to my world of modern software development!
The whole point of SnakeoilOS is you do not have to know and learn all these things. The aim of SnakeoilOS is to shield you from all these computer/geek stuffs. All you need to know is to learn how to:
- Setup your music library
- Play music
- Tweak and customise SnakeoilOS to get the best out of your audio system
That is all. Everything else leave it to the Snakeoil Squad.
The plan is to spend 12 months to develop Snakeoil 1.0.0. Thanks to the use of ESXi, GIT, MantisBT, NodeJS, Angular and RESTful - it is nearly finished in just three months. May not seem like much, but for me personally the move to 1.0.0 is very ambitious - almost suicidal really. A lot of software projects have crashed and burned due to poor management or poor planning (e.g. Mantl, Corral, MontageJS).
Choosing the tools required for this project requires some experience, a bit of luck and a lot of research. Yes, a lot of research, lots and lots of research.
If you are interested to see what the new SnakeoilOS is like, download the preview edition in the forums and have a try now. Already I'm liking the audio quality the preview is bringing.
The best is yet to come.
[adsense:]
Add new comment