Building the Magus firmware

[mars]

Thank you @mcc for this rundown, it helped me try to pinpoint the Windows driver issue/s.

For a long while I thought it was due to library changes in the STM32Cube_FW_F4_V1.25.1 updates. But no, the libs check out alright.

Turns out Windows doesn’t like audio devices with zero channels. And that there’s some issue with the new audio driver, so that compiling it in MIDI-only mode caused it to hang whenever there was USB activity. And that for some reason Windows doesn’t like our driver with output channels configured.

And most annoying of all, because I knew about it but still didn’t realise that’s why my testing failed: Windows has issues with MIDI devices resetting (as when you’ve just flashed it) while there’s a program connected to it (such as Chrome or Edge). Mostly it works, but sometimes it hangs the browser and does all sorts of weird things. Frequently it looks as if the device is connected, but no data is sent or received. Restarting Chrome doesn’t help. Sometimes doing a ‘hard’ restart works, ie going to Task Manager and killing all the background Chrome processes. Suffice to say, Windows and I don’t enjoy each other’s company.

Long story short: Magus firmware, develop branch should now work on Windows.

Key configuration settings:

#define AUDIO_CHANNELS              2
#define USE_USBD_AUDIO
#define USE_USBD_AUDIO_TX  // microphone
/* #define USE_USBD_AUDIO_RX // speaker */

Upshot is, you’ve now got a USB Audio interface that gives you the Magus stereo inputs as 48kHz 16-bit audio.

[antisvin]

Interesting. I’ve seen occasional freezes on Daisy after boot for first few seconds with USB audio enabled. I thought this could be semihosting printf slowing things down, don’t remember if this issue was present in release builds too.

[AlWared]

Hi !

I’ve been reading this discussion a few times as I’m trying to update my magus firmware for the first time over linux.
You’ve been through many things and I probably didn’t get everything so please correct me if I’m wrong. I will try to synthetize what’s been done.
So on one side you’re talking about general firmware update and on another side how to custom your own firmware. You managed to correct encoders, usb host perf (midi?) and in the latest release (v20.10) you’ve opened audio over usb. Does that make magus a DC coupled audio interface ?

I understood you first need to build the firmware and then flash with midi sys message over usb-B. I managed to build but I don’t understand how to flash.

What I did:
clone v20-10 branch
git submodule init
git submodule update
TOOLROOT ?= /home/me/bin/gcc-arm-non-eabi-9-2020-q2-update/bin directory in common.mk
cd /home/me/Document/RebelTechnology/OpenWare
make clean
make magus (this builds the Magus.bin in the Build folder)
clone FirmwareSender
make (in FirmwareSender linux folder)

I reckon I still need to generate Magus.sys from Magus.bin and then send it to Magus via USB-B.
So a few questions:
Do I still need STMcubeMX to generate files from Magus.ioc ? If yes, when do I have to do this ?
How and when to use FirmwareSender ?
Do I need any other connection than USB-B ? Such as hardware programmer ?
Stupid question: Does Magus have to be recognized as a USB device when updating firmware ?

@antisvin In your first message, you mentioned a few command line:

@antisvin

#!/bin/bash
PROJECT=Magus

make clean
TOOLROOT=pwd/…/Tools/gcc-arm-none-eabi-9-2019-q4-major/bin/ make -j17 CONFIG=Release PLATFORM=MagusBus || exit 1
FirmwareSender -in Build/${PROJECT}.bin -flash crc32 Build/${PROJECT}.bin -save ${PROJECT}.syx || exit 1
echo “Entering bootloader mode”
amidi -p hw:3,0,0 -S f07d527ef7 || exit 1
sleep 1
echo “Sending firmware”
amidi -p hw:3,0,0 -s ${PROJECT}.syx || exit 1
sleep 1
echo “Restart”
amidi -p hw:3,0,0 -S f07d527df7 || exit 1
echo “Done!”

I’m sure there are elements I need here but I just don’t understand the firmwaresender line. Is it using FirmwareSender.jucer ?

[antisvin]

Hi @AlWared

FirmwareSender is the program that can convert binary firmware image to MIDI SySex file (Magus.syx in this case) that you can load. So you need it if you’re building firmare. When loading official releases you can just download SySex file and send (play) to the OWL MIDI device. This is the project repository - GitHub - pingdynasty/FirmwareSender: Utility program to package and send firmware and dynamic prog

I think that right now you shouldn’t need to run CubeMX just to build firmware. At least that was the intention of recent updates to project infrastructure. So when you run “git submodule init/update” commands, you should get all necessary libraries to build FW. CubeMX should only be needed for new devices or if you’re making some kind of hardware modifications, etc.

But if you want to use USB Audio, you’ll have to build from develop branch - it’s not released yet, but I think Martin will be making a release very soon. It will be the version 21.0. Currently there’s one known problem with it - new USB MIDI host code stops working if you disconnect and reconnect a device to it. This requires a reboot if you want to switch controllers, so not a huge issue. V20.10 was a release made for the new Lich module, that’s why there was no release made for other devices. It doesn’t have USB audio yet.

Currently Magus configs enables capturing from its audio inputs - so yes, it can be used as DC coupled audio interface. MIDI still works on it at the same time. I didn’t spend much time testing them, but it seems pretty stable to me. I think Martin did an excellent job here, because USB code is quite complicated.

You don’t need to use programmer just to load firmware from a SySex file. But if you will load firmware that doesn’t work on Magus, you will need a programmer to restore firmware that works. There’s a newer version of bootloader that can detect this situation, so if firmware doesn’t boots you’ll be back to bootloader mode. However, you also need a programmer to update bootloader - there’s no bootloader update procedure on device itself. That is why I would suggest to get programmer if you want to install any experimental code (modified firmware or development branch).

[mcc]

AlWared, does this document help at all?

magus-tutorial

It gives instructions for Windows but I have “if you’re on Linux”… branches at some point.

The document describes the various cases of installing an official firmware, installing a self-built firmware, installing the bootloader (needed for custom firmwares), and running STMCube/building the IOC (not needed anymore)

[AlWared]

Thanks for your help !
I think I misunderstood with the usb audio. So now the master branch is at release 20.7
So I cloned openware. I git init and git update. Then I change the common.mk file.
And then when I try make magus I get this FreeRTOS.h missing file

./Src/freertos.c:52:10: fatal error: FreeRTOS.h: No such file or directory
52 | #include “FreeRTOS.h”
| ^~~~~~~~~~~~

Do I even need to run this make magus ? Where do I find the syx file ?

[mars]

Sysex releases are available here: https://github.com/pingdynasty/OpenWare/releases/tag/release
You only need to build from source if you are modifying the firmware, or want to be on the bleeding edge of development (there could be some pain involved).

If so, do git submodule init (once) and git submodule update (if you change branch) to get the right version of the libraries.

[mcc]

Got two more Magus questions…

1. I suddenly discovered firmware sender can no longer save patches with my currently installed firmware. -run works but -store does not. I’m trying to send with:

    (cd .. && make patch TOOLROOT= PATCHNAME=Midi2CVTriplet PLATFORM=Magus && ../FirmwareSender/Builds/MacOSX/build/Debug/FirmwareSender -in Build/patch.bin -out OWL-MIDI -store 2)
   

   I’m running 732ceac17cb merged with a447765258d9 on the device. I’m using d9e0d958b3fb of FirmwareSender

   Other oddities: Can’t seem to upload from rebeltech/patch-library either now, frequently when I interact “INVALID SYSEX” prints on one side or the other. I at one point recently was able to upload but they only seemed to upload with the name “Template”.

   Any idea what I could do here? Is USB HOST working with master head right now?

2. I got a module called a “Chainsaw” that’s 3-voice oscillator with individual CV control. I made a Magus patch to convert MIDI into 3-channel paraphonic CV specially for use with this. Since there are only 2 audio-style outputs, I set the 3 voices on parameter outputs. The parameter outputs appear to use 0.0 for 5V and 1.0 for 10V. Testing with this patch, the 3 voice inputs sound “wrong”. It is hard to describe, they just don’t seem to be harmonizing correct. The problem here could be at the Magus end or could be a problem with the Chainsaw. A couple of questions I’ve got: (1) is my notion of the range (0=0, 1=10V) correct? (2) Looking in the scope it seems like the parameter outputs are slightly noisier than the audio outputs. I don’t know if it’s significant enough to effect perceived pitch but putting my scope in tuner mode it seems to have more of a problem getting a lock on a a pitch. Is there anything different between the audio and parameter outs that is controllable from software? (3) There’s this whole “calibration” feature— I have not used it. Does this apply or to audio only or to the parameters also?

[antisvin]

That error has been frequently happening on MacOS lately. Sysex error messages where disabled in latest firmware release for this reason.

USB host is working, unless you have some unknown problem. It still gives you an error if you reconnect device to it, device restart is required. I had previously ran into some problems, turns out I had a faulty cable inside my Magus. So at least I can use it now.

Calibration is for audio channels only and Magus is using 0-10V output range, correct. DAC is 12 bit vs 24 bit audio codec. As I’ve mentioned a few times, currently Magus has its UI update code locked to display update rate, which means that effectively it’s sampled at 50Hz. I would consider it one step above broken. I’ve made some updates to resolves this issue on other hardware in my forked code, might be good time to try to backport it to upstream.

[mars]

@mcc

Is USB HOST working with master head right now?

Yes everything should be working reliably afaik.
Best thing to do is to upgrade to v21.0 then if your problems persist I will try to reproduce.

[AlWared]

I first had issues with my ubuntu vm: sysex program errors, buffer overflow. I could neither load patches, nor upload firmware.
Then I uploaded v21.0 through chrome on windows and it’s all working properly now !
Thanks for helping

[mcc]

I have got around to installing tag v21.0, I compiled the sysex myself… I think it works because it says “Magus v21.0” on the screen. Oddly I am noticing now that I’ve uploaded, I forgot to turn on the watchdog before building, but it’s working anyway (and not randomly rebooting) for some reason.

But, unfortunately, this very strange error where I can no longer store anything onto the device is persisting If I run ../FirmwareSender/Builds/MacOSX/build/Debug/FirmwareSender -in Build/patch.bin -out OWL-MIDI -store 6 it is ignored, if I use the “store” button on the Rebeltech “patches” page it does not work. I have five patches installed and I can select any of them fine but it seems to ignore any attempt to upload a new patch.

What could possibly be wrong? Any idea how to regain the ability to save patches?

I did the above tests on my mac. I notice that although when I plug in to my Windows machine it says “setting up OWL-MIDI”, the Rebeltech “patches” page is unable to connect to the magus on the Windows machine. This makes me worry the v21.0 did not actually upload, somehow, if USB DEVICE is expected to work with Windows now. The firmware uploader page said “Firmware upload complete” though.

[antisvin]

IIRC, watchdog doesn’t reboot randomly, it would do that after 8 seconds it gets enabled and not reset. Initially it was turned on in newer version of bootloader, but now we do that in FW too. So it’s safe to always enable it in firmware by default, which was done a while ago. You still need to have an updated bootloader in case if something totally broken is flashed, because obviously you’ll get reset to bootloader only if WD was enabled before it happens.

Also, I’ve made a bootloader update procedure that allows flashing it by MIDI - no programmer required. That includes toggling flash lock options. But we don’t have a web UI for that yet.

So if you’ve seen random lockups when device was running, I would expect that to mean that your patch was hanging and you were triggering watchdog reset.

Regarding patch storing problems, we have a new command for loading patches by name rather than by slot number. FW code for storing patches had some changes, but the new command is not used and I think that the storage should not require formatting just because of this. You may still try that in case if nothing else helps.

One obvious problem in your post is that you’re using OWL-MIDI as device name, but now each device its own name in FW, so it should be -out OWL-MAGUS. Specifically:

• OWL-MIDI used by old booltoader and firmware
• OWL-$DEVICE is unique for each device, OWL-MAGUS in this case
• OWL-BOOT is used by newer bootloader builds

[mcc]

Okay, so I just need to update FirmwareSender to load a patch by name…?

[mars]

@antisvin

we have a new command for loading patches by name

Actually we have a new command for storing resources by name, not patches. Things like samples. Patches are still stored under a slot number, so that they can be sequentially accessed.

[antisvin]

My bad, haven’t realized that it doesn’t apply to patches (since they were handled the same way as resources before). But either way, old version of FirmwareSender still works, I haven’t rebuild mine for ages. I’ve only mentioned this as one of candidates for breaking something.

It’s kind of hard to understand what exactly is wrong with that FW update. The most reasonable approach would be to erase patch storage and flash official firmware release. Then you’ll have a firmware version that is expected to work and no possibly corrupt storage.

[mcc]

v21 did not break patch storage, it broke sometime before that when I was rapidly reflashing custom firmwares.

How do I erase patch storage? I have access to the web pages and “FirmwareSender”.

[mars]

Click Erase Storage on the device page: OWL Patch Library – Rebel Technology

[mcc]

An update with v21: After running Erase Storage I can upload patches again.

Remaining oddities:

• Everything I upload has the name “Template”. When the patch starts running its name is not printed big the way it used to be.
• REALLY weird thing I’ve just noticed: I have two USB-B cables, a long one and a short one. I have discovered the Magus cannot talk to or power a USB MIDI keyboard when the MIDI keyboard is plugged in to the Magus HOST port with the short cable. The MIDI keyboard works when I use the long cable. If I plug the Magus into a Macintosh with the short cable via the DEVICE port, this works.

I am uploading with

(cd .. && make patch TOOLROOT= PATCHNAME=PatchNameGoesHere PLATFORM=Magus && ../FirmwareSender/Builds/MacOSX/build/Debug/FirmwareSender -in Build/patch.bin -out OWL-MIDI -run)

In OwlProgram repo 2918c483c53c subfolder PatchSource.

[antisvin]

Generally, there’s no need to use FirmwareSender directly - everything you need is doable with makefile. So normally I would just run something like “make patch PATCHNAME=PatchNameGoesHere clean load”.
FirmwareSender can be passed as env variable too, but I prefer to make a symlink in default path (Tools/FirmwareSender). TOOLROOT variable is set to empty string by default in recent versions of OwlProgram.

The requirement to use PLATFORM variable is removed, you can build patches with display without using it. Actually we’ll probably have to reintroduce it later, but for a different purpose - to build patches for other MCU families.

“Template” is default path name value, typically you see it in one particular case - when you have mistyped “PATCHNAME” variable name. Other than that, there’s been a lot of changes to OwlProgram in last few months, there’s no point to troubleshoot its version from October.

I have no idea what’s wrong with that USB cable, but you may be interested to know that serial MIDI output is enabled in latest development branch. Currently it just gets a copy of all MIDI output, but there are plans to allow accessing USB and serial MIDI separately.