Building the Magus firmware

I have had a Magus for about a year but not really been able to use it because of some problems*. I think maybe the problems either are fixed in a newer firmware version (the website says my device is running v20.5) or I could maybe fix myself if I had the source. I could not find any instructions on where to get upgrade firmware or how to install firmware. I decided to build the source myself. I could not get it to compile (I got as far as linker errors). Here are the steps I followed:

(Steps performed on a Windows 10 machine, with Ubuntu for Linux for Windows already installed.)

  • I downloaded and installed ā€œ64-bit java for windowsā€. https://www.java.com/en/download/windows-64bit.jsp
  • I rebooted.
  • I signed up for an account on st.com.
  • I installed STM32CubeMX. STM32CubeMX - STM32Cube initialization code generator - STMicroelectronics
  • I checked out GitHub - pingdynasty/OpenWare: Firmware for OWL devices to my Windows home directory, using a git client in Ubuntu for Linux. I got git revision 10a981bf68a2 from the develop branch.
  • I ran STM32CubeMX.
    • I selected an existing project and opened Magus/Magus.ioc. I migrated the project to the current version. I clicked ā€œgenerateā€
      • I got a ā€œWARNING: When RTOS used strongly recommended to use HAL timebase instead of Systickā€. I ignored this.
  • In Ubuntu for Windows:
    • I installed gcc (with sudo apt install gcc-arm-none-eabi). (I closed and reopened the Ubuntu terminal at this point.)
    • I DID THIS STEP, BUT ANYONE ELSE READING THESE STEPS SHOULDNā€™T: I edited checkout directory common.mk and replaced the GIT_REVISION line with:
      GIT_REVISION = $(hg id)
      This step is a byproduct of the git client I use.
    • I changed magus: to _magus: in checkout directory Makefile. (This is because ā€œmake magusā€ will instantly succeed because the Windows partition is case insensitive and so it treats the directory ā€œMagusā€ like a file called ā€œmagusā€. I think .PHONY might also be a way to fix this.)
    • Ran make _magus TOOLROOT= in checkout directory. (TOOLROOT= is becuase the eabi tools are already in PATH thanks to apt.)

This is where things went wrongā€¦

I got this error:

make[1]: Entering directory '/mnt/c/Users/Andi/work/gh/OpenWare/Magus'
./Src/usb_host.c:26:24: fatal error: usbh_audio.h: No such file or directory

I figured, the Makefile forgot to set one of the include directories. So I tried running again with an -I in CPPFLAGS. This lead to more include-not-found errors, which again I was able to correct by adding -Is.

make clean
PWD=`pwd` make _magus TOOLROOT= CPPFLAGS="-I$PWD/Magus/Inc -I$PWD/Biosignals/Inc -I$PWD/Magus/Middlewares/ST/STM32_USB_Host_Library/Class/AUDIO/Inc -I$PWD/Source -I$PWD/LibSource"

This gave me the errorsā€¦

Build/usb_host.o: In function `MX_USB_HOST_Init':
usb_host.c:(.text+0xac): undefined reference to `AUDIO_Class'
Build/usbd_audio_if.o: In function `TransferComplete_CallBack_FS':
usbd_audio_if.c:(.text+0x20): undefined reference to `USBD_AUDIO_Sync'
Build/usbd_audio_if.o: In function `HalfTransfer_CallBack_FS':
usbd_audio_if.c:(.text+0x2c): undefined reference to `USBD_AUDIO_Sync'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0xc): undefined reference to `realCoefAQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x10): undefined reference to `realCoefBQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x24): undefined reference to `realCoefAQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x28): undefined reference to `realCoefBQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x3c): undefined reference to `realCoefAQ15'

Many more errors of the same type as the last five follow.

I am not sure how to proceed. Grep suggests that realCoefAQ15 could be supplied either by a source file named arm_const_structs.c, or any of a variety of lib files in ./Magus/Drivers/CMSIS/Lib/ARM/.

My questions are:

  • Is there a way to make this build work?
  • Would the build work out of the box if I were building some other revision? For example instead of 10a981bf68a2 should I be building master or one of the release/ tags?
  • Is there a way to upgrade the firmware without building it myself?
  • Once I have completed a build, how do I upload it to the device? (The README references an ā€œopenocdā€?)

Thanks.


* The problems Iā€™m currently having with Magus that prevent me from using it as an instrument:

  • MIDI input via the ā€œHOSTā€ USB port is often dropped or late.
  • When I turn a knob (any knob) a tick in one direction, it always registers one increment ā€œbackwardā€. If I turn it a second tick in the same direction, it will turn forward and return me to the starting point. I have to tick three times to move a value in the intended direction. This makes ā€œlive playā€ almost impossible.
  • ā€œNice to haveā€: I would like to be able to set the refresh rate and voltage range on the ā€œbottom patch panelā€ ports.
  • ā€œNice to haveā€: I would like to be able to draw to the screen. Apparently the website doesnā€™t have the headers for this but the OpenWare git repo does.
./Src/usb_host.c:26:24: fatal error: usbh_audio.h: No such file or directory

Thatā€™s actually expected. Itā€™s a side effect due to CubeMX generating some files that we donā€™t want to use (i.e. that usbd_audio_if.c file). Thereā€™s a script ā€œcube-update.shā€ in Magus directory that you should run to restore everything in order.

PWD=`pwd` make _magus TOOLROOT= CPPFLAGS="-I$PWD/Magus/Inc -I$PWD/Biosignals/Inc -I$PWD/Magus/Middlewares/ST/STM32_USB_Host_Library/Class/AUDIO/Inc -I$PWD/Source -I$PWD/LibSource"

Iā€™m not sure why is there include for Biosignals, but youā€™re not suppose to use it when building Magus files.

- Is there a way to upgrade the firmware without building it myself?

Yes, certainly. This (and many other stuff) is exposed to MIDI protocol, example usage for Linux is here . Youā€™ll need to build this firmware sender program to convert firmware to SySex, then call bootloader mode, upload firmware and restart device.

Hereā€™s script that I use (you probably will have to change device HW id for ALSA):

#!/bin/bash
PROJECT=Magus

make clean
TOOLROOT=`pwd`/../Tools/gcc-arm-none-eabi-9-2019-q4-major/bin/ make -j17 CONFIG=Release PLATFORM=MagusBus || exit 1
FirmwareSender -in Build/${PROJECT}.bin -flash `crc32 Build/${PROJECT}.bin` -save ${PROJECT}.syx || exit 1
echo "Entering bootloader mode"
amidi -p hw:3,0,0 -S f07d527ef7 || exit 1
sleep 1
echo "Sending firmware"
amidi -p hw:3,0,0 -s ${PROJECT}.syx || exit 1
sleep 1
echo "Restart"
amidi -p hw:3,0,0 -S f07d527df7 || exit 1
echo "Done!"
The README references an "openocd"?

Openocd is the program that you would use for debugging with a programmer. You can also use it to burn firmware.

Canā€™t comment on USB issues as I donā€™t use it myself. Other than that:

  • the encoder issue is something that Iā€™ve fixed myself, I think it was included in one of the recent firmware releases
  • youā€™ll probably have to do it yourself, but the code for accessing MAX11300 that Magus uses is present in HAL_MAX11300.* files. Also, thereā€™s an example of access to that GPIO peripheral from Owl internals. It definitely possible to expose it to user patches if you want to add a custom service call for that.
  • you donā€™t need a custom firmware to draw or use any device-specific functions - just build code for your device in OwlProgram repo

Besides this, you should get St-link v2 programmer before trying any sort of custom firmware on Magus. Otherwise you can end up with unbootable device that you canā€™t recover (without programmer). Iā€™ve made a guide about connecting it a while ago.

Also, itā€™s possible to update bootloader to a newer version that would return to bootloader mode if you boot unusable firmware. Again, this needs a programmer.

Thanks, this is helpful! I now have a compiled firmware. Now that I have working build steps (with your -update.sh step) I would like to post my steps somewhere as a tutorial. I will probably submit them (with the -update.sh step) as a pull request update to the OpenWare README.

Is there a way to upgrade the firmware without building it myself?..

ā€¦Youā€™ll need to build this firmware sender programā€¦

Thanks. But, where are the precompiled firmwares? Iā€™m sorry, I couldnā€™t find them.

If there is a risk of a bad firmware bricking the device (as you said below), Iā€™d rather upload a known-good release than one I compiled myself off the develop branch.

Besides this, you should get St-link v2 programmer before trying any sort of custom firmware on Magus. Otherwise you can end up with unbootable device that you canā€™t recover (without programmer). Iā€™ve made a guide about connecting it a while ago.

Also, itā€™s possible to update bootloader to a newer version that would return to bootloader mode if you boot unusable firmware

Hm, this is a big problem for me. I do not have an objection to buying an St-link programmer, but your steps involve soldering and that is not possible for me right now. Because of COVID I do not have access to the maker space where I normally solder, and I probably do not have space to set up a soldering iron at home. (I am also worried the complexity of the solder job you describe might be beyond my abilitiesā€¦) It is because I am not so good at soldering and such that I was seeking an all-in-one device like the Magus in the first placeā€¦

Maybe this is silly, but would it be possible to mail my device in to Rebel Technology, get the bootloader with USB bad-firmware-emergency-recovery capability installed and sent back to me? I could pay for return post. I am in Toronto.

Alternately, if I did not need to close the Magus back while I was programming the new bootloader, could I use those pin-grabby-wire things in lieu of soldering the angled pins?

Releases are here - Releases Ā· pingdynasty/OpenWare Ā· GitHub

Iā€™m not working for Rebel Tech, so I canā€™t answer if they would do the soldering. I mean, thereā€™s no reason for them to refuse this. But itā€™s a very trivial job, thereā€™s just no reason to send it to UK rather than try to find some local electronics repair shop.

Also, the soldering was required just because I wanted to be able to close that lid. But you can take out digital board from Magus and power it from programmer. Itā€™s not very convenient and you wonā€™t have access to the IO board, but itā€™s possible to flash firmware or bootloader if necessary.

Regarding trying to use those pins - thereā€™s some chance that it would work, but Iā€™d avoid something without a reliable electrical contact.

One more thing about the way bootloader works. Like Iā€™ve said, it does prevent bricking the device from flashing bad firmware. But in order to debug firmware you have to flash and boot it without bootloader - and trying to hack firmware without debugger would be very time consuming. So just updating bootloader once wonā€™t be sufficient if you plan to add something non-trivial to firmware.

Thanks so much, I think I can work with this! I will experiment with all of this next weekendā€¦

Iā€™ve made some progress with the firmware sender tool. However I have one deeply silly question:
How do I boot into bootloader mode? Apparently for the Wizard and Alchemist it is ā€œhold bottom right button at bootā€ but this does not work with my Magus. (I see you seem to have SysEx codes that force this but it would be nice to know the ā€œnormalā€ procedure.)

Related questions: FirmwareSender.exe has a ā€œflash NUMā€ input that accepts a ā€œchecksum NUMā€. What checksum algorithm, and what format should the checksum be in (hexadecimal? should there be a leading ā€œ0xā€?). The release page lists "checksum"s but doesnā€™t say what type of checksum. I tried FirmwareSender with 0x8329dec8, and it accepted that, but it also accepted 0x8329dec7 without an error, which was worrisome.

There was a one-page double-sided printout that originally came with the Magus that explained various specific usage details probably including the bootloader mode instructions. Since the move I cannot find this piece of paper. It isnā€™t on the product page. Can I download this cheatsheet from somewhere?

Sounds like you should take another look at update script Iā€™ve posted above.

This sends MIDI Sysex to trigger bootloader mode. You could use any other tool to send the same data (f07d527ef7 in hex).

So checksum is the CRC32 of firmware .bin file. This program can also be used for FW uploads (at least as far as I can tell from quick look at its sources), in this case it would not upload anything that doesnā€™t have expected checksum. But the checksum wonā€™t make much differences if you just want to convert files to Sysex format (itā€™s checked after this conversion - FirmwareSender/FirmwareLoader.hpp at master Ā· pingdynasty/FirmwareSender Ā· GitHub) .

Thereā€™s another option - to use Web updater, which should work in Chromium via WebMidi - OpenWareLaboratory . It somehow doesnā€™t work for me on Linux - I think it needs some extra permissions and I canā€™t figure what exactly is the issue. But itā€™s a problem on my side, the updater itself should be usable.

Itā€™s in docs repo , but thereā€™s no info about bootloader.

OK. I was wondering if there was a way to go into bootloader mode with the buttons. Iā€™ll try sysex. Thanks for the doc link!

There isnā€™t a way to force bootloader mode easily, thatā€™s why watchdog support was added to prevent device bricking if FW hangs on startup, for instance. Some devices check BOOT1 pin and stay in bootloader mode if itā€™s set, but I think this is disabled in Magus bootloader build, because that pin is not connected to any button.

Strictly speaking, erasing firmware with a programmer would also prevent bootloader from trying to load anything, but thatā€™s not the kind of answer that would help here.

Thank you for the explanations.

Based on these instructions I was able to install first release v20.7, then a self-built Magus.bin from git 10a981bf68a2.

In my testing, the encoder problem went away as soon as v20.7 was installed (thanks for that too by the way). The USB HOST MIDI problem however persists with host v20.7 and the git build. Testing with a SQ-1, about 50% of incoming MIDI messages seem to be ignored totally.

I am happy with this progress, next weekend maybe I will get a chance to actually look at the code :slight_smile:

You should definitely try building FW from develop branch. It has considerable changes to USB and MIDI stacks - different driver for USB MIDI, serial MIDI support and thereā€™s also USB audio. Iā€™m not sure if everything works in there, but at least thereā€™s a chance that this issue youā€™re having is fixed. Thatā€™s if the issue is in Magus/OpenWare and not somewhere else.

Another thing that is worth trying is ā€œdowngradingā€ USB host from USB High Speed to Full Speed. Thereā€™s no benefit from extra bandwidth of High Speed interface for just MIDI. This would require changing device type in Cube project and editing Inc/device.h in Magus directory and maybe some changes to Makefile. Thereā€™s a chance to end up with non-bootable firmware, so it should only be flashed if you have a programmer and confirmed that you can use it for flashing.

Also, if you donā€™t need audio inputs, Martin merged my V/Oct calibration UI code to that branch. So youā€™ll be able to calibrate Magus and use SQ1 as CV sequencer.

10a981bf is current head of develop branchā€¦the USB MIDI drop problem still reproduces. So apparently it is still present after the new USB MIDI driver. I do see the calibration feature in my develop-branch build.

I think the USB drop problem must be in OpenWare because I experienced it with multiple different USB MIDI devices and multiple ā€œpatchesā€ by different authors.

I will plan to test the USB Full Speed mode once Iā€™ve got a programmer.

1 Like

Iā€™ve tried testing USB host with OP-1 as a controller. Power has to be supplied separately in such case, because Magus wonā€™t provide enough voltage over USB.

There was no lost notes if I played it monophonically, every note was played without noticeable delay. But playing a chord a few times made it consistently stop processing MIDI altogether, which resulted in either silence or a stuck note. I will probably try fixing this myself eventually, but this sounds like a different issue - I donā€™t think youā€™d be able to make chords on SQ-1.

Iā€™ve not got an SQ-1 to test with, but I know that sending lots of MIDI through the host interface can lead to dropped messages, and hung notes.
I did a bit of a rewrite of the USB handling already to improve this (and to work around an MCU hardware issue with the USB OTG peripheral which I think has been discussed elsewhere on the forum). Itā€™s in a bit of a monster branch though, and it includes lots of other stuff. Iā€™ll review it now and see if I canā€™t merge it to develop, orā€¦ wait, Iā€™ll make a PR:

Iā€™ve got a suggestion:
instead of having to run STM32CubeMX to generate the library code for each subproject, how about we just bung a recent version of the libraries in a shared directory?
@antisvin and @mcc this is mostly a question for you!

Letā€™s say in Libraries we put STM32F4/Drivers and whatever else we might need.
Then if/when we make some H7 hardware, we add an STM32H7 folder, et c.
And a single copy of CMSIS!

It should make it easier for new people wanting to build their own firmware, and easier to maintain with respect to patching the libraries, such as with the USB Host NAK fix.

ST used to have a weird license on their firmware libs that explicitly didnā€™t allow open source projects, but it seems to have been replaced with BSD, and CMSIS appears to be Apache.

oops sorry I might be confused, and the USB MIDI changes I was thinking of may already be merged.
Which means that the USBH messages are buffered before they are processed, which should lead to less risk of them being lost. But looking into the STM32_USB_Host_Library (in usbh_core.c) it seems that the callback that receives the data is only called from the USBH_Process(), which we invoke in the main thread which also handles the screen updates. Ie itā€™s not interrupt driven.

It would be interesting to see if we can move the USBH_Process() call to a timer instead, to make it run more frequently. But it looks horribly inefficient!

Martin, running OpenWare on H7 is not a thing that belongs to the future - Iā€™m already doing it :wink:

Regarding including drivers, this is the same approach that MI takes - stmlib/third_party/STM at master Ā· pichenettes/stmlib Ā· GitHub . So look like licensing issues shouldnā€™t be present for FLOSS.

Also, Iā€™d say this is makes signifant change not for me or @mcc, but for potential new users that would want to mess. Not requiring installing CubeMX would lower barries for entry here, so this sounds like a welcomed addition.

Another thing is that it would make dealing with Cube FW updates easier for yourself, as they typically wonā€™t require re-exporting every project manually.

Actually, I think that this may not be necessary. Currently this thread is blocked to run every 20ms when screen is enabled. What we should do is block for 1ms or so, then update stuff that needs to run frequently (USB, probably digitalbus too) and skip display update until it needs to be refreshed.

This is the general idea, in reality we would probably need to precalculate next screen refresh time and use shorter wait period for last update to avoid jitter. But from your description, Iā€™m pretty sure that display is blocking USB host in this case. Also, if we use this approach, it makes sense to enable DMA for SPI to minimize blocking in this thread, I havenā€™t noticed any issues after enabling it on Magus.

Hi, sorry, I havenā€™t looked at this in a week or two.

I got an STM programmer and am interested to try reprogramming the bootloader soon. Are there instructions on replacing the bootloader (to get the new bootloader with the hang-at-boot protections)? Where can I find a copy of the improved bootloader? I see only the full packages at Releases Ā· pingdynasty/OpenWare Ā· GitHub .

Sorry, when I mentioned the SQ-1, I didnā€™t necessarily mean I was using it as a sequencerā€” I saw the problem even in ā€œStep Jumpā€ mode, which basically transforms the SQ-1 into a keyboard with sixteen keys.

I tested just now (on git 10a981bf) with a Korg microKEY Air, so a bog-standard USB keyboard. Testing with my AndiSaw4 patch, if I just sit there and play scales a little under half the notes seem to be missing. It didnā€™t seem like ā€œlots of MIDIā€. It seemed like the problem was triggering with as few as six or seven notes over a period of four to five seconds. I can also test with the YM2413 synth patch (which is polyphonic) in which case I see something more insidious: NOTE OFF messages are frequently dropped, so after hitting random keys for ten seconds or so there is a drone of three continuous notes that never turn off. I could try to record a video if it helps.

I would be happy to test with your ā€œmonsterā€ branch (especially if I can get the safety bootloader installed firstā€¦ :slight_smile: ). Which commit would be most useful for me to test? 45bf2e2?

instead of having to run STM32CubeMX to generate the library code for each subproject, how about we just bung a recent version of the libraries in a shared directory?

In my opinion, running STM32CubeMX was one of the least onerous parts of the firmware installation process. The hardest part of the entire build actually was that the cube-update.sh step was not not documented and I had to find out about it here. I am sure removing the STM32CubeMX step would make it easier in general but it wouldnā€™t really make a difference to me.

Here is a question: are the libraries large? Storing libraries in git can get unpleasant, over time. Maybe you could put the libraries in a zip file or something and add a script that automatically downloads the correct version for whatever project the user is trying to install and unpacks it.

I did write up a tutorial on how to do the build from scratch on Windows. I will try to submit that soon, perhaps this week.


Incidentally, I have been listening to ā€œYM2413 Synthā€ for about five or six minutes now with seven random notes held down infinitely because of the NOTE OFF breakage. It is actually very pleasant once you set some operators. Maybe ā€œbreak notes offā€ should be a firmware option :stuck_out_tongue:

I donā€™t think that SQ-1 would be sending too much MIDI data, so maybe weā€™re looking at the wrong issue here. Neither should a keyboard played manually do.

Also, Martin mentioned that MIDI receive runs in main thread (that deals with display), but actually itā€™s transmit that goes there:

Receive runs in audio loop, so should be running every 1.33 ms : OpenWare/ProgramManager.cpp at develop Ā· pingdynasty/OpenWare Ā· GitHub

Now, in case of the first patch - it wonā€™t correctly handle processing multiple messages that arrive at once. The callback would be called multiple times, so all but the last note that arrives would be ā€œlostā€ because patch would do nothing with them. Which doesnā€™t mean that thereā€™s no MIDI issues in OpenWare, but in this case the patch itself has bugs. So we canā€™t distinguish those issues.

Martinā€™s patch looks like it handles MIDI note state properly. It would still have stuck notes if MIDI data arrives out of order (note off before note on). A good test for this would be to count how much note on and off events have arrived. If they differ when nothing is pressed, then this could be considered a confirmation that MIDI data is lost.