Building the Magus firmware

mcc 2020-08-23 01:12:39 UTC #1

I have had a Magus for about a year but not really been able to use it because of some problems*. I think maybe the problems either are fixed in a newer firmware version (the website says my device is running v20.5) or I could maybe fix myself if I had the source. I could not find any instructions on where to get upgrade firmware or how to install firmware. I decided to build the source myself. I could not get it to compile (I got as far as linker errors). Here are the steps I followed:

(Steps performed on a Windows 10 machine, with Ubuntu for Linux for Windows already installed.)

I downloaded and installed "64-bit java for windows". https://www.java.com/en/download/windows-64bit.jsp
I rebooted.
I signed up for an account on st.com.
I installed STM32CubeMX. https://www.st.com/en/development-tools/stm32cubemx.html
I checked out https://github.com/pingdynasty/OpenWare to my Windows home directory, using a git client in Ubuntu for Linux. I got git revision 10a981bf68a2 from the develop branch.
I ran STM32CubeMX.
- I selected an existing project and opened Magus/Magus.ioc. I migrated the project to the current version. I clicked "generate"
  - I got a "WARNING: When RTOS used strongly recommended to use HAL timebase instead of Systick". I ignored this.
In Ubuntu for Windows:
- I installed gcc (with sudo apt install gcc-arm-none-eabi). (I closed and reopened the Ubuntu terminal at this point.)
- I DID THIS STEP, BUT ANYONE ELSE READING THESE STEPS SHOULDN'T: I edited checkout directory common.mk and replaced the GIT_REVISION line with:
  GIT_REVISION = $(hg id)
  This step is a byproduct of the git client I use.
- I changed magus: to _magus: in checkout directory Makefile. (This is because "make magus" will instantly succeed because the Windows partition is case insensitive and so it treats the directory "Magus" like a file called "magus". I think .PHONY might also be a way to fix this.)
- Ran make _magus TOOLROOT= in checkout directory. (TOOLROOT= is becuase the eabi tools are already in PATH thanks to apt.)

This is where things went wrong...

I got this error:

make[1]: Entering directory '/mnt/c/Users/Andi/work/gh/OpenWare/Magus'
./Src/usb_host.c:26:24: fatal error: usbh_audio.h: No such file or directory

I figured, the Makefile forgot to set one of the include directories. So I tried running again with an -I in CPPFLAGS. This lead to more include-not-found errors, which again I was able to correct by adding -Is.

make clean
PWD=`pwd` make _magus TOOLROOT= CPPFLAGS="-I$PWD/Magus/Inc -I$PWD/Biosignals/Inc -I$PWD/Magus/Middlewares/ST/STM32_USB_Host_Library/Class/AUDIO/Inc -I$PWD/Source -I$PWD/LibSource"

This gave me the errors...

Build/usb_host.o: In function `MX_USB_HOST_Init':
usb_host.c:(.text+0xac): undefined reference to `AUDIO_Class'
Build/usbd_audio_if.o: In function `TransferComplete_CallBack_FS':
usbd_audio_if.c:(.text+0x20): undefined reference to `USBD_AUDIO_Sync'
Build/usbd_audio_if.o: In function `HalfTransfer_CallBack_FS':
usbd_audio_if.c:(.text+0x2c): undefined reference to `USBD_AUDIO_Sync'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0xc): undefined reference to `realCoefAQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x10): undefined reference to `realCoefBQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x24): undefined reference to `realCoefAQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x28): undefined reference to `realCoefBQ15'
./Drivers/CMSIS/DSP/Source/CommonTables/arm_const_structs.o:(.rodata+0x3c): undefined reference to `realCoefAQ15'

Many more errors of the same type as the last five follow.

I am not sure how to proceed. Grep suggests that realCoefAQ15 could be supplied either by a source file named arm_const_structs.c, or any of a variety of lib files in ./Magus/Drivers/CMSIS/Lib/ARM/.

My questions are:
- Is there a way to make this build work?
- Would the build work out of the box if I were building some other revision? For example instead of 10a981bf68a2 should I be building master or one of the release/ tags?
- Is there a way to upgrade the firmware without building it myself?
- Once I have completed a build, how do I upload it to the device? (The README references an "openocd"?)

Thanks.

* The problems I'm currently having with Magus that prevent me from using it as an instrument:

MIDI input via the "HOST" USB port is often dropped or late.
When I turn a knob (any knob) a tick in one direction, it always registers one increment "backward". If I turn it a second tick in the same direction, it will turn forward and return me to the starting point. I have to tick three times to move a value in the intended direction. This makes "live play" almost impossible.
"Nice to have": I would like to be able to set the refresh rate and voltage range on the "bottom patch panel" ports.
"Nice to have": I would like to be able to draw to the screen. Apparently the website doesn't have the headers for this but the OpenWare git repo does.

antisvin 2020-08-23 09:12:13 UTC #2

./Src/usb_host.c:26:24: fatal error: usbh_audio.h: No such file or directory

That's actually expected. It's a side effect due to CubeMX generating some files that we don't want to use (i.e. that usbd_audio_if.c file). There's a script "cube-update.sh" in Magus directory that you should run to restore everything in order.

PWD=`pwd` make _magus TOOLROOT= CPPFLAGS="-I$PWD/Magus/Inc -I$PWD/Biosignals/Inc -I$PWD/Magus/Middlewares/ST/STM32_USB_Host_Library/Class/AUDIO/Inc -I$PWD/Source -I$PWD/LibSource"

I'm not sure why is there include for Biosignals, but you're not suppose to use it when building Magus files.

- Is there a way to upgrade the firmware without building it myself?

Yes, certainly. This (and many other stuff) is exposed to MIDI protocol, example usage for Linux is here . You'll need to build this firmware sender program to convert firmware to SySex, then call bootloader mode, upload firmware and restart device.

Here's script that I use (you probably will have to change device HW id for ALSA):

#!/bin/bash
PROJECT=Magus

make clean
TOOLROOT=`pwd`/../Tools/gcc-arm-none-eabi-9-2019-q4-major/bin/ make -j17 CONFIG=Release PLATFORM=MagusBus || exit 1
FirmwareSender -in Build/${PROJECT}.bin -flash `crc32 Build/${PROJECT}.bin` -save ${PROJECT}.syx || exit 1
echo "Entering bootloader mode"
amidi -p hw:3,0,0 -S f07d527ef7 || exit 1
sleep 1
echo "Sending firmware"
amidi -p hw:3,0,0 -s ${PROJECT}.syx || exit 1
sleep 1
echo "Restart"
amidi -p hw:3,0,0 -S f07d527df7 || exit 1
echo "Done!"

The README references an "openocd"?

Openocd is the program that you would use for debugging with a programmer. You can also use it to burn firmware.

Can't comment on USB issues as I don't use it myself. Other than that:

the encoder issue is something that I've fixed myself, I think it was included in one of the recent firmware releases
you'll probably have to do it yourself, but the code for accessing MAX11300 that Magus uses is present in HAL_MAX11300.* files. Also, there's an example of access to that GPIO peripheral from Owl internals. It definitely possible to expose it to user patches if you want to add a custom service call for that.
you don't need a custom firmware to draw or use any device-specific functions - just build code for your device in OwlProgram repo

Besides this, you should get St-link v2 programmer before trying any sort of custom firmware on Magus. Otherwise you can end up with unbootable device that you can't recover (without programmer). I've made a guide about connecting it a while ago.

Also, it's possible to update bootloader to a newer version that would return to bootloader mode if you boot unusable firmware. Again, this needs a programmer.

mcc 2020-08-23 18:55:30 UTC #3

Thanks, this is helpful! I now have a compiled firmware. Now that I have working build steps (with your -update.sh step) I would like to post my steps somewhere as a tutorial. I will probably submit them (with the -update.sh step) as a pull request update to the OpenWare README.

Is there a way to upgrade the firmware without building it myself?...

...You'll need to build this firmware sender program...

Thanks. But, where are the precompiled firmwares? I'm sorry, I couldn't find them.

If there is a risk of a bad firmware bricking the device (as you said below), I'd rather upload a known-good release than one I compiled myself off the develop branch.

Besides this, you should get St-link v2 programmer before trying any sort of custom firmware on Magus. Otherwise you can end up with unbootable device that you can't recover (without programmer). I've made a guide about connecting it a while ago.

Also, it's possible to update bootloader to a newer version that would return to bootloader mode if you boot unusable firmware

Hm, this is a big problem for me. I do not have an objection to buying an St-link programmer, but your steps involve soldering and that is not possible for me right now. Because of COVID I do not have access to the maker space where I normally solder, and I probably do not have space to set up a soldering iron at home. (I am also worried the complexity of the solder job you describe might be beyond my abilities...) It is because I am not so good at soldering and such that I was seeking an all-in-one device like the Magus in the first place...

Maybe this is silly, but would it be possible to mail my device in to Rebel Technology, get the bootloader with USB bad-firmware-emergency-recovery capability installed and sent back to me? I could pay for return post. I am in Toronto.

Alternately, if I did not need to close the Magus back while I was programming the new bootloader, could I use those pin-grabby-wire things in lieu of soldering the angled pins?

antisvin 2020-08-23 19:51:04 UTC #4

Releases are here - https://github.com/pingdynasty/OpenWare/releases

I'm not working for Rebel Tech, so I can't answer if they would do the soldering. I mean, there's no reason for them to refuse this. But it's a very trivial job, there's just no reason to send it to UK rather than try to find some local electronics repair shop.

Also, the soldering was required just because I wanted to be able to close that lid. But you can take out digital board from Magus and power it from programmer. It's not very convenient and you won't have access to the IO board, but it's possible to flash firmware or bootloader if necessary.

Regarding trying to use those pins - there's some chance that it would work, but I'd avoid something without a reliable electrical contact.

One more thing about the way bootloader works. Like I've said, it does prevent bricking the device from flashing bad firmware. But in order to debug firmware you have to flash and boot it without bootloader - and trying to hack firmware without debugger would be very time consuming. So just updating bootloader once won't be sufficient if you plan to add something non-trivial to firmware.

mcc 2020-08-24 04:23:00 UTC #5

Thanks so much, I think I can work with this! I will experiment with all of this next weekend...

mcc 2020-08-30 17:01:51 UTC #6

I've made some progress with the firmware sender tool. However I have one deeply silly question:
How do I boot into bootloader mode? Apparently for the Wizard and Alchemist it is "hold bottom right button at boot" but this does not work with my Magus. (I see you seem to have SysEx codes that force this but it would be nice to know the "normal" procedure.)

Related questions: FirmwareSender.exe has a "flash NUM" input that accepts a "checksum NUM". What checksum algorithm, and what format should the checksum be in (hexadecimal? should there be a leading "0x"?). The release page lists "checksum"s but doesn't say what type of checksum. I tried FirmwareSender with 0x8329dec8, and it accepted that, but it also accepted 0x8329dec7 without an error, which was worrisome.

There was a one-page double-sided printout that originally came with the Magus that explained various specific usage details probably including the bootloader mode instructions. Since the move I cannot find this piece of paper. It isn't on the product page. Can I download this cheatsheet from somewhere?

antisvin 2020-08-30 18:53:51 UTC #7

Sounds like you should take another look at update script I've posted above.

This sends MIDI Sysex to trigger bootloader mode. You could use any other tool to send the same data (f07d527ef7 in hex).

So checksum is the CRC32 of firmware .bin file. This program can also be used for FW uploads (at least as far as I can tell from quick look at its sources), in this case it would not upload anything that doesn't have expected checksum. But the checksum won't make much differences if you just want to convert files to Sysex format (it's checked after this conversion - https://github.com/pingdynasty/FirmwareSender/blob/master/Source/FirmwareLoader.hpp#L91-L96) .

There's another option - to use Web updater, which should work in Chromium via WebMidi - https://pingdynasty.github.io/OwlWebControl/firmware.html . It somehow doesn't work for me on Linux - I think it needs some extra permissions and I can't figure what exactly is the issue. But it's a problem on my side, the updater itself should be usable.

It's in docs repo , but there's no info about bootloader.

mcc 2020-08-30 19:02:20 UTC #8

OK. I was wondering if there was a way to go into bootloader mode with the buttons. I'll try sysex. Thanks for the doc link!

antisvin 2020-08-30 19:23:25 UTC #9

There isn't a way to force bootloader mode easily, that's why watchdog support was added to prevent device bricking if FW hangs on startup, for instance. Some devices check BOOT1 pin and stay in bootloader mode if it's set, but I think this is disabled in Magus bootloader build, because that pin is not connected to any button.

Strictly speaking, erasing firmware with a programmer would also prevent bootloader from trying to load anything, but that's not the kind of answer that would help here.

mcc 2020-08-31 03:19:44 UTC #10

Thank you for the explanations.

Based on these instructions I was able to install first release v20.7, then a self-built Magus.bin from git 10a981bf68a2.

In my testing, the encoder problem went away as soon as v20.7 was installed (thanks for that too by the way). The USB HOST MIDI problem however persists with host v20.7 and the git build. Testing with a SQ-1, about 50% of incoming MIDI messages seem to be ignored totally.

I am happy with this progress, next weekend maybe I will get a chance to actually look at the code

antisvin 2020-08-31 11:14:30 UTC #11

You should definitely try building FW from develop branch. It has considerable changes to USB and MIDI stacks - different driver for USB MIDI, serial MIDI support and there's also USB audio. I'm not sure if everything works in there, but at least there's a chance that this issue you're having is fixed. That's if the issue is in Magus/OpenWare and not somewhere else.

Another thing that is worth trying is "downgrading" USB host from USB High Speed to Full Speed. There's no benefit from extra bandwidth of High Speed interface for just MIDI. This would require changing device type in Cube project and editing Inc/device.h in Magus directory and maybe some changes to Makefile. There's a chance to end up with non-bootable firmware, so it should only be flashed if you have a programmer and confirmed that you can use it for flashing.

Also, if you don't need audio inputs, Martin merged my V/Oct calibration UI code to that branch. So you'll be able to calibrate Magus and use SQ1 as CV sequencer.

mcc 2020-08-31 15:10:34 UTC #12

10a981bf is current head of develop branch...the USB MIDI drop problem still reproduces. So apparently it is still present after the new USB MIDI driver. I do see the calibration feature in my develop-branch build.

I think the USB drop problem must be in OpenWare because I experienced it with multiple different USB MIDI devices and multiple "patches" by different authors.

I will plan to test the USB Full Speed mode once I've got a programmer.

antisvin 2020-08-31 17:43:56 UTC #13

I've tried testing USB host with OP-1 as a controller. Power has to be supplied separately in such case, because Magus won't provide enough voltage over USB.

There was no lost notes if I played it monophonically, every note was played without noticeable delay. But playing a chord a few times made it consistently stop processing MIDI altogether, which resulted in either silence or a stuck note. I will probably try fixing this myself eventually, but this sounds like a different issue - I don't think you'd be able to make chords on SQ-1.

mars 2020-09-13 00:13:27 UTC #14

I've not got an SQ-1 to test with, but I know that sending lots of MIDI through the host interface can lead to dropped messages, and hung notes.
I did a bit of a rewrite of the USB handling already to improve this (and to work around an MCU hardware issue with the USB OTG peripheral which I think has been discussed elsewhere on the forum). It's in a bit of a monster branch though, and it includes lots of other stuff. I'll review it now and see if I can't merge it to develop, or... wait, I'll make a PR:

mars 2020-09-13 01:13:42 UTC #15

I've got a suggestion:
instead of having to run STM32CubeMX to generate the library code for each subproject, how about we just bung a recent version of the libraries in a shared directory?
@antisvin and @mcc this is mostly a question for you!

Let's say in Libraries we put STM32F4/Drivers and whatever else we might need.
Then if/when we make some H7 hardware, we add an STM32H7 folder, et c.
And a single copy of CMSIS!

It should make it easier for new people wanting to build their own firmware, and easier to maintain with respect to patching the libraries, such as with the USB Host NAK fix.

ST used to have a weird license on their firmware libs that explicitly didn't allow open source projects, but it seems to have been replaced with BSD, and CMSIS appears to be Apache.

mars 2020-09-13 02:19:32 UTC #16

oops sorry I might be confused, and the USB MIDI changes I was thinking of may already be merged.
Which means that the USBH messages are buffered before they are processed, which should lead to less risk of them being lost. But looking into the STM32_USB_Host_Library (in usbh_core.c) it seems that the callback that receives the data is only called from the USBH_Process(), which we invoke in the main thread which also handles the screen updates. Ie it's not interrupt driven.

It would be interesting to see if we can move the USBH_Process() call to a timer instead, to make it run more frequently. But it looks horribly inefficient!

antisvin 2020-09-13 09:45:37 UTC #17

Martin, running OpenWare on H7 is not a thing that belongs to the future - I'm already doing it

Regarding including drivers, this is the same approach that MI takes - https://github.com/pichenettes/stmlib/tree/master/third_party/STM . So look like licensing issues shouldn't be present for FLOSS.

Also, I'd say this is makes signifant change not for me or @mcc, but for potential new users that would want to mess. Not requiring installing CubeMX would lower barries for entry here, so this sounds like a welcomed addition.

Another thing is that it would make dealing with Cube FW updates easier for yourself, as they typically won't require re-exporting every project manually.

antisvin 2020-09-13 11:09:44 UTC #18

Actually, I think that this may not be necessary. Currently this thread is blocked to run every 20ms when screen is enabled. What we should do is block for 1ms or so, then update stuff that needs to run frequently (USB, probably digitalbus too) and skip display update until it needs to be refreshed.

This is the general idea, in reality we would probably need to precalculate next screen refresh time and use shorter wait period for last update to avoid jitter. But from your description, I'm pretty sure that display is blocking USB host in this case. Also, if we use this approach, it makes sense to enable DMA for SPI to minimize blocking in this thread, I haven't noticed any issues after enabling it on Magus.

mcc 2020-09-14 02:38:31 UTC #19

Hi, sorry, I haven't looked at this in a week or two.

I got an STM programmer and am interested to try reprogramming the bootloader soon. Are there instructions on replacing the bootloader (to get the new bootloader with the hang-at-boot protections)? Where can I find a copy of the improved bootloader? I see only the full packages at https://github.com/pingdynasty/OpenWare/releases .

Sorry, when I mentioned the SQ-1, I didn't necessarily mean I was using it as a sequencer— I saw the problem even in "Step Jump" mode, which basically transforms the SQ-1 into a keyboard with sixteen keys.

I tested just now (on git 10a981bf) with a Korg microKEY Air, so a bog-standard USB keyboard. Testing with my AndiSaw4 patch, if I just sit there and play scales a little under half the notes seem to be missing. It didn't seem like "lots of MIDI". It seemed like the problem was triggering with as few as six or seven notes over a period of four to five seconds. I can also test with the YM2413 synth patch (which is polyphonic) in which case I see something more insidious: NOTE OFF messages are frequently dropped, so after hitting random keys for ten seconds or so there is a drone of three continuous notes that never turn off. I could try to record a video if it helps.

I would be happy to test with your "monster" branch (especially if I can get the safety bootloader installed first… ). Which commit would be most useful for me to test? 45bf2e2?

instead of having to run STM32CubeMX to generate the library code for each subproject, how about we just bung a recent version of the libraries in a shared directory?

In my opinion, running STM32CubeMX was one of the least onerous parts of the firmware installation process. The hardest part of the entire build actually was that the cube-update.sh step was not not documented and I had to find out about it here. I am sure removing the STM32CubeMX step would make it easier in general but it wouldn't really make a difference to me.

Here is a question: are the libraries large? Storing libraries in git can get unpleasant, over time. Maybe you could put the libraries in a zip file or something and add a script that automatically downloads the correct version for whatever project the user is trying to install and unpacks it.

I did write up a tutorial on how to do the build from scratch on Windows. I will try to submit that soon, perhaps this week.

Incidentally, I have been listening to "YM2413 Synth" for about five or six minutes now with seven random notes held down infinitely because of the NOTE OFF breakage. It is actually very pleasant once you set some operators. Maybe "break notes off" should be a firmware option

antisvin 2020-09-14 07:22:09 UTC #20

I don't think that SQ-1 would be sending too much MIDI data, so maybe we're looking at the wrong issue here. Neither should a keyboard played manually do.

Also, Martin mentioned that MIDI receive runs in main thread (that deals with display), but actually it's transmit that goes there:

github.com

pingdynasty/OpenWare/blob/develop/Source/Owl.cpp#L904


#ifdef USE_SCREEN
graphics.draw();
graphics.display();
#endif /* USE_SCREEN */
#ifdef OLED_DMA
// When using OLED_DMA this must delay for a minimum amount to allow screen to update
vTaskDelay(xFrequency);
#else
vTaskDelayUntil(&xLastWakeTime, xFrequency);
#endif
midi_tx.transmit();
#ifdef USE_IWDG
IWDG->KR = 0xaaaa; // reset the watchdog timer (if enabled)
#endif
}


extern "C"{


#if 0 // ifdef USE_ENCODERS
int16_t getEncoderValue(uint8_t encoder){
  if(encoder == 0)

Receive runs in audio loop, so should be running every 1.33 ms : https://github.com/pingdynasty/OpenWare/blob/develop/Source/ProgramManager.cpp#L244

Now, in case of the first patch - it won't correctly handle processing multiple messages that arrive at once. The callback would be called multiple times, so all but the last note that arrives would be "lost" because patch would do nothing with them. Which doesn't mean that there's no MIDI issues in OpenWare, but in this case the patch itself has bugs. So we can't distinguish those issues.

Martin's patch looks like it handles MIDI note state properly. It would still have stuck notes if MIDI data arrives out of order (note off before note on). A good test for this would be to count how much note on and off events have arrived. If they differ when nothing is pressed, then this could be considered a confirmation that MIDI data is lost.