Building the Magus firmware

mars · September 13, 2020, 2:19am

oops sorry I might be confused, and the USB MIDI changes I was thinking of may already be merged.
Which means that the USBH messages are buffered before they are processed, which should lead to less risk of them being lost. But looking into the STM32_USB_Host_Library (in usbh_core.c) it seems that the callback that receives the data is only called from the USBH_Process(), which we invoke in the main thread which also handles the screen updates. Ie it’s not interrupt driven.

It would be interesting to see if we can move the USBH_Process() call to a timer instead, to make it run more frequently. But it looks horribly inefficient!

antisvin · September 13, 2020, 9:45am

Martin, running OpenWare on H7 is not a thing that belongs to the future - I’m already doing it

Regarding including drivers, this is the same approach that MI takes - stmlib/third_party/STM at master · pichenettes/stmlib · GitHub . So look like licensing issues shouldn’t be present for FLOSS.

Also, I’d say this is makes signifant change not for me or @mcc, but for potential new users that would want to mess. Not requiring installing CubeMX would lower barries for entry here, so this sounds like a welcomed addition.

Another thing is that it would make dealing with Cube FW updates easier for yourself, as they typically won’t require re-exporting every project manually.

antisvin · September 13, 2020, 11:09am

Actually, I think that this may not be necessary. Currently this thread is blocked to run every 20ms when screen is enabled. What we should do is block for 1ms or so, then update stuff that needs to run frequently (USB, probably digitalbus too) and skip display update until it needs to be refreshed.

This is the general idea, in reality we would probably need to precalculate next screen refresh time and use shorter wait period for last update to avoid jitter. But from your description, I’m pretty sure that display is blocking USB host in this case. Also, if we use this approach, it makes sense to enable DMA for SPI to minimize blocking in this thread, I haven’t noticed any issues after enabling it on Magus.

mcc · September 14, 2020, 2:50am

Hi, sorry, I haven’t looked at this in a week or two.

I got an STM programmer and am interested to try reprogramming the bootloader soon. Are there instructions on replacing the bootloader (to get the new bootloader with the hang-at-boot protections)? Where can I find a copy of the improved bootloader? I see only the full packages at Releases · pingdynasty/OpenWare · GitHub .

Sorry, when I mentioned the SQ-1, I didn’t necessarily mean I was using it as a sequencer— I saw the problem even in “Step Jump” mode, which basically transforms the SQ-1 into a keyboard with sixteen keys.

I tested just now (on git 10a981bf) with a Korg microKEY Air, so a bog-standard USB keyboard. Testing with my AndiSaw4 patch, if I just sit there and play scales a little under half the notes seem to be missing. It didn’t seem like “lots of MIDI”. It seemed like the problem was triggering with as few as six or seven notes over a period of four to five seconds. I can also test with the YM2413 synth patch (which is polyphonic) in which case I see something more insidious: NOTE OFF messages are frequently dropped, so after hitting random keys for ten seconds or so there is a drone of three continuous notes that never turn off. I could try to record a video if it helps.

I would be happy to test with your “monster” branch (especially if I can get the safety bootloader installed first… ). Which commit would be most useful for me to test? 45bf2e2?

instead of having to run STM32CubeMX to generate the library code for each subproject, how about we just bung a recent version of the libraries in a shared directory?

In my opinion, running STM32CubeMX was one of the least onerous parts of the firmware installation process. The hardest part of the entire build actually was that the cube-update.sh step was not not documented and I had to find out about it here. I am sure removing the STM32CubeMX step would make it easier in general but it wouldn’t really make a difference to me.

Here is a question: are the libraries large? Storing libraries in git can get unpleasant, over time. Maybe you could put the libraries in a zip file or something and add a script that automatically downloads the correct version for whatever project the user is trying to install and unpacks it.

I did write up a tutorial on how to do the build from scratch on Windows. I will try to submit that soon, perhaps this week.

Incidentally, I have been listening to “YM2413 Synth” for about five or six minutes now with seven random notes held down infinitely because of the NOTE OFF breakage. It is actually very pleasant once you set some operators. Maybe “break notes off” should be a firmware option

antisvin · September 14, 2020, 7:22am

I don’t think that SQ-1 would be sending too much MIDI data, so maybe we’re looking at the wrong issue here. Neither should a keyboard played manually do.

Also, Martin mentioned that MIDI receive runs in main thread (that deals with display), but actually it’s transmit that goes there:

github.com

pingdynasty/OpenWare/blob/develop/Source/Owl.cpp#L904

    
      
          #include <string.h>
          #include <stdlib.h>
          #include "device.h"
          #include "Owl.h"
          #ifdef USE_CODEC
          #include "Codec.h"
          #endif
          #include "MidiReceiver.h"
          #include "MidiStreamReader.h"
          #include "MidiController.h"
          #include "ProgramVector.h"
          #include "ProgramManager.h"
          #include "ApplicationSettings.h"
          #include "VersionToken.h"
          #include "cmsis_os.h"
          #include "BitState.hpp"
          #include "errorhandlers.h"
          #include "message.h"
          #include "Storage.h"
          #include "PatchRegistry.h"
          
          
#ifdef OWL_BIOSIGNALS
          #include "ads.h"
          #ifdef USE_KX122
          #include "kx122.h"
          #endif
          #include "ble_midi.h"
          #endif
          
          
#if defined USE_RGB_LED
          #include "rainbow.h"
          #endif /* USE_RGB_LED */
          
          
#ifdef USE_USB_HOST
          #include "usbh_core.h"
          extern "C"{
            void MX_USB_HOST_Process(void);
          }
          #endif /* USE_USB_HOST */
          
          
#ifdef USE_DIGITALBUS
          #include "bus.h"
          #endif /* USE_DIGITALBUS */
          
          
Owl owl;
          #ifdef USE_RGB_LED
          uint32_t ledstatus;
          #endif
          MidiController midi_tx;
          MidiReceiver midi_rx;
          ApplicationSettings settings;
          #ifdef USE_CODEC
          Codec codec;
          #endif
          #ifdef USE_ADC
          uint16_t adc_values[NOF_ADC_VALUES] DMA_RAM;
          #endif
          #ifdef USE_DAC
          extern DAC_HandleTypeDef hdac;
          #endif
          
          
int16_t getAnalogValue(uint8_t ch){
          #ifdef USE_ADC
            if(ch < NOF_ADC_VALUES)
              return adc_values[ch];
          #endif
            return 0;
          }
          
          
void midiSetInputChannel(int8_t channel){
            midi_rx.setInputChannel(channel);
          }
          
          
void midiSetOutputChannel(int8_t channel){
            settings.midi_output_channel = channel;
            midi_tx.setOutputChannel(channel);
          }
          
          
__weak void initLed(){
            // Initialise RGB LED PWM timers
          #if defined OWL_TESSERACT
            extern TIM_HandleTypeDef htim2;
            extern TIM_HandleTypeDef htim3;
            // Red
            HAL_TIM_Base_Start(&htim2);
            HAL_TIM_PWM_Start(&htim2, TIM_CHANNEL_1);
            // Green
            HAL_TIM_PWM_Start(&htim2, TIM_CHANNEL_2);
            // Blue
            HAL_TIM_Base_Start(&htim3);
            HAL_TIM_PWM_Start(&htim3, TIM_CHANNEL_4);
          #elif defined OWL_BIOSIGNALS
          #ifdef USE_LED_PWM
            extern TIM_HandleTypeDef htim1;
            HAL_TIM_Base_Start(&htim1);
            HAL_TIM_PWM_Start(&htim1, TIM_CHANNEL_1);
            HAL_TIM_PWM_Start(&htim1, TIM_CHANNEL_2);
            HAL_TIM_PWM_Start(&htim1, TIM_CHANNEL_3);
          #else
            /*Configure GPIO pin : LED_GREEN_Pin, LED_RED_Pin */
            GPIO_InitTypeDef GPIO_InitStruct = {0};
            GPIO_InitStruct.Pin = LED_GREEN_Pin | LED_RED_Pin;
            GPIO_InitStruct.Mode = GPIO_MODE_OUTPUT_PP;
            GPIO_InitStruct.Pull = GPIO_NOPULL;
            GPIO_InitStruct.Speed = GPIO_SPEED_FREQ_LOW;
            HAL_GPIO_Init(LED_GREEN_GPIO_Port, &GPIO_InitStruct);
          #endif
          #endif
          }
          
          

          
extern "C" {
          
          
void HAL_GPIO_EXTI_Callback(uint16_t pin){
            onChangePin(pin);
          }
          }
          
          
static TickType_t xLastWakeTime;
          static TickType_t xFrequency;
          
          
void Owl::setup(void){
          #ifdef USE_IWDG
          #ifdef STM32H7xx
            IWDG1->KR = 0xCCCC; // Enable IWDG and turn on LSI
            IWDG1->KR = 0x5555; // ensure watchdog register write is allowed
            IWDG1->PR = 0x05;   // prescaler 128
            IWDG1->RLR = 0x753; // reload 8 seconds
          #else
            IWDG->KR = 0xCCCC; // Enable IWDG and turn on LSI
            IWDG->KR = 0x5555; // ensure watchdog register write is allowed
            IWDG->PR = 0x05;   // prescaler 128
            IWDG->RLR = 0x753; // reload 8 seconds
          #endif
          #endif
          #ifdef USE_BKPSRAM
            HAL_PWR_EnableBkUpAccess();
          #endif
          #ifdef USE_RGB_LED
            ledstatus = 0;
          #endif
            storage.init();
            registry.init();
            settings.init(); // settings need the registry to be initialised first
            onResourceUpdate();
          #ifdef USE_CODEC
            codec.init();
            codec.set(0);
            codec.bypass(false);
            codec.setInputGain(settings.audio_input_gain);
            codec.setOutputGain(settings.audio_output_gain);
          #endif /* USE_CODEC */
          
          
  program.startManager();
          
          
#ifdef USE_DAC
            HAL_DAC_Start(&hdac, DAC_CHANNEL_1);
            HAL_DAC_Start(&hdac, DAC_CHANNEL_2);
            setAnalogValue(PARAMETER_F, 0);
            setAnalogValue(PARAMETER_G, 0);
          #endif
          
          
#if defined USE_ADC && !defined OWL_WAVETABLE
            extern ADC_HandleTypeDef ADC_PERIPH;
            HAL_StatusTypeDef ret = HAL_ADC_Start_DMA(&ADC_PERIPH, (uint32_t*)adc_values, NOF_ADC_VALUES);
            if(ret != HAL_OK)
              error(CONFIG_ERROR, "ADC Start failed");
          #endif /* USE_ADC */
          
          
  midi_rx.init();
            midiSetInputChannel(settings.midi_input_channel);
            midiSetOutputChannel(settings.midi_output_channel);
          
          
  xLastWakeTime = xTaskGetTickCount();
            xFrequency = MAIN_LOOP_SLEEP_MS / portTICK_PERIOD_MS; // 20mS = 50Hz refresh rate
          
          
#ifdef USE_DIGITALBUS
            bus_setup();
            bus_set_input_channel(settings.midi_input_channel);
          #endif /* USE_DIGITALBUS */
          }
          
          
#ifdef USE_DIGITALBUS
          int busstatus;
          #endif
          
          
#ifdef USE_RGB_LED
          void updateLed(){
            uint32_t colour =
              (adc_values[ADC_A]>>3)+
              (adc_values[ADC_B]>>3)+
              (adc_values[ADC_C]>>3)+
              (adc_values[ADC_D]>>3);
          #ifdef ADC_E
            colour += (adc_values[ADC_E]>>3);
          #endif
          #ifdef FASCINATION_MACHINE
            extern float audio_envelope;
            colour = colour*(1+audio_envelope);
          #endif
            colour &= 0x3ff;
            setLed(0, ledstatus | rainbow[colour]);
            // setLed(4095-adc_values[0], 4095-adc_values[1], 4095-adc_values[2]);
          }
          #endif /*USE_RGB_LED */
          
          
OperationMode Owl::getOperationMode(){
            return operationMode;
          }
          
          
void Owl::setOperationMode(OperationMode mode){
            OperationMode old_mode = operationMode;
            operationMode = mode;
            onChangeMode(mode, old_mode);
          }
          
          
void Owl::loop(){
          #ifdef USE_DIGITALBUS
            busstatus = bus_status();
          #endif
          #ifdef OLED_DMA
            // When using OLED_DMA this must delay for a minimum amount to allow screen to update
            vTaskDelay(xFrequency);
          #else
            vTaskDelayUntil(&xLastWakeTime, xFrequency);
          #endif
          #ifdef USE_IWDG
          #ifdef STM32H7xx
            IWDG1->KR = 0xaaaa; // reset the watchdog timer (if enabled)
          #else
            IWDG->KR = 0xaaaa; // reset the watchdog timer (if enabled)
          #endif
          #endif
            if(backgroundTask != NULL)
              backgroundTask->loop();
          }
          
          
void Owl::setBackgroundTask(BackgroundTask* bt){
            if(backgroundTask != NULL)
              backgroundTask->end();
            backgroundTask = bt;
            if(backgroundTask != NULL)
              backgroundTask->begin();
          }
          
          
void jump_to_bootloader(void){
          #ifdef USE_USB_HOST
          #if defined USB_HOST_PWR_EN_GPIO_Port && defined USB_HOST_PWR_EN_Pin
            HAL_GPIO_WritePin(USB_HOST_PWR_EN_GPIO_Port, USB_HOST_PWR_EN_Pin, GPIO_PIN_RESET);
          #endif
          #endif
          // #ifdef USE_USBD_MIDI
          //   extern USBD_HandleTypeDef USBD_HANDLE;
          //   USBD_DeInit(&USBD_HANDLE);
          // #endif
            *OWLBOOT_MAGIC_ADDRESS = OWLBOOT_MAGIC_NUMBER;
          #ifdef USE_BKPSRAM
            extern RTC_HandleTypeDef hrtc;
            HAL_RTCEx_BKUPWrite(&hrtc, RTC_BKP_DR1, 0);
          #endif
            /* Disable all interrupts */
          #ifdef STM32H7xx
            RCC->CIER = 0x00000000;
          #else
            RCC->CIR = 0x00000000;
          #endif
            NVIC_SystemReset();
            /* Shouldn't get here */
            while(1);
          }
          
          
void device_reset(){
            *OWLBOOT_MAGIC_ADDRESS = 0;
          #ifdef USE_BKPSRAM
            extern RTC_HandleTypeDef hrtc;
            HAL_RTCEx_BKUPWrite(&hrtc, RTC_BKP_DR1, 0);
          #endif
            /* Disable all interrupts */
          #ifdef STM32H7xx
            RCC->CIER = 0x00000000;
          #else
            RCC->CIR = 0x00000000;
          #endif
            NVIC_SystemReset();
            /* Shouldn't get here */
            while(1);
          }
          
          
// called from patch program: Patch::sendMidi(MidiMessage)
          void midi_send(uint8_t port, uint8_t status, uint8_t d1, uint8_t d2){
            midi_tx.send(MidiMessage(port, status, d1, d2));
          }
          
          
const char* getFirmwareVersion(){ 
            return (const char*)(HARDWARE_VERSION " " FIRMWARE_VERSION) ;
          }
          
          
extern char _BOOTLOADER, _ISR_VECTOR_SIZE;
          VersionToken* bootloader_token = reinterpret_cast<VersionToken*>(
            (uint32_t)&_BOOTLOADER + (uint32_t)&_ISR_VECTOR_SIZE);
          
          
const char* getBootloaderVersion(){
            if (bootloader_token->magic == BOOTLOADER_MAGIC){
              return bootloader_token->version;
            }
            else {
              return "N/A";
            }
          }
          
          
const char* getDeviceName(){
            static char name[22];
            static char* ptr = 0;
            if(ptr == 0){
              uint32_t* id = (uint32_t*)UID_BASE; // get a pointer to the 96-bit unique device id
              unsigned int hash = id[0]^id[1]^id[2]; // hash into 32-bit value
              hash = (hash>>16)^(hash&0xffff); // hash into 16-bit value
              char* p = stpcpy(name, "OWL-" HARDWARE_VERSION "-");
              stpcpy(p, msg_itoa(hash, 16));
              ptr = name;
            }
            return ptr;
          }

This file has been truncated. show original

Receive runs in audio loop, so should be running every 1.33 ms : OpenWare/ProgramManager.cpp at develop · pingdynasty/OpenWare · GitHub

Now, in case of the first patch - it won’t correctly handle processing multiple messages that arrive at once. The callback would be called multiple times, so all but the last note that arrives would be “lost” because patch would do nothing with them. Which doesn’t mean that there’s no MIDI issues in OpenWare, but in this case the patch itself has bugs. So we can’t distinguish those issues.

Martin’s patch looks like it handles MIDI note state properly. It would still have stuck notes if MIDI data arrives out of order (note off before note on). A good test for this would be to count how much note on and off events have arrived. If they differ when nothing is pressed, then this could be considered a confirmation that MIDI data is lost.

antisvin · September 14, 2020, 7:42am

There’s no separate repo for this, it’s been added to bootloader project itself. Just not distributed to end users yet. Bootloader itself is built similar to Magus firmware. So what you should do:

Build MidiBoot with USE_IWDG enabled in MidiBoot/Inc/hardware.h
Build whatever firmware you want to use for Magus with USE_IWDG enabled in Magus/Inc/hardware.h . Not doing this would cause your device to return to bootloader - for watchdog there’s no difference between device that just hangs or is built without this option.

And you’ll have to do your own research for using openocd. MidiBoot has all the necessary commands in its Makefile, but something may need tweaking in openocd config. IIRC, my Chinese clones of ST-Link don’t have a working reset andt require commenting out this line in openocd config: https://github.com/pingdynasty/OpenWare/blob/master/Hardware/openocd.cfg#L10

mars · September 15, 2020, 9:01am

Yes, sorry! I think it is part of the same problem, of having to first generate then customise the libraries. I’m not keen on maintaining a patch-script for every subproject, and trying to keep them up to date with whatever the Cube might throw at us.

Around 50M in source, after pruning. Not small.

antisvin · September 15, 2020, 10:36am

We could have a fork of cube drivers with custom patches and FreeRTOS as git submodules. Then main repo won’t be slowed down from extra code.

I think CubeMX can be configured to only generate source for project and not fetch all dependencies. So running cubeMX would be required only for setting up new projects.

mcc · September 15, 2020, 3:17pm

Hm, so to be clear, does this mean if I install the upgraded/safe bootloader, and then I install an official Magus build from the releases page, that the official builds WON’T work with the safe bootloader? Or are the official builds being built with USE_IWDG now?

antisvin · September 15, 2020, 4:11pm

Yes, without watchdog-enabled FW you will be returned to bootloader. And I think that doing it the other way around (stock bootloader + watchdog FW) may leave you with device that hangs after booting. Currently there are no FW releases that enable it (it was actually added not so long ago). I suspect that it would only be used officially by later devices.

Btw, it’s not mandatory to use bootloader at all if you’re using programmer to flash FW. FW can run on its own and you will probably end up not using bootloader to run debugger.

mcc · September 20, 2020, 5:38pm

@antisvin , I’m trying to follow your instructions on attaching the programmer so I can install the watchdog bootloader… I am trying for now to do it without soldering. The programmer comes with the female version of “jump wires”, I am not sure the term for them, so I just need to affix those to the pins.

Uh oh. So I think I need to remove the white card and attach the programmer wires to it. But the white card is basically fixed in place by all the wires above it. I’m not sure where to pull.

I assume the white card “on top” is the “IO card”, which you said I might have to remove? But I am not sure how to remove that, either…

How should I proceed? Maybe I could test the USB branch without installing the new bootloader.

Sorry for all these basic questions, I’m pretty helpless with hardware

mcc · September 20, 2020, 6:48pm

Hi @mars, I tried checking out the current head of develop branch so I could test if it fixes my USB-MIDI woes. I got the following build error

make[1]: Entering directory '/mnt/c/Users/Andi/work/gh/OpenWare/Magus'
/mnt/c/Users/Andi/work/gh/OpenWare/Source/ProgramManager.cpp: In function 'void onProgramReady()':
/mnt/c/Users/Andi/work/gh/OpenWare/Source/ProgramManager.cpp:245:20: error: 'parameter_values' was not declared in this scope
   updateParameters(parameter_values, NOF_PARAMETERS, adc_values, NOF_ADC_VALUES);
                    ^~~~~~~~~~~~~~~~
/mnt/c/Users/Andi/work/gh/OpenWare/Source/ProgramManager.cpp:245:54: error: 'adc_values' was not declared in this scope
   updateParameters(parameter_values, NOF_PARAMETERS, adc_values, NOF_ADC_VALUES);
                                                      ^~~~~~~~~~

I tried doing a make clean, “git clean xffd” and a reset and I re-ran STMcube. I got the same error.

I see adc_values is defined behind a USE_ADC ifdef guard and parameter_values is behind a USE_SCREEN guard. Am I doing something wrong in my build, that these two defines are not being set? I don’t know what changed but I did not see this error with builds of older commits.

In any case I think it is a bug that when USE_ADC and USE_SCREEN are undefined, the updateParameters() call is still included. Probably that call should be behind the USE_ADC call (for example) also.

By the way, I have sent you a minor pull request.

antisvin · September 20, 2020, 7:19pm

It’s fairly straightforward to remove the digital board from Magus:

unscrew audio jacks on the outside
remove IO panel (the narrow panel that hold all jacks) - there’s no need to unplug cables, just move the panel outside the case
now there’s enough space to pull out digital board

Do it a few times and you’ll really want to solder headers for quick access.

And yes, you don’t have to install the bootloader. I mostly tried to explain that you should have a programmer ready when installing unstable FW. Updated bootloader is an alternative to that, but it’s not mandatory to use it.

antisvin · September 20, 2020, 7:34pm

Speaking of PRs, I’ve made one for fixing this very issue just yesterday - Develop branch won't build on devices with screen by antisvin · Pull Request #15 · pingdynasty/OpenWare · G

But I haven’t run it on Magus, just made sure that it builds. Actually, recent updates in develop branch broke something related to USB interrupts on device on which I’ve ran it. ~~So there’s a good chance that it won’t work on Magus, since it’s obvious that Martin never built latest commits on it.~~ Nevermind, seems to run fine.

mcc · September 20, 2020, 11:42pm

Thanks @antisvin. Can confirm your PR#15 makes it build.

When you say “unscrew the audio jacks on the outside”, what do you mean? I see a little hex screw thing on the side. Or am I supposed to find a hex wrench thing and physically unscrew the jacks…? Again sorry about these basic questions.

mcc · September 20, 2020, 11:21pm

@mars: I did a test with ddfe79cf55f1 (current develop head) merged with a447765258d9 (antisvin’s PR#15). Testing with the Korg microKEY Air, I am still seeing the MIDI drops in both patches I tested. If anything the success rate may have dropped a little.

What would you recommend? Should I try the USB Full Speed setting as antisvin suggests? The USB-MIDI drop issue is the main thing preventing me from using the Magus for music right now.

mcc · September 20, 2020, 11:41pm

A second thing for @mars – Here is my attempt at a “how to build firmware for Magus” tutorial.

I’m not sure where the proper place to submit tutorials is.

I think the tutorial is unfinished because (1) I am still confused how to attach the programmer by these instructions… and (2) I would like to add a final section explaining how to build a patch locally. But the main part should be good reproducible steps.

antisvin · September 21, 2020, 10:12am

Yup, those 3 hex nuts for audio jacks should be unscrewed using a wrench or pliers. A good engineering solution would be to try fingernails in case if they’re not too tight Teeth are not recommended, because device warranty won’t cover potential dental damage.

mars · September 21, 2020, 1:33pm

Sorry I’ve not had time to dig deeper into this. I don’t think changing between USB FS/HS will work, tbh, at least I don’t see how. We use the internal FS PHY (physical interface), to run in HS requires an external PHY.

There are a couple of other things we can try: USB DMA, and a dedicated IRQ for one of the USBH endpoints. But the latter will probably required circumventing the USB Host middleware.

What I really want to try is running the USBH process on a 1ms timer - I’ve got some code for it, but need a bit of dedicated time to test it. Definitely will happen this week.

mars · September 21, 2020, 1:53pm

A bit more digging reveals that there is (I think) an interrupt associated with USBH IN and OUT requests, in the HCD handlers - see HCD_HC_IN_IRQHandler and HCD_HC_OUT_IRQHandler. So perhaps there’s a more elegant and efficient solution to this.