The esp32 port moves to CMake and has S2 support, new features for rp2 This release of MicroPython adds general support in the core for using CMake as a build system. The rp2 port is consolidated to use the new CMake files, and the esp32 and zephyr ports have switched to build as pure CMake projects. These three ports have SDKs which are built around CMake and this change should make them easier to maintain and use. As part of this work, CMake based ports now have support for user C modules. Authors of user C modules should now provide both .mk and .cmake configuration files (following the documentation and examples). A bug was fixed in the multiple precision integer library, an arithmetic overflow in the long division routine. Prior to this fix certain integer divisions would take excessive time and produce incorrect results. See commit 0a59938574502b19b3d685133084399c090d3c11 for details. There was also a fix for regular expressions, to check and report byte overflow errors when compiling expressions. See commit 172fb5230a3943eeb6fbbb4de1dc56b16e2a7637. In the uasyncio library, a new MicroPython extension has been added, ThreadSafeFlag, which can be set from outside the asyncio event loop, such as other threads, IRQs or scheduler context. It allows preemptive code like IRQs to signal asyncio tasks, which are by nature cooperative (non-preemptive). asyncio.current_task() has also been added, with the same semantics as CPython. As mentioned above, the esp32 port has switched to a full CMake-based project, and traditional make capability has been removed (although a simple helper Makefile remains to keep top-level build/deploy commands consistent with other ports). Because of the move to CMake, network.LAN support has been removed, to be added back in the future (use a prior release if LAN is needed). Basic support for Non-Volatile-Storage is added to the esp32 module. And there is also preliminary support for ESP32S2 SoCs and USB, with a GENERIC_S2 board defined. On the mimxrt and samd ports, USB CDC TX handling has been fixed so that it now works reliably. The rp2 port has had many more core Python features enabled, along with the enabling of ubinascii.crc32(), the uos.VfsFat and machine.Signal classes, and the uerrno module. uos.urandom() has been added and machine.freq() can now change the CPU clock frequency. The machine.UART class now has support for timeout/timeout_char, inverted TX/RX lines, and buffered TX/RX with configurable sized buffers. For PIO, StateMachine has added restart(), rx_fifo() and tx_fifo() helper functions, and support for FIFO joining. There is now support for user C modules (via CMake) and for building different board configurations (the default remains the PICO board). USB reliability has been improved. For the stm32 port, there is now more configuration options for boards, such as USBD VID/PID and fine-grained selection of modules. The UART class now supports LPUART on L0, L4, H7 and WB MCUs. WB MCUs have a fix for a race condition accessing the BLE ACL free buffer list, and a workaround for a low-level BLE bug. The zephyr port has been updated to use zephyr v2.5.0, and now builds MicroPython as a CMake target. What follows is a detailed list of changes, generated from the git commit history, and organised into sections. Main components =============== all: - rename BYTES_PER_WORD to MP_BYTES_PER_OBJ_WORD - add .git-blame-ignore-revs for fixing up git blame output py core: - gc: don't include mpconfig.h and misc.h in gc.h - remove BITS_PER_WORD definition - rename BITS_PER_BYTE to MP_BITS_PER_BYTE - rename WORD_MSBIT_HIGH to MP_OBJ_WORD_MSBIT_HIGH - gc: change include of stdint.h to stddef.h - mpz: fix overflow of borrow in mpn_div - add core cmake rule files - expand lists in core cmake custom commands - mkrules.cmake: rename QSTR_DEFS variables to QSTRDEFS - mkrules.cmake: add MICROPY_QSTRDEFS_PORT to qstr build process - nlr: implement NLR for AArch64 - nlrx64: fix typo in comment - vm: for tracing use mp_printf, and print state when thread enabled - rename remaining object types to be of the form mp_type_xxx - py.cmake: move qstr helper code to micropy_gather_target_properties - py.cmake: introduce MICROPY_INC_CORE as a list with core includes - profile: resolve name collision with STATIC unset - runtime: make sys.modules preallocate to a configurable size extmod: - vfs_posix_file: allow closing an already closed file - btstack: add HCI trace debugging option in btstack_hci_uart - btstack: add stub functions for passkey, l2cap bindings - btstack: enable SYNC_EVENTS, PAIRING_BONDING by default - uasyncio: add asyncio.current_task() - add core cmake rule files - nimble: ensure handle is set on read error - moduselect: fix unsigned/signed comparison for timeout!=-1 - uasyncio: add ThreadSafeFlag - nimble/hal/hal_uart: fix HCI_TRACE format specifiers - modbluetooth: allow NimBLE to use Zephyr static address - modussl: fix ussl read/recv/send/write errors when non-blocking - btstack: use MICROPY_HW_BLE_UART_BAUDRATE for first UART init - modbluetooth: separate enabling of "client" from "central" - extmod.cmake: add modonewire.c to MICROPY_SOURCE_EXTMOD list - modbluetooth: free temp arrays in gatts register services - re1.5: check and report byte overflow errors in _compilecode - extmod.cmake: add support to build btree module with CMake lib: - tinyusb: update to version 0.8.0 - mbedtls: switch to currently latest commit of LTS branch v2.16 - utils/gchelper_generic: implement AArch64 support - pico-sdk: update to latest version 1.1.0 drivers: no changes specific to this component/port mpy-cross: no changes specific to this component/port Support components ================== docs: - library/uasyncio.rst: add docs for ThreadSafeFlag - develop/cmodules.rst: document C-modules and micropython.cmake - develop: improve user C modules to properly describe how to build examples: - usercmodule: add micropython.cmake to the C and CPP examples - usercmodules: simplify user C module enabling - embedding: fix example so it compiles again tests: - extmod/vfs_posix.py: add more tests for VfsPosix class - extmod: add test for ThreadSafeFlag - multi_bluetooth: add basic performance tests - rename run-tests to run-tests.py for consistency - run-tests.py: reformat with Black - multi_bluetooth: skip tests when BLE features are unsupported - extmod/vfs_fat_fileio2.py: close test file at end of test - run-tests.py: provide more info if script run via pyboard crashes - feature_check: check for lack of pass result rather than failure - net_inet: add 'Strict-Transport-Security' to exp file tools: - add filesystem action examples to pyboard.py help - ci.sh: change esp32 CI to work with idf.py and IDF v4.0.2 - makemanifest.py: allow passing option args to include() - ci.sh: update zephyr docker image to v0.11.13 - verifygitlog.py: show required format regexp in error message - pydfu.py: support DFU files with elements of zero size - ci.sh: add CI for CMake USER_C_MODULE support - ci.sh: build user C modules for esp32 - metrics.py: fix esp32 output filename due to move to CMake - ci.sh: build esp32 using IDF v4.0.2 and v4.3 The ports ========= all ports: - remove def of MP_PLAT_PRINT_STRN if it's the same as the default - update to build with new tinyusb bare-arm port: - clean up the code, make it run on an F405, and add a README cc3200 port: no changes specific to this component/port esp8266 port: - modules: fix fs_corrupted() to use start_sec not START_SEC esp32 port: - add support to build using IDF with cmake - esp32_rmt: don't do unnecessary check for unsigned less than zero - add explicit initialisers to silence compiler warnings - remove traditional "make" capability - boards: remove old IDF v3 sdkconfig values - boards: enable BLE on all boards - README: update based on new IDF v4 cmake build process - add support to build with ESP-IDF v4.1.1 - add support to build with ESP-IDF v4.2 - remove obsolete IDF v3 code wrapped in MICROPY_ESP_IDF_4 - modsocket: remove unix socket error code translation - set MICROPY_USE_INTERNAL_ERRNO=0 to use toolchain's errno.h - boards: enable size optimisation for builds - add support to build with ESP-IDF v4.3 pre-release - add basic support for Non-Volatile-Storage in esp32 module - make machine.soft_reset() work in main.py and reset_cause() - define MICROPY_QSTRDEFS_PORT to include special qstrs - Makefile: specify port and baud on erase_flash command - machine_hw_spi: use default pins when making SPI if none given - restore USER_C_MODULE support with new CMake build system - fix multiple definition errors with mp_hal_stdout_tx functions - enable btree module - modsocket: correctly handle poll/read of unconnected TCP socket - add initial support for ESP32S2 SoCs - add support for USB with CDC ACM - boards: add GENERIC_S2 board definition - machine_pin: use rtc_gpio_deinit instead of gpio_reset_pin javascript port: no changes specific to this component/port mimxrt port: - fix USB CDC handling so it works reliably - boards: add MIMXRT1050_EVK board, based on MIMXRT1060_EVK - enable CPYTHON_COMPAT, PY_ASYNC_AWAIT, PY_ATTRTUPLE options minimal port: no changes specific to this component/port nrf port: - drivers/usb: add USBD_IRQHandler which calls tud_int_handler pic16bit port: no changes specific to this component/port powerpc port: no changes specific to this component/port qemu-arm port: no changes specific to this component/port rp2 port: - machine_adc: only initialise the ADC periph if not already enabled - micropy_rules.cmake: fix makemoduledefs vpath to work with abs path - use local tinyusb instead of the one in pico-sdk - enable MICROPY_PY_UBINASCII_CRC32 to get ubinascii.crc32() - enable VfsFat class for FAT filesystem support - machine_uart: add timeout/timeout_char to read and write - machine_uart: add support for inverted TX and RX lines - rp2_pio: allow more than 8 consecutive pins for PIO out/set/sideset - rp2_pio: fix sm.get(buf) to not wait after getting last item - modmachine: allow changing CPU clock frequency - modmachine: re-init UART for REPL on frequency change - rp2_flash: prevent MICROPY_HW_FLASH_STORAGE_BASE being set negative - enable uerrno module - enabled more core Python features - modmachine: enable machine.Signal class - use core-provided cmake fragments instead of custom ones - mpthreadport.h: cast core_state to _mp_state_thread_t - add support for USER_C_MODULES to CMake build system - don't advertise remote wakeup for USB serial - CMakeLists.txt: enable USB enumeration fix - import uarray instead of array in rp2 module - rp2_pio: validate state machine frequency in constructor - moduos: implement uos.urandom() - rp2_pio: add StateMachine restart,rx_fifo,tx_fifo helper functions - machine_uart: add buffered transfer of data with rxbuf/txbuf kwargs - add support for building different board configurations - rp2_pio: add fifo_join support for PIO samd port: - mphalport: fix USB CDC tx handling to work reliably stm32 port: - uart: add uart_set_baudrate function - mpbthciport: only init the uart once, then use uart_set_baudrate - mboot: add unpack-dfu command to mboot_pack_dfu.py tool - usb: allow a board to configure USBD_VID and all PIDs - make pyb, uos, utime, machine and onewire modules configurable - boards: disable onewire module on boards with small flash - mpbthciport: fix initial baudrate to use provided value - mpbthciport: use mp_printf instead of printf for error message - mpbtstackport: allow chipset and secondary baudrate to be set - mboot: after sig verify, only write firmware-head if latter valid - uart: add support for LPUART1 on L0, L4, H7 and WB MCUs - boards/NUCLEO_WB55: enable LPUART1 on PA2/PA3 - enable MICROPY_PY_UBINASCII_CRC32 to get ubinascii.crc32() - rfcore: allow BLE settings to be changed by a board - storage: prevent attempts to read/write invalid block addresses - make-stmconst.py: allow "[]" chars when parsing source comments - main: fix passing state.reset_mode to init_flash_fs - powerctrl: save and restore EWUP state when configuring standby - spi: fix baudrate calculation for H7 series - boardctrl: add MICROPY_BOARD_STARTUP hook - Makefile: fix C++ linker flags when toolchain has spaces in path - Makefile: allow QSTR_DEFS,QSTR_GLOBAL_DEPENDENCIES to be extended - include .ARM section in firmware for C++ exception handling - powerctrl: allow a board to configure AHB and APB clock dividers - powerctrl: support using PLLI2C on STM32F413 as USB clock source - boards/pllvalues.py: relax PLLQ constraints on STM32F413 MCUs - mpconfigport.h: add support for a board to specify root pointers - boardctrl: give boards control over execution of boot.py,main.py - boards/NUCLEO_L476RG: add 5 remaining UARTs - rfcore: fix race condition with C2 accessing free buffer list - rfcore: intercept addr-resolution HCI cmd to work around BLE bug - boards: split UARTx_RTS_DE into UARTx_RTS/UARTx_DE in pin defs - uart: use LL_USART_GetBaudRate to compute baudrate - sdram: make MICROPY_HW_FMC_BA1,MICROPY_HW_FMC_A11 optional pins teensy port: no changes specific to this component/port unix port: - mpbtstackport_common: implement mp_bluetooth_hci_active - moduselect: don't allow both posix and non-posix configurations - improve command line argument processing - main: make static variable that's potentially clobbered by longjmp windows port: no changes specific to this component/port zephyr port: - update to zephyr v2.5.0 - disable frozen source modules - remove unused build files - build MicroPython as a cmake target - boards: add support for the nucleo_h743zi board - modusocket: fix parameter in calls to net_context_get_XXX()