New rp2 port, BLE pairing, bonding and l2cap APIs, and docs on internals
This release of MicroPython sees the addition of a new "rp2" port for the
new Raspberry Pi RP2040 microcontroller, a dual-core Cortex-M0+ MCU with
programmable IO. This port is built on the pico-sdk and MicroPython sits
very close to the bare metal. The programmable IO is supported via the
@rp2.asm_pio decorator which allows writing inline assembler to be run on
the PIO state machines.
The ubluetooth module has seen significant improvements, fixes and
additions. BLE events are now synchronous on unix and stm32 so that user
Python callbacks are called directly from the BLE stack (from NimBLE).
This helps with the new pairing and bonding API. There is also a new l2cap
API for much faster raw data transfer between BLE devices.
Many new sections have been added to the existing "Developing and building
MicroPython" chapter to make it all about the internals of MicroPython,
including details about the MicroPython compiler, and how to port to a new
system. This documentation work was done as part of Google's Season of
Docs 2020.
The native emitter and inline assembler now fully support ARMv6M, for
Cortex-M0+ CPUs. There is also support added for C++ user modules on
unix, stm32 and esp32. And reproducible builds can now be achieved by
setting SOURCE_DATE_EPOCH in the shell build environment. Qstr
preprocessing is now done in parallel to speed up building, and there is
a new tools/ci.sh script with all functionality to run the continuous
integration tests.
The machine.I2C and machine.SPI classes have been modified so they no
longer construct both software- and hardware-based peripheral instances.
Such construction is now split to explicit soft and non-soft types by the
introduction of machine.SoftI2C and machine.SoftSPI classes. See below for
more information.
The wait_for function in uasyncio now handles cancellation correctly,
VfsLfs supports mounting in read-only mode, and the urandom module will
randomise its seed on import on stm32, esp8266, esp32 and rp2 ports.
A new raw-paste mode is added to the REPL which includes flow control
between the device and host, as well as a mechanism where the device pulls
in data from the host as it parses the input. This makes pasting code
faster, more reliable and use less memory. This feature is supported on
all ports that have a raw REPL and still supports previous REPL behaviour.
See the documentation for more detail on this raw-paste mode.
The stm32 port sees improved rfcore support for WB MCUs, including flash
locking for writes when BLE is active, and a script for automatic update of
FUS/WS firmware. New hooks are added for boards to configure the top level
behaviour of the system in stm32_main, and there is now support for
Ethernet on H7 processors. Mboot now allows signed and encrypted firmware
updates.
Breaking changes in this release are:
- The "sys" module is renamed to "usys". Ports that do not enable module
weak links must replace "import sys" with "import usys as sys".
See commit 40ad8f1666b265dafc7844d765f45cfae4b6299f
- machine.I2C and machine.SPI constructors are changed when using
software-based I2C and SPI. Code that constructed I2C/SPI peripherals in
the following way will need to be changed (or else their use will emit a
warning which will eventually be an error):
machine.I2C(-1, ...) -> machine.SoftI2C(...)
machine.I2C(scl=scl, sda=sda) -> machine.SoftI2C(scl=scl, sda=sda)
machine.SPI(-1, ...) -> machine.SoftSPI(...)
machine.SPI(sck=sck, mosi=mosi, miso=miso)
-> machine.SoftSPI(sck=sck, mosi=mosi, miso=miso)
Code which uses machine.I2C and machine.SPI classes to access hardware
peripherals does not need to change.
See commits 39d50d129ce428858332523548f0594503d0f45b and
98182a97c5a9229938406beb722966eacceeb823
- In ubluetooth, the BLE.irq() method now has positional only arguments.
See commit 6a6a5f9e151473bdcc1d14725d680691ff665a82
- In ubluetooth, arguments passed to the BLE.irq() callback that were
previously bytes objects are now memoryview objects.
See commit 81f2162ca0e926c9a4a1787a3863d94d86be0b36
What follows is a detailed list of changes, generated from the git commit
history, and organised into sections.
Main components
===============
all:
- rename "sys" module to "usys"
- rename absolute time-based functions to include "epoch"
py core:
- objfloat: fix handling of negative float to power of nan
- fix handling of NaN in certain pow implementations
- showbc: pass in an mp_print_t struct to all bytecode-print functions
- parse: pass in an mp_print_t to mp_parse_node_print
- dynruntime.h: add mp_import_* and mp_load/store_*
- objstr: make bytes(bytes_obj) return bytes_obj
- objarray.h: add mp_obj_memoryview_init() helper function
- makeversionhdr.py: match only git tags which look like versions
- parse: expose rule-name printing as MICROPY_DEBUG_PARSE_RULE_NAME
- objdict: add mp_const_empty_dict_obj, use it for mp_const_empty_map
- objtype: handle __dict__ attribute when type has no locals
- scope: name and use id_kind_type_t
- objexcept: compare mp_emergency_exception_buf_size signed
- use unsigned comparison of chars
- add explicit initializers for default values
- vmentrytable: ignore GCC -Woverride-init
- introduce and use MP_FALLTHROUGH macro
- makeqstrdefs.py: fix beaviour when scanning non-C preprocessed files
- makeqstrdefs.py: process C++ files as well
- mkrules.mk: add target for compiling C++ files
- makeqstrdefs.py: support preprocessing C++ files for QSTR generation
- workaround clang error when building misc.h with C++ compiler
- py.mk: support C++ code for user C modules
- mpz: do sign extension in mpz_as_bytes for negative values
- binary: fix sign extension setting wide integer on 32-bit archs
- mpprint: prevent case fall-through when assert is disabled
- mpprint: fix length calculation for strings with precision-modifier
- modmath: simplify handling of positional args to reduce code size
- mkrules.mk: remove stray vpath and unused -Itmp, add $(Q) for $(AR)
- objfun: support fun.__globals__ attribute
- asmthumb: add support for ARMv6M in native emitter
- emitnative: ensure encoding to load prelude_offset doesn't change sz
- emitinlinethumb: exclude code using #if when ARMV7M disabled
- emitnative: support binary ops on ARMv6M without use of ite instr
- qstr.h: remove QSTR_FROM_STR_STATIC macro
- gc: fix debug printing of pointer
- persistentcode: introduce MICROPY_PERSISTENT_CODE_SAVE_FILE option
- makeversionhdr: honor SOURCE_DATE_EPOCH if present
extmod:
- modbluetooth: rename logging macro to be just DEBUG_printf
- modbluetooth: refactor stack/hci/driver/port bindings
- nimble: set struct alignment correctly on 64-bit arch
- nimble: make nimble_malloc work with allocated size
- nimble: implement NimBLE mutex
- modbluetooth: implement configuration of address modes
- modbluetooth: allow using mp_hal_get_mac as a static address
- btstack: add btstack support for _IRQ_GATTS_READ_REQUEST
- btstack: detect HCI UART init failure
- nimble: add timeout for HCI sync on startup
- modussl_axtls: reduce size of code that makes exception
- modbluetooth: fix handling of optional data/uuid args
- modbluetooth: make BLE.irq() method positional only
- modbluetooth: print UUIDs correctly
- modbluetooth: implement MTU
- vfs: fix lookup of entry in root dir so it fails correctly
- modbluetooth: change module-owned bytes objects to memoryview
- modure: allow \\ in re.sub replacements
- machine_i2c: rename type to SoftI2C and add custom print method
- machine_i2c: remove "id" arg in SoftI2C constructor
- machine_spi: remove "id" arg in SoftSPI constructor
- utime_mphal: add generic utime.time_ns() function
- add explicit initializers for default values
- disable -Wmissing-field-initializers for lfs2
- introduce and use MP_FALLTHROUGH macro
- vfs_lfs: support mounting LFS filesystems in read-only mode
- modurandom: support urandom.seed() without an argument
- btstack/btstack.mk: add -Wimplicit-fallthrough=0
- nimble/nimble.mk: add -Wno-old-style-declaration
- moductypes: fix storing to (U)INT64 arrays on 32-bit archs
- machine_mem: only allow integers in machine.memX subscript
- modbluetooth: make UUID type accessible outside modbluetooth.c
- modbluetooth: combine gattc-data-available callbacks into one
- nimble: poll startup directly rather than using NimBLE sem
- modbluetooth: re-instate optional no-ringbuf modbluetooth
- nimble: make stm32 and unix NimBLE ports use synchronous events
- machine_i2c: add init protocol method for generic I2C bindings
- modbluetooth: add API for L2CAP channels
- nimble/modbluetooth_nimble: fix build when l2cap unavailable
- uasyncio: delay calling Loop.call_exception_handler by 1 loop
- uasyncio: add Task.done() method
- uasyncio: fix cancellation handling of wait_for
- nimble: fail read if the characteristic is too big
- modbluetooth: add _IRQ_CONNECTION_UPDATE event
- modbluetooth: allow setting char/desc enc/auth options
- modbluetooth: allow user-specified reason in read request IRQ
- modbluetooth: add compile-config flag to enable pairing/bonding
- modbluetooth: add _IRQ_ENCRYPTION_UPDATE event
- modbluetooth: allow configuration of pairing/bonding parameters
- modbluetooth: add gap_pair(conn_handle) func to intiate pairing
- modbluetooth: add support for bonding (key persistence)
- modbluetooth: simplify synchronous invoke_irq_handler signature
- modbluetooth: add support for passkey authentication
- nimble: generate and persist a unique IRK
- modubinascii: update code, docs for hexlify now CPython has sep
- vfs: raise OSError(ENODEV) if mounting bdev without a filesystem
- nimble: reset NimBLE BSS in mp_bluetooth_init
- nimble: don't assert on save-IRK failure
- modbluetooth: add ble.hci_cmd(ogf, ocf, req, resp) function
- vfs: check block 0 and 1 when auto-detecting littlefs
- modframebuf: change int to unsigned int in format methods args
- modonewire: use pin_od_high/pin_od_low instead of pin_write
- nimble: improve the flow control for l2cap recv path
lib:
- libm: reduce size of static two_over_pi array
- utils/mpirq: add mp_irq_init func, and clean up unused init method
- utils/pyexec: add MICROPY_BOARD hooks before/after executing code
- utils/pyexec: add stdin-reader on raw REPL with flow control
- littlefs: update littlefs2 to v2.3.0
- littlefs: guard lfs2_mlist_isopen with LFS2_NO_ASSERT
- stm32lib: update library for WB v1.10.0
- libhydrogen: add new libhydrogen submodule
- timeutils: provide simple impl of extra funcs when Epoch is 1970
- pico-sdk: add new pico-sdk submodule, for the rp2 port
- utils/pyexec: remove obsolete LCD initialization
- pico-sdk: update to latest version v1.0.1
drivers:
- memory/spiflash: add MICROPY_HW_SPIFLASH_ENABLE_CACHE option
mpy-cross:
- enable more warnings
Support components
==================
docs:
- library/ubluetooth.rst: document BLE address modes
- library/ubluetooth.rst: clarify position/kw arguments
- library/ubluetooth.rst: clarify peripheral/central vs server/client
- library/ubluetooth.rst: add docs for MTU API
- develop: add notes on prerequisite tools for building native .mpy
- reference/packages.rst: fix typo, remove duplicate "port"
- update I2C and SPI docs to add reference to SoftI2C and SoftSPI
- library/btree.rst: correct method typo: __detitem__ to __delitem__
- library/machine.rst: correct minor typo: timout to timeout
- library/machine.Signal.rst: correct typo: usecases to use cases
- library/machine.Timer.rst: add mention of constructor arguments
- reference/glossary.rst: fix minor grammar error, An -> A
- fix reference to QSTR_GEN_CFLAGS Makefile flag
- develop/cmodules.rst: add link to source code for user C example
- library/ubluetooth.rst: add docs for L2CAP channels
- reference/repl.rst: add information about new raw-paste mode
- library/ubluetooth.rst: add _IRQ_CONNECTION_UDPATE docs
- library/ubluetooth.rst: update char/desc flags
- library/ubluetooth.rst: update read request IRQ docs
- library/ubluetooth.rst: add pairing/bonding config docs
- library/ubluetooth.rst: add gap_pair() docs
- library/ubluetooth.rst: add bonding docs
- library/ubluetooth.rst: add passkey docs
- develop: add MicroPython Internals chapter
- fix minor typos in RTC docs, and->an
- esp8266/quickref: add warning block about NeoPixel timing
- develop/natmod: fix a small typo, con->can
- update copyright year range to include 2021
examples:
- bluetooth: update to use positional-only args to irq()
- add example code for user C modules, both C and C++
- bluetooth: add bonding/passkey demo
- rp2: add pio_uart_rx.py example
tests:
- multi_bluetooth: update to new config('mac') behaviour
- multi_bluetooth: make ble_gap_connect robust against event timing
- multi_bluetooth: update UUID format in .exp files
- run-multitests.py: show test/truth diff
- multi_bluetooth/ble_mtu.py: add multitest for BLE MTU
- basics: enable == and != special-method tests now that they work
- basics: add test for MICROPY_PY_ALL_INPLACE_SPECIAL_METHODS ops
- run-tests: use -BS flags when running CPython
- thread/stress_schedule.py: assign globals before running test
- micropython/extreme_exc.py: unlink alloc'd lists earlier in chain
- run-multitests.py: fix diff order, show changes relative to truth
- multi_bluetooth: improve reliability of event waiting
- run-multitests.py: add a -p flag to run permutations of instances
- multi_bluetooth: change dict index-and-del to pop, to clear event
- multi_bluetooth: add a test for WB55 concurrent flash access
- multi_bluetooth: add L2CAP channels multi-test
- extmod: add vfs_posix.py test for uos.VfsPosix class
- run-tests: update skipped tests on CI for GitHub Actions
- multi_bluetooth: add multitests for BLE pairing and bonding
- misc/sys_settrace_features.py: ignore CPython zipimport traces
- extmod: add test to try and mount a block device directly
- misc/sys_settrace: make test output independent of invoked path
- misc/sys_settrace_features.py: fix running with non-dflt encoding
- extmod: add test for the precision of utime functions
- move native for test from pybnative to micropython
- extmod/utime_time_ns.py: relax bounds on time_ns measurement
- run-tests: change default Python command used on Windows
tools:
- mpy-tool.py: fix merge of multiple mpy files to POP_TOP correctly
- write msvc-compatible frozen content
- upip.py: support explicit port number in host
- makeqstrdefs.py: run qstr preprocessing in parallel
- ci.sh: add helper script to run CI tasks
- pyboard.py: add fast raw-paste mode
- ci.sh: put echo of CI path in a separate function
- ci.sh: use pip-install to get latest version of esptool.py
- ci.sh: pip install pyhy for stm32 builds
- ci.sh: for code size build, fetch history of master branch only
- ci.sh: for ci_code_size_setup, update apt to install gcc-multilib
- mpy-tool.py: list frozen modules in MICROPY_FROZEN_LIST_ITEM
- verifygitlog.py: add script for verifying commit message format
- makemanifest.py: add check that freeze path is a directory
- remove obsolete upip bootstrap script
CI:
- install setuptools for black code formatting
- update zephyr build to v2.4.0
- stop using Travis for CI
- workflows: add workflows for all CI tasks, builds and tests
- workflows: run unix and qemu-arm workflows when tests change
- workflows: fix code-size CI workflow
- workflows: add CI workflow for rp2 port
- workflows: add workflow to verify commit message format
- workflows/ports_unix.yml: add job for a reproducible build
The ports
=========
all ports:
- add utime.gmtime() function
- add SoftI2C and SoftSPI to machine module where appropriate
- support legacy soft I2C/SPI construction via id=-1 arg
- use correct in/out endpoint size in TUD_CDC_DESCRIPTOR
cc3200 port:
- ftp: add quotes to PWD response and allow FEAT prior to login
- fix debug build
esp8266 port:
- remove release-specific manifest, disable osdebug by default
- mpconfigport.h: seed the urandom module on import
- remove "FAT" from warning message in inisetup.py
- modules/neopixel.py: add timing param to NeoPixel constructor
esp32 port:
- pin MicroPython and NimBLE tasks to core 0
- mpconfigport.h: remove duplicate uhashlib registration
- boards/sdkconfig.base: set default IDF log level to ERROR
- modnetwork: re-enable PPP support for IDF-SDK >=v4
- use path relative to root for netutils/timeutils headers
- mpconfigport.h: enable MICROPY_PY_DELATTR_SETATTR
- mpconfigport.h: seed the urandom module on import
- support building C++ code
- machine_pin: reset pin if init sets mode
- remove "FAT" from warning message in inisetup.py
- modnetwork: synchronize WiFi AUTH_xxx constants with IDF values
javascript port: no changes specific to this component/port
mimxrt port:
- boards: add MIMXRT1064_EVK board
minimal port: no changes specific to this component/port
nrf port:
- main: make mp_builtin_open signature match that in py/builtin.h
- remove unnecessary includes of mpconfigport.h and its header guard
- README: describe Pin numbering scheme for nRF52840
- boards: update memory.ld to include bootloader offsets
- Makefile: add bootloader specific section
- Makefile: add support for flashing with nrfutil
- boards: add linker script for nrf52840 Open Bootloader 1.2.0
- change selected boards to utilize pre-flashed bootloader
- README: add use of "make submodules" in alternative build paragraph
pic16bit port: no changes specific to this component/port
powerpc port: no changes specific to this component/port
qemu-arm port: no changes specific to this component/port
rp2 port:
- add new port to Raspberry Pi RP2 microcontroller
- machine_pin: change N_GPIOS to NUM_BANK0_GPIOS for pico-sdk compat
- memmap_mp.ld: update for latest SDK
- rp2_pio: add JMP PIN support for PIO
- modmachine: implement additional functions incl unique_id and idle
samd port: no changes specific to this component/port
stm32 port:
- mpconfigport.h: enable MICROPY_PY_REVERSE_SPECIAL_METHODS
- uart: allow static IRQ handler registration
- modbluetooth_hci: use a static mp_irq_obj_t for BT HCI UART IRQ
- mpbthciport: increase char timeout of BT HCI UART
- make-stmconst.py: add support for WB55 header files
- rfcore: update rfcore.c to match how ST examples work
- boards/USBDONGLE_WB55: add USE_MBOOT support
- powerctrlboot: acquire HSEM5 on STM32WB during SystemClock_Config
- rfcore: fix length matching in HCI parser
- rfcore: refactor some helper funcs, and remove some magic numbers
- rfcore: enable RX IRQ on BLE IPCC channel for better performance
- boards/NUCLEO_WB55: add Python helper code for rfcore
- usb: don't nul pyb_hid_report_desc if MICROPY_HW_USB_HID disabled
- rtc.h: include py/obj.h to make header self contained
- servo: fix angle and speed methods to work again with -ve args
- led: support PWM output without TIM3
- rfcore: fix FUS layout and size of ipcc_device_info_table_t
- Makefile: allow boards to extend SRC_C, SRC_O and OBJ variables
- sdio: don't change any DMA2 settings on H7 MCUs
- sdcard: fix H7 build when using SDMMC2
- rfcore: add Python API for basic rfcore operations
- boards/NUCLEO_WB55: add standalone WB55 FUS/WS firmware updater
- rfcore: update to support WS=1.9.0.0.4
- boards/NUCLEO_WB55/rfcore_firmware.py: fix flash unlock
- boards/NUCLEO_WB55/rfcore_firmware.py: fix bad variable name
- boards/NUCLEO_WB55/rfcore_firmware.py: increase GET_STATE timeout
- fix broken build when FAT FS multi-partition is disabled
- usbd_cdc_interface: check and handle CDC TX wrap-overflow
- machine_adc: fix ADC auto-calibration to run when ADC not enabled
- mpconfigport.h: seed the urandom module on import
- rng: use SysTick+RTC+unique-id to seed pRNG for MCUs without RNG
- boards: factor out common data/bss/heap/stack linker sections
- support C++ code and user C modules written in C++
- main: move update_reset_mode to outside the soft-reset loop
- add MICROPY_BOARD calls in various places in stm32_main
- boardctrl: define MICROPY_BOARD_EARLY_INIT alongside others
- Makefile: make the generation of `firmware.bin` explicit
- Makefile: allow boards to extend all SRC variables
- rtc: validate the RTC prescaler on boot and change if incorrect
- rfcore: make RX IRQ schedule the NimBLE handler
- flash: implement WB55 flash locking
- rfcore: don't send HCI ACL cmds while another is pending
- rfcore: depend on NimBLE only when BLE enabled
- modmachine: add device and revision ids to machine.info()
- Makefile: disable text compression in debug builds
- powerctrl: define RCC_SR_SFTRSTF flag for H747
- powerctrl: fix STOP mode voltage scaling on H7 REV V devices
- powerctrl: on H7, re-enable disabled OSCs/PLLs on exit from STOP
- powerctrl: disable RTC write protection before changing flags
- powerctrl: set H7 RTC wakeup flags
- powerctrl: ensure SysTick is disabled on STOP mode entry for H7
- i2c: factor I2C finding code to i2c_find_peripheral function
- add support for a board to reserve certain peripherals
- Makefile: change -O0 to -Og for DEBUG=1 builds
- boards/stm32h743.ld: enable D2 RAM and add eth-buffer section
- eth: add support for H7 processors
- boards/NUCLEO_H743ZI: enable ethernet peripheral
- mboot: enable LFS2_READONLY for mboot builds with littlefs
- usb: allocate 128 bytes to CDC data out EPs on non-multi-OTG MCUs
- pyb_can: add ability to calculate CAN bit timing from baudrate
- system_stm32: enable DBGMCU in low-power modes for debug builds
- sdram: add SDRAM enter/leave self-refresh mode functions
- adc: deselect VBAT after reading to prevent battery drain
- main: do extended readblocks call when auto-detecting littlefs
- boards/PYBD_SF2: disable SPIFLASH_ENABLE_CACHE for mboot builds
- mboot/gzstream: fix lost data decompressing final part of file
- mboot: add support for signed and encrypted firmware updates
- boards/NUCLEO_WB55: enable MBOOT with packing mode
- Makefile: use MBOOT_PACK_KEYS_FILE as depedency of .pack.dfu
- main: check block 0 and 1 when auto-detecting littlefs
- mboot: don't auto-detect littlefs block size
- mboot: add ELEM_TYPE_STATUS element so application can get status
- mboot: introduce MBOOT_ERRNO_xxx constants and use them
- mboot/fwupdate.py: refactor update_mpy with support for STATUS
- main: introduce MICROPY_HW_FLASH_MOUNT_AT_BOOT config option
- rfcore: fix flow control for IPCC RX IRQ
- fix minor typos in RTC docs, and->an
- usbd_cdc_interface: don't wait in usbd_cdc_tx_always if suspended
- mboot: change debug compiler optimisation from -O0 to -Og
teensy port:
- fix build errors and warnings and enable -Werror
unix port:
- Makefile: always enable -f*-sections regardless of DEBUG setting
- implement BLE H4 HCI UART for btstack/nimble
- implement mp_hal_time_ns using gettimeofday
- variants: enable MICROPY_DEBUG_PARSE_RULE_NAME on coverage build
- mpconfigport.h: enable MICROPY_PY_DELATTR_SETATTR
- enable more warnings
- support building C++ code
- Makefile: move coverage.c and coveragecpp.cpp to coverage variant
- handle pending events/scheduler in MICROPY_EVENT_POLL_HOOK
- make mp_hal_delay_ms run MICROPY_EVENT_POLL_HOOK
- modtime: fix time() precision on unix ports with non-double floats
windows port:
- msvc: support freezing modules
- Makefile: support freezing modules
- show test failures in the Appveyor builds
- update build instructions in README
- implement mp_hal_time_ns using gettimeofday
- msvc: use same default python command as core
zephyr port:
- replace zephyr integer types with C99 types
- const-ify struct device instance pointers
- update build instructions to v2.4.0
- replace broken shell_net_iface() with more general shell_exec()
- guard I2C code with appropriate ifdef config
- add basic UART functionality to machine module
- machine_uart: fix arg of machine_uart_ioctl to make it uintptr_t