SolvedOctoPrint [Brainstorming] Making use of ADVANCED_OK in MarlinFirmware to speed up serial

I've been experiencing slowdowns in printing small complex models due to octoprint not streaming commands to the printer fast enough to stop the buffer for emptying. see: #2799

I've been following #2346 in hopes that it will provide a solution, but it's obvious that the upstream issues with software flow control in the linux usb-serial stack are not something that are likely to be fixed soon, and it's not something i feel competent to take a look at myself.

I have however come across a feature in MarlinFirmware that looks like it could provide a solution: ADVANCED_OK
This feature changes the format of 'ok' messages to include two new fields:

• P: the remaining space in the move planner buffer
• B: the remaining space in the unprocessed command buffer

With this information I believe we should be able to send multiple commands to the printer between ok responses as long as we don't send more than the available unprocessed buffer.

I've had a play and done a hacky implementation in comm.py and the initial results seem great, I've been testing purposfully over detailed gcode that always stalled with the normal communications code and I've been unable to drain the planned buffer regardless of complexity or print speed. It also increased sd transfer speeds although not enough to make it quicker than just unplugging and copying.

So far my implementation is hack and unsuitable for a pull request, and i've only tested with a pi3 on my printer at 250000 baud rate and a fairly large buffer, however if this is something you'd consider adding I'd be happy to spend the time to turn it into an actual feature and PR it.

Things to think about:

• Feature detection: Marlin does't currently report ADVANCED_OK in it's M115 response, however the presence of the P and B parameters in the ok responses is something we could check for along with a setting in the connection settings to manually turn on and off (we can't assume the presence of the P and B parameters anyway as not every ok message includes them even with the feature turned on). Alternatively, maybe MarlinFirmware would accept a PR to add it.
• Resends seem to work from my limited testing, however with a large buffer on the printer it's possible to have sent more commands than the history contains which can kill a print if we recieve a resend request for one of those commands.
• Implementation: The code I've been plaing with so far just skips calling _clear_to_send.clear() if we've sent less than B (space in the unprocessed buffer) commands since the last ok. it's current form is hacky and maybe there is a better way to do it using the functinality of CountedEvent (_clear_to_send), I'm really open to suggestions, python isn't my native programming language so I'm a bit unfamiliar with the tools at my disposal.

Sorry for the wall of text, I'd be glad to hear if this is something useful.

i believe now my patch is production ready, it automatically recognize is marlin with advanced_ok and if so send commands asynchronously,while marlin buffers is free, but not all, commands M109, M190, G4, G28, G29, G30, G32, M400, M226, M600 always running synchronously.

if marlin without advanced_ok then octoprint working as before.

user may configure this exception list under settings->Serial connection->Firmware and protocol->advanced options->Synchronously running commands

how to install manually
following files must be replaced with files from my repository

src/octoprint/server/api/settings.py
src/octoprint/settings.py
src/octoprint/static/js/app/viewmodels/settings.js
src/octoprint/templates/dialogs/settings/serialconnection.jinja2
src/octoprint/util/comm.py

if octoprint was installed in env then local files will be placed under env directory, something like
/home/octoprint/OctoPrint/venv/lib/python2.7/site-packages/octoprint

for best performance following marlin options is mandatory:

#define BAUDRATE 500000
#define BLOCK_BUFFER_SIZE 64
#define BUFSIZE 32
#define TX_BUFFER_SIZE 32
#define RX_BUFFER_SIZE 2048
#define ADVANCED_OK

Additional recommendations:

1. be sure Serial logging in octoprint is disabled, (settings -> Serial connection-> general->Serial logging)
2. disconnect from usb host controller, to which printer connected , all other usb devices

@foosel done