SolvedOctoPrint [Brainstorming] Making use of ADVANCED_OK in MarlinFirmware to speed up serial

I've been experiencing slowdowns in printing small complex models due to octoprint not streaming commands to the printer fast enough to stop the buffer for emptying. see: #2799

I've been following #2346 in hopes that it will provide a solution, but it's obvious that the upstream issues with software flow control in the linux usb-serial stack are not something that are likely to be fixed soon, and it's not something i feel competent to take a look at myself.

I have however come across a feature in MarlinFirmware that looks like it could provide a solution: ADVANCED_OK
This feature changes the format of 'ok' messages to include two new fields:

  • P: the remaining space in the move planner buffer
  • B: the remaining space in the unprocessed command buffer

With this information I believe we should be able to send multiple commands to the printer between ok responses as long as we don't send more than the available unprocessed buffer.

I've had a play and done a hacky implementation in comm.py and the initial results seem great, I've been testing purposfully over detailed gcode that always stalled with the normal communications code and I've been unable to drain the planned buffer regardless of complexity or print speed. It also increased sd transfer speeds although not enough to make it quicker than just unplugging and copying.

So far my implementation is hack and unsuitable for a pull request, and i've only tested with a pi3 on my printer at 250000 baud rate and a fairly large buffer, however if this is something you'd consider adding I'd be happy to spend the time to turn it into an actual feature and PR it.

Things to think about:

  • Feature detection: Marlin does't currently report ADVANCED_OK in it's M115 response, however the presence of the P and B parameters in the ok responses is something we could check for along with a setting in the connection settings to manually turn on and off (we can't assume the presence of the P and B parameters anyway as not every ok message includes them even with the feature turned on). Alternatively, maybe MarlinFirmware would accept a PR to add it.
  • Resends seem to work from my limited testing, however with a large buffer on the printer it's possible to have sent more commands than the history contains which can kill a print if we recieve a resend request for one of those commands.
  • Implementation: The code I've been plaing with so far just skips calling _clear_to_send.clear() if we've sent less than B (space in the unprocessed buffer) commands since the last ok. it's current form is hacky and maybe there is a better way to do it using the functinality of CountedEvent (_clear_to_send), I'm really open to suggestions, python isn't my native programming language so I'm a bit unfamiliar with the tools at my disposal.

Sorry for the wall of text, I'd be glad to hear if this is something useful.

17 Answers

✔️Accepted Answer

i believe now my patch is production ready, it automatically recognize is marlin with advanced_ok and if so send commands asynchronously,while marlin buffers is free, but not all, commands M109, M190, G4, G28, G29, G30, G32, M400, M226, M600 always running synchronously.

if marlin without advanced_ok then octoprint working as before.

user may configure this exception list under settings->Serial connection->Firmware and protocol->advanced options->Synchronously running commands

how to install manually
following files must be replaced with files from my repository

src/octoprint/server/api/settings.py
src/octoprint/settings.py
src/octoprint/static/js/app/viewmodels/settings.js
src/octoprint/templates/dialogs/settings/serialconnection.jinja2
src/octoprint/util/comm.py

if octoprint was installed in env then local files will be placed under env directory, something like
/home/octoprint/OctoPrint/venv/lib/python2.7/site-packages/octoprint

for best performance following marlin options is mandatory:

#define BAUDRATE 500000
#define BLOCK_BUFFER_SIZE 64
#define BUFSIZE 32
#define TX_BUFFER_SIZE 32
#define RX_BUFFER_SIZE 2048
#define ADVANCED_OK

Additional recommendations:

  1. be sure Serial logging in octoprint is disabled, (settings -> Serial connection-> general->Serial logging)
  2. disconnect from usb host controller, to which printer connected , all other usb devices

Other Answers:

Related Issues:

4
OctoPrint Connecting to Printer
So.. here's something to test with It's a minimal plugin that changes the serial opening to what @yg...
4
OctoPrint [Brainstorming] Making use of ADVANCED_OK in MarlinFirmware to speed up serial
i believe now my patch is production ready I've been experiencing slowdowns in printing small comple...
3
OctoPrint Failure To Resume From Pause
serial.log log created with standard configuration (I did not change Log position on pause option) I...
3831
axios Axios catch error returns javascript error not server response
I have exactly the same environment Try this: Modify from console.log(error) to console.log(error.re...
731
scrapy ' error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 '
@euler16 for scrapy with Python 3 you'll need with Python 2 you'll need I wanted to install scrapy i...
684
laradock Mysql. The server requested authentication method unknown to the client [caching_sha2_password]
alter user 'username'@'localhost' identified with mysql_native_password by 'password'; would fix it....
517
react navigation screenIsActive prop / componentDidFocus event for TabNavigator items
It probably makes sense to add lifecycle hooks to screens In one of my Tabs i need to load Data from...
474
meteor [1.4.2.1] Error: ENFILE: file table overflow
I was getting the same after an upgrade to macOS Sierra Turns out macOS have a harsh limit on number...
423
ipython Last jedi release (0.18.0) is incompatible with ipython (7.19 and 7.18 tested); reason - column arg was deprecated, and now removed
As a temporary fix for anyone just trying to get things working again: It would be really nice if yo...
397
material ui Module not found: Can't resolve 'material-ui-icons/Menu' Martial Next
For anyone else experiencing this issue: npm install @material-ui/icons https://www.npmjs.com/packag...
395
laravel dompdf (1/1) ErrorException Non-static method Barryvdh\DomPDF\PDF::loadView() should not be called statically
This happens because you are namespacing the wrong PDF class You are namespacing Barryvdh\DomPDF\PDF...
378
webpacker localIdentName option moved in css-loader configuration
I faced same issue after upading css-loader but I solved it If you check css-loader readme ...
364
react navigation Send data back from child screen?
@itswaze You can do something along these lines to pass back from the child screen ...
358
react navigation How to goBack from nested StackNavigator?
@dhruvparmar372 According to the NOTE in the doc a navigator's navigation prop may not have the help...
352
axios POST request works in Browser but not on Node
This might be considered a duplicate of #789 I was able to use the form-data package with Axios in n...
317
react navigation Best pattern for a 'Save' button in the header
Try setting your component instance's handleSave function as a navigation state parameter after the ...
310
DefinitelyTyped [@types/react] RefObject.current should no longer be readonly
It's not It'a intentionally left readonly to ensure correct usage even if it's not frozen ...
306
react native navigation [V3][Android] FATAL EXCEPTION: create_react_context
OK after a good night of sleep I've found why I was having this issue In the MainApplication.java I ...
303
ts node Custom typings not working with ts-node 8.0.2
When using with ts-node you have to add --files flag After updating to ts-node version 8.0.2 the cus...
294
ohmyzsh compinit:503: no such file or directory: /usr/local/share/zsh/site-functions/_brew
Per #9602 (comment) brew cleanup fixed it for me I am using Apple M1 When I added this line: export ...
292
laradock SQLSTATE[HY000] [2054] The server requested authentication method unknown to the client
+1 I'm having the same problem here. Info: Docker version ($ docker --version): Docker version 17.12...
289
vagrant vagrant box update - Fails with 404 Not Found error
A workarround to add in your Vagrantfile: Vagrant version Host operating system Ubuntu 16.04.3 LTS G...
263
vagrant vagrant --help displays a rubygems error
To fix this error: Vagrant version Host operating system Expected behavior vagrant and vagrant --hel...
262
date fns Can't resolve 'date-fns/_lib/format/longFormatters'
You probably forgot to install date-fns or Code: import DateFnsUtils from '@date-io/date-fns'; ...
259
virtualenv Error creating virtualenv with python3.6
The original poster's problem is due to not having the 'python3.6-venv' package installed ...
252
provider A Product was used after being disposed. flutter: Once you have called dispose() on a Product, it can no longer be used.
Oh I see what you're doing Don't: DO: i have a ChangeNotifireProvider that such that i do pushReplac...
248
react navigation Reset to nested route, "There is no route defined for..."
A quick workaround (not heavily tested): set the key property to null on your action. ...
238
axios Adding headers to axios.post method
Edit: I had to add Authorization to allowed headers in my CORS filter @jffernandez I'm having the sa...
231
axios Adding Retry Parameter
@mericsson I am too in need of exponential backoff when retrying I've put together the following whi...
224
homebrew openjdk Cask adoptopenjdk8 exists in multiple taps
I think AdoptOpenJDK8 should be removed from the homebrew-cask-versions repo Somone(TM) should proba...
222
nativescript cli TNS doctor doesn't recognize Xcode (High Sierra)
HI @philipfeldmann Can yhou please run xcodebuild -version and paste the output? May be Xcode is ins...
215
axios BaseURL not being used
Please IGNORE THIS ISSUE Found the problem: I was setting baseUrl but it should be baseURL. ...
205
axios How to ignore SSL issues
You can configure axios to use a custom agent and set rejectUnauthorized to false for that agent: Ho...
202
ipython NameError: name 'sys' is not defined
Ok that was easy The Jedi release on 12/25 version 0.18.0 is breaking tab completion pip install --u...
190
axios How to send Raw http for Instagram Authentication?
Yes StringClient uses the application/x-www-form-urlencoded format by default while axios uses appli...
186
vuetify [Bug Report] Could not find a declaration file for module 'vuetify/lib'
@KaelWD Thank you for the link More specifically this comment helped me out to properly update my ts...
182
axios CORS error No 'Access-Control-Allow-Origin' header is present on the requested resource
My solution is axios.get('https://cors-anywhere.herokuapp.com/' + yourUrl) update: doesn't work ...
181
runtime An assembly specified in the application dependencies manifest (RhWeb.deps.json) was not found
I also hit this For me it had to do with the use of in my .csproj when running in an environment tha...
179
react navigation Drawer changes in #1803 causes undocumented breaking changes to drawernavigators
Adding to the drawer configs fixed the issue for me. Current Behavior create a single drawer navigat...
178
laradock MySQL Container fails to start
I had the same issue last night I think it's the mysql version problem What I did was edited laradoc...
177
webpacker Compilation Failed without error message
In case anyone else runs into this I had to set webpack_compile_output: true in webpacker.yml for ra...
175
matplotlib 'NSWindow drag regions should only be invalidated on the Main Thread!' - macos/python
If you are running a webserver and using it to save Matplotlib make sure to set the backend to a non...
174
axios request progress?
@slim12kg Depending on the way you coded your project it may change if my code confuses you ...
174
ruby build Cannot install Ruby versions < 2.4 because of openssl@1.1 dependency
@jyr There are multiple things wrong with what you are doing First of all you are using rvm ...
169
NativeBase Needs to tap twice to fire onPress function when keyboard is open
@bm-software Use keyboardShouldPersistTaps={'handled'} and will works fine. react-native ...
167
react navigation Android header is overlapped
Yes @rockingskier I had a chat with the Expo guys Looks like we need to explicitly add padding for A...
165
DefinitelyTyped node_modules/@types/react-native/globals.d.ts (36,15): Duplicate identifier 'FormData'.
Fixed by set compilerOptions.types manually If you know how to fix the issue make a pull request ins...
163
ts node ts-node fails when ES Modules are in the dependency graph in Node.js 13+
It seems that ts-node fails to run with module: esnext in the tsconfig.json and type: module in the ...
161
typeorm Error: RepositoryNotFoundError: No repository for [Enitity] was found. Looks like this entity is not registered in current "default" connection?
https://typeorm.io/#/connection-options/common-connection-options Issue type: [x] question [x] bug r...