Solvedgutenberg Block API: Server-side awareness of block types

Previously: #2529, #104
Related: #886
Related Slack conversation: https://wordpress.slack.com/archives/C5UNMSU4R/p1505677036000104 (cc @jasonbahl)

When initially developing the API for implementing a block (#104), there were some competing objectives:

  • The editor representation of a block must occur in the client to preserve an ideal user experience
  • A block should ideally be easy to implement for quick prototyping (#27, e.g. defining block in a browser console)
  • General information about a block should be context unaware (title, attributes definition, category, etc need not specifically be defined in a client, server, or an other external static file)

Where we have fallen short is in:

  • The third of these objectives; As a need has arisen for server-side attribute validation (#2529), we have further fragmented the definition of a block (its attributes may be defined on the server or in the client).
  • It is not possible for the server to have reliable awareness of all blocks which would be registered in the editor.

Proposal:

To improve consistency, we should move block attributes (and other general information) to a server block registration. This would likely behave exactly as it does in #2529, including client-side attributes bootstrapping and validation, but would be applied consistently across all blocks. Therefore, every block would have a companion PHP file associated with it, which in addition to current supports defines:

  • Title
  • Icon
  • Category
  • Attributes

The trade-off here is that it is not as simple to implement a block in a single location. We could potentially leave client-defined attributes support as-is, but this could also entice developers to avoid the server registration and reintroduce all the inconsistencies/drawbacks therein.

Some previous discussion had considered the idea of a third JSON "manifest" file, similar to a package.json or composer.json, describing the static general information about a block. While this would be a simple format for what is essentially overview data, is problematic because (a) it is yet another file that a developer could need to become familiar with, (b) it is difficult to bring into the client without a build step understanding JSON imports, and (c) localization of strings would need to implement some amount of "magic" (hard-coding to properties expected to be localized).

It's worth pointing out that a plugin author already needs some amount of server-specific logic for each of their blocks, specifically that of enqueueing the JavaScript files responsible for registering the blocks in the client. By requiring block registration in the server, there may be some added benefit in handling scripts and styles.

  • A block could define its companion scripts and styles as properties of the block type (e.g. script_handle, style_handle).
  • Or: These files could be "discovered" adjacent the block registering file by naming conventions (e.g. find style.css, edit.css, block.js adjacent myblock/index.php). This could encourage developers to define their blocks in a standalone folder/file structure for easier separation overview.

A remaining challenge in moving all block attributes definitions to the server is support for attribute sources. With advent of additional source types (meta, options), we will likely need to expand this concept to cover more than just DOM-based attribute sourcing. Therefore, I would propose creating an alternative structure for representing what currently exists as the assignment of source functions:

Before:

{
	url: {
		source: attr( 'img', 'src' )
	}
}

After:

[
	'url' => [
		'source' => [
			'type'     => 'attribute',
			'selector' => 'img',
			'name'     => 'src',
		],
	],
]

The intent here is not necessarily that the server would become responsible for parsing attributes from the markup of a block (although there is nothing that precludes it from doing so), but rather to be able to represent the structure in a static format, which could then be bootstrapped into the client to recreate the current behavior.

As an example for how this extends into the concept of additional sources:

'author' => [
	'type' => 'string',
	'source' => [
		'type' => 'meta',
		'name' => 'author', // Inferred from attribute name if not specifically defined?
	],
],
'siteTitle' => [
	'type' => 'string',
	'source' => [
		'type' => 'option',
		'name' => 'name',
	],
]

Open questions:

  • While there is justification for defining block attributions in the server for the purpose of improving consistency, there is still a lack of general understanding of specific use cases for server-specific awareness of blocks, aside from a "nice to have" reasonings. Lack of these use cases is not intended as an argument for a server definition not to exist, but it would help guide specific implementation decisions.

cc @westonruter

19 Answers

✔️Accepted Answer

@aduth thanks for creating this ticket and CCing me!

there is still a lack of general understanding of specific use cases for server-specific awareness of blocks, aside from a "nice to have" reasonings. Lack of these use cases is not intended as an argument for a server definition not to exist, but it would help guide specific implementation decisions.

I'll share some use cases (some might still be considered nice to have but in my opinion should not be dismissed).

General WordPress Consistency

WordPress thrives on the hook/filter system, and defining many pieces of the block Schema on just the client limits the power of the system that essentially makes WordPress what it is.

I know there's been work on a JS implementation of similar hook/filter functionality, but that's only useful in the context of WordPress itself, and WordPress is used to power much more than just itself (I'll talk more about decoupled apps below).

Take a step back and imagine if the entire admin was rewritten in JS (think Calypso). And we statically defined Post Type and Taxonomy Registration only in JS and not on the server.

Now think about how much of the WP Ecosystem at large would not exist at all because of the lack of a server-side registration. . .a LOT. There would be no way to generate per-post-type feeds or REST endpoints. Calypso wouldn't be able to work with external WordPress sites, because it wouldn't know what post_types and taxonomies exist in the sites, because that info wouldn't be available via REST, because the REST server wouldn't have any knowledge of it. . .

Without comprehensive server-side registration, we drastically reduce the power of WordPress as we know it to interact with the blocks.

Same goes with most extendable parts of WordPress. . .image sizes, post_stati, post_types, taxonomies, admin_menus, etc. . .so many things are statically defined on the server, so not having a solid registration for Blocks server-side seems like it goes agains WP's history. I know there is a minimal schema, but think if register_post_type was just register_post_type('post') and didn't accept any additional args for labels, capabilities, etc. . .

Also consider existing conversations about how troublesome Meta Boxes in Gutenberg are going to be, mostly due to the lack of a true Fields API (server side schema for fields).

Ideally, one should be able to register a block on the server and that becomes the source of truth for the block's capabilities. The REST API could make use of it, other plugins could make use of it, other APIs (XML-RPC, WPGraphQL, etc) could make use of it, and of course the Gutenberg JS itself could make use of it.

Alternative UIs
Let's be honest. Gutenberg is not going to solve everyones's problems. No matter how awesome it is or will become, it simply will not meet everyone's needs. There will still be a market for creating content in different ways. Page builders currently completely dismiss the Post Edit screen, and these alternative post creation solutions will continue to exist post-Gutenberg.

With a solid Server Side schema, hopefully the Page Builders could at least start to adhere to a standard way of interacting with "Blocks". . .that way, if Page Builder A thinks using Modals to edit content makes more sense than the fixed-right sidebar of Gutenberg, they could build their own UI to interact with Gutenblocks, but still ensure the blocks are saved in the same way.

I think having a solid server-side schema for blocks will help standardize the Page Builder market. Page builders could essentially become "Gutenberg Themes" where the editing experience looks and feels different, but at the end of the day the data produced is the same. This probably isn't impossible without a comprehensive server side schema, but it would be much easier.

If there was a comprehensive Block schema on the server, page builders could use that to hydrate their view layer and ensure compatibility with Gutenberg (avoid the lock-in, at least to a degree, that everyone talks about with page builders).

Headless CMS / Decoupled Apps

Headless CMS with WordPress is nothing new. . .one of my first projects with WordPress was populating a Flash site with data from WordPress via XML-RPC back in ~2009. . .but now with the REST API in core (and exciting plugins like WPGraphQL), it's becoming more and more common to see folks interacting with WordPress in non-traditional Theme / WP-Admin contexts.

For any of these decoupled applications to interact with Gutenblocks, they have to have knowledge of the blocks in the same way Gutenberg itself does. . .not just the blocks that have already been created by Gutenberg inside the WP-Admin, but also the capabilities of what can be done to create/modify blocks. That way these decoupled applications can provide an experience comparable in some fashion to Gutenberg. If you edit a post on a Native Mobile App, Calypso or some other decoupled app, the data should be able to be saved back in the same format that Gutenberg saves it. But in order for said decoupled app to provide an experience that allows for data to be saved in the same format as Gutenberg, said decoupled app needs to know what the capabilities of each block are.

If everything is defined in client-side code that lives in the WP-Admin, we're limiting interaction with the blocks to just the WP-Admin.

The way I see this, is that blocks will have a statically defined registry which will act as the source of truth for what blocks are available, what their capabilities are, the attributes they can have, the types (string, int, bool, array, etc) of data their attributes are expecting to store, etc.

With this knowledge, a decoupled application could request this Block registration from the server, hydrate a client and render out a UI based on the Block registration.

The Gutenberg editor could also use this same registration to Hydrate what blocks are available, what UI elements are needed to provide editing of the available attributes per-block, etc.

Having the configuration defined on the server should also help reduce code duplication. If the server is the source of truth, then the client and server can both use that truth for their needs. The client wouldn't need to define block attributes and fields in addition to the server, just know how to read the source of truth from the server and do something with it, which should actually reduce JS file sizes because there's less info in there defining things, and instead just rendering data that gets served from the client.

Alternate Storage Options
Right now the block data is stored as a large string in the post_content (which is genius for backward compatibility), but I see alternative storage being explored by both core and plugin developers.

For example, where I work we will likely at least experiment with offloading Blocks and their associated meta to ElasticSearch, so that we can read/write individual blocks without having to alter the entire post_content to fix a typo in one block, for example. We will also use ElasticSearch to to aggregations, etc to get insight into how blocks are being used across our sites, etc (how we use ES is beyond the point, though). The point is that the server will need to know about the blocks so validation can be properly made when interacting with external systems.

ElasticSearch has a type mapping for the data that gets sent to it, and if there's a server side schema for blocks, plugins like ElasticPress, or even Jetpack's ElasticSearch implementation would be able to provide mapping for blocks to ES to ensure block data gets indexed properly and efficiently. Without a server side schema, the mapping would be created only as data is sent to ES and ES would give it's best guess on things, but we'd all lose out on the potential ES could give us if there were a server side Block registry that could be used to define mapping.

ElasticSearch aside, I imagine the core team or other developers will explore options for storing Block data in WordPress in some fashion. . .whether it's new wp_blocks and wp_blocks_meta tables or something else, I'm sure folks will experiment with storing blocks outside of a massive post_content string and as we've learned with the evolution of post_meta, the lack of a schema makes things "interesting" to say the least.

WPGraphQL

I'm the creator and maintainer of WPGraphQL (https://github.com/wp-graphql/wp-graphql) which brings a GraphQL API to WordPress. . .so I'll pitch my case, which I think is applicable to all the above in some fashion.

WPGraphQL allows for data to be declaratively fetched and for just the requested data to be sent to the client from the server.

An example of a WPGraphQL query right now would be:

{
  post(id:"someId") {
     id
     title
  }
}

And that produces a JSON response like so:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World"
    }
  }
}

As you can see, unlike REST where a request sends back a pretty large payload with the entire post object (unless you've written up some custom feature endpoints 🤢 ), the minimal amount of data needed is transferred from server to client, which is super beneficial, especially on slow mobile networks where data transfer can hold things up.

Anyway, with Gutenberg I envision WPGraphQL being a super powerful tool, especially for decoupled applications. I can picture folks using Gutenblocks with various block meta to indicate whether a block should be viewable on desktop or mobile, (or in our case WordPress even powers our newspaper print content). With WPGraphQL, a native mobile client could ask for just blocks that are meant for use in a native context.

For example, Let's say I've extended all of my Gutenblocks to have a "Context" multi-select which would allow me to apply various contexts for where my blocks should display (Web / Native Mobile / Print, for example).

I could have a post created by Gutenberg that has this HTML:

<!-- wp:core/heading {"align":"center", contexts:[web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show only on web</h3>
<!-- /wp:core/heading -->

<!-- wp:core/heading {"align":"center", contexts:[mobile-native, web]} -->
<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>
<!-- /wp:core/heading -->

I could query from a React native (or similar) app, and ask for just the blocks that were tagged with "mobile-native" context. . .something to this tune:

{
   post(id:"someId") {
      id
      title
      contentBlocks( where: { context: MOBILE } ) {
           ...on HeadingBlock {
               rawContent
               __typename
           }
      }
   }
}

I would receive a payload like:

{
  data: {
    post: {
       id: "someId",
       title: "Hello World",
       contentBlocks: [
         {
             rawContent: "<h3 style="text-align:center" id="some-anchor">Heading to show ONLY on mobile</h3>",
              __typename: "HeadingBlock"
          }
       ]
    }
  }
}

My post contains multiple blocks, but my Native client was able to request just the blocks that were intended to be rendered in the Native context. This reduces network bandwidth as the data send to the client is limited to what should be sent to the client, reduces bugs in code as it's a schema with defined type system, reduces transformation on the client as the payload contains what's needed so the client doesn't need to parse/transform the Gutenblocks for all contexts to determine which ones it should use, etc. . .

With the current implementation of gutenberg_parse_blocks my hypothetical scenario is already partly possible with WPGraphQL, but the big missing piece (in this example), would be WPGraphQL knowing what Enum options there are for the Context select. . . WPGraphQL would have no knowledge of what options exist for the Context select field if it's defined in the client, not the server. It wouldn't even know that "context" was a possible attribute of a block/all blocks if the attributes weren't defined on the server.

=====

I'll stop here, as I probably just sound like I'm spouting a bunch of nonsense. . .but happy to elaborate, discuss, clarify anything I've pointed out. . .and who knows, maybe some things I've pointed out already have been addressed since the last time I had to dig through things??

Other Answers:

@aduth you pointed out that this preloads server-registered blocks:

// Preload server-registered block schemas.
$block_registry = WP_Block_Type_Registry::get_instance();
$schemas = array();
foreach ( $block_registry->get_all_registered() as $block_name => $block_type ) {
if ( isset( $block_type->attributes ) ) {
$schemas[ $block_name ] = $block_type->attributes;
}
}
wp_localize_script( 'wp-blocks', '_wpBlocksAttributes', $schemas );

However, it's a completely optional thing. Gutenberg still loads up all 40 (or however many) blocks whether there are any blocks registered server side or not.

With plugin developers being asked to migrate functionality to Gutenblocks, the server-side API needs to be more of a first-class citizen.

Let's take Matias demo from WCUS as an example. He showcased editing a "Book" page that had some content and an image.

Let's say the workflow for an organization is that certain user roles can edit the content, and certain user roles can edit the image.

If all block logic is handled on the client, that means any user role can effectively edit any content, as the restriction to what they can edit is just a client-side prop that could easily be changed via the browser console giving any user power to do whatever they want. . .

The server needs to be the contract between the client and the persistence layer. We can't trust the client 💯 .

I get that the current TinyMCE is basically free-reign, and any content can be placed in there without server-side validation, but that's 1 big text input. Gutenberg is introducing 100's of new text inputs, but not introducing server-side validation/auth checks for any of them.

I feel like for a long time, when learning best practices folks would say: "Look how core does it", but when folks "look how core does it" and see Gutenberg doing things exclusively on the client with no contract on the server determining what's safe and expected, folks will follow suit, and will start writing code that trusts the client explicitly, encouraging bad practices and all sorts of security vulnerabilities down the road.

If Gutenberg is trying to be "the best editor" out there, and "leapfrog medium and squarespace", etc. . .then let's make it "the best editor" by also making sure it adheres to best practices (never trust the client, handle validation on the server, etc)

@aduth I highly support what you've proposed here. By having a language-agnostic schema defined for what a block's attributes look like, this can only open opportunities for improvements for portability of blocks across systems. If source types can all be defined in a schema then it would open the possibility for a server-side PHP processor to be able to parse and apply transformations to a block, or to parse a block in order to extract data for an indexing service, for example. It could also then be used in mobile apps for native code to do the same. The blocks API written in JS then in Gutenberg would serve as a reference implementation of a standardized/common format.

+1 if we could register blocks via PHP, this would greatly improve the ability for plugins like Pods to register blocks etc. Otherwise we're stuck building our own interface for it that then outputs JS objects that we have another JS file loaded to register those JS objects as blocks in the editor.

Seems like this should happen sooner than later, as I'd imagine many are already taking advantage of the JavaScript registration API to add support in plugins etc - it would be a shame to cause duplicative work for any 3rd parties.

Is anyone owning this / have a write up on specifically what we need to do? If the code is written, perhaps it's mainly a documentation effort needed next?

Related Issues:

179
gutenberg Can't NPM Install: ENOENT: no such file or directory .staging/core-js-c2a9e69c (solved with rm -rf node_modules && npm install)
Delete package-lock.json and run npm install Hey @mintplugins which version of node and npm do you u...
78
gutenberg What is the proposed way to update blocks?
I am facing the same issue and absolutely agree with Fabian on this What if the frontend switches to...
49
gutenberg Are iframes a viable long-term solution for meta boxes?
Do all metaboxes have to work on mobile? Yes why wouldn't they? What are the cases (if any) that wou...
38
gutenberg Supporting Metaboxes
I also want to emphasize that many plugins use custom post types that rely on meta boxes without a c...
36
gutenberg Block API: Server-side awareness of block types
@aduth thanks for creating this ticket and CCing me! there is still a lack of general understanding ...
31
gutenberg Immediate crash of editor upon opening new post page (solved by updating nginx config)
Fixed tldr: if you run Nginx you might need to update your nginx config I have the same problem With...
25
gutenberg Gutenberg breaks completely if site URL is not the same as wordpress URL
Here's what I did to make it work (I only have 1 site): Issue Overview I have wordpress installed on...
23
gutenberg The editor has encountered an unexpected error. (solved by updating the try_files nginx configuration)
Just to confirm the following worked for me: try_files $uri $uri/ /index.php$is_args$args; @Zenexer ...
21
gutenberg Block Transforms should be more discoverable/obvious
I'd like to propose an alternative as I think there is already an over-reliance on iconography in Gu...
20
gutenberg Can't style embedded videos properly due to inline width & height
Here's basic front-end css: You are welcome. Describe the bug There's numerous issues open that disc...
16
gutenberg Is it possible to disable default block styles?
Removing works fine for me with the following example: This script is enqueued via wp_enqueue_script...
16
gutenberg Introduce concept of block templates for full site editing
Since we already did some Full Site Editing work on WordPress.com I’ll try to convey some of the thi...
16
gutenberg Allow multiple InnerBlocks per block
Imagine something simple as this example... Currently InnerBlocks can only be rendered once into any...
14
gutenberg Event or callback after block loaded on editor
I was also looking for the same options but could not find any solutions anywhere It seems like no e...
13
gutenberg @wordpress/env: A zero-config, self contained local WordPress environment for development and testing.
We've iterated a lot on @wordpress/env these past few weeks and I think we're now at the point where...
13
gutenberg Classic + Custom HTML blocks: Convert to Blocks removes valid inline formatting
@ZebulanStanphill @danielbachhuber F.Y.I I tried some fix phrasingContentReducer for ruby tag Now it...
12
gutenberg Metaboxes overlap Gutenberg in Chrome 77
Confirmed that the CSS solution worked for us I dropped this into our theme, ~~ Added by @nerrad ~~ ...
12
gutenberg Improve behaviour when saving content while offline
It would indeed be a great improvement to show a warning when offline and to store the changes in lo...
11
gutenberg How to replace the Document Settings panels or controls?
@jonathanstegall The removeEditorPanel() method only supports the removal of 5 built-in meta boxes: ...
9
gutenberg Support for Responsive Columns
Just adding my 2 cents.. In my own theme I simply add a media query and then set each column (.wp-bl...
7
gutenberg How to add blocks is not obvious enough and only getting worse.
Please I'm submitting this issue because I think that issue #5074 doesn't quite go far enough in out...
7
gutenberg My Parent Pages Are Still Missing In Gutenberg
I started work on implementing an accessible-autocomplete for sites with a large number of pages in ...
6
gutenberg How to server render block with nested blocks?
Hey sorry all for the delay but it is possible to have an InnerBlocks and render your block on the s...
3
gutenberg Dynamic reusable blocks: technical details
I might be missing a piece of this puzzle so apologies for wading into this late! The above REST spe...
139
ddev In WSL2 ddev start fails at docker-credential-desktop.exe, "error listing credentials"
I had to set credsStore: in my ~/.docker/config.json .. it was previously set to credentials.exe ...
41
woocommerce The Header Mini Cart does not update after a product is removed
Here is my fix EXPLANATION OF THE ISSUE After removing a product on the Shopping cart page /cart/ ...
39
FoundationPress Missing modules when running npm run build or npm install
The error in the original post is from running npm run build Hi I am working on a foundation press p...
34
woocommerce Rethinking 3.6's Dashboard Ads (#22857 )
​It appears what you're doing here is tracking usage reporting that back to your API and returning t...
31
FoundationPress After Node update, Issue running npm run production
Try this: npm uninstall gulp --save Then once that is complete run: npm install then: npm run build ...
24
lando Unhandled rejection Error: problem parsing null.tooling.cache. Ensure it is valid JSON!
Hi I had the same issue just now and apparently I only missed adding the name on my .lando.yml file ...
20
wp graphql Get Single Nodes (posts, terms, users, etc) by more than ID
The next release I'm working on (#1086) addresses this (the ability to fetch single nodes by various...
16
lando lando does not start
On ubuntu 16.04 docker rm landoproxyhyperion5000gandalfedition_proxy_1 helped me to ged rid of this ...
16
sage Sage 9.0.10 - postcss.config.js not working
To anyone else hung up on this I found a workaround My version of this issue was not font related bu...
16
woocommerce Negative amount as fee and tax calculation in checkout
If someone is still looking for an answer here is mine It works like a charm Describe the bug I have...
15
wp calypso Editor: Permission Errors when trying to Update/Publish Post/Pages
The fix has been deployed today to wpcom as D57128 after the first attempt had to be reverted due to...
15
lando Permissions issue with Docker
I also ran into this on: macOS 11.1 (Big Sur) Lando v3.0.25 I fixed with sudo chown -R username /Use...
14
acf to rest api How to get ACF post object field with ACF fields
Hi @travis-zookacreative Thanks for using my plugin bellow I wrote an example most generic V3 V2 Fun...
14
lando Unable to 'lando pull' pantheon site
This worked for me: lando terminus auth:login --machine-token=ABCDEFGHIJKLMNOPQRSTUVQXYZ (don't stea...
13
lando Please add Docs for Drupal's "Drush" CLI Command Debugging via XDebug
This is what worked for me: Then turning on Listen for PHP Debug Connections in PHPStorm and calling...
11
lando mysql stop connecting after lando restart
I think I found a solution I had same issue multiple times and I was forced to completely uninstall ...
11
woocommerce Attributes lost all parents
Hi @ABCPortugal like Claudio wrote you need to use the filter woocommerce_taxonomy_args_{$name} for ...
8
lando Support Xdebug 3
This worked for me PHPStorm 2020.3 drupal9 Xdebug 3 is out and comes with a lot of performance optim...
8
wp graphql How to create query/mutation for custom table in WP
WPGraphQL can work with any data source It's optimized to work with WordPress core data (Posts Terms...
7
woocommerce PayPal Intermittent Transaction cancelled or Internal server error on guest checkout
Thanks @mikejolley EXPLANATION OF THE ISSUE Using WooCommerce 3.0.5 and guest checkout ...
6
woocommerce product slider not working for rtl on woocommerce update
all you need to do is just set direction for gallery wrapper to rtl EXPLANATION OF THE ISSUE I updat...
5
jetpack Jetpack JSON API access crashes PHP-FPM running PHP 7.1
The Jetpack people noticed me about the freshly baked 5.3 update today: https://jetpack.com/2017/09/...
5
acf to rest api Apply same post format as WP JSON if ACF return format is Post Object
Allright thanks! The ticket on core is picked up and it will be fixed in WP 4.9.5 For the people who...
5
ddev Create a service configuration for varnish
Hi this is how we are doing this at wegewerk: What happened (or feature request): Edit: @stborchert ...
4
wp calypso Calypso: Deep links redirecting
This is a real bug and it's probably been around for some time Not a recent regression ...