Solvedselenium Shadow DOM traversal support

I've been gradually trying to push support of shadow DOM v1 into selenium (see #4230, #5762).

However, we are still missing one major piece: traversal.

There needs to be some ability to traverse shadow DOM when calling FindElement and what not.

So maybe we can discuss some possible implementations here?

What we can't/shouldn't do:

• Alter how existing selectors work (CSS, XPath, etc. they should all behave as they do now)
• Implement deprecated things like /deep/
• Implement our own selector syntax/mechanism

The way I see this working is that we should treat the DOM as we would in the browser, meaning there will not be a way to select a deep element in one call:

element
  .FindElement(By.CssSelector("foo"))
  .ShadowRoot?
  .FindElement(By.CssSelector("bar"));

Which, in the browser, would be structured the same (no shortcuts):

element
  .querySelector('foo')
  .shadowRoot
  .querySelector('bar');

So maybe we just need to implement ShadowRoot on WebElement? Which can be null (just like in the browser).

The idea is to conceptually treat the shadow DOM in the same way we treat frames.

I think a huge difference between ShadowRoots and frames is the sheer pervasiveness and number of them. Apps are likely to have 100s to 1000s of ShadowRoots, basically every Custom Element will.

And they're not nearly as special as iframes - they're in the same window and document, share the same globals, many style properties inherit through them, some events bubble up past them, light-DOM children are projected into them, and they're always attached to a host.

Many tests need to check on some interaction between elements in the ShadowRoot and outside of it - ie, set a property on the host and make sure it's reflected in the shadow; click on a button in the shadow and many sure that an event is fired on the host, or an attribute is added; add a child and make sure it's assigned to a slot, style a CSS ::part and check that it's applied, etc. The idea that a ShadowRoot is a different state just doesn't really hold. It's merely a different branch in the DOM tree (the tree of trees).

Iframes on the other hand have none of these APIs/interactions, and interactions between the iframe and it's host document are very limited, so these types of tests are probably non-existent and the pain that a switching-based API would cause haven't been felt.

ShadowRoots also very commonly nest, whereas with frames testing through arbitrary testing nesting levels is probably pretty rare. So we'll want the ability to traverse deeply into ShadowRoots, possibly multiple branches of the tree of trees in a single test to test cross-components interactions.

Consider a search form with three custom components - a text field, a button, and an output area - that all have shadow roots, all composed inside a shadow root.

One way the test looks like this:

const root = element.shadowRoot;
root.findElement(By.css('x-text-field')).shadowRoot.findElement(By.css('input')).sendKeys('hello');
root.findElement(By.css('x-button')).shadowRoot.findElement(By.css('button')).click();
assert.equals(root.findElement(By.css('x-output')).shadowRoot.getText(), 'hello');

Another way is like this presumably:

driver.switchToShadowRoot(element);
driver.switchToShadowRoot(driver.findElement(By.css('x-text-field')));
driver.findElement(By.css('input')).sendKeys('hello');
driver.switchToShadowRoot(element);
driver.switchToShadowRoot(driver.findElement(By.css('x-button')));
driver.findElement(By.css('button')).click();
driver.switchToShadowRoot(element);
driver.switchToShadowRoot(driver.findElement(By.css('x-output')));
// I'm not sure how to do this - how would we get non-Element Nodes from a Driver instance?
assert.equals(driver.getText(), 'hello');
// And make sure you switch back at the end of the test!
driver.switchToShadowRoot(element);

I personally find that much harder to follow because of the internal state of the driver, and there are certain things that are just not possible because you can't get a reference to a ShadowRoot, ie, you can't pass a root to a helper function, you're IDE won't help identify what root you're referencing on a line because it's not a reference...

Also, now that we're using WebDriver to interact with ShadowRoots, what do all the other APIs on WebDriver do? Why do I care about get(url), getCurrentUrl(), getTitle(), etc.? These can only return the window's state, but it's confusing to me at least to intermix global state like that and the state of an individual element. In contrast, these APIs do apply to Iframes.

Also if the shadowRoot is removed from that node

It's not possible to remove a ShadowRoot, even if you delete the shadowRoot reference. The only way a ShadowRoot is destroyed is if its host is - so the danger is exactly the same as if an Element was removed from the document.

@justnpT Maybe it is not appropriate to mention it here, but for in our project, it was reason #1 to move to Playwright (https://playwright.dev). It has everything you could dream of for shadow dom support.