Conquering Shadow DOM for a better autofill experience with Dashlane

The usage of Shadow DOM on a given webpage has historically presented problems for password managers because “shadow” fields are hidden from normal DOM calls, meaning autofill can’t work for a login credential or address form, or any other data which might be relevant for the given page. Thanks to the engineers at Dashlane, this limitation has been overcome, making way for even better usability on a wider range of websites than before.

Definitions: Shadow DOM and Dashlane

The Shadow DOM is a standard Web technology implemented by all modern browsers (Chrome, Firefox, Edge, Opera, Safari, on both desktop and mobile) which allows you to componentize Web markup and behaviors. In short, each component (with its inner markup, styles and JavaScript-based behaviors) is attached to a host element (the Shadow host) that “hides” all inner contents (the Shadow DOM) from regular DOM calls. For instance, calls to document.querySelectorAll().

Dashlane is a leading Password Manager and data protection provider. Users can install Dashlane on their mobile devices or as an extension to their Web browsers on desktop and laptop computers.

The problem

How does a Password Manager like Dashlane work inside a Web page?

• We analyse your Web pages and detect forms and their form fields through DOM calls. We recently switched our analysis engine to Machine Learning and we take pride in the incredible speed of that new core.
• We add our Dashlane icon to the fields we understand and can autofill with data (if you have relevant data, of course) or in which you can generate a password.
• When you place the caret inside such a field, we display a webcard offering various choices: autofill from your data vault (credentials, identity, addresses, payment information, passport, etc), generate a safe password, and more.
• We detect new or updated site credentials and we show another kind of webcard allowing users to store the data in their data vault.

All of that is based on the Open Web Platform and in particular on Document Object Model (DOM) queries which allow us to traverse a document tree, query elements in that tree, etc.

As we said above in our introduction, the inner contents of a Shadow DOM are completely “hidden” from regular DOM calls. If we take a Shadow DOM of host <div id="host"> and containing for instance an <input id="username" class="loginField"> field,

• document.querySelector(".loginField") will reply null
• document.getElementById("username") will reply null
• document.getElementById("host").firstElementChild will reply null

You can easily see Shadow DOM in action using the Inspector in your preferred browser. In the screenshot below (Chrome), you can see the login form of a well-known bank and a view of its markup. It uses Shadow DOM which is visible in the screenshot of the Inspector.

That’s why all Password Managers on the market —(until recently), including Dashlane— silently but completely fail at analyzing and autofilling Web pages using Shadow DOM for their login or signup pages. We just don’t see these forms through regular DOM calls.

/deep/ anyone?

Long ago, CSS Selectors had a “shadow-piercing combinator”. What was that beast?

In a regular Web page, meaning a Web page that does not use Shadow DOM at all, you can select all text input fields inside a div element through the following CSS selector:

div input[type="text"]

But, as we saw above, if your document has a <div> containing a shadow host itself containing a shadow <input type="text">, the selector above will not see it. The /deep/ combinator was fixing that hole:

div /deep/ input[type="text"]

That selector is “shadow-piercing” because of the /deep/ combinator. Shadow boundaries, ie. the DOM-visibility walls between Shadow hosts and their inner Shadow DOM, are traversed by /deep/.

/deep/ was then perfect to find forms and form fields in all Web pages, including the ones using Shadow DOM. Unfortunately, /deep/ was formally abandoned and removed from implementations (web browsers) back in 2017, citing performance reasons and, I quote, “violations of encapsulation.”

That deprecation left third-party app providers like Dashlane with no native option to query elements inside a Shadow DOM.

But…

But Shadow DOM is trending these days because it is quite convenient to architecture a web site in terms of reusable components.

The number of popular Web sites using Shadow DOM is now slowly but steadily increasing, meaning there were more and more Web pages that Dashlane could not autofill. Because we care about our customers, and because we have to deal with the Web in all its glorious (and sometimes weird) diversity, we at Dashlane had to try again to deal with Web sites based on Shadow DOM.

If native shadow-piercing queries are not possible any more, is there anything we can do through JavaScript and other regular calls? In short, the answer is yes.

PoC #1

We then started writing code simulating the /deep/ combinator mentioned above to query elements even if they are contained inside a Shadow DOM, with some minor and harmless restrictions. The basic algorithm was simple:

• Given sel a CSS selector valid in querySelectorAll() scope and root a Document, DocumentFragment or Element, let retArray be the array-ification of root.querySelectorAll(sel)
• Let hostArray be an array of all Shadow hosts in the subtree rooted at root, without traversing any Shadow boundary.
• For each shadowHost in hostArray, concatenate the result of the current algorithm on sel and shadowHost.shadowRoot to retArray
• Return retArray

Similarly, we had to write our own code to make other DOM calls like Node.parentNode or Node.firstChild pierce Shadow boundaries.

That worked. Pretty well, actually. But the performance of that attempt was suboptimal on Web pages with a very complex markup or using nested Shadow DOMs (components inside components) because the detection of all Shadow hosts inside a document is not easy. There is no CSS pseudo-class that could help us detect Shadow hosts at native speed, for instance. A NodeIterator retrieving all nodes being also an Element and having a non-null shadowRoot attribute is then the best choice and that choice can be expensive, very expensive.

Shadow DOM then remained a problem for us, and for our customers.

PoC #2

A few months passed and a new idea emerged to ease the pain. We considerably optimised the algorithm above to make it much faster while also having a negligible impact on Web pages not using Shadow DOM. To describe our current landscape, our Machine Learning engine usually analyses and classifies a regular Web form in 20 milliseconds on average and in 5 milliseconds in peak… The classification of a unique form field is performed after analysis in 160 microseconds on average… We’re fast, really fast.

• ✅ Mutations inside a Shadow DOM are pretty hard to observe from outside of that Shadow DOM because Mutation events don’t traverse Shadow boundaries… An attribute change, a subtree change for instance modifying the visibility of a Web form or adding a field to a Web form inside a Shadow DOM are painful to detect at document level and WebExtensions’ content code lives at document level. We succeeded in working around the issue.
• ✅ On Web pages that do use Shadow DOM, the overhead is well contained and lower than 5% (often much lower) in all cases we could find. That means that a Web form made of regular DOM elements and classified in 20 milliseconds would be classified in 21 milliseconds max if it used Shadow DOM.
• ✅ On Web pages that don’t use Shadow DOM, the overhead is negligible (far under 1 millisecond).

All in all, that new implementation reached a performance level that made it very acceptable from the only point of view that really matters, the customer’s point of view: negligible impact on regular Web pages and more than reasonable speed on pages using Shadow DOM.

What it tells us

A few important conclusions must be drawn from that two-step exploration:

1. The Shadow DOM specifications lack a few extras allowing third-party WebExtension vendors like Dashlane to interact with all Web pages:
   1. The shadow-piercing combinator, allowing to find all elements matching a given CSS selector at native speed wherever they are (in the regular DOM tree or inside a Shadow DOM), is the most important hole, for us, in the Standard. The performance issue, cited by browser vendors as one of the two reasons for the deprecation of that feature, does not really exist, at least on DOM API side (querySelector, querySelectorAll, matches, closest). Implementations in both Gecko’s matching.rs and Blink’s selector_checker.cc seem easily doable with no performance impact compared to a regular non-shadow-piercing call.
   2. The second reason given for that deprecation (violation of encapsulation), is not really valid. It’s still relatively easy, but expensive, to implement a JavaScript-based shadow-piercing version of querySelectorAll(), closest() or matches().
   3. We clearly miss a CSS pseudo-class selecting elements that are a Shadow host, even if it does not pierce shadow boundaries. Availability of such a rather easy to implement pseudo-class would drastically increase our performance.
2. A configuration value for Mutation Observers allowing us to detect DOM mutations inside (possibly nested) Shadow DOMs was proposed and abandoned long ago. Lesson being: browser vendors cannot think of all possible use cases of a new technology. Only users (read:  implementors who will use that new technology) will discover what’s well or badly designed, what’s missing or what needs to be improved. In particular when that new tech changes the way Web pages are made, proactivity is crucial. WebExtension vendors need to be much more active in Web Standardisation.

Now available in your browser

We’re excited to share that Dashlane now goes even deeper in Web pages, finding and recognizing Web forms inside Web Components and Shadow DOM to provide you with an even better autofill experience!