Re-expose jQuery.find.tokenize

[mgol]

Is this [the jQuery.find.tokenize method - added by @mgol] missing now? jQuery 3.7.0.

$.find.tokenize('.test');
Uncaught TypeError: $.find.tokenize is not a function
    at <anonymous>:1:8

We use this in pie6k/jquery.initialize. We watch the DOM for mutations that match a selector, and being able tokenise the selector allows us to ignore certain mutations that the selector would never match against.

Originally posted by @bezborodow in jquery/sizzle#242 (comment)

[mgol]

This is my omission - my goal was to preserve all the Sizzle API that was exposed via jQuery.find in jQuery 3.7.0 and it looks I missed this one.

I re-checked all Sizzle.* APIs now and all the others I did expose.

We'll fix it in 3.7.1.

Now, I am wondering whether not to preserve this for 4.0.0 as well. This is the current jQuery 3.7.0 selector.js ending:

jQuery.find = find;

// Deprecated
jQuery.expr[ ":" ] = jQuery.expr.pseudos;
jQuery.unique = jQuery.uniqueSort;

// These have always been private, but they used to be documented
// as part of Sizzle so let's maintain them in the 3.x line
// for backwards compatibility purposes.
find.compile = compile;
find.select = select;
find.setDocument = setDocument;

find.escape = jQuery.escapeSelector;
find.getText = jQuery.text;
find.isXML = jQuery.isXMLDoc;
find.selectors = jQuery.expr;
find.support = jQuery.support;
find.uniqueSort = jQuery.uniqueSort;

The latter part that aliases jQuery APIs back to jQuery.find I would definitely not keep in 4.0 - we are, in the end, removing all other deprecated APIs and these have been private from the perspective of jQuery. I wonder about this block, though:

find.compile = compile;
find.select = select;
find.setDocument = setDocument;

(together with tokenize that's so far missing from this list). They are not currently exposed on main. With the selector engine rewrite in 5.0 they may need to go but 4.0 doesn't go that far yet so maybe we could keep exposing them for advanced use cases? I'm marking this part for the team discussion.

[mgol]

PR: #5260

[timmywil]

Tokenize is important and will stick around beyond the selector rewrite, so I think it's okay to leave exposed and document in the long term. I can see how it would remain useful. As for which other methods to keep exposed when we do the rewrite, I don't know if I'll be able to say that for sure until we get to the rewrite.

[mgol]

@timmywil Yeah, I can imagine it's hard to judge what will still be there after the rewrite. But the decision we need to take now is what to still expose in 4.0. I'd propose to re-add the following block to main:

find.compile = compile;
find.select = select;
find.setDocument = setDocument;
find.tokenize = tokenize;

where find is jQuery.find. If some of them stop making sense, we can remove in 5.0.

The only from this list usefulness of which is not so obvious to me is setDocument but even on main it triggers event handler attachment in IE so maybe potentially removing that can also wait for 5.0.

[timmywil]

I agree with all of that. We can take our time and wait to remove anything until 5.0.

[bezborodow]

Would any consideration be given to releasing the tokeniser as a subpackage in its own right? Users may want to use the tokeniser directly, given its usefulness for analysing selectors. For example, there are other packages such as css-selector-tokenizer to which the jQuery tokeniser could provide a trusted (and widely-tested) alternative.

[mgol]

Publishing a new package incurs additional maintenance cost and we don't have a lot of free cycles for such things. Not to mention that this is just a small part of jQuery and if we considered publishing it separately, there would potentially be many other parts warranting similar treatment.

The community is free to publish something by themselves based on jQuery code. Alternatively, in jQuery 4.0 you'll be able to do something like:

import tokenize from 'jquery/src/selector/tokenize.js';

as the tokenizer has been extracted to a separate file and jQuery 4.x source will be exposed and using ES modules.

[mgol]

@bezborodow Also, remember that the jQuery tokenizer only handles what jQuery needs and e.g. doesn't support pseudo-elements as they cannot be used in element queries, only in stylesheets.

[mgol]

PRs:

• main: Selector: Re-expose jQuery.find.{tokenize,select,compile,setDocument} #5263
• 3.x-stable: Selector: Re-expose jQuery.find.tokenize (3.x version) #5260