Feature Suggestion - Offer a Firewall-Enabled Mode

[bgribaudo]

My understanding is that the new SDK operates without Power Query's data protection firewall running.

From Power Query's perspective, everything inside a connector (well, technically, inside a data source kind, I believe) is considered a single data source for data protection partitioning purposes. So, within a data source kind, the fact that the firewall isn't present with the SDK is a moot point.

However, the SDK allows M code outside of the connector/outside of a data source kind to run, specifically the code in the MyConnector.query.pq test file. The code in this test file can pull from multiple data source kinds that are defined in the project's connector, as well as from other connectors that ship with Power Query.

In order for behavior to match between the SDK and other Power Query environments (e.g. PQ in Power BI or Excel), it would be nice to have the option in the SDK to have the firewall enabled and operating normally so that code inside the .query.pq test file is properly partitioned and firewalled.

Thank you for your consideration!

[mattmasson]

Exposing privacy levels makes sense if we expand the scope of the SDK to cover general PQ scenarios, but at this time it is focused on connector development.

The SDK lets you run test queries for logic you'll be putting into your data source, and data source calls made from inside a connector are done within a single firewall partition. To replicate this behavior, all SDK evaluations are done with Fast Combine / "Ignore Privacy Levels" enabled.

[bgribaudo]

[...] data source calls made from inside a connector are done within a single firewall partition

@mattmasson, in "regular" Power Query (e.g. the PQ that ships with Power BI or Excel), if a single connector exposes multiple data source kind methods and those methods are used from a single query, I'm prompted to set privacy levels separately for each kind (so 1 connector w/2 data source kinds potentially equals 2 privacy level settings).

Are you saying that behind the scenes those separate privacy level settings are ignored and instead all the data source methods from the connector are put into a single firewall partition?

If they are not, but instead each data source kind is partitioned separately, it would be nice to (eventually) be able to test in the SDK with the same behavior as users would encounter in regular PQ.

Example: If the test file's query sources data by calling a method from each of the connector's two data source kinds then preforms a join, it would be nice for the evaluation of that test query to work just like it would in Power BI (where the firewall being in play can have performance impacts, add extra evaluations for data analysis into the mix, etc.—this way, surprises aren't encountered when the connector is first run from regular PQ).

[mattmasson]

For data sources to be considered the same, their Kind + Path have to match. So if the data source kinds differ it means they'll always be treated as different data sources (even if they are loaded from the same module/extension).

I can see it being useful for reproducing issues. But that said, there really isn't much a connector developer can do about firewall issues. Either the data sources match, or they don't.

If we extend the SDK to cover general / advanced PQ performance tuning scenarios, then this functionality would definitely be appropriate.

Longer term, we'd like to integrate/embed PQ editor components into this VS Code experience. Right now, things like the data grid and credential entry/"management" are rough because we'd prefer to use the same PQ components. We'd likely get privacy level support for "free" if/when we get this deep into our SDK functionality.