Skip to content

Generate fields for markers in REPL so compiler plugin can extract the schema #1154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

koperagen
Copy link
Collaborator

without this change plugin cannot see schemas from previous cells, generated by REPL:

val df = DataFrame.read("...")
%%
val df1 = df.add("a") { 42 } 
// only df1.a is available here, because from plugin POV df has empty schema

After this PR generated markers (DataFrame<T>) will have unified structure, so schema can be extracted from both REPL and plugin generated ones

@koperagen koperagen requested a review from zaleslaw April 28, 2025 12:20
@koperagen koperagen self-assigned this Apr 28, 2025
@@ -4,5 +4,9 @@ public interface DataFrameSchema {

public val columns: Map<String, ColumnSchema>

public fun compare(other: DataFrameSchema): CompareResult
/**
* By default generated markers for leafs aren't used as supertypes: @DataSchema(isOpen = false)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we use the word "leaf" for DataFrames. Value/Frame columns are used for this purpose I believe.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"strictlyEqualValueCols" maybe?

Copy link
Collaborator Author

@koperagen koperagen Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, maybe "nested schemas" is a better wording, idk. By leaf here i meant.

interface A {
  val col: Int
  val nested: B <== Leaf
  val nestedFrame: List<B> <== Leaf
}

What do you think?

Copy link
Collaborator

@Jolanrensen Jolanrensen Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you actually meant "node"? because the schema continues downwards in a different schema? (B is a schema too right?) a "leaf" is normally a node with no children
Yes, then nested schema is better definitely

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see here the gravitation of Graph Theory with presenting dataframe as a tree.
I am fine with both schema - oriented and graph theory oriented wording, but... have a guess that our users closer to the hierarchical schemas wording

@koperagen koperagen added this to the 1.0.0-Beta1 (0.16) milestone Apr 29, 2025
@koperagen koperagen added the Compiler plugin Anything related to the DataFrame Compiler Plugin label Apr 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Compiler plugin Anything related to the DataFrame Compiler Plugin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants