add pseudocode for feature of inter data type and sharding data

[Yzing]

PR includes

• add pseudocode for feature of inter data type and sharding data

[gemini-code-assist]

Summary of Changes

Hello @Yzing, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request lays crucial groundwork for enhancing the data processing and chart advising capabilities by introducing a robust system for inferring various data shapes (plain, tree, graph, flow) and preparing for data sharding. It involves significant refactoring of data models, adding advanced analytical utilities, and integrating these changes into the existing advising pipeline. The overall goal is to enable more intelligent and context-aware data analysis and visualization recommendations.

Highlights

• Data Shape Inference: Introduced new functionality to infer the shape of input data, distinguishing between plain, tree, graph, and flow data structures. This is a foundational step for tailoring data processing and visualization recommendations based on the inherent structure of the data.
• Data Sharding Preparation: Added pseudocode and initial structures for data sharding, which will allow for breaking down large datasets into smaller, more manageable pieces for analysis. This includes new modules for shard and prompt generation for potential LLM integration in sharding.
• Refactored Data Models and Utilities: Existing data models like Graph and DataStore have been updated, and new models for Flow and Tree data types have been introduced. A suite of advanced data utility functions, including K-Means clustering and vector variance calculations, were added to support sophisticated data analysis.
• Pipeline Integration: The DataPlugin and ExtractPlugin in the chart advising pipeline have been significantly updated to leverage the new data shape inference and sharding capabilities, ensuring that data is correctly processed and prepared for subsequent advising stages.
• Type System Enhancements: Extensive updates to the type definitions (types/data.ts, types/pipeline.ts, types/insight.ts) to accommodate the new data shapes, sharding concepts, and purpose-driven analysis, improving type safety and clarity across the codebase.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant amount of scaffolding and pseudocode for handling various data structures like trees and graphs, and for data sharding. While this is a good step towards the new features, there are several issues in the current implementation, including critical bugs in tests and utility functions, regressions in existing logic, and areas where error handling is missing. The review comments highlight these issues, which should be addressed to ensure the new foundation is solid. Key areas for improvement include fixing the k-means implementation, correcting data shape inference logic, and improving error handling in plugins.