Import Design and File Type Detection

[leecalcote]

Current Behavior

There's a lack of central and holistically written logic that is singularly used for detecting and processing. The import of infrastructure is code files, as well as infrastructure is design (Meshery) files.

Example: There's duplication in the identification of Docker Compose files. The https://github.com/meshery/meshkit/blob/master/utils/detect_pattern_file_type.go is poorly written and needs to be displaced with the following logic. We're in need of consolidating file detection and file import functions.

Desired Behavior

Refer to the https://github.com/meshery/meshery/pull/13392/files and see the logical flow in pseudo code below.

Implementation

Sample client code in Meshery Server

/server/handlers/design/import.go

# This logic should really be the same or highly reused with 
# the  importModel( *file handle, allow-duplicate-registration bool) (_existing function_) flow.

const supportedFileTypes := types from meshery schemas for "filetype-moniker"

importDesign(* filehandle, identify-only bool) {

  sanitization(file)
  identification(file)
  conversion(file)

  return mesheryDesignContents, fileType
}

Sample MeshKit Logic

Sanitization

/utils/files/sanitize.go

sanitization() {

  1. Process single file or single file bundle (compressed file).
  1. File must have filename extension.
  1. Accepts only files with the following extensions: json, yaml, yml, tar, tar.gz, tgz
  1. Create temp space. Uncompress file, if necessary. 
  1. Apply yaml and/or json validation for each file. Only continue processing valid yaml/json.
  1. Fail forward; e.g. if an unsupported file type is included in a bundle, continue forward despite the single unsupported file.
  
return filehandle
}

Identification

/utils/files/identify.go

# Largely the bulk import use case (with GitHub repo sync/import) justifies this step, 
# which would otherwise not be necessary (just overhead).
# Sequencing of which file type identification is ONLY important when more than one file is being identified.
# This routine makes under the assumption that all supported file types, are identifiable based on authoritative (3rd-party) project schema.
  
 IdentificationSubroutine(filehandle) {
  # Run schema validation against each file, while respecting the
  validateAgainst3rdPartySchema() {
    In a concurrent goroutine (optional) for each supported file type validate schema {
      
     return (filetype-moniker, temp-file handle)
   }
}

Conversion

/utils/files/convert.go

# Invoke this function once for each file to be converted to Meshery Design.
Conversion subroutines (filehandle) {
 switch identifiedType {

   case "DockerCompose"
    convertDockerComposetoMesheryDesign()

   case "KubernetesManifest"
     convertKubernetesManifesttoMesheryDesign()
  ...
 default
}

# Leverage the conversion logic of the supported file provider.
func convertHelmChartToMesheryDesign(filehandle) {
    loader.Load(helm chart)
   helm.client(file).convert()
}

func convertKustomizeToMesheryDesign(filehandle) {
  krusty.import(kustomization)
}

return Meshery patternID

Acceptance Tests

1. When a .png is submitted in a tar.gz, fail forward and continue processing the rest of the file bundle.
2. Meshery design import REST API should return a pattern ID with the need for prompting users to specify what file type is being imported.

Notes:

1. Remote Providers are dependent upon the IdentifyFileType function.
2. This logic should very much mirror model import behavior and vice-versa.
   1. Future: Add support for Meshery Views.

Mockups

---

Contributor Guides and Handbook

• 🛠 Meshery Build & Release Strategy
• 📚 Instructions for contributing to documentation
  • Meshery documentation site and source
• 🎨 Wireframes and designs for Meshery UI in Figma
• 🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack