Skip to content
This repository has been archived by the owner on Mar 27, 2023. It is now read-only.

rafaelvp-db/dbx-insurance-qa-hugging-face

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

35 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Insurance Q&A Intent Classification with Databricks & Hugging Face

This repo has been deprecated. Please refer to this one


TLDR; this repo contains code that showcases the process of:

  • Ingesting data related to Insurance questions and answers (Insurance QA Dataset) into Delta Lake
  • Basic cleaning and preprocessing
  • Creating custom PyTorch Lightning DataModule and LightningModule to wrap, respectively, our dataset and our backbone model (distilbert_en_uncased)
  • Training with multiple GPUs while logging desired metrics into MLflow and registering model assets into Databricks Model Registry
  • Running inference both with single and multiple nodes

Additional Reference

  1. Minwei Feng, Bing Xiang, Michael R. Glass, Lidan Wang, Bowen Zhou. Applying Deep Learning to Answer Selection: A Study and An Open Task
  2. Fine-tune Transformers Models with PyTorch Lightning
  3. PyTorch Lightning MLflow Logger
  4. dbx by Databricks Labs

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published