Software 3.0 stack.md

high quality guides and tutorial

full stack deep learning - llm bootcamp: https://fullstackdeeplearning.com/llm-bootcamp/spring-2023/

no code

• prototyping
  • nat.dev
  • https://play.vercel.ai/
• prompt engineering
  • https://github.com/ianarawjo/ChainForge An open-source visual programming environment for LLM experimentation and prompt evaluation.
    • alternatives
    • https://github.com/logspace-ai/langflow a UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows.
    • https://github.com/FlowiseAI/Flowise - visual langchain buidler

highest code level

• owning the endpoint
  • OpenLM - https://github.com/r2d4/openlm OpenAI-compatible Python client that can call any LLM
• SDK wrappers
  • https://github.com/minimaxir/simpleaichat
  • https://github.com/vercel-labs/ai
• prompt tooling
  • langchain
    • https://www.pinecone.io/learn/langchain/
  • llamaindex
  • deepset haystack
  • guardrails
  • scale spellbook
• vector databases

llmops

• portkey https://twitter.com/jumbld/status/1648684887988117508?s=46&t=90xQ8sGy63D2OtiaoGJuww
• helicone
• Ozone - prompt unit testing https://twitter.com/at_sushi_/status/1667004844153131008
• https://log10.io/ - pivoting to llm quality monitoring
• eval
  • https://github.com/BerriAI/bettertest https://twitter.com/ishaan_jaff/status/1665105582804832258
  • https://github.com/AgentOps-AI/agentops
  • https://www.ycombinator.com/launches/JFc-baserun-ai-ship-llm-features-with-confidence
  • https://github.com/mr-gpt/deepeval
  • Hegel AI Prompttools
  • https://github.com/promptfoo/promptfoo
  • https://benchllm.com/ - https://www.getscorecard.ai - https://arxiv.org/abs/2308.03688 - https://withmartian.com - https://aihero.studio/
• evals
  • scorecard
  • https://www.arthur.ai/blog/introducing-arthur-bench
• data quality
  • cleanlab.ai
  • deepchecks <- bigger
  • lilac ai
  • gallileo

typing/json structure libraries

• Microsoft TypeChat https://news.ycombinator.com/item?id=36803124
• jsonformer
• lmql

lower code level

• hugginface transformers https://huggingface.co/learn/nlp-course/chapter0/1?fw=pt
• lightning https://twitter.com/_willfalcon/status/1665826619200614401

vector databases

https://news.ycombinator.com/item?id=36943318

• chroma
• pinecone
• weaviate
• qdrant
• marqo vector search
• postgres
  • supabase vector
  • problems with it:
    https://twitter.com/nirantk/status/1674110063286571008?s=46

ETL

• psychic.dev

infra

• https://mlfoundry.com/
• together.ai

misc

• AI relational database https://github.com/georgia-tech-db/eva
• finetune industry
  • https://predibase.com/
• AI devtools
  • codegen.ai