v1.14.0 #4291

vblagoje started this conversation in General

vblagoje
Feb 28, 2023
Maintainer

⭐ Highlights

PromptNode enhancements

PromptNode just rolled out prompt logging (pipeline debug), run_batch, and model_kwargs support. More updates to PromptNode and PromptTemplates coming soon!

Shaper

We're introducing the Shaper, PromptNode's helper. Shaper unlocks the full potential of PromptNode and ensures its seamless integration with Haystack. But Shaper's scope and functionality are not limited to PromptNode; you can also use it independently, opening up a whole new world of possibilities.

IVF and Product Quantization support for OpenSearchDocumentStore

We've added support for IVF and IVF with Product Quantization to OpenSearchDocumentStore. You can train the IVF index by calling train_index method (same as in FAISSDocumentStore) or by setting ivf_train_size when initializing OpenSearchDocumentStore and take your search to the next level.

What's Changed

Breaking Changes

refactor: Updated rest_api schema for tables to be consistent with Document.to_dict by @sjrl in refactor: Updated rest_api schema for tables to be consistent with Document.to_dict #3872
feat: Support multiple document_ids in Answer object (for generative QA) by @tstadel in feat: Support multiple document_ids in Answer object (for generative QA) #4062
feat: Update OpenAIAnswerGenerator defaults and with learnings from PromptNode by @sjrl in feat: Update OpenAIAnswerGenerator defaults and with learnings from PromptNode #4038
build: cache nltk models into the docker image by @mayankjobanputra in build: cache nltk models into the docker image #4118
feat: Add IVF and Product Quantization support for OpenSearchDocumentStore by @bogdankostic in feat: Add IVF and Product Quantization support for OpenSearchDocumentStore #3850

Pipeline

feat: add frontmatter to meta in MarkdownConverter by @TuanaCelik in feat: add frontmatter to meta in MarkdownConverter #3953
fix: removing code block in MarkdownConverter by @TuanaCelik in fix: removing code block in MarkdownConverter #3960
feat: Add page range support to PDF converters. by @danielbichuetti in feat: Add page range support to PDF converters. #3965
fix: Update telemetry to not serialize Pipeline if disabled. by @sjrl in fix: Update telemetry to not serialize Pipeline if disabled. #4000
feat: add Shaper by @ZanSara in feat: add Shaper #3880
fix: Event sending for RayPipeline crashing Haystack by @zoltan-fedor in fix: Event sending for RayPipeline crashing Haystack #3971
fix: document retrieval metrics for non-document_id document_relevance_criteria by @tstadel in fix: document retrieval metrics for non-document_id document_relevance_criteria #3885
fix: make the crawler more robust on Windows by @anakin87 in fix: make the crawler more robust on Windows #4049
fix: use correct count of outgoing edges in RayPipeline by @zoltan-fedor in fix: use correct count of outgoing edges in RayPipeline #4066
feat: Allow all training options for training a SentenceTransformers EmbeddingRetriever by @sjrl in feat: Allow all training options for training a SentenceTransformers EmbeddingRetriever #4026
refactor: replace mutable default arguments by @julian-risch in refactor: replace mutable default arguments #4070
feat: Support multiple RayPipelines by @zoltan-fedor in feat: Support multiple RayPipelines #4078
Remove double batching in retrieve_batch by @sjrl in Remove double batching in retrieve_batch #4014
style: Update black by @silvanocerza in style: Update black #4101
fix: Fix TableTextRetriever for input consisting of tables only by @jackapbutler in fix: Fix TableTextRetriever for input consisting of tables only #4048
fix: Deduplicate same Documents in isolated evaluation of Reader by @bogdankostic in fix: Deduplicate same Documents in isolated evaluation of Reader #4114
Docs: Fix code block formatting by @agnieszka-m in Docs: Fix code block formatting #4162
refactor: Remove the pin from the espnet module and fix the audio node tests. by @danielbichuetti in refactor: Remove the pin from the espnet module and fix the audio node tests. #4128
fix: change tiktoken fallback mechanism to support Windows amd64 by @danielbichuetti in fix: change tiktoken fallback mechanism to support Windows amd64 #4175
feat: Add OpenAIError to retry mechanism by @sjrl in feat: Add OpenAIError to retry mechanism #4178

DocumentStores

refactor: use weaviate client to build BM25 query by @hsm207 in refactor: use weaviate client to build BM25 query #3939
fix: fixed InMemoryDocumentStore.get_embedding_count to return correct number by @sjrl in fix: fixed InMemoryDocumentStore.get_embedding_count to return correct number #3980
fix: Add inner query for mysql compatibility by @julian-risch in fix: Add inner query for mysql compatibility #4068
feat: add support for custom headers by @hsm207 in feat: add support for custom headers #4040
feat: Add BM25 support for tables in InMemoryDocumentStore by @bogdankostic in feat: Add BM25 support for tables in InMemoryDocumentStore #4090
refactor: InMemoryDocumentStore - manage documents without embedding & fix mypy errors by @anakin87 in refactor: InMemoryDocumentStore - manage documents without embedding & fix mypy errors #4113
refactor: complete the document stores test refactoring by @masci in refactor: complete the document stores test refactoring #4125
feat: include testing facilities into haystack package by @masci in feat: include testing facilities into haystack package #4182

Documentation

Align with the docs install guide + correct lg by @agnieszka-m in Align with the docs install guide + correct lg #3950
docs: Update Crawler docstring for correct usage in Google colab by @silvanocerza in docs: Update Crawler docstring for correct usage in Google colab #3979
Docs: Update docstrings by @agnieszka-m in Docs: Update docstrings #4119
docs: Update Annotation Tool README.md by @bogdankostic in docs: Update Annotation Tool README.md #4123
feat: Add model_kwargs option to PromptNode by @sjrl in feat: Add model_kwargs option to PromptNode #4151
fix: Remove logging statement of setting ID manually in Document by @bogdankostic in fix: Remove logging statement of setting ID manually in Document #4129
chore: Fixing PromptNode .prompt() docstring to include the PromptTemplate object as an option by @TuanaCelik in chore: Fixing PromptNode .prompt() docstring to include the PromptTemplate object as an option #4135
chore: de-couple the telemetry events for each tutorial from the dataset on AWS that is used by @TuanaCelik in chore: de-couple the telemetry events for each tutorial from the dataset on AWS that is used #4155
feat: Implement run_batch for PromptNode by @sjrl in feat: Implement run_batch for PromptNode #4072

Other Changes

fix: add option to not override results by Shaper fix: add option to not override results by Shaper #4231
fix: Shaper store all outputs from function fix: Shaper store all outputs from function #4223
fix: allowing file-upload api to write files to disk fix: allowing file-upload api to write files to disk #4221
fix: Fix bug in prompt template check of OpenAIAnswerGenerator fix: Fix bug in prompt template check of OpenAIAnswerGenerator #4220
feat: add top_k to PromptNode feat: add top_k to PromptNode #4159
feat: Add JsonConverter node feat: Add JsonConverter node #4130
feat: adding secure loading of models by default for haystack by @mayankjobanputra in feat: adding secure loading of models by default for haystack #3901
fix: add tiktoken fallback mechanism. by @danielbichuetti in fix: add tiktoken fallback mechanism. #3929
fix: change model in distillation test by @ZanSara in fix: change model in distillation test #3944
feat: Expose output_variable in PromptNode result, adjust unit tests by @vblagoje in feat: Expose output_variable in PromptNode result, adjust unit tests #3892
fix: Fix type in FARMReader's save_to_remote by @bogdankostic in fix: Fix type in FARMReader's save_to_remote #3952
refactor: Remove PromptNode hash and equality functions by @vblagoje in refactor: Remove PromptNode hash and equality functions #3923
ci: Remove mypy deps install step in python_cache action by @silvanocerza in ci: Remove mypy deps install step in python_cache action #3956
fix: overwrite params with environment variables even if there are no params in the pipeline definition; make mypy ignore REST API tests by @anakin87 in fix: overwrite params with environment variables even if there are no params in the pipeline definition; make mypy ignore REST API tests #3930
Docs: Update ImageToText docstrings by @agnieszka-m in Docs: Update ImageToText docstrings #3963
Docs: Add TransformersImageToText API doc by @agnieszka-m in Docs: Add TransformersImageToText API doc #3966
ci: Add Docker images testing by @silvanocerza in ci: Add Docker images testing #3943
feat: Allow users to set a timeout for remote APIs by @danielbichuetti in feat: Allow users to set a timeout for remote APIs #3949
ci: Fix docker image testing on release by @silvanocerza in ci: Fix docker image testing on release #3976
Fix: Fix quotation marks by @agnieszka-m in Fix: Fix quotation marks #3973
fix: PromptNode doesn't have run_batch support (yet) by @vblagoje in fix: PromptNode doesn't have run_batch support (yet) #3972
chore: increased timeout for loading pipelines through API by @mayankjobanputra in chore: increased timeout for loading pipelines through API #3977
Missing import for TransformersImageToText by @ZanSara in Missing import for TransformersImageToText #3984
test: CI on py3.8 by @ZanSara in test: CI on py3.8 #3926
Simplifies and fix docker images tests on release by @silvanocerza in Simplifies and fix docker images tests on release #3982
feat: Add use_prefiltering parameter to DeepsetCloudDocumentStore by @bogdankostic in feat: Add use_prefiltering parameter to DeepsetCloudDocumentStore #3969
ci: Delete Docker images after testing to prevent workflow failure by @silvanocerza in ci: Delete Docker images after testing to prevent workflow failure #4004
fix: Add a verbose option to PromptNode to let users understand the prompts being used Wrong offsets returned by FARMReader #2 by @zoltan-fedor in fix: Add a verbose option to PromptNode to let users understand the prompts being used #2 #3898
fix: prevent posthog from sending errors to stderr by @julian-risch in fix: prevent posthog from sending errors to stderr #4008
fix: extend schema for prompt node results by @tstadel in fix: extend schema for prompt node results #3891
proposal: TableCell by @sjrl in proposal: TableCell #3875
refactor: In PromptNode reuse tokenizer instead of loading new one for stop words by @sjrl in refactor: In PromptNode reuse tokenizer instead of loading new one for stop words #4016
ci: Automate release on PyPi by @silvanocerza in ci: Automate release on PyPi #4015
ci: Fix PyPi release workflow by @silvanocerza in ci: Fix PyPi release workflow #4029
ci: Bump act10ns/slack from v1 to v2 by @silvanocerza in ci: Bump act10ns/slack from v1 to v2 #4031
ci: latest version of pylint is failing, ignore new errors by @masci in ci: latest version of pylint is failing, ignore new errors #4035
ci: Add linting of workflow and related pre-commit hook by @silvanocerza in ci: Add linting of workflow and related pre-commit hook #4032
ci: Fix pylint version to prevent crash by @silvanocerza in ci: Fix pylint version to prevent crash #4043
ci: Add missing env var in PyPi release slack notification by @silvanocerza in ci: Add missing env var in PyPi release slack notification #4052
fix: allow Biadaptive & Triadaptive to work with EarlyStopping by @jackapbutler in fix: allow Biadaptive & Triadaptive to work with EarlyStopping #4033
proposal: Add Agents for extended LLM support by @julian-risch in proposal: Add Agents for extended LLM support #3925
ci: Fix pylint workflow check running on tests files by @silvanocerza in ci: Fix pylint workflow check running on tests files #4076
fix: Add PromptTemplate repr method by @vblagoje in fix: Add PromptTemplate __repr__ method #4058
ci: Change actionlint pre-commit hook to use Dockerized tool by @silvanocerza in ci: Change actionlint pre-commit hook to use Dockerized tool #4060
ci: Make tests run conditionally in CI by @silvanocerza in ci: Make tests run conditionally in CI #4086
feat: OpenAI - warn users if max_tokens is too short by @anakin87 in feat: OpenAI - warn users if max_tokens is too short #4094
Docs: Add shaper to api docs by @agnieszka-m in Docs: Add shaper to api docs #4083
feat: Update allowed models to be used with Prompt Node by @sjrl in feat: Update allowed models to be used with Prompt Node #4018
ci: Add missing env vars in rest_api CI tests by @silvanocerza in ci: Add missing env vars in rest_api CI tests #4098
ci: Fix pylint CI check running with no files by @silvanocerza in ci: Fix pylint CI check running with no files #4097
Proposal: Add a JsonConverter node by @bglearning in Proposal: Add a JsonConverter node #3959
fix: query filters in REST API by @oryx1729 in fix: query filters in REST API #4105
fix: fix torchaudio version by @mayankjobanputra in fix: fix torchaudio version #4102
ci: Exclude .github folder from triggering tests by @silvanocerza in ci: Exclude .github folder from triggering tests #4120
ci: Add workflow to label PRs that edit docstrings by @silvanocerza in ci: Add workflow to label PRs that edit docstrings #4115
Update PromptTemplate unit tests by @vblagoje in Update PromptTemplate unit tests #4131
ci: Add load arg to docker/bake-action before testing Docker images by @silvanocerza in ci: Add load arg to docker/bake-action before testing Docker images #4124
Revert changes introduced in PR ci: Add load arg to docker/bake-action before testing Docker images #4124 by @silvanocerza in Revert changes introduced in PR #4124 #4137
ci: Fix Docker images test on release by @silvanocerza in ci: Fix Docker images test on release #4153
ci: Update docstring-labeler.yml workflow to safely run in PRs from forks by @silvanocerza in ci: Update docstring-labeler.yml workflow to safely run in PRs from forks #4146
Docs: Add filter to hide entity post processor by @agnieszka-m in Docs: Add filter to hide entity post processor #4160
ci: Use larger runner for Docker release workflow by @silvanocerza in ci: Use larger runner for Docker release workflow #4185
fix: make all OpenAI API params in PromptNode and PromptModel controllable via model_kwargs by @tstadel in fix: make all OpenAI API params in PromptNode and PromptModel controllable via model_kwargs #4183

New Contributors

@jackapbutler made their first contribution in fix: allow Biadaptive & Triadaptive to work with EarlyStopping #4033

Full Changelog: v1.13.2...v1.14.0

This discussion was created from the release v1.14.0.

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment