Evaluation timeout for promptflow using Azure AI Evaluation SDK

[kshitijcode]

• azure-ai-evaluation:
• 1.1.0:
• MAC:
• 3.12:

Describe the bug
We are trying to run evaluations using azure ai evaluation SDK. We have create a custom evaluator which takes more than 10 minutes to complete. We get the error

{"message": "Failed to run 1/13 lines. First error message is: Line 0 execution timeout for exceeding 600 seconds", "messageFormat": "", "messageParameters": {}, "referenceCode": "Runtime", "code": "UserError", "innerError": {"code": "SDKError", "innerError": {"code": "BulkRunException", "innerError": null}}, "additionalInfo": [{"type": "BulkRunErrors", "info": {"errors": [{"line_number": 0, "error": {"message": "Line 0 execution timeout for exceeding 600 seconds", "messageFormat": "Line {line_number} execution timeout for exceeding {timeout} seconds", "messageParameters": {"line_number": "0", "timeout": "600"}, "referenceCode": "Executor", "code": "UserError", "innerError": {"code": "LineExecutionTimeoutError", "innerError": null}, "debugInfo": {"type": "LineExecutionTimeoutError", "message": "Line 0 execution timeout for exceeding 600 seconds", "stackTrace": "Traceback (most recent call last):\n", "innerException": null}}}]}}], "debugInfo": {"type": "BulkRunException", "message": "Failed to run 1/13 lines. First error message is: Line 0 execution timeout for exceeding 600 seconds", "stackTrace": "Traceback (most recent call last):\n", "innerException": null}}

This indicates that the prompt flow run in the backend is taking more than 600 seconds. Is there a way I can increase the timeout?

To Reproduce
Steps to reproduce the behavior:

1. Create any custom evalutor which takes more than 600 seconds to execute. We can simply place sleep of 600 seconds in evalator code.
2. Run the evaluator and timeout issue will occur.

Expected behavior
A clear and concise description of what you expected to happen.
Evaluation should run succesfully.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @luigiw @needuv @singankit.