FEAT: Integrate XPIATestOrchestrator with the AI Recruiter

[KutalVolkan]

The AI Recruiter is now fully functional with a FastAPI server, allowing us to upload PDFs and compare candidates’ résumés against job descriptions.

The previous raw HTTP approach struggled with parsing, formatting, and multipart uploads, making integration a challenge. I couldn’t get the old feature to work properly, so I did the next best thing—added a new feature instead! 😅 But don’t worry, I kept backward compatibility—no features were harmed in the process!

Now, HTTPTarget fully supports AI Recruiter, enabling seamless automated CV uploads and candidate evaluation.

I also updated the Docker setup to simplify deployment—be sure to run it before testing the ai_recruiter_demo.ipynb. You can find it on GitHub: https://github.com/KutalVolkan/ai_recruiter/tree/main/docker_setup

Next Steps:

• Ensure full functionality of XPIAOrchstrator (this may require organizing ai_recruiter_demo.ipynb).
• Code clean up and update docstrings.
• Convert the notebook into a .py script.
• Modify the prompt injection technique:
  • Update injection_items and insert relevant skills, education, and qualifications based on the job description.
• Write tests for the new HTTPTarget features.
• Write a PyRIT blog post covering the setup, the idea behind it, and the results.
• Perform integration testing for the AI Recruiter Demo. If it's wished 😄

Related Issue:
#541

---

More Information about the AI Recruiter:

• Vulnerabilities: https://github.com/KutalVolkan/ai_recruiter/tree/main/owasp_top_ten
• General README: https://github.com/KutalVolkan/ai_recruiter

[rlundeen2]

This is all so cool @KutalVolkan! I love this scenario!

I left some comments. One additional non-comment, I'd love to have this written out on the blog if you're interested in either writing it or having us writing it!

[romanlutz]

I think the entire example is super cool. We should 100% have it here. Amazing work!

A few thoughts, though:

• maybe we should have a xpia directory? @rlundeen2 wdyt?
• This heavily depends on the AI recruiter application. That means running this notebook depends on having that repo's code downloaded and running (and perhaps certain resources set up?). We need instructions for that in the notebook and I'm thinking an integration test. You might have seen other PRs with integration tests recently. If you don't know what I mean please lmk and I'll point you the right way.

[KutalVolkan]

Hello @rlundeen2 & @romanlutz ,

When running pre-commit run --all, I encountered a MyPy type-checking error in doc/code/orchestrators/3_xpia_orchestrator.py at line 192. The issue is an incompatible type assignment—HTTPXApiTarget is being assigned to a variable that expects SemanticKernelPluginAzureOpenAIPromptTarget.

I’ve tried troubleshooting, but I haven't been able to resolve it yet. Any suggestions?

Thanks!

[KutalVolkan]

• If you don't know what I mean please lmk and I'll point you the right way.

Hello Roman,

If you don’t mind, could you provide some pointers on where to look? Otherwise, I’ll figure it out myself. :)

[romanlutz]

• If you don't know what I mean please lmk and I'll point you the right way.

Hello Roman,

If you don’t mind, could you provide some pointers on where to look? Otherwise, I’ll figure it out myself. :)

The tests under tests/unit all run locally only. The integration tests are under tests/integration and we run these separately with actual LLM endpoints etc. It would be really cool if we could have an integration test that runs this scenario, but it would require starting this service locally, of course.

FWIW integration tests are brand new here and we're just in the process of adding a bunch of them to cover as much as we can, including notebook examples.

[romanlutz]

Hello @rlundeen2 & @romanlutz ,

When running pre-commit run --all, I encountered a MyPy type-checking error in doc/code/orchestrators/3_xpia_orchestrator.py at line 192. The issue is an incompatible type assignment—HTTPXApiTarget is being assigned to a variable that expects SemanticKernelPluginAzureOpenAIPromptTarget.

I’ve tried troubleshooting, but I haven't been able to resolve it yet. Any suggestions?

Thanks!

My guess is that you're reusing the same variable name for the processing_target. Can we try calling them something specific for each of the examples? semantic_kernel_processing_target and httpx_api_processing_target or something like that.

[KutalVolkan]

• If you don't know what I mean please lmk and I'll point you the right way.

Hello Roman,
If you don’t mind, could you provide some pointers on where to look? Otherwise, I’ll figure it out myself. :)

The tests under tests/unit all run locally only. The integration tests are under tests/integration and we run these separately with actual LLM endpoints etc. It would be really cool if we could have an integration test that runs this scenario, but it would require starting this service locally, of course.

FWIW integration tests are brand new here and we're just in the process of adding a bunch of them to cover as much as we can, including notebook examples.

Hello Roman,

I uploaded the integration test. It is ready for review. You can go into the path PyRIT\tests\integration\ai_recruiter and run:

pytest .\test_ai_recruiter.py -s

Note: I use OpenAI models and endpoints for the AI recruiter.
Update: Switched to using AzureOpenAI endpoints and deployments.

[romanlutz]

Amazing work, @KutalVolkan !

IMO this is already in a very good state. I left one comment about how we'll deal with Docker in the integration tests that I want the other maintainers to chime in on, but otherwise I think this is ready to go (assuming the smaller comments are addressed, of course).

[KutalVolkan]

Hello Roman,

I tried to resolve every comment you gave. Hopefully to your needs! If not, feel free to provide feedback, and I'll address them. I’ll probably have more time next weekend. :)

During the pre-commit run --all, I encountered an error related to UnicodeDecodeError in check_links.py, which was caused by the default Windows encoding (cp1252). I fixed it by explicitly setting UTF-8 encoding when reading files.

However, after fixing that, I ran into another issue with the Jupyter Book Build Check regarding a failed import of the function fetch_decoding_trust_stereotypes_examples in pyrit.datasets. I didn’t have much time today to debug it further, so I had to skip fixing it for now.

Let me know if you need any other changes!

[KutalVolkan]

Hey Roman & Hello Rich,

I ran the test, and somehow the injected PDF gets a score of 0. It’s still selected as the best match, but only due to the low distance. I’ll need to investigate further and come back with more details on why it’s not working as expected.

Update: The culprit was the api_version of the chat_client, which is responsible for the scoring!

[romanlutz]

This looks great to me. I want to at least run the integration test locally once before merging it, though. I should have a little bit of time to do that on Friday or next week. Thanks for your patience!