generateContent throws 500 Internal Server Error when attempting to ground with VertexAISearch

[jwilson-tower]

All requests using the RetrievalTool fail with the following error...

[VertexAI.GoogleGenerativeAIError]: got status: 500 Internal Server Error. {"error":{"code":500,"message":"Internal error encountered.","status":"INTERNAL"}}"

The same requests, or rather requests with the same parameters, work from the console.

Requests using the GoogleSearchRetrievalTool work in the same code but I am specifically looking to ground my responses against my own data set in Vertex AI Search

[jwilson-tower]

const vertexAI = new VertexAI({
	project: 'xxx',
	location: 'xxx',
	googleAuthOptions: {
		keyFile: 'xxx'
	}
});
const.generativeModel = this.vertexAI.preview.getGenerativeModel({
	model: 'gemini-1.5-flash-001',
	generationConfig: {
		"max_output_tokens": 8192,
		"temperature": 0,
		"top_p": 1,
	},
	tools: [
		{
			retrieval: {
				vertexAiSearch: {
					datastore: 'projects/xxx/locations/global/collections/default_collection/dataStores/xxx',
				},
				disableAttribution: false,
			},
		}
	]
});
const result = await generativeModel.generateContent({
    contents: [{
        role: 'user',
        parts: [{
            text: 'xxx'
        }]
    }]
});

[keepforever]

This is happening to me too, using the node sdk

GoogleGenerativeAIError: [VertexAI.GoogleGenerativeAIError]: got status: 500 Internal Server Error. {"error":{"code":500,"message":"Internal error encountered.","status":"INTERNAL"}}
    at throwErrorIfNotOK (/Users/xxx/Desktop/projects/xxx/node_modules/@google-cloud/vertexai/build/src/functions/post_fetch_processing.js:34:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async generateContent (/Users/xxx/Desktop/projects/xxx/node_modules/@google-cloud/vertexai/build/src/functions/generate_content.js:51:5)

import { invariantResponse } from '@epic-web/invariant'
import { Part } from '@google-cloud/vertexai'
import {
  ActionFunctionArgs,
  json,
  unstable_createMemoryUploadHandler,
  unstable_parseMultipartFormData,
} from '@remix-run/node'
import { MAX_BRAND_LOGO_UPLOAD_SIZE } from '~/constants/constants'
import { deleteFile, uploadFile } from '~/utils/cloud-storage.server'
import { generativeModelForPdf, parseAndConcatText } from '~/utils/vertex.server'

export async function action(args: ActionFunctionArgs) {
  const { request } = args

  const formData = await unstable_parseMultipartFormData(
    request,
    unstable_createMemoryUploadHandler({ maxPartSize: MAX_BRAND_LOGO_UPLOAD_SIZE }),
  )

  const guidelinesFile = formData.get('pdf') as File
  invariantResponse(guidelinesFile, 'Guidelines file is required')

  const brandId = formData.get('brandId') as string
  invariantResponse(brandId, 'Brand ID is required')

  const { uri, fileName } = await uploadFile(parseInt(brandId), guidelinesFile)
  console.log('\n', `uri = `, uri, '\n')

  const filePart: Part = {
    fileData: {
      fileUri: uri,
      mimeType: 'application/pdf',
    },
  }

  const textPart = {
    text: `
    You are a very professional document summarization specialist.
    Please summarize the given document.`,
  }

  const vertexRequest = {
    contents: [{ role: 'user', parts: [filePart, textPart] }],
  }

  let resp
  try {
    resp = await generativeModelForPdf.generateContent(vertexRequest)
  } catch (error) {
    console.log('\n', `error = `, error, '\n')
    throw new Error('Failed to generate content')
  }

  const contentResponse = resp.response
  console.log('Generated content response: ', JSON.stringify(contentResponse))

  const concatenatedText = parseAndConcatText(contentResponse)

  console.log('\n', `concatenatedText = `, concatenatedText, '\n')

  // delete file after analysis
  try {
    const resp = await deleteFile(fileName)
    console.log('\n', `resp = `, resp, '\n')
  } catch (error) {
    console.log('\n', `failed to delete file ${fileName} `, '\n')
  }

  return json({ resp: concatenatedText })
}

export const generativeModelForPdf = vertexAI.getGenerativeModel({
  model: 'gemini-1.5-flash-001',
  generationConfig: { maxOutputTokens: 256 },
})

[neoromantic]

Have same problem with very minimal setup.

I'm using demo project from Vercel and trying to use it with Gemini.

As soon as I enable tools (with @ai/tools from vercel) every request fails with 500.

import { createVertex } from "@ai-sdk/google-vertex"

import { convertToCoreMessages, streamText, tool } from "ai"

const vertex = createVertex({
  project: process.env.GOOGLE_VERTEX_PROJECT,
  location: "us-east1",
})

export const maxDuration = 30

export async function POST(req: Request) {
  const { messages } = await req.json()

  const result = await streamText({
    model: vertex("gemini-1.5-pro"),
    messages: convertToCoreMessages(messages),
    system: `You are a helpful assistant.`,
    // tools: {
    // getInformation: tool({
    //   description: `get information from your knowledge base to answer questions.`,
    //   parameters: z.object({
    //     question: z.string().describe("the users question"),
    //   }),
    //   execute: async ({ question }) => findRelevantContent(question),
    // }),
  })

  return result.toAIStreamResponse()
}

Works with tools commented out, fails if enable tools.

[happy-qiao]

Cannot reproduce 500 error now. I use the following script and it returns 200

import {
    VertexAI
  } from '@google-cloud/vertexai';

(async function() {
    const project = 'vertexsdk';
    const datastoreId = 'hello_1724863936046';
    const vertexAI = new VertexAI({
        project: project,
        location: 'us-central1'
    });
    const generativeModel = vertexAI.preview.getGenerativeModel({
        model: 'gemini-1.5-flash-001',
        generationConfig: {
            maxOutputTokens: 8192,
            temperature: 0,
            topP: 1,
        },
        tools: [
            {
                retrieval: {
                    vertexAiSearch: {
                        datastore: `projects/${project}/locations/global/collections/default_collection/dataStores/${datastoreId}`,
                    },
                    disableAttribution: false,
                },
            }
        ]
    });
    const result = await generativeModel.generateContent({
        contents: [{
            role: 'user',
            parts: [{
                text: 'what\'s the weather today?'
            }]
        }]
    });
    console.log(JSON.stringify(result, null, 2));
}
)();

[SebSchroen]

Even though the error message didn't point in that direction, it essentially was a permission issue for me. Giving the Service Account who is calling the model the role "Discovery Agent Viewer" fixed the problems. You also get the internal error if the location of the project and the datastore don't match (e.g. europe-west1 + global). @jwilson-tower

[jwilson-tower]

Thank you @SebSchroen! I added the "Discovery Engine Viewer" to the service account and it fixed the issue.