502 Bad Gateway #284
Replies: 18 comments 27 replies
-
|
Did you get 502 when you went to the cloud console, or when you went to https://project_id.region_id.r.appspot.com/? If you got the 502 when you went to ...appspot..., check the logs in the cloud console. You might see a stack trace there. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your response. I got the 502 fixed. And I am able to login
using the FTC fmltc login page. However, I am getting the following error
that I cannot fix for 2 days. I am new to Google Cloud and I appreciate
your help. I got this error right after I sign in in fmltc login page:
Thank you again for your help.
…On Mon, Oct 10, 2022 at 12:55 PM Liz Looney ***@***.***> wrote:
Did you get 502 when you went to the cloud console, or when you went to
https://project_id.region_id.r.appspot.com/?
If you got the 502 when you went to ...appspot..., check the logs in the
cloud console. You might see a stack trace there.
—
Reply to this email directly, view it on GitHub
<#284 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDR265F5L4W2M7EXD3DWCRJ7RANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
I don't see what error you got. Did you forget to put it in the message here? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Liz:
I got the following error message:
iam.gserviceaccount.com does not have storage.buckets.get access to the
Google Cloud Storage bucket.'
<https://serverfault.com/questions/1112762/error-iam-gserviceaccount-com-does-not-have-storage-buckets-get-access-to-the-g>
Please see the attached screenshot. Thank you very much!
…On Mon, Oct 10, 2022 at 8:26 PM Liz Looney ***@***.***> wrote:
I don't see what error you got. Did you forget to put it in the message
here?
—
Reply to this email directly, view it on GitHub
<#284 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDR47YI5MJJU4VV6RMTWCS63NANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
This is helpful! So the name of the storage bucket is ftc10138powerplay-blobs. That's good. Hold on a second. It will take me a few minutes to look up the URL you need to go to on the cloud console to fix this. |
Beta Was this translation helpful? Give feedback.
-
|
Go to https://console.cloud.google.com/iam-admin/iam?project=ftc10138powerplay. In the center area there should be a heading "VIEW BY PRINCIPALS" |
Beta Was this translation helpful? Give feedback.
-
|
It is "Computer Storage Admin". I cannot find any "Storage Legacy Bucket.XXX" role in IAM & Admin's role selection box. I filtered "storage". Apparently "Computer Storage Admin" is one of the option. I just double checked again. I will switch to "Storage Admin" and try again. Thank you very much! |
Beta Was this translation helpful? Give feedback.
-
|
I tried again. Now I got another error: FailedPrecondition('no matching index found.'). I retype the Deploy step1 in ReadMe: scripts/deploy_indexes.sh. I still got the FailedPrecondition('no matching index found.'). Attached are the screenshot for current updated IAM & Admin principal's role "ftc10138powerplay-serv-acct@ftc10138powerplay.iam.gservice.account.com." and the login error screenshot. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Liz:
This morning I got a different error after I logged in. Please advise or
give some hint where I should start to fix it. Attached is the screenshot:
Thank you very much!
…On Mon, Oct 10, 2022 at 8:26 PM Liz Looney ***@***.***> wrote:
I don't see what error you got. Did you forget to put it in the message
here?
—
Reply to this email directly, view it on GitHub
<#284 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDR47YI5MJJU4VV6RMTWCS63NANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
It looks like you didn't create the database indexes. |
Beta Was this translation helpful? Give feedback.
-
|
Apparently, github does not accept email attachments. So if you want to attach screenshots, you need to go to this discussion and drag/drop or paste them into a reply message. |
Beta Was this translation helpful? Give feedback.
-
|
Liz:
So I opened the "Cloud Storage/Bucket" site and try to assign the roles to the principals. Here is my current bucket list: These buckets were created in ReadMe step #10. Please review the list as they are correctly created as instructed in ReadMe. I attached the screen print of the permission in the "ftc10138powerplay" bucket.
Please advise and let me know how can I fix this issue. Thank you very much! |
Beta Was this translation helpful? Give feedback.
-
|
In your first screenshot, you are looking at the permissions for the wrong bucket. The bucket that is the problem is ftc10138powerplay-blobs. In the second screenshot, I can see that the principal ftc10138powerplay-serv-acct@ftc10138powerplay.iam.gserviceaccount.com does NOT have the Storage Admin Role. You need to do this: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you Liz. I modified the roles. The error is still here: FailedPrecondition('no matching index found.'). Do I need to rerun the source env_setup.sh & scripts/deploy_indexes.sh before each "gcloud app browser"? Here is the screen print for IAM & Admin: Here is the screen print for ftc10138powerplay-blobs's bucket: |
Beta Was this translation helpful? Give feedback.
-
|
OK. So I think Google Cloud Storage is good now. The indexes are for the firestore database. Unfortunately, I don't know how to see those in the cloud console. (In my own fmltc projects, the Cloud Console page for Firestore Indexes says "No rows to display". I've never understood why.) Instead, let's look at the logs from appengine. Here's what to do: |
Beta Was this translation helpful? Give feedback.
-
|
Here is the log message: |
Beta Was this translation helpful? Give feedback.
-
|
Or, instead of a screenshot, can you copy and paste the text? That way I copy the text to an editor where I can reformat it to see it better. |
Beta Was this translation helpful? Give feedback.
-
|
Here is the log message: |
Beta Was this translation helpful? Give feedback.
-
|
Great! That error is exactly the problem. Here's how I investigate this. Then I look at that code and see that it's doing a query with filters on the Then I look in the index file and see that the index we are trying to use is declared on line 4. So, that's all good. Now, when you ran was there any output or errors? |
Beta Was this translation helpful? Give feedback.
-
|
I got the following for source env_setup.sh: I got no output for scripts/deploy_indexes.sh |
Beta Was this translation helpful? Give feedback.
-
|
OK. It looks like the deploy_indexes.sh script doesn't deploy if it thinks it already did it. Please do The output should be something like this: |
Beta Was this translation helpful? Give feedback.
-
|
Please Wait for 20 mins I am outside. Will do asap.
…On Tue, Oct 11, 2022 at 3:30 PM Liz Looney ***@***.***> wrote:
OK. It looks like the deploy_indexes.sh script doesn't deploy if it thinks
it already did it.
Please do
source env_setup.sh
gcloud -q datastore indexes create index.yaml
The output should be something like this:
Configurations to update:
descriptor: [index.yaml]
type: [datastore indexes]
target project: [ftc10138powerplay]
....done.
—
Reply to this email directly, view it on GitHub
<#284 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDRPDOW2OAQ57RIBOH3WCXE6BANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
Sure! |
Beta Was this translation helpful? Give feedback.
-
|
Liz, I got the screen: Thank you for your help. I will do the testing tonight. Have a nice evening. Really appreciate your help, Liz! |
Beta Was this translation helpful? Give feedback.
-
|
That's fantastic!!! I will update the readme for these two problems:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, Liz: |
Beta Was this translation helpful? Give feedback.
-
|
The further down error message: Both of Cloud Firestore/Datastore API shown the activity around 10/14, both displayed 200 as color blue(middle of page, far right, )... I manually create database Firestore on Native mode in fmltc's ReadMe. I never create Firestore on Datastore mode. I do not know why both Cloud Firestore/Datastore API displayed and recorded activities:
|
Beta Was this translation helpful? Give feedback.
-
|
It looks like there were changes database configuration in google side after fmltc was build in last year. Firestore can be created by command line in 2022 and default to Native mode. https://cloud.google.com/sdk/gcloud/reference/firestore/databases/create. Does creating index is no longer required in Firestore...? Besides the deploy index command line, there are multiple places in current fmltc have Datastore information. For example in main.tf, My question is, the current fmltc on FTC site works fine. What is the difference between the current instructions on ReadMe.md and the actual configuration and implementation on the fmltc application? I also knew someone who was able to train the datasets from fmltc last year (2021), any documented changes about connect to database in fmltc? Really appreciate if you have any idea how to fix this issue. Thank you very much! |
Beta Was this translation helpful? Give feedback.
-
|
Understand now. It is great to learn from you and this processing . There
is one last thing if you might interested:
There is error after I rerun the actual command in deploy_indexes.sh,
“Failed Precondition-the index for the query is not ready to serve…”. The
was solved by assigning all principles the role of owner. I googled and it
works for me. I am not sure why… is there a document you can recommend to
me for brief details of the what we did today? It’s okay if you don’t, I
can try to google. Really appreciate your time and help today!
…On Tue, Oct 11, 2022 at 5:12 PM Liz Looney ***@***.***> wrote:
That's fantastic!!!
I will update the readme for these two problems:
1. I'll add a command to give the Storage Admin role to the service
account.
2. I'll document that deploying the indexes takes a long time and is
asynchronous, so the indexes might not be ready until several hours (a
day?) after running deploy_indexes.sh.
—
Reply to this email directly, view it on GitHub
<#284 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDVJVR7YLLR4TW5ISHTWCXQ3HANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
I think that the “Failed Precondition-the index for the query is not ready to serve…”. error just indicates that (as it says) the index is not ready. I believe that adding the role of owner to all principles didn't make it work... It's just that by the time you'd done that, the index was ready. For documentation about what we did today: I tried to find documentation about the indexes, but I can't find any documentation that makes sense. I think when I created fmltc, I wanted to use Datastore and then Google replaced that with Firestore. I think for a while I was using "Firestore in Datastore Mode", but the cloud console says I'm using "Firestore in Native Mode". I think the index.yaml file and the command that is in deploy_indexes.sh are typically used for Datastore. I'm not even sure why that command works if the database is actually Firestore in Native Mode. I'll see if I can figure that out. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Liz: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the update and the document , Liz.
Just wondering, do you work with FTC or Google Cloud? Our team want to
train dataset more than team element this year. Is there a limit on how
many model we can train on google cloud? Second question is once we get
the models (assume more than one), does the running and detecting object
using tensor flow fast enough in the human drive period?
It’s interesting to see how the index.yaml file work with database.
Thank you again. Have a nice evening,
…On Tue, Oct 11, 2022 at 7:07 PM Liz Looney ***@***.***> wrote:
I think that the “Failed Precondition-the index for the query is not ready
to serve…”. error just indicates that (as it says) the index is not ready.
I believe that adding the role of owner to all principles didn't make it
work... It's just that by the time you'd done that, the index was ready.
For documentation about what we did today:
IAM roles for Cloud Storage
<https://cloud.google.com/storage/docs/access-control/iam-roles>
I tried to find documentation about the indexes, but I can't find any
documentation that makes sense. I think when I created fmltc, I wanted to
use Datastore and then Google replaced that with Firestore. I think for a
while I was using "Firestore in Datastore Mode", but the cloud console says
I'm using "Firestore in Native Mode". I think the index.yaml file and the
command that is in deploy_indexes.sh are typically used for Datastore. I'm
not even sure why that command works if the database is actually Firestore
in Native Mode. I'll see if I can figure that out.
—
Reply to this email directly, view it on GitHub
<#284 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDSGPTBTYCVIPGF4KGTWCX6KJANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
I am on the FTC Tech Team, a group of volunteers who help the engineers at FIRST to create the FTC SDK and fmltc. When you use your own google cloud project (ftc10138powerplay), the limit is money. You may have received a certain amount of free credits from Google when you created the project. When those run out, it will start costing real money. As to whether it is fast enough during the tele-op part of the matches, that's a good question. In the FTC SDK, there's an object tracker that is used to track objects on the camera frames that come in while the tensorflow object detection is busy. That makes it pretty fast. You'll have to try it out and see. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome, thanks for the info! Again, really appreciate you time and
help with this!
…On Tue, Oct 11, 2022 at 8:01 PM Liz Looney ***@***.***> wrote:
I am on the FTC Tech Team, a group of volunteers who help the engineers at
FIRST to create the FTC SDK
<https://github.com/FIRST-Tech-Challenge/FtcRobotController> and fmltc
<https://github.com/FIRST-Tech-Challenge/fmltc>.
When you use your own google cloud project (ftc10138powerplay), the limit
is money. You may have received a certain amount of free credits from
Google when you created the project. When those run out, it will start
costing real money.
In my experience, it isn't that expensive. In the project I use for
developing fmltc, the cost of using the service to upload videos, label
frames, create datasets, and train about 5 models was about $37. I'm not
sure if my usage is typical though because I usually don't care if my
models work very well. I don't usually have as many video frames as some
people do.
As to whether it is fast enough during the tele-op part of the matches,
that's a good question. In the FTC SDK, there's an object tracker that is
used to track objects on the camera frames that come in while the
tensorflow object detection is busy. That makes it pretty fast. You'll have
to try it out and see.
—
Reply to this email directly, view it on GitHub
<#284 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDVEG3Z6X6FQ4LXLTS3WCYEVTANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
I am glad I am able to run the fmltc now after found out what cause the errors existed in past couple days. However, I got new errors when I clicked the "Start training" button. I am still not very familiar with the IAM and the error seems came from the IAM. I have more than 5 principals in IAM. I am not sure which one I need to check the permission. Just in case you have any clue about the following error message: CRITICAL:root:capture_exception traceback: Traceback (most recent call last): ... File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request ... rv = self.dispatch_request() ... File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request ... return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) ... File "/workspace/wrappers.py", line 85, in wrapper ... return func(*args, **kwargs) ... File "/workspace/wrappers.py", line 45, in wrapper ... return func(*args, **kwargs) ... File "/workspace/app_engine.py", line 1290, in start_training_model ... model_entity = model_trainer.start_training_model(team_uuid, description, dataset_uuids, ... File "/workspace/model_trainer.py", line 307, in start_training_model ... eval_job_response = ml.projects().jobs().create(parent=parent, body=eval_job).execute() ... File "/layers/google.python.pip/pip/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper ... return wrapped(*args, **kwargs) ... File "/layers/google.python.pip/pip/lib/python3.9/site-packages/googleapiclient/http.py", line 907, in execute ... raise HttpError(resp, content, uri=self.uri) ... googleapiclient.errors.HttpError: <HttpError 403 when requesting https://ml.googleapis.com/v1/projects/XXXXpowerplay/jobs?alt=json returned "Access to project denied. This might be a transient error and a retry may succeed. If the error persists, please check the IAM permissions on your project."> . Thank you very much. |
Beta Was this translation helpful? Give feedback.
-
|
I sounds like the app engine service account does not have permission to create a machine learning job. |
Beta Was this translation helpful? Give feedback.
-
|
I was spending a while on searching on the API, permission and fmltc project for any clue... I reviewed the ftc fmlt instruction again and I was able to find something that I didn't understand when built up the whole application for deployment. It was up and running now. Thank you for your reply, Liz! |
Beta Was this translation helpful? Give feedback.
-
|
Liz: |
Beta Was this translation helpful? Give feedback.
-
Barbara: As we move to more general questions about model training, and less specifically about hosting your own instance of fmltc, it is probably more appropriate to ask them over in the community forum where we have dedicated categories for machine learning. This question in particular is one that I think many people may have and the answer would be a useful addition to the forum's knowledge base. The ftc community forums are here: https://ftc-community.firstinspires.org/ |
Beta Was this translation helpful? Give feedback.
-
|
You cannot combine TFLITE files. If you think about it like neural networks, which is exactly what it is, one neural network might be incredibly tuned for one thing but they need special training to not recognize other similar objects as that thing. That's why you need to create a dataset with ALL of the labels you want - during training the model will then learn to differentiate between objects. If you have videos labeled with individual labels, try creating a dataset with ALL of your videos. That's how I did it with all 3 images. That's assuming, of course, that your videos individually don't include other images. If so, you need to label them or else you might confuse the network training. Anything not labeled is specifically marked to be ignored, and if one video frame has something ignored and another video frame has something labeled, that's going to lead to a bad model. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your reply and advice. That is very helpful!
…On Fri, Oct 21, 2022 at 11:44 AM Danny Diaz ***@***.***> wrote:
You cannot combine TFLITE files. If you think about it like neural
networks, which is exactly what it is, one neural network might be
incredibly tuned for one thing but they need special training to not
recognize other similar objects as that thing. That's why you need to
create a dataset with ALL of the labels you want - during training the
model will then learn to differentiate between objects. If you have videos
labeled with individual labels, try creating a dataset with ALL of your
videos. That's how I did it with all 3 images.
That's assuming, of course, that your videos individually don't include
other images. If so, you need to label them or else you might confuse the
network training. Anything not labeled is specifically marked to be
ignored, and if one video frame has something ignored and another video
frame has something labeled, that's going to lead to a bad model.
—
Reply to this email directly, view it on GitHub
<#284 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABU4ZDVROQIL4CZ62IBQ6ITWELB5FANCNFSM6AAAAAARALYLDA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, Liz: We tested on private fmltc instance and when the model failed, we tested on FTC's fmltc instance and it worked. The batch size is different between the two instances. Maybe when we switch from TPU to GPU in private fmltc instance, the smaller batch size/ more steps work better? Would you please advice if we can do anything in IAM and assign the admin role to firestore? Thank you |
Beta Was this translation helpful? Give feedback.
-
|
@lizlooney Specifically, I think @barbaralau3 is trying to edit the Firestore to be able to create a Configuration, whereas she can then set the |
Beta Was this translation helpful? Give feedback.
-
I am new the google cloud so I followed the https://github.com/FIRST-Tech-Challenge/fmltc instructions as careful as I can. I got 502 Bad Gateway after I went to my https://console.cloud.google.com/appengine?project=YOUR-PROJECT-ID. The ending url is https://project_id.region_id.r.appspot.com/ with 502 Bad Gateway in website. Does anyone know what I can do to correct this mistake?
Thank you very much!
Beta Was this translation helpful? Give feedback.
All reactions