Issue with not detecting objects on IOS

[Tamer7]

Hi, so i cloned the IOS repo to my mac and opened the project workfile, it downloads smoothly to my ios device however anytime i try to make a detection it does not detect any object, and when i try to see what the console logs out, it just shows an empty bracked list [ ]. Any idea what i can do?

[megatunger]

You need to start the backend first in here: https://github.com/iamvon/viBlind/tree/master/Backend/AI

[Tamer7]

Anytime i try to run the server, without docker i keep facing the same error, which says "invalid syntax" and points to async=false. I attached a picture above not sure if you can see it. Not sure what went wrong there as i installed the requirements.txt.

[megatunger]

You should use a lower python version (<3.7) or you can try upgrade pyexpect pexpect/pexpect#453

[Tamer7]

Thanks for the information, i upgraded to the latest version of pyexpect, however another error came upon as i tried to run the server, firstly im not exactly sure why your path is showing instead of mine, and i don't think i see the yolov3.weights anywhere in the source code? Any idea how i can fix this?

[iamvon]

yolov3.weights is a pre-trained model for real-time object detection. You can download it at this link. I do not push it in this source because of the large size of the file.
If you want to learn more about it, feel free to read it here: https://pjreddie.com/darknet/yolo/
Hope this may help.

[Tamer7]

Thanks for the comment, so basically i add this file to the object_detection folder, where the yoloModel.py is? Is that correct?

[iamvon]

That's is correct.

[Tamer7]

Hey, sorry for bothering, i included the yolo.weights in the object_detection folder, but it seems it did not change anything. Im still getting the same error as earlier, which is this below? Any idea what i could do?

Here is the error.

**_[INFO] loading YOLO from disk...
Traceback (most recent call last):
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/bin/flask", line 8, in
sys.exit(main())
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 894, in main
cli.main(args=args, prog_name=name)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 557, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 767, in run_command
app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 293, in init
self._load_unlocked()
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 317, in _load_unlocked
self.app = rv = self.loader()
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 372, in load_app
app = locate_app(self, import_name, name)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 235, in locate_app
import(module_name)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/server/api.py", line 26, in
net, ln = yoloModel.load_model()
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/ml_core/object_detection/yoloModel.py", line 26, in load_model
net = cv2.dnn.readNetFromDarknet("yolov3.cfg", "yolov3.weights")
cv2.error: OpenCV(4.1.0) /Users/travis/build/skvark/opencv-python/opencv/modules/dnn/src/darknet/darknet_importer.cpp:214: error: (-212:Parsing error) Failed to parse NetParameter file: yolov3.weights in function 'readNetFromDarknet'

[Tamer7]

I think i might have figured the above issue out, by replacing the yolov3.weights name with the path of the download. However it seems i get another error i assume now coming from the answering_question folder which is a file called single.mdl. Which i cannot find anywhere. Error is below.

[iamvon]

You need to import another pre-trained model for the module question answering. Here we use DrQA from https://github.com/facebookresearch/DrQA.
You can download it at this link and add this file to folder https://github.com/iamvon/viBlind/tree/master/Backend/AI/ml_core/question_answering/data/reader.

[Tamer7]

Thanks for that, that issue is now solved, however i've faced another issue were it gives me a NameError that SpacyTokenizer is not defined. I checked the requirements.txt and it seems there is a pip for "spacy". Not sure if i missing something. Below is the error.

_**[INFO] loading YOLO from disk...
Done
Traceback (most recent call last):
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/bin/flask", line 8, in
sys.exit(main())
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 894, in main
cli.main(args=args, prog_name=name)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 557, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, kwargs)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 767, in run_command
app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 293, in init
self._load_unlocked()
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 317, in _load_unlocked
self._app = rv = self.loader()
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 372, in load_app
app = locate_app(self, import_name, name)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/venv/lib/python3.7/site-packages/flask/cli.py", line 235, in locate_app
import(module_name)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/server/api.py", line 30, in
predictions = predictor.Predictor(model=None, tokenizer=token, normalize=True, embedding_file=None, num_workers=None)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/ml_core/question_answering/drqa/reader/predictor.py", line 74, in init
tokenizer_class = tokenizers.get_class(tokenizer)
File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/ml_core/question_answering/drqa/tokenizers/init.py", line 33, in get_class
return SpacyTokenizer
NameError: name 'SpacyTokenizer' is not defined

[iamvon]

You need at least a tokenizer for DrQA works, refer to: https://github.com/facebookresearch/DrQA#tokenizers.
Maybe spacy missing in the setup, in that case, you need to install it manually: https://pypi.org/project/spacy/

[Tamer7]

Done that, however keep on coming up with errors, im sorry for constantly bugging, but really want to start the backend and im not so familiar with the errors. So im getting this error now. It saya unexpected argument 'parser' in the spacy_tokenizer.py

File "/Users/tamerjar/Desktop/viBlind-master/Backend/AI/ml_core/question_answering/drqa/tokenizers/spacy_tokenizer.py", line 32, in init
self.nlp = spacy.load(model, **nlp_kwargs)
TypeError: load() got an unexpected keyword argument 'parser'

[Tamer7]

Hey, i think i might have figured it out by installing a lower version of spacy, however, is this the correct url of the server? This is what is shows up when i run the server, haven't yet tested with the app.

[iamvon]

Glad to hear that you've found a way to fix it yourself. It's always a version error.
For testing the backend, you can follow my small docs here: https://github.com/iamvon/viBlind/tree/master/Backend/AI#api-references

[iamvon]

Maybe you'll need to run this python client for testing the Flask backend: https://github.com/iamvon/viBlind/blob/master/Backend/AI/client.py

[Tamer7]

Thanks for the information, i did so and the back end seems to work perfectly, it displays the json output as it should. However, while running the server and the mobile application simultaneously, it does not seem to be calling the api. As when im trying to take a picture on the device it does the same thing it was doing earlier. When checking xcode terminal it still shows empty bracket. So not sure if im supposed to do something in the xcode file.

This is the output xcode gives me when i try taking a picture, please note the api is running and does not get any requests sent to it.

[megatunger]

Did you change the endpoint in here https://github.com/iamvon/viBlind/blob/master/App/iOS/BlindVision/BlindVision/Resources/Extensions/1%20-%20Vision%20View/CallAPI.swift

let APIEndpoint: String = "http://52.163.230.167:5000/v1/api/predict"

https://github.com/iamvon/viBlind/blob/master/App/iOS/BlindVision/BlindVision/Resources/Extensions/2%20-%20News%20View/CallNewsAPI.swift

https://github.com/iamvon/viBlind/blob/master/App/iOS/BlindVision/BlindVision/Resources/Extensions/3%20-%20Voice%20Control/CallAnswerAPI.swift

[Tamer7]

Yeah, so i changed each api url in the code to have "http://0.0.0.0:5000/" this on the start rather than "http://52.163.230.167:5000". But still nothing happens, i check the terminal were the api is running but still no requests seem to be going through, and in the console of xcode it shows me up with this error, which im not really sure what it means, any idea if im doing anything wrong?

[megatunger]

You should change to your current machine IP (eg: 192.168.2.104:5000) rather than 0.0.0.0

[Tamer7]

Okay, so for example my mac ip address is : 192.168.1.6, so instead of having "http://0.0.0.0:5000/v1/api/predict" how do i connect the ip address to the api? As i tried to format it in this way "http://192.168.1.6/v1/api/predict", but still didnt work it gave me this, i think i might be doing wrong so any help would be appreciated.

[iamvon]

It seems that the client application is not pointing correctly to the backend, have you tried replacing 0.0.0.0 with 127.0.0.1 or localhost yet ?

[Tamer7]

Hi again! thanks for the response, i managed to configure the IP successfully. And the Object detection in the mobile application works smoothly. However i seem to be getting issues when clicking on the news tab. It shows me the following error.

And once checking the backend server, i can see that a request is trying to be made, but do't really understand the probelm.

Lastly i also have question, i noticed there is a microphone button next to the news tab, what is the major function of that? As i can see from my xcode console that it detects the words im saying once clicked, but does not do anything. Your help will be appreciated for those matters. Thanks in advance.

[iamvon]

It looks like you handled it by configuring the phone and the backend on the same network, I almost forgot about this. Great!
About the News Reader feature, you need to start another backend. This backend is written in golang and I use it to scrape and get data from The New York Times. You can find out the way to run that backend in here: https://github.com/iamvon/viBlind/tree/master/Backend/News/nyTimes

[Tamer7]

Thanks for the response, so trying to run the nytimes backend but getting faced with this error, Am I missing any files here?

[iamvon]

Document of this backend is missing, so I'll explain it.
Firstly, you must create a database (MySQL in this case) and run the crawler to scrape data from The New York Times. After you have data in the database then you can run the golang server to send news data to the application.
This can be a bit complicated, you can do it simply by writing a Flask server in python by yourself that returns a list of articles (mock data for testing) via API follow this format.

[Tamer7]

So basically to get this in order,

• First i create a database in mysql, with colums (id, date, topic, description etc) according to the api format. My question here is where do i connect this database to?
• Then i run the crawler which is the (./runCrawler.sh), when running it now without creating the database i encounter this error.
  Crawler/rssCrawler.go:8:2: no required module provides package github.com/go-sql-driver/mysql: go.mod file not found in current directory or any parent directory; see 'go help modules'
  Crawler/rssCrawler.go:9:2: no required module provides package github.com/gocolly/colly: go.mod file not found in current directory or any parent directory; see 'go help modules'
  Crawler/rssCrawler.go:10:2: no required module provides package github.com/mmcdole/gofeed: go.mod file not found in current directory or any parent directory; see 'go help modules'
  Crawler/rssCrawler.go:13:2: package nyTimes/Rss is not in GOROOT (/usr/local/go/src/nyTimes/Rss)
  Crawler/rssCrawler.go:14:2: package nyTimes/db_helper is not in GOROOT (/usr/local/go/src/nyTimes/db_helper)
  mv: rename rssCrawler to Output/: No such file or directory
  ./runCrawler.sh: line 6: ./Output/rssCrawler: No such file or directory

So what i'm assuming is to link the database to the crawler file is that correct, and a question can i use (phpmyadmin) for the database?

[Tamer7]

Hey again guys, im trying to run the client_for_nlpAPI.py file here : https://github.com/iamvon/viBlind/blob/master/Backend/AI/client_for_nlpAPI.py

But i seem to be getting this error down here, this is the mic button in the application that answers questions, is that correct?

Traceback (most recent call last):
File "client_for_nlpAPI.py", line 33, in
res = post_response.json()
File "/Users/tamerjar/opt/anaconda3/envs/tester/lib/python3.6/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/Users/tamerjar/opt/anaconda3/envs/tester/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/Users/tamerjar/opt/anaconda3/envs/tester/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Users/tamerjar/opt/anaconda3/envs/tester/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[Tamer7]

Hey guys i was wondering if you have any sort of Readme or detailed description of how the backend works, as I'm trying to understand how the backend works and connected to each other. That would be helpful if available! Thanks in advance!