MediCrawl

A Web Search Engine for Diseases.

Prerequisites

Git: https://git-scm.com/
Git-LFS: https://com.puter.tips/2020/04/install-git-lfs-in-macos.html
Solr: https://lucene.apache.org/solr/downloads.html
Nutch: http://nutch.apache.org/downloads.html

Components of a web search engine

Functionality	Module Type	Software
Crawling	Web crawler	Nutch
Indexing	Database	Solr
Ranking	Database	Solr
Searching	User interface	HTML / JS / CSS

Procedure

To launch the preconfigured setup just run this command: solr/bin/solr start -Dsolr.ltr.enabled=true or you can download Nutch and Solr manually and perform the following operations:

Download and extract Nutch and Solr in nutch and solr directories
Modify nutch/conf/nutch-site.xml: set http.agent.name and indexr-solr property
Modify nutch/conf/schema.xml
Create a URL seed list: nutch/urls/seed.txt
cp -r solr/server/solr/configsets/_default solr/server/solr/configsets/nutch
cp nutch/conf/schema.xml solr/server/solr/configsets/nutch/conf
rm solr/server/solr/configsets/nutch/conf/managed-schema
Modify solr/server/solr/configsets/nutch/conf/solrconfig.xml to enable LTR and change _text_ to text
start solr with LTR: solr/bin/solr start -Dsolr.ltr.enabled=true
set JAVA_HOME environment variable
Crawl and Index simultaneously: nutch/bin/crawl -i -D solr.server.url=http://localhost:8983/solr/nutch -s nutch/urls/ Crawl 3
Test if it succeeded: curl 'http://localhost:8983/solr/nutch/query?q=breast+cancer&fl=url,title'
Create /path/features.json and /path/model.json
Upload features to Solr: curl -XPUT 'http://localhost:8983/solr/nutch/schema/feature-store' --data-binary "@./path/features.json" -H 'Content-type:application/json'
View uploaded features: curl 'http://localhost:8983/solr/nutch/schema/feature-store/_DEFAULT_'
Test feature extraction: curl 'http://localhost:8983/solr/nutch/query?q=test&fl=id,score,[features]'
Upload model to Solr: curl -XPUT 'http://localhost:8983/solr/nutch/schema/model-store' --data-binary "@./path/model.json" -H 'Content-type:application/json'
View uploaded model: curl 'http://localhost:8983/solr/nutch/schema/model-store'
Test model: curl -g 'http://localhost:8983/solr/nutch/query?q=breast+cancer&rq={!ltr%20model=myModel%20efi.query=breast+cancer}&fl=url,title,[features]'

Examples

Plain Query: http://localhost:8983/solr/nutch/query?q=breast+cancer&fl=url,title

{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"breast cancer",
      "fl":"url,title"}},
  "response":{"numFound":1122,"start":0,"docs":[
      {
        "title":"Breast cancer - Symptoms and causes - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/breast-cancer/symptoms-causes/syc-20352470"},
      {
        "title":"Breast cancer - Diagnosis and treatment - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/breast-cancer/diagnosis-treatment/drc-20352475"},
      {
        "title":"Breast cancer - Care at Mayo Clinic - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/breast-cancer/care-at-mayo-clinic/mac-20352479"},
      {
        "title":"Breast cancer: Symptoms, causes, and treatment",
        "url":"https://www.medicalnewstoday.com/articles/37136"},
      {
        "title":"Breast Clinic - Overview - Mayo Clinic",
        "url":"https://www.mayoclinic.org/departments-centers/breast-clinic/sections/overview/ovc-20459469"},
      {
        "title":"Bring Your Brave.",
        "url":"https://bringyourbrave.tumblr.com/"},
      {
        "title":"Breast Cancer News from Medical News Today",
        "url":"https://www.medicalnewstoday.com/categories/breast-cancer"},
      {
        "title":"Breast Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/breast/"},
      {
        "title":"Breast cancer - Doctors and departments - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/breast-cancer/doctors-departments/ddc-20352478"},
      {
        "title":"Breast Cancer | Disease of the Week | CDC",
        "url":"https://www.cdc.gov/dotw/breastcancer/index.html"}]
  }}

Features model: http://localhost:8983/solr/nutch/query?q=breast+cancer&rq={!ltr%20model=myModel%20efi.query=breast+cancer}&fl=url,title,[features]

{
  "responseHeader":{
    "status":0,
    "QTime":8,
    "params":{
      "q":"breast cancer",
      "fl":"url,title,[features]",
      "rq":"{!ltr model=myModel efi.query=breast cancer}"}},
  "response":{"numFound":1122,"start":0,"docs":[
      {
        "title":"Science Clips - Volume 12, Issue 10, March 23, 2020",
        "url":"https://www.cdc.gov/library/sciclips/issues/index.html",
        "[features]":"originalScore=2.2747984,titleLength=9.0,contentLength=22552.0"},
      {
        "title":"Science Clips - Volume 12, Issue 10, March 23, 2020",
        "url":"https://www.cdc.gov/library/sciclips/issues/",
        "[features]":"originalScore=2.2747984,titleLength=9.0,contentLength=22552.0"},
      {
        "title":"Breast cancer - Diagnosis and treatment - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/breast-cancer/diagnosis-treatment/drc-20352475",
        "[features]":"originalScore=3.7579455,titleLength=7.0,contentLength=5144.0"},
      {
        "title":"The Topic Is Cancer | Blogs | CDC",
        "url":"https://blogs.cdc.gov/cancer/",
        "[features]":"originalScore=3.5015173,titleLength=6.0,contentLength=5144.0"},
      {
        "title":"All Issues - Mayo Clinic Health Letter",
        "url":"https://healthletter.mayoclinic.com/issues",
        "[features]":"originalScore=3.071907,titleLength=6.0,contentLength=5144.0"},
      {
        "title":"Menopause Treatment, Signs, Symptoms & Age",
        "url":"https://www.medicinenet.com/menopause/article.htm",
        "[features]":"originalScore=2.6457264,titleLength=5.0,contentLength=5144.0"},
      {
        "title":"Hormone Therapy for Women: Side Effects, Cancer Risks",
        "url":"https://www.medicinenet.com/hormone_therapy/article.htm",
        "[features]":"originalScore=3.3211832,titleLength=8.0,contentLength=4632.0"},
      {
        "title":"Cáncer de mama - Atención en Mayo Clinic - Mayo Clinic",
        "url":"https://www.mayoclinic.org/es-es/diseases-conditions/breast-cancer/care-at-mayo-clinic/mac-20352479",
        "[features]":"originalScore=3.6176624,titleLength=9.0,contentLength=4120.0"},
      {
        "title":"Hot Flashes Causes, Symptoms & Treatment Medicine for Men & Women",
        "url":"https://www.medicinenet.com/hot_flashes/article.htm",
        "[features]":"originalScore=2.8397057,titleLength=9.0,contentLength=3864.0"},
      {
        "title":"Breast cancer - Care at Mayo Clinic - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/breast-cancer/care-at-mayo-clinic/mac-20352479",
        "[features]":"originalScore=3.7574568,titleLength=8.0,contentLength=3608.0"}]
  }}

Pagination - First page twenty queries: http://localhost:8983/solr/nutch/query?q=cancer&fl=url,title&start=0&rows=20

{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"cancer",
      "fl":"url,title",
      "start":"0",
      "rows":"20"}},
  "response":{"numFound":1083,"start":0,"docs":[
      {
        "title":"Key Statistics for Prostate Cancer | Prostate Cancer Facts",
        "url":"https://www.cancer.org/cancer/prostate-cancer/about/key-statistics.html"},
      {
        "title":"Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/"},
      {
        "title":"Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/index.htm"},
      {
        "title":"Print Materials About Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/publications/"},
      {
        "title":"Other Cancer Data Sources | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/data/other.htm"},
      {
        "title":"Cancer Data and Statistics | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/data/"},
      {
        "title":"Kinds of Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/kinds.htm"},
      {
        "title":"Cancer Center: Types, Symptoms, Causes, Tests, and Treatments, Including Chemo and Radiation",
        "url":"https://www.webmd.com/cancer/default.htm"},
      {
        "title":"Risk Factors and Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/risk_factors.htm"},
      {
        "title":"Cancer Resource Library | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/resources/"},
      {
        "title":"Cancer Article Summaries | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/index.htm"},
      {
        "title":"Cancer Article Summaries Published in 2017 | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/cancer-article-summaries-2017.html"},
      {
        "title":"Educational Campaigns About Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/about/campaigns.htm"},
      {
        "title":"National Programs to Prevent and Control Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/about/programs.htm"},
      {
        "title":"Annual Report to the Nation on the Status of Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/annual-report/index.htm"},
      {
        "title":"How to Prevent Cancer or Find It Early | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/prevention/"},
      {
        "title":"Cancer Research | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/index.htm"},
      {
        "title":"Healthy People 2020 Targets | Annual Report to the Nation | CDC",
        "url":"https://www.cdc.gov/cancer/annual-report/healthy-people-2020-targets.htm"},
      {
        "title":"Cancer Screening Tests | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/prevention/screening.htm"},
      {
        "title":"Cancer Survival in the United States | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/concord-2-supplement.htm"}]
  }}

Pagination - Second page twenty queries: http://localhost:8983/solr/nutch/query?q=cancer&fl=url,title&start=21&rows=20

{
  "responseHeader":{
    "status":0,
    "QTime":0,
    "params":{
      "q":"cancer",
      "fl":"url,title",
      "start":"21",
      "rows":"20"}},
  "response":{"numFound":1083,"start":21,"docs":[
      {
        "title":"Cancer Death Rates | Annual Report to the Nation | CDC",
        "url":"https://www.cdc.gov/cancer/annual-report/death-rates.htm"},
      {
        "title":"Cancer Incidence Rates | Annual Report to the Nation | CDC",
        "url":"https://www.cdc.gov/cancer/annual-report/incidence-rates.htm"},
      {
        "title":"Cancer Resources for Health Care Providers | CDC",
        "url":"https://www.cdc.gov/cancer/health-care-providers/"},
      {
        "title":"Cancer Among Children, Adolescents, and Young Adults | Annual Report to the Nation | CDC",
        "url":"https://www.cdc.gov/cancer/annual-report/children-aya.htm"},
      {
        "title":"Cancer Prevention Works Newsletter | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/resources/newsletter.htm"},
      {
        "title":"Cancer Data and Statistics Tools | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/data/tools.htm"},
      {
        "title":"CDC - Rates of Children and Teens Getting Cancer by State or Region",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/rates-children-teens-state-region.htm"},
      {
        "title":"Cancer in American Indians and Alaska Natives in the United States | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/cancer-AIAN-US.htm"},
      {
        "title":"Use of Colorectal Cancer Screening Tests by State | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/use-colorectal-screening-tests-state.htm"},
      {
        "title":"About CDC’s Division of Cancer Prevention and Control | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/about/"},
      {
        "title":"Few People Are Being Screened for Lung Cancer as Recommended | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/lung-cancer-screening.htm"},
      {
        "title":"Healthy Choices to Lower Your Cancer Risk | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/prevention/other.htm"},
      {
        "title":"Vaccines (Shots) Against Cancer | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/prevention/vaccination.htm"},
      {
        "title":"HPV Vaccines and Cervical Precancers | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/hpv-vaccines-cervical-precancers.htm"},
      {
        "title":"Images to Share | Annual Report to the Nation | CDC",
        "url":"https://www.cdc.gov/cancer/annual-report/images.htm"},
      {
        "title":"Breast cancer - Symptoms and causes - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/breast-cancer/symptoms-causes/syc-20352470"},
      {
        "title":"Many Older Adults Don’t Protect Their Skin From the Sun | CDC",
        "url":"https://www.cdc.gov/cancer/dcpc/research/articles/older-adults-protect-skin-sun.htm"},
      {
        "title":"Colon cancer - Symptoms and causes - Mayo Clinic",
        "url":"https://www.mayoclinic.org/diseases-conditions/colon-cancer/symptoms-causes/syc-20353669"},
      {
        "title":"Chemotherapy: Success rates for different cancers",
        "url":"https://www.medicalnewstoday.com/articles/326031"},
      {
        "title":"Metastatic prostate cancer: Treatment and prognosis",
        "url":"https://www.medicalnewstoday.com/articles/320120"}]
  }}

Name		Name	Last commit message	Last commit date
Latest commit History 44 Commits
.github/workflows		.github/workflows
.vscode		.vscode
UI		UI
docs		docs
nutch		nutch
path		path
solr		solr
.DS_Store		.DS_Store
.gitignore		.gitignore
README.md		README.md
app.js		app.js
package-lock.json		package-lock.json
package.json		package.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

MediCrawl

Prerequisites

Components of a web search engine

Procedure

Examples

Plain Query: http://localhost:8983/solr/nutch/query?q=breast+cancer&fl=url,title

Features model: http://localhost:8983/solr/nutch/query?q=breast+cancer&rq={!ltr%20model=myModel%20efi.query=breast+cancer}&fl=url,title,[features]

Pagination - First page twenty queries: http://localhost:8983/solr/nutch/query?q=cancer&fl=url,title&start=0&rows=20

Pagination - Second page twenty queries: http://localhost:8983/solr/nutch/query?q=cancer&fl=url,title&start=21&rows=20

References

About

Releases

Packages

Contributors 4

Languages

devharsh/MediCrawl

Folders and files

Latest commit

History

Repository files navigation

MediCrawl

Prerequisites

Components of a web search engine

Procedure

Examples

Plain Query: http://localhost:8983/solr/nutch/query?q=breast+cancer&fl=url,title

Features model: http://localhost:8983/solr/nutch/query?q=breast+cancer&rq={!ltr%20model=myModel%20efi.query=breast+cancer}&fl=url,title,[features]

Pagination - First page twenty queries: http://localhost:8983/solr/nutch/query?q=cancer&fl=url,title&start=0&rows=20

Pagination - Second page twenty queries: http://localhost:8983/solr/nutch/query?q=cancer&fl=url,title&start=21&rows=20

References

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages 0

Contributors 4

Languages

Packages