This is the official codebase of the paper BEACON: Benchmark for Comprehensive RNA Tasks and Language Models
- [07/25]🔥 Updating models list and usage!
- [06/11]🔥 BEACON is coming! We release the paper, code, data, and models for BEACON!
important libs: torch==1.13.1+cu117, transformers==4.38.1
git clone https://github.com/terry-r123/RNABenchmark.git
cd RNABenchmark
conda create -n beacon python=3.8
pip install -r requirements.txt
Datasets of RNA tasks can be found in Google Drive
Model checkpoints of opensource RNA language models and BEACON-B can be found in Google Drive
RNABenchmark
├── checkpoint
│ └── opensource
| ├── rna-fm
| ├── rnabert
| ├── rnamsm
| ├── splicebert-human510
| ├── splicebert-ms510
| ├── splicebert-ms1024
| ├── utr-lm-mrl
| ├── utr-lm-te-el
| ├── utrbert-3mer
| ├── utrbert-4mer
| ├── utrbert-5mer
| └── utrbert-6mer
│ └── baseline
| ├── BEACON-B
| └── BEACON-B512
├── data
│ ├── ContactMap
│ ├── CRISPROffTarget
│ ├── CRISPROnTarget
│ ├── Degradation
│ ├── DistanceMap
│ ├── Isoform
│ ├── MeanRibosomeLoading
│ ├── Modification
│ ├── NoncodingRNAFamily
│ ├── ProgrammableRNASwitches
│ ├── Secondary_structure_prediction
│ ├── SpliceAI
│ └── StructuralScoreImputation
├── downstream
│ └── structure
├── model
| |── rna-fm
| ├── rnabert
| ├── rnamsm
| ├── splicebert
| ├── utrlm
| ├── utrbert
| └── rnalm
├── tokenizer
└── scripts
│ ├── BEACON-B
│ └── opensource
The full list of current task names are :
Secondary_structure_prediction
ContactMap
DistanceMap
StructuralScoreImputation
SpliceAI
Isoform
NoncodingRNAFamily
Modification
MeanRibosomeLoading
Degradation
ProgrammableRNASwitches
CRISPROnTarget
CRISPROffTarget
And the list of available embedders/models used for training on the tasks are :
rna-fm
rnabert
rnamsm
utr-lm-mrl
utr-lm-te-el
splicebert-human510
splicebert-ms510
splicebert-ms1024
utrbert-3mer
utrbert-4mer
utrbert-5mer
utrbert-6mer
Models | name | token | pos | length |
---|---|---|---|---|
RNA-FM | rna-fm | single | ape | 1024 |
RNABERT | rnabert | single | ape | 440 |
RNA-MSM | rnamsm | single | ape | 1024 |
SpliceBERT-H510 | splicebert-human510 | single | ape | 510 |
SpliceBERT-MS510 | splicebert-ms510 | single | ape | 510 |
SpliceBERT-MS510 | splicebert-ms510 | single | ape | 1024 |
UTR-LM-MRL | utr-lm-mrl | single | rope | 1026 |
UTR-LM-TE&EL | utr-lm-te-el | single | rope | 1026 |
UTRBERT-3mer | utrbert-3mer | 3mer | ape | 512 |
UTRBERT-4mer | utrbert-4mer | 4mer | ape | 512 |
UTRBERT-5mer | utrbert-5mer | 5mer | ape | 512 |
UTRBERT-6mer | utrbert-6mer | 6mer | ape | 512 |
BEACON-B | rnalm | single | alibi | 1026 |
BEACON-B512 | rnalm | single | alibi | 512 |
To evalute on all RNA tasks, you can run the bash scripts in the scripts
folder, for example:
cd RNABenchmark
bash ./scripts/BEACON-B/all_task.sh
Embeddings from a dummy RNA sequence can be used as follows:
import os, sys
current_path = os.path.dirname(os.path.abspath(__file__))
parent_dir = os.path.dirname(current_path)
sys.path.append(parent_dir)
from model.utrlm.modeling_utrlm import UtrLmModel
from tokenizer.tokenization_opensource import OpenRnaLMTokenizer
tokenizer = OpenRnaLMTokenizer.from_pretrained('./checkpoint/opensource/utr-lm-mrl', model_max_length=1026, padding_side="right", use_fast=True,)
model = UtrLmModel.from_pretrained('./checkpoint/opensource/utr-lm-mrl')
sequences = ["AUUCCGAUUCCGAUUCCG"]
output = tokenizer.batch_encode_plus(sequences, return_tensors="pt", padding="longest", max_length = 1026, truncation=True)
input_ids = output["input_ids"]
attention_mask = output["attention_mask"]
embedding = model(input_ids=input_ids,attention_mask=attention_mask)[0] # shape [bz,length, hidden_size]
print(embedding.shape)
This codebase is released under the Apache License 2.0 as in the LICENSE file.
If you find this repo useful for your research, please consider citing the paper
@misc{ren2024beacon,
title={BEACON: Benchmark for Comprehensive RNA Tasks and Language Models},
author={Yuchen Ren and Zhiyuan Chen and Lifeng Qiao and Hongtai Jing and Yuchen Cai and Sheng Xu and Peng Ye and Xinzhu Ma and Siqi Sun and Hongliang Yan and Dong Yuan and Wanli Ouyang and Xihui Liu},
year={2024},
eprint={2406.10391},
archivePrefix={arXiv},
primaryClass={id='q-bio.QM' full_name='Quantitative Methods' is_active=True alt_name=None in_archive='q-bio' is_general=False description='All experimental, numerical, statistical and mathematical contributions of value to biology'}
}