← 工作项目← Work

GSA-Ingest 文献自动入库流水线GSA-Ingest Literature Pipeline

2025.10 – 至今 · 中国科学院北京基因组研究所(国家生物信息中心)Oct 2025 – present · Beijing Institute of Genomics, CAS

面向国家级基因组数据库(GSA / GSA Human / OMIX)。将散在邮件、PMC、期刊页里的文献,变成可审计、可入库的结构化数据。展示平台数据集价值,提升平台数据集申请和使用量Serving the national genomics databases (GSA / GSA Human / OMIX). Turns papers scattered across mailboxes, PMC and journal pages into auditable, database-ready structured data. Surfaces how the platform's datasets get used, driving dataset requests and reuse.

PythonFastAPIMySQL DockerLLM语义分类LLM classificationAI WorkflowEvaluation

1三条数据源汇进一条流水线Three sources, one pipeline

每日采集邮件订阅、PMC、期刊页,非结构化的文献文本经清洗、解析、富化后入库。Daily intake from mail subscriptions, PMC and journal pages; unstructured paper text is cleaned, parsed, enriched, then ingested.

2缺失元信息逐级补齐Missing metadata filled in, tier by tier

来源自带 → 落地页 → PubMed/PMC → Crossref 四级降级,任一级失败不中断后续。Source payload → landing page → PubMed/PMC → Crossref: four tiers of fallback, and a failure at any tier does not stop the rest.

3用 LLM 判断引用类型An LLM decides the citation type

一篇论文与一个数据集的关系有三种:提交、复用、只是引用。由 LLM 做语义识别并交叉验证,准确率由规则版 64.1% 提升至 94.2%。A paper relates to a dataset in one of three ways: deposited, reused, or merely referenced. The LLM reads for semantics and cross-checks itself — accuracy up from 64.1% (rule-based) to 94.2%.

4审核系统The review console

所有记录先进审核队列:校验通过的自动入库,其余转人工复核。Every record enters the review queue first: those that pass validation are ingested automatically, the rest go to a human.

定时采集Scheduled intake邮件 · PMC · 期刊Email · PMC · Journals清洗 + 解析Clean + parse富化Enrich类型判定(LLM)Classify (LLM)审核队列Review queue自动入库Auto-ingest综合入库率 80.6%80.6% overall审核系统 · 网页端Web review console公开发布库Public release DB 校验通过passes校验不通过fails复核后入库after review手动补录manual fix