Natural Language Processing for the next decade. Tokenization, Part-of-Speech Tagging, Named Entity Recognition, Syntactic & Semantic Dependency Parsing, Document Classification
This report summarizes scanner evidence for hankcs/HanLP, a Python project with 36,505 GitHub stars. SynthScan examined 68,636 lines of code across 652 source files, recording 863 pattern matches distributed across 15 syntactic categories. The legacy adjusted pattern signal is 18.2 (Elevated pattern signal); it is not an authorship estimate.
Use the findings to ask whether maintainers can explain intent, verify dependencies, simplify unnecessary scaffolding, and focus review effort. Risk Payload v1 adds ownership, supply-chain, over-engineering, and review-effort vectors when available. These are prompts for human verification, not probabilities, security findings, or measured maintenance costs.
Use each vector to guide manual verification of the underlying findings.
Can maintainers explain intent, assumptions, and trade-offs? 627 finding(s).
Are imports declared and package/API names verified? 127 finding(s).
Are wrappers, guards, and repeated structures necessary? 353 finding(s).
Which changes need focused human review? 863 finding(s).
Run additional scans to build a history of the legacy pattern signal. Score changes are not a measure of AI adoption or code quality.
Groups findings by configured severity weight. Severity is not a confidence estimate, security rating, or proof that a package is missing. Verify imports against the project's manifests and inspect each finding in its local design context.
This horizontal bar chart shows where configured pattern scores accumulate across directories. Treat the bars as locations to inspect, not as evidence of authorship or code quality. Use the file-level findings to decide whether a directory needs a focused ownership, dependency, complexity, or review check.
The scanner identified 863 distinct pattern matches across 15 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a detector match to inspect in context. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the configured finding weight (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — legacy comment-context matches receive a ×1.5 score weight. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier under the legacy scoring model.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | plugins/hanlp_common/hanlp_common/util.py | 82 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/util.py | 110 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/visualization.py | 26 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/visualization.py | 145 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/document.py | 18 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/document.py | 104 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/document.py | 188 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/document.py | 436 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 144 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 359 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 529 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 549 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 724 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 890 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 904 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/dictionary.py | 68 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/dictionary.py | 106 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/dictionary.py | 184 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/trie.py | 99 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/trie.py | 123 | CODE | |
| LOW | hanlp/metrics/chunking/conlleval.py | 151 | CODE | |
| LOW | hanlp/metrics/parsing/conllx_eval.py | 40 | CODE | |
| LOW | hanlp/metrics/parsing/semdep_eval.py | 28 | CODE | |
| LOW | hanlp/layers/embeddings/word2vec_tf.py | 95 | CODE | |
| LOW | hanlp/layers/embeddings/util_tf.py | 23 | CODE | |
| LOW | hanlp/layers/transformers/encoder.py | 19 | CODE | |
| LOW | hanlp/layers/transformers/utils.py | 101 | CODE | |
| LOW | hanlp/datasets/ner/loaders/json_ner.py | 141 | CODE | |
| LOW | hanlp/datasets/ner/loaders/json_ner.py | 34 | CODE | |
| LOW | hanlp/datasets/lm/loaders/lm_dataset.py | 66 | CODE | |
| LOW | hanlp/datasets/qa/hotpotqa.py | 147 | CODE | |
| LOW | hanlp/datasets/tokenization/loaders/chunking_dataset.py | 32 | CODE | |
| LOW | hanlp/datasets/parsing/semeval15.py | 16 | CODE | |
| LOW | hanlp/datasets/parsing/semeval15.py | 48 | CODE | |
| LOW | hanlp/datasets/parsing/pmt1.py | 20 | CODE | |
| LOW | hanlp/datasets/parsing/amr.py | 83 | CODE | |
| LOW | hanlp/datasets/parsing/amr.py | 160 | CODE | |
| LOW | hanlp/datasets/parsing/amr.py | 210 | CODE | |
| LOW | hanlp/datasets/parsing/amr.py | 259 | CODE | |
| LOW | hanlp/datasets/parsing/amr.py | 362 | CODE | |
| LOW | hanlp/datasets/parsing/amr.py | 370 | CODE | |
| LOW | hanlp/datasets/parsing/semeval16.py | 33 | CODE | |
| LOW | hanlp/datasets/parsing/ud/ud210.py | 19 | CODE | |
| LOW | hanlp/datasets/parsing/ud/__init__.py | 12 | CODE | |
| LOW | hanlp/datasets/parsing/ud/ud23.py | 9 | CODE | |
| LOW | hanlp/datasets/parsing/ud/ud27.py | 19 | CODE | |
| LOW | hanlp/datasets/parsing/loaders/constituency_dataset.py | 52 | CODE | |
| LOW | hanlp/datasets/parsing/loaders/_ctb_utils.py | 107 | CODE | |
| LOW | hanlp/datasets/parsing/loaders/_ctb_utils.py | 160 | CODE | |
| LOW | hanlp/datasets/parsing/loaders/_ctb_utils.py | 218 | CODE | |
| LOW | hanlp/datasets/parsing/loaders/conll_dataset.py | 90 | CODE | |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 147 | CODE | |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 269 | CODE | |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 432 | CODE | |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 484 | CODE | |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 499 | CODE | |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 608 | CODE | |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 112 | CODE | |
| LOW | hanlp/datasets/srl/loaders/ontonotes_loader.py | 463 | CODE | |
| LOW | hanlp/datasets/srl/loaders/ontonotes_loader.py | 586 | CODE | |
| 119 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | plugins/hanlp_common/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_common/hanlp_common/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_common/hanlp_common/visualization.py | 181 | spans.extend([''] * (b - offset)) | COMMENT |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 1 | # MIT License | COMMENT |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 61 | COMMENT | |
| LOW | plugins/hanlp_trie/tests/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_restful/tests/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/mul/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/mul/train/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/ja/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/demo_mtl.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/train/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/tf/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/tf/train/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | …demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py | 21 | # masked_lm_dropout=.33, | COMMENT |
| LOW | …/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | …/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert.py | 21 | # # early_stopping_patience=10, | COMMENT |
| LOW | …p_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py | 21 | # masked_lm_dropout=.33, | COMMENT |
| LOW | …/zh/tf/train/train_ptb_dep_biaffine_bert_positional.py | 21 | d_positional=128, | COMMENT |
| LOW | …gins/hanlp_demo/hanlp_demo/zh/tf/train/cws/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | plugins/hanlp_demo/hanlp_demo/en/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | docs/conf.py | 81 | html_sourcelink_suffix = "" | COMMENT |
| LOW | docs/annotations/ner/pku.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/ner/msra.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/ner/ontonotes.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/constituency/ctb.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/constituency/npcmj.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/constituency/ptb.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/pos/pku.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/pos/ud.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/pos/863.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/pos/ctb.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/pos/npcmj.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/dep/sd_zh.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/dep/sd_en.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/dep/pmt.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/dep/ud.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/sdp/semeval16.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/srl/propbank.md | 1 | <!-- | COMMENT |
| LOW | docs/annotations/srl/cpb.md | 1 | <!-- | COMMENT |
| LOW | hanlp/metrics/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/metrics/amr/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/metrics/amr/smatch_eval.py | 61 | # Smatch -> P: 0.136, R: 0.107, F: 0.120 | COMMENT |
| LOW | hanlp/metrics/chunking/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/metrics/chunking/sequence_labeling.py | 1 | # MIT License | COMMENT |
| LOW | hanlp/metrics/parsing/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/metrics/parsing/semdep_eval.py | 1 | #!/usr/bin/env python | COMMENT |
| LOW | hanlp/metrics/parsing/span.py | 1 | # MIT License | COMMENT |
| LOW | hanlp/metrics/parsing/attachmentscore.py | 1 | # MIT License | COMMENT |
| LOW | hanlp/metrics/srl/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/losses/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/layers/scalar_mix.py | 1 | # This file is modified from udify, which is licensed under the MIT license: | COMMENT |
| LOW | hanlp/layers/weight_normalization.py | 1 | # Copyright 2019 The TensorFlow Authors. All Rights Reserved. | COMMENT |
| LOW | hanlp/layers/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/layers/embeddings/__init__.py | 1 | # -*- coding:utf-8 -*- | COMMENT |
| LOW | hanlp/layers/transformers/utils.py | 221 | split_input_ids[-1] = F.pad(split_input_ids[-1], pad=[0, padding_amount], value=0) | COMMENT |
| LOW | hanlp/layers/transformers/utils_tf.py | 81 | logger.warning( | COMMENT |
| LOW | hanlp/layers/crf/crf_tf.py | 1 | # Copyright 2019 The TensorFlow Authors. All Rights Reserved. | COMMENT |
| 91 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | docs/annotations/ner/pku.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/ner/pku.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/ner/msra.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/ner/msra.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/ner/ontonotes.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/ner/ontonotes.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/constituency/ctb.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/constituency/ctb.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/constituency/npcmj.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/constituency/npcmj.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/constituency/ptb.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/constituency/ptb.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/pku.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/pku.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/ud.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/ud.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/863.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/863.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/ctb.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/ctb.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/npcmj.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/pos/npcmj.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/sd_zh.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/sd_zh.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/sd_en.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/sd_en.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/pmt.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/pmt.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/ud.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/dep/ud.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/sdp/semeval16.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/sdp/semeval16.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/srl/propbank.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/srl/propbank.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/srl/cpb.md | 2 | # ======================================================================== | COMMENT |
| MEDIUM | docs/annotations/srl/cpb.md | 19 | # ======================================================================== | COMMENT |
| MEDIUM | hanlp/metrics/parsing/semdep_eval.py | 27 | # =============================================================== | COMMENT |
| MEDIUM | hanlp/metrics/parsing/semdep_eval.py | 121 | # =============================================================== | COMMENT |
| MEDIUM | hanlp/layers/weight_normalization.py | 14 | # ============================================================================= | COMMENT |
| MEDIUM | hanlp/layers/crf/crf_tf.py | 14 | # ============================================================================== | COMMENT |
| MEDIUM | hanlp/optimizers/adamw/optimization.py | 14 | # ============================================================================== | COMMENT |
| MEDIUM | hanlp/components/parsers/parse_alg.py | 40 | # ============================================================= | COMMENT |
| MEDIUM | hanlp/components/parsers/parse_alg.py | 76 | # ====================== | COMMENT |
| MEDIUM | hanlp/components/parsers/alg_tf.py | 120 | # ============================================================= | COMMENT |
| MEDIUM | hanlp/components/parsers/alg_tf.py | 156 | # ====================== | COMMENT |
| MEDIUM | hanlp/components/parsers/biaffine_tf/alg.py | 120 | # ============================================================= | COMMENT |
| MEDIUM | hanlp/components/parsers/biaffine_tf/alg.py | 156 | # ====================== | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 581 | def trim_very_long_tgt_tokens(tgt_tokens, head_tags, head_indices, node_to_idx): | CODE |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 626 | def add_source_side_tags_to_target_side(_src_tokens, _src_tags): | CODE |
| LOW⚡ | plugins/hanlp_restful/tests/test_client.py | 44 | def test_coreference_resolution(self): | CODE |
| LOW⚡ | plugins/hanlp_restful/tests/test_client.py | 52 | def test_abstract_meaning_representation(self): | CODE |
| LOW⚡ | plugins/hanlp_restful/tests/test_client.py | 59 | def test_keyphrase_extraction(self): | CODE |
| LOW⚡ | plugins/hanlp_restful/tests/test_client.py | 64 | def test_extractive_summarization(self): | CODE |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 203 | def semantic_textual_similarity(self, text: Union[Tuple[str, str], List[Tuple[str, str]]], language: str = None) \ | CODE |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 330 | def abstract_meaning_representation(self, | STRING |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 443 | def abstractive_summarization( | STRING |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 475 | def grammatical_error_correction(self, text: Union[str, List[str]], language: str = None) \ | STRING |
| LOW | tests/test_string_util.py | 10 | def test_enumerate_tokenization(self): | CODE |
| LOW | tests/test_mtl.py | 42 | def test_sdp_as_the_first_task(self): | CODE |
| LOW | tests/test_mtl.py | 73 | def test_unicode_removed_by_hf(self): | CODE |
| LOW | hanlp/layers/embeddings/util_tf.py | 78 | def embeddings_require_string_input(embeddings): | CODE |
| LOW | hanlp/layers/embeddings/util_tf.py | 85 | def embeddings_require_char_input(embeddings): | CODE |
| LOW | hanlp/layers/transformers/encoder.py | 128 | def build_transformer_tokenizer(config_or_str, use_fast=True, do_basic_tokenize=True) -> PreTrainedTokenizer: | CODE |
| LOW | hanlp/layers/transformers/utils.py | 81 | def pick_tensor_for_each_token(h, token_span, average_subwords): | CODE |
| LOW | hanlp/layers/transformers/utils.py | 101 | def transformer_sliding_window(transformer: PreTrainedModel, | CODE |
| LOW | hanlp/layers/transformers/utils.py | 227 | def restore_from_sliding_window(all_encoder_layers, batch_size, max_pieces, full_seq_len, start_tokens, end_tokens): | CODE |
| LOW | hanlp/layers/transformers/utils.py | 253 | def build_optimizer_for_pretrained(model: torch.nn.Module, | CODE |
| LOW | hanlp/layers/transformers/utils.py | 296 | def build_optimizer_scheduler_with_transformer(model: torch.nn.Module, | CODE |
| LOW | hanlp/layers/transformers/utils_tf.py | 13 | def convert_examples_to_features( | CODE |
| LOW | hanlp/layers/transformers/utils_tf.py | 173 | def adjust_tokens_for_transformers(sentence): | CODE |
| LOW | hanlp/layers/crf/crf_tf.py | 75 | def crf_multitag_sequence_score(inputs, tag_bitmap, sequence_lengths, | CODE |
| LOW | hanlp/datasets/tokenization/loaders/txt.py | 85 | def generate_tags_for_subtokens(sample: dict, tagging_scheme='BMES'): | CODE |
| LOW | hanlp/datasets/tokenization/loaders/txt.py | 117 | def subtoken_offsets_to_subtokens(text, token_subtoken_offsets): | CODE |
| LOW | hanlp/datasets/tokenization/loaders/txt.py | 124 | def subtokens_group_to_subtokens(tokens, subtoken_offsets_group): | CODE |
| LOW | hanlp/datasets/parsing/semeval15.py | 16 | def unpack_deps_to_head_deprel(sample: dict, pad_rel=None, arc_key='arc', rel_key='rel'): | CODE |
| LOW | hanlp/datasets/parsing/amr.py | 316 | def remove_unconnected_components(concept: List, edge: List): | CODE |
| LOW | hanlp/datasets/parsing/amr.py | 342 | def largest_connected_component(triples: List): | CODE |
| LOW | hanlp/datasets/parsing/amr.py | 362 | def reverse_edge_for_levi_bfs(concept, edge): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 463 | def make_ner_tsv_if_necessary(json_file): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 471 | def batch_make_ner_tsv_if_necessary(json_files): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 476 | def make_pos_tsv_if_necessary(json_file): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 518 | def make_con_txt_if_necessary(json_file): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 526 | def batch_make_con_txt_if_necessary(json_files): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 531 | def batch_remove_empty_category_if_necessary(json_files): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 544 | def make_dep_conllx_if_necessary(con_txt_file: str, language='en'): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 552 | def batch_make_dep_conllx_if_necessary(con_txt_files, language='en'): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 557 | def make_ner_json_if_necessary(json_file): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 565 | def batch_make_ner_json_if_necessary(json_files): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 574 | def make_srl_json_if_necessary(json_file): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 582 | def make_coref_json_if_necessary(json_file): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 590 | def batch_make_srl_json_if_necessary(json_files): | CODE |
| LOW⚡ | hanlp/datasets/srl/ontonotes5/_utils.py | 599 | def batch_make_coref_json_if_necessary(json_files): | CODE |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 339 | def ontonotes_document_generator(input_path, language, labels, stats): | CODE |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 371 | def make_ontonotes_language_jsonlines(conll12_ontonotes_path, output_path=None, language='english'): | CODE |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 494 | def batch_make_pos_tsv_if_necessary(json_files): | CODE |
| LOW | hanlp/datasets/srl/ontonotes5/_utils.py | 642 | def make_raw_text_if_necessary(home: str): | CODE |
| LOW | hanlp/datasets/srl/loaders/ontonotes_loader.py | 205 | def dataset_document_iterator(self, file_path: str) -> Iterator[List[OntonotesSentence]]: | CODE |
| LOW | hanlp/datasets/srl/loaders/ontonotes_loader.py | 369 | def _process_coref_span_annotations_for_word( | CODE |
| LOW | hanlp/datasets/srl/loaders/ontonotes_loader.py | 420 | def _process_span_annotations_for_word( | CODE |
| LOW | hanlp/datasets/coref/loaders/conll12coref.py | 16 | def dataset_document_iterator(self, file_path: str) -> Iterator[List[OntonotesSentence]]: | CODE |
| LOW | hanlp/utils/io_util.py | 455 | def generate_words_tags_from_tsv(tsv_file_path, lower=False, gold=True, max_seq_length=None, sent_delimiter=None, | CODE |
| LOW | hanlp/utils/io_util.py | 588 | def get_exitcode_stdout_stderr(cmd): | CODE |
| LOW | hanlp/utils/io_util.py | 700 | def get_latest_info_from_pypi(package='hanlp', repository_url='https://pypi.python.org/pypi/%s/json'): | CODE |
| LOW | hanlp/utils/file_read_backwards/buffer_work_space.py | 156 | def _remove_trailing_new_line(l): | CODE |
| LOW | hanlp/utils/file_read_backwards/buffer_work_space.py | 189 | def _is_partially_read_new_line(b): | CODE |
| LOW | hanlp/components/lemmatizer.py | 13 | def add_lemma_rules_to_sample(sample: dict): | CODE |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 513 | def _set_gradient_checkpointing(self, module, value=False): | CODE |
| 71 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | plugins/hanlp_common/hanlp_common/conll.py | 360 | Checks if a dependency tree is projective. This also works for partial annotation. Besides the obvious cro | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 70 | Parse a piece of text. Args: text: A document (str), or a list of sentences (List[str]). | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 172 | Text style transfer aims to change the style of the input text to the target style while preserving its content | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 205 | Semantic textual similarity deals with determining how similar two pieces of texts are. Args: text | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 235 | Coreference resolution is the task of clustering mentions in text that refer to the same underlying real world | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 276 | Split a document into sentences and tokenize them. Note that it is always faster to tokenize a whole document t | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 336 | Abstract Meaning Representation (AMR) captures “who is doing what to whom” in a sentence. Each sentence is repre | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 375 | Keyphrase extraction aims to identify keywords or phrases reflecting the main topics of a document. Args: | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 408 | Single document summarization is the task of selecting a subset of the sentences which best represents a summar | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 448 | Abstractive Summarization is the task of generating a short and concise summary that captures the salient ideas | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 477 | Grammatical Error Correction (GEC) is the task of correcting different kinds of errors in text such as spelling | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 525 | Sentiment analysis is the task of classifying the polarity of a given text. For instance, a text-based | STRING |
| HIGH | plugins/hanlp_restful/hanlp_restful/__init__.py | 555 | Identify the language of a given text. Args: text: A document or a list of documents. | STRING |
| HIGH | hanlp/__init__.py | 19 | Load a pretrained component from an identifier. Args: save_dir (str): The identifier to the saved component. | STRING |
| HIGH | hanlp/metrics/chunking/sequence_labeling.py | 323 | Build a text report showing the main classification metrics. Args: y_true: 2d array. Ground truth (correct) t | STRING |
| HIGH | hanlp/layers/weight_normalization.py | 26 | This wrapper reparameterizes a layer by decoupling the weight's magnitude and direction. This speeds up con | STRING |
| HIGH | hanlp/datasets/parsing/loaders/constituency_dataset.py | 53 | Conducts binarization over the tree. First, the tree is transformed to satisfy `Chomsky Normal Form (CNF)`_. | STRING |
| HIGH | hanlp/datasets/parsing/loaders/constituency_dataset.py | 106 | Factorizes the tree into a sequence. The tree is traversed in pre-order. Args: tree (tree.Tree): | STRING |
| HIGH | hanlp/datasets/parsing/loaders/constituency_dataset.py | 165 | Builds a constituency tree from the sequence. The sequence is generated in pre-order. During building the tree, | STRING |
| HIGH | hanlp/components/parsers/alg.py | 30 | KMeans algorithm for clustering the sentences by length. Args: x (list[int]): The list of | STRING |
| HIGH | hanlp/components/parsers/alg.py | 103 | First-order Eisner algorithm for projective decoding. References: - Ryan McDonald, Koby Crammer and Fe | STRING |
| HIGH | hanlp/components/parsers/alg.py | 247 | The implementation of `Cocke-Kasami-Younger`_ (CKY) algorithm to parse constituency trees. References: | STRING |
| HIGH | hanlp/components/parsers/alg.py | 319 | Checks if the arcs form an valid dependency tree. Args: sequence (list[int]): A list of he | STRING |
| HIGH | hanlp/components/parsers/alg.py | 353 | Tarjan algorithm for finding Strongly Connected Components (SCCs) of a graph. Args: sequence (list): | STRING |
| HIGH | hanlp/components/parsers/alg.py | 409 | ChuLiu/Edmonds algorithm for non-projective decoding. Some code is borrowed from `tdozat's implementation`_. | STRING |
| HIGH | hanlp/components/parsers/alg.py | 508 | MST algorithm for decoding non-pojective trees. This is a wrapper for ChuLiu/Edmonds algorithm. The algori | STRING |
| HIGH | hanlp/components/parsers/alg.py | 567 | Second-order Eisner algorithm for projective decoding. This is an extension of the first-order one that further | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | plugins/hanlp_common/hanlp_common/document.py | 6 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/__init__.py | 4 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/__init__.py | 5 | CODE | |
| LOW | plugins/hanlp_trie/hanlp_trie/__init__.py | 5 | CODE | |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/demo_ner_dict.py | 6 | CODE | |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/demo_pos_dict.py | 7 | CODE | |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/train/open_small.py | 4 | CODE | |
| LOW | plugins/hanlp_demo/hanlp_demo/zh/train/open_base.py | 4 | CODE | |
| LOW | …hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert_topk.py | 4 | CODE | |
| LOW | …hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert_topk.py | 7 | CODE | |
| LOW | …hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert_topk.py | 9 | CODE | |
| LOW | …demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py | 4 | CODE | |
| LOW | …demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py | 6 | CODE | |
| LOW | …demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py | 6 | CODE | |
| LOW | …demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py | 6 | CODE | |
| LOW | …demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py | 7 | CODE | |
| LOW | …demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py | 9 | CODE | |
| LOW | …/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert.py | 4 | CODE | |
| LOW | …/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert.py | 6 | CODE | |
| LOW | …/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert.py | 6 | CODE | |
| LOW | …/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert.py | 6 | CODE | |
| LOW | …p_demo/zh/tf/train/train_ptb_dep_biaffine_bert_96.6.py | 5 | CODE | |
| LOW | …p_demo/zh/tf/train/train_ptb_dep_biaffine_bert_96.6.py | 5 | CODE | |
| LOW | …p_demo/zh/tf/train/train_ptb_dep_biaffine_bert_96.6.py | 5 | CODE | |
| LOW | …anlp_demo/zh/tf/train/train_ptb_dep_biaffine_albert.py | 4 | CODE | |
| LOW | …p_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py | 4 | CODE | |
| LOW | …p_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py | 6 | CODE | |
| LOW | …p_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py | 6 | CODE | |
| LOW | …p_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py | 6 | CODE | |
| LOW | …p_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py | 7 | CODE | |
| LOW | …p_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py | 9 | CODE | |
| LOW | …mo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_pos_bert.py | 4 | CODE | |
| LOW | …mo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_pos_bert.py | 6 | CODE | |
| LOW | …mo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_pos_bert.py | 6 | CODE | |
| LOW | …mo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_pos_bert.py | 6 | CODE | |
| LOW | …mo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_pos_bert.py | 7 | CODE | |
| LOW | …mo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_pos_bert.py | 9 | CODE | |
| LOW | …/zh/tf/train/train_ptb_dep_biaffine_bert_positional.py | 4 | CODE | |
| LOW | plugins/hanlp_demo/hanlp_demo/en/demo_sdp.py | 5 | CODE | |
| LOW | hanlp/__init__.py | 13 | CODE | |
| LOW | hanlp/metrics/parsing/semdep_eval.py | 18 | CODE | |
| LOW | hanlp/metrics/parsing/semdep_eval.py | 19 | CODE | |
| LOW | hanlp/metrics/parsing/semdep_eval.py | 20 | CODE | |
| LOW | hanlp/layers/scalar_mix.py | 30 | CODE | |
| LOW | hanlp/layers/weight_normalization.py | 15 | CODE | |
| LOW | hanlp/layers/weight_normalization.py | 16 | CODE | |
| LOW | hanlp/layers/weight_normalization.py | 17 | CODE | |
| LOW | hanlp/layers/embeddings/util.py | 11 | CODE | |
| LOW | hanlp/layers/embeddings/fast_text.py | 20 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| LOW | hanlp/layers/transformers/tf_imports.py | 4 | CODE | |
| 66 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | hanlp/datasets/tokenization/sighan2005/msr.py | 0 | training set (first 90% of the full official training set). | STRING |
| HIGH | hanlp/datasets/tokenization/sighan2005/cityu.py | 0 | training set (first 90% of the full official training set). | STRING |
| HIGH | hanlp/datasets/tokenization/sighan2005/as_.py | 0 | training set (first 90% of the full official training set). | STRING |
| HIGH | hanlp/datasets/tokenization/sighan2005/pku.py | 0 | training set (first 90% of the full official training set). | STRING |
| HIGH | …nents/amr/amrbart/model_interface/tokenization_bart.py | 0 | ?<[a-z]+:?\d*>| ?:[^\s]+|'s|'t|'re|'ve|'m|'ll|'d| ?\p{l}+| ?\p{n}+| ?[^\s\p{l}\p{n}]+|\s+(?!\s)|\s+ | STRING |
| HIGH | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 0 | ?<[a-z]+:?\d*>| ?:[^\s]+|'s|'t|'re|'ve|'m|'ll|'d| ?\p{l}+| ?\p{n}+| ?[^\s\p{l}\p{n}]+|\s+(?!\s)|\s+ | STRING |
| HIGH | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 0 | ?<[a-z]+:?\d*>| ?:[^\s]+|'s|'t|'re|'ve|'m|'ll|'d| ?\p{l}+| ?\p{n}+| ?[^\s\p{l}\p{n}]+|\s+(?!\s)|\s+ | STRING |
| HIGH | …nents/amr/amrbart/model_interface/tokenization_bart.py | 0 | tokenize a string. modified in order to handle sentences with recategorization pointers | STRING |
| HIGH | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 0 | tokenize a string. modified in order to handle sentences with recategorization pointers | STRING |
| HIGH | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 0 | tokenize a string. modified in order to handle sentences with recategorization pointers | STRING |
| HIGH | …nents/amr/amrbart/model_interface/tokenization_bart.py | 0 | line = linearized # make sure parentheses match # copied from https://github.com/rikvn/amr/blob/master/restoreamr/restor | STRING |
| HIGH | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 0 | line = linearized # make sure parentheses match # copied from https://github.com/rikvn/amr/blob/master/restoreamr/restor | STRING |
| HIGH | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 0 | line = linearized # make sure parentheses match # copied from https://github.com/rikvn/amr/blob/master/restoreamr/restor | STRING |
| HIGH | hanlp/components/parsers/parse_alg.py | 0 | computes tarjan's algorithm for finding strongly connected components (cycles) of a graph | STRING |
| HIGH | hanlp/components/parsers/alg_tf.py | 0 | computes tarjan's algorithm for finding strongly connected components (cycles) of a graph | STRING |
| HIGH | hanlp/components/parsers/biaffine_tf/alg.py | 0 | computes tarjan's algorithm for finding strongly connected components (cycles) of a graph | STRING |
| HIGH | hanlp/components/parsers/parse_alg.py | 0 | parameters ---------- prediction : numpy.ndarray a predicted dependency tree where prediction[dep_idx] = head_idx tokens | STRING |
| HIGH | hanlp/components/parsers/alg_tf.py | 0 | parameters ---------- prediction : numpy.ndarray a predicted dependency tree where prediction[dep_idx] = head_idx tokens | STRING |
| HIGH | hanlp/components/parsers/biaffine_tf/alg.py | 0 | parameters ---------- prediction : numpy.ndarray a predicted dependency tree where prediction[dep_idx] = head_idx tokens | STRING |
| HIGH | hanlp/components/parsers/parse_alg.py | 0 | adopted from timothy dozat https://github.com/tdozat/parser/blob/master/lib/models/nn.py args: parse_probs(ndarray): seq | STRING |
| HIGH | hanlp/components/parsers/alg_tf.py | 0 | adopted from timothy dozat https://github.com/tdozat/parser/blob/master/lib/models/nn.py args: parse_probs(ndarray): seq | STRING |
| HIGH | hanlp/components/parsers/biaffine_tf/alg.py | 0 | adopted from timothy dozat https://github.com/tdozat/parser/blob/master/lib/models/nn.py args: parse_probs(ndarray): seq | STRING |
| HIGH | …lp/components/classifiers/transformer_classifier_hf.py | 0 | classify text. args: text: a document or a list of documents. topk: ``true`` or ``int`` to return the top-k labels. prob | STRING |
| HIGH | …lp/components/classifiers/transformer_regression_hf.py | 0 | classify text. args: text: a document or a list of documents. topk: ``true`` or ``int`` to return the top-k labels. prob | STRING |
| HIGH | hanlp/components/classifiers/fasttext_classifier.py | 0 | classify text. args: text: a document or a list of documents. topk: ``true`` or ``int`` to return the top-k labels. prob | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | plugins/hanlp_common/hanlp_common/conll.py | 11 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/conll.py | 88 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/document.py | 188 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/document.py | 408 | CODE | |
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 85 | CODE | |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 201 | CODE | |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 230 | CODE | |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 273 | CODE | |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 502 | CODE | |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 522 | CODE | |
| LOW | plugins/hanlp_restful/hanlp_restful/__init__.py | 551 | CODE | |
| LOW | hanlp/layers/embeddings/contextual_word_embedding.py | 18 | CODE | |
| LOW | hanlp/layers/embeddings/contextual_word_embedding.py | 79 | CODE | |
| LOW | hanlp/layers/embeddings/word2vec.py | 94 | CODE | |
| LOW | hanlp/layers/embeddings/char_cnn_tf.py | 14 | CODE | |
| LOW | hanlp/layers/embeddings/word2vec_tf.py | 19 | CODE | |
| LOW | hanlp/layers/embeddings/word2vec_tf.py | 114 | CODE | |
| LOW | hanlp/layers/embeddings/word2vec_tf.py | 180 | CODE | |
| LOW | hanlp/layers/transformers/encoder.py | 19 | CODE | |
| LOW | hanlp/layers/transformers/utils_tf.py | 13 | CODE | |
| LOW | hanlp/datasets/tokenization/loaders/txt.py | 13 | CODE | |
| LOW | …okenization/loaders/multi_criteria_cws/mcws_dataset.py | 12 | CODE | |
| LOW | hanlp/datasets/parsing/amr.py | 83 | CODE | |
| LOW | hanlp/utils/time_util.py | 165 | CODE | |
| LOW | hanlp/utils/io_util.py | 660 | CODE | |
| LOW | hanlp/utils/lang/ja/bert_tok.py | 17 | CODE | |
| LOW | hanlp/optimizers/adamw/optimization.py | 121 | CODE | |
| LOW | hanlp/components/rnn_language_model_tf.py | 20 | CODE | |
| LOW | hanlp/components/amr/amrbart/bart_amr_parser.py | 170 | CODE | |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 382 | CODE | |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 925 | CODE | |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 1181 | CODE | |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 1315 | CODE | |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 1464 | CODE | |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 1590 | CODE | |
| LOW | …omponents/amr/amrbart/model_interface/modeling_bart.py | 1731 | CODE | |
| LOW | …nents/amr/amrbart/model_interface/tokenization_bart.py | 36 | CODE | |
| LOW | hanlp/components/amr/seq2seq/seq2seq_amr_parser.py | 297 | CODE | |
| LOW | hanlp/components/amr/seq2seq/seq2seq_amr_parser.py | 399 | CODE | |
| LOW | hanlp/components/tokenizers/tok_tf.py | 54 | CODE | |
| LOW | hanlp/components/tokenizers/tok_tf.py | 105 | CODE | |
| LOW | …omponents/tokenizers/multi_criteria_cws_transformer.py | 80 | CODE | |
| LOW | hanlp/components/tokenizers/tok.py | 48 | CODE | |
| LOW | hanlp/components/tokenizers/transformer.py | 240 | CODE | |
| LOW | hanlp/components/parsers/biaffine_parser_tf.py | 60 | CODE | |
| LOW | hanlp/components/parsers/biaffine_parser_tf.py | 197 | CODE | |
| LOW | hanlp/components/parsers/biaffine_parser_tf.py | 359 | CODE | |
| LOW | hanlp/components/parsers/biaffine_parser_tf.py | 441 | CODE | |
| LOW | hanlp/components/parsers/biaffine_parser_tf.py | 556 | CODE | |
| LOW | …onents/parsers/constituency/crf_constituency_parser.py | 265 | CODE | |
| LOW | hanlp/components/parsers/ud/ud_parser.py | 185 | CODE | |
| LOW | hanlp/components/parsers/biaffine/biaffine_2nd_dep.py | 103 | CODE | |
| LOW | hanlp/components/parsers/biaffine/biaffine_sdp.py | 153 | CODE | |
| LOW | …nlp/components/parsers/biaffine/structual_attention.py | 129 | CODE | |
| LOW | hanlp/components/parsers/biaffine/biaffine_dep.py | 146 | CODE | |
| LOW | hanlp/components/ner/rnn_ner.py | 33 | CODE | |
| LOW | hanlp/components/ner/transformer_ner.py | 123 | CODE | |
| LOW | hanlp/components/ner/ner_tf.py | 37 | CODE | |
| LOW | hanlp/components/ner/ner_tf.py | 58 | CODE | |
| LOW | hanlp/components/ner/ner_tf.py | 80 | CODE | |
| 53 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | hanlp/utils/torch_util.py | 195 | except IOError: | CODE |
| HIGH | hanlp/utils/torch_util.py | 234 | except IOError: | CODE |
| HIGH | hanlp/utils/component_util.py | 186 | except: | CODE |
| HIGH | hanlp/utils/io_util.py | 583 | except: | CODE |
| HIGH | hanlp/components/amr/amrbart/bart_amr_parser.py | 198 | except: | CODE |
| HIGH | hanlp/components/amr/amrbart/common/postprocessing.py | 458 | except: | CODE |
| HIGH | hanlp/components/amr/seq2seq/seq2seq_amr_parser.py | 250 | except: | CODE |
| HIGH | hanlp/components/amr/seq2seq/dataset/postprocessing.py | 580 | except: | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | plugins/hanlp_common/hanlp_common/amr.py | 816 | except Exception as e: | CODE |
| LOW | hanlp/datasets/parsing/loaders/_ctb_utils.py | 268 | except Exception as e: | CODE |
| LOW | hanlp/utils/torch_util.py | 48 | except Exception as e: | CODE |
| LOW⚡ | hanlp/utils/component_util.py | 125 | except Exception as e: | CODE |
| LOW⚡ | hanlp/utils/component_util.py | 130 | except Exception as check_e: | CODE |
| LOW | hanlp/utils/component_util.py | 173 | except Exception as tf_e: | CODE |
| LOW | hanlp/utils/io_util.py | 226 | except Exception as e: | CODE |
| LOW | hanlp/utils/io_util.py | 260 | except Exception as e: | CODE |
| LOW | …nents/amr/amrbart/model_interface/tokenization_bart.py | 144 | except Exception as e: | CODE |
| LOW | …nents/amr/amrbart/model_interface/tokenization_bart.py | 152 | except Exception as e: | CODE |
| LOW | …nents/amr/amrbart/model_interface/tokenization_bart.py | 166 | except Exception as e: | CODE |
| MEDIUM | …nents/amr/amrbart/model_interface/tokenization_bart.py | 141 | def decode_amr(self, tokens, restore_name_ops=None): | CODE |
| LOW | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 216 | except Exception as e: | CODE |
| LOW | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 224 | except Exception as e: | CODE |
| LOW | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 239 | except Exception as e: | CODE |
| LOW | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 639 | except Exception as e: | CODE |
| LOW | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 647 | except Exception as e: | CODE |
| LOW | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 662 | except Exception as e: | CODE |
| MEDIUM | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 213 | def decode_amr(self, tokens, restore_name_ops=False): | CODE |
| MEDIUM | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 630 | def decode_amr(self, tokens, restore_name_ops=None): | CODE |
| LOW | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 226 | except Exception as e: | CODE |
| LOW | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 234 | except Exception as e: | CODE |
| LOW | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 248 | except Exception as e: | CODE |
| LOW | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 643 | except Exception as e: | CODE |
| LOW | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 651 | except Exception as e: | CODE |
| LOW | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 665 | except Exception as e: | CODE |
| MEDIUM | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 223 | def decode_amr(self, tokens, restore_name_ops=False): | CODE |
| MEDIUM | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 635 | def decode_amr(self, tokens, restore_name_ops=None): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | plugins/hanlp_common/hanlp_common/document.py | 29 | # Create a document | COMMENT |
| MEDIUM | plugins/hanlp_demo/hanlp_demo/zh/demo_document.py | 6 | # Create a document or get a document from HanLP.parse | COMMENT |
| MEDIUM | plugins/hanlp_demo/hanlp_demo/zh/demo_document.py | 34 | # Create a document from a dict | COMMENT |
| MEDIUM | hanlp/layers/scalar_mix.py | 1 | # This file is modified from udify, which is licensed under the MIT license: | COMMENT |
| MEDIUM | hanlp/layers/dropout.py | 50 | # Create a uniformly random mask selecting either the original words or OOV tokens | COMMENT |
| MEDIUM | hanlp/components/parsers/ud/tag_decoder.py | 1 | # This file is modified from udify, which is licensed under the MIT license: | COMMENT |
| MEDIUM | hanlp/components/parsers/ud/udify_util.py | 1 | # This file is modified from udify and allennlp, which are licensed under the MIT license: | COMMENT |
| MEDIUM | …nlp/components/parsers/biaffine/structual_attention.py | 108 | # Create a uniformly random mask selecting either the original words or OOV tokens | COMMENT |
| MEDIUM | hanlp/components/mtl/multi_task_learning.py | 128 | # This method is only called during training or evaluation but not prediction | COMMENT |
| MEDIUM | hanlp/transform/transformer_tokenizer.py | 325 | # The following block will tokenize each empty string (space) into an unk token | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | …ns/hanlp_demo/hanlp_demo/zh/demo_parse_constituency.py | 40 | # If you need to parse raw text, simply add a tokenizer into this pipeline. | COMMENT |
| LOW | …ugins/hanlp_demo/hanlp_demo/zh/tf/demo_multiprocess.py | 17 | # Important! The python multiprocessing package defaults to just call fork when creating a child process. | COMMENT |
| LOW | hanlp/layers/cnn_encoder.py | 99 | # pooling is simple, we just use `torch.max`. The resultant tensor of has shape | COMMENT |
| LOW | hanlp/layers/crf/crf_tf.py | 481 | # If max_seq_len is 1, we skip the algorithm and simply return the argmax tag | COMMENT |
| LOW | hanlp/datasets/srl/loaders/ontonotes_loader.py | 297 | # we just set the parse piece to be None which will result | COMMENT |
| MEDIUM | …nents/amr/amrbart/model_interface/tokenization_bart.py | 67 | """ Tokenize a string. Modified in order to handle sentences with recategorization pointers""" | STRING |
| MEDIUM | hanlp/components/amr/seq2seq/dataset/tokenization_t5.py | 80 | """ Tokenize a string. Modified in order to handle sentences with recategorization pointers""" | STRING |
| MEDIUM | …lp/components/amr/seq2seq/dataset/tokenization_bart.py | 88 | """ Tokenize a string. Modified in order to handle sentences with recategorization pointers""" | STRING |
| MEDIUM | hanlp/components/taggers/tagger.py | 209 | >>> pos.dict_tags = {'HanLP': 'state-of-the-art-tool'} # Force 'HanLP' to be 'state-of-the-art-tool' | CODE |
| MEDIUM | hanlp/components/taggers/tagger.py | 211 | # HanLP/state-of-the-art-tool 为/P 生产/NN 环境/NN 带来/VV 次世代/NN 最/AD 先进/VA 的/DEC 多语种/NN NLP/NR 技术/NN 。/PU | COMMENT |
| LOW | hanlp/transform/transformer_tokenizer.py | 499 | # f'Or simply set truncate_long_sequences = False to enable sliding window.') | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| CRITICAL⚡ | hanlp/utils/component_util.py | 126 | if not e.__traceback__.tb_frame.f_code.co_filename.startswith(os.path.dirname(os.path.dirname(__file__))): | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | hanlp/metrics/accuracy.py | 115 | # Check if it's actually a tensor in case something else was passed. | COMMENT |
| LOW | hanlp/metrics/accuracy.py | 226 | # Check if it's actually a tensor in case something else was passed. | COMMENT |
| LOW | hanlp/layers/crf/crf.py | 266 | # Set score to the next score if this timestep is valid (mask == 1) | COMMENT |
| LOW | hanlp/layers/crf/crf.py | 321 | # Set score to the next score if this timestep is valid (mask == 1) | COMMENT |
| LOW | hanlp/components/parsers/chu_liu_edmonds.py | 168 | # Check if this solution has a cycle. | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | hanlp/datasets/srl/loaders/ontonotes_loader.py | 10 | logger = logging.getLogger(__name__) | CODE |