小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 | 评论爬虫、微博帖子 | 评论爬虫、百度贴吧帖子 | 百度贴吧评论回复爬虫 | 知乎问答文章|评论爬虫
This report presents the forensic synthetic code analysis of NanmiCoder/MediaCrawler, a Python project with 56,390 GitHub stars. SynthScan v2.0 examined 50,437 lines of code across 267 source files, recording 888 pattern matches distributed across 15 syntactic categories. The overall adjusted score of 28.4 places this repository in the Moderate AI signal band.
The scanner applied 160+ deterministic lexical heuristics, multi-line block detectors, abstract syntax tree depth profilers, and a cross-file Jaccard similarity matrix to construct a statistically normalised synthetic code estimate. All matches are individually weighted by severity coefficient and contextual multiplier before summation, and the resulting headline score is temporally discounted to account for the repository's development history relative to the commercial emergence of large language model coding tooling (November 2022 onward).
This chart maps the temporal evolution of the adjusted synthetic code score across successive scan runs. An upward trajectory indicates ongoing incorporation of AI-generated code or expanding LLM-assisted scaffolding; a stable or declining trajectory may reflect active human refactoring, code removal, or the adoption of stricter authorship policies. The dashed secondary line (right axis) independently tracks total raw pattern hit count, which can diverge from the normalised score when codebase size changes significantly between scans.
Classifies detected patterns by their diagnostic confidence and structural impact. CRITICAL patterns (coefficient 10) represent definitive synthetic signatures — hallucinated imports, explicit LLM attribution metadata — virtually never produced by human authors. HIGH (5) indicates strong structural tells such as cross-file repetition or cross-linguistic idioms. MEDIUM (2) covers recognisable conversational padding and AI-specific vocabulary. LOW (1) captures subtle indicators like tautological comments and generic boilerplate that require density to carry independent signal.
This horizontal bar chart decomposes the repository's raw synthetic code score by top-level directory, allowing you to pinpoint precisely which modules or components carry the highest AI authorship density. Directories with disproportionately high scores relative to their size warrant targeted manual review: concentrated AI signatures often trace back to mass-generated configuration layers, auto-ported test suites, LLM-scaffolded boilerplate classes, or entire subsystems authored under heavy copilot assistance. Use this view to prioritise your human code-review effort.
The scanner identified 888 distinct pattern matches across 15 syntactic categories. Each entry below represents a discrete location in the source code where the engine recorded a statistically significant AI authorship indicator. Expand any category row to inspect the individual file paths, line numbers, code snippets, and the lexical context (CODE, COMMENT, or STRING) in which each match was detected.
Reading the findings table: The Severity column indicates the diagnostic confidence level (CRITICAL / HIGH / MEDIUM / LOW). The Context column identifies whether the match occurred inside executable code, an inline comment, or a string literal — comment-context matches receive a ×1.5 weight because LLMs systematically over-annotate. The ⚡ bolt icon marks clustered matches: three or more patterns within a 10-line window, each receiving an additional ×1.5 density multiplier as dense clusters constitute far stronger evidence of synthetic authorship than isolated hits.
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | recv_sms.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | var.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | main.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | database/db.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | database/models.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | database/mongodb_store_base.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | database/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | database/db_session.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/crawler_util.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/async_file_writer.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/easing.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/utils.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/slider_util.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/words.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/browser_launcher.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/cdp_browser.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/app_runner.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/time_util.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/file_header_manager.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tools/file_header_manager.py | 81 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/types.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/proxy_mixin.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/proxy_ip_pool.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/base_proxy.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/providers/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/providers/kuaidl_proxy.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/providers/jishu_http_proxy.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | proxy/providers/wandou_http_proxy.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | cache/local_cache.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | cache/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | cache/abs_cache.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | cache/cache_factory.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | cache/redis_cache.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | test/test_utils.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | test/test_db_sync.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | test/test_expiring_local_cache.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | test/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | test/test_redis_cache.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | test/test_proxy_ip_pool.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | test/test_mongodb_integration.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/zhihu_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/base_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/ks_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/dy_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/tieba_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/xhs_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/db_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/weibo_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | config/bilibili_config.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tests/conftest.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tests/test_store_factory.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tests/test_excel_store.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | tests/test_excel_store.py | 267 | # Create an instance | COMMENT |
| MEDIUM | cmd_arg/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | cmd_arg/arg.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | media_platform/__init__.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| MEDIUM | media_platform/bilibili/field.py | 4 | # This file is part of MediaCrawler project. | COMMENT |
| 102 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | media_platform/bilibili/login.py | 0 | check if the current login status is successful and return true otherwise return false retry decorator will retry 20 tim | STRING |
| HIGH | media_platform/weibo/login.py | 0 | check if the current login status is successful and return true otherwise return false retry decorator will retry 20 tim | STRING |
| HIGH | media_platform/kuaishou/login.py | 0 | check if the current login status is successful and return true otherwise return false retry decorator will retry 20 tim | STRING |
| HIGH | media_platform/xhs/client.py | 0 | update cookies method provided by api client, typically called after successful login args: browser_context: browser con | STRING |
| HIGH | media_platform/tieba/client.py | 0 | update cookies method provided by api client, typically called after successful login args: browser_context: browser con | STRING |
| HIGH | media_platform/zhihu/client.py | 0 | update cookies method provided by api client, typically called after successful login args: browser_context: browser con | STRING |
| HIGH | media_platform/xhs/core.py | 0 | search for notes and retrieve their comment information. returns: | STRING |
| HIGH | media_platform/zhihu/core.py | 0 | search for notes and retrieve their comment information. returns: | STRING |
| HIGH | media_platform/tieba/core.py | 0 | search for notes and retrieve their comment information. returns: | STRING |
| HIGH | media_platform/xhs/core.py | 0 | concurrently obtain the specified post list and save the data | STRING |
| HIGH | media_platform/douyin/core.py | 0 | concurrently obtain the specified post list and save the data | STRING |
| HIGH | media_platform/kuaishou/core.py | 0 | concurrently obtain the specified post list and save the data | STRING |
| HIGH | media_platform/weibo/core.py | 0 | get creator's information and their notes and comments returns: | STRING |
| HIGH | media_platform/tieba/core.py | 0 | get creator's information and their notes and comments returns: | STRING |
| HIGH | media_platform/zhihu/core.py | 0 | get creator's information and their notes and comments returns: | STRING |
| HIGH | store/bilibili/_store_impl.py | 0 | content json storage implementation args: content_item: returns: | STRING |
| HIGH | store/weibo/_store_impl.py | 0 | content json storage implementation args: content_item: returns: | STRING |
| HIGH | store/douyin/_store_impl.py | 0 | content json storage implementation args: content_item: returns: | STRING |
| HIGH | store/kuaishou/_store_impl.py | 0 | content json storage implementation args: content_item: returns: | STRING |
| HIGH | store/zhihu/_store_impl.py | 0 | content json storage implementation args: content_item: returns: | STRING |
| HIGH | store/bilibili/_store_impl.py | 0 | comment json storage implementation args: comment_item: returns: | STRING |
| HIGH | store/weibo/_store_impl.py | 0 | comment json storage implementation args: comment_item: returns: | STRING |
| HIGH | store/douyin/_store_impl.py | 0 | comment json storage implementation args: comment_item: returns: | STRING |
| HIGH | store/kuaishou/_store_impl.py | 0 | comment json storage implementation args: comment_item: returns: | STRING |
| HIGH | store/zhihu/_store_impl.py | 0 | comment json storage implementation args: comment_item: returns: | STRING |
| HIGH | store/bilibili/_store_impl.py | 0 | creator json storage implementation 教学版:不采集/持久化创作者个人信息,空操作。 args: creator: returns: | STRING |
| HIGH | store/douyin/_store_impl.py | 0 | creator json storage implementation 教学版:不采集/持久化创作者个人信息,空操作。 args: creator: returns: | STRING |
| HIGH | store/weibo/_store_impl.py | 0 | creator json storage implementation 教学版:不采集/持久化创作者个人信息,空操作。 args: creator: returns: | STRING |
| HIGH | store/bilibili/_store_impl.py | 0 | store content to mongodb args: content_item: content data | STRING |
| HIGH | store/douyin/_store_impl.py | 0 | store content to mongodb args: content_item: content data | STRING |
| HIGH | store/kuaishou/_store_impl.py | 0 | store content to mongodb args: content_item: content data | STRING |
| HIGH | store/xhs/_store_impl.py | 0 | store content to mongodb args: content_item: content data | STRING |
| HIGH | store/weibo/_store_impl.py | 0 | store content to mongodb args: content_item: content data | STRING |
| HIGH | store/tieba/_store_impl.py | 0 | store content to mongodb args: content_item: content data | STRING |
| HIGH | store/zhihu/_store_impl.py | 0 | store content to mongodb args: content_item: content data | STRING |
| HIGH | store/bilibili/_store_impl.py | 0 | store comment to mongodb args: comment_item: comment data | STRING |
| HIGH | store/xhs/_store_impl.py | 0 | store comment to mongodb args: comment_item: comment data | STRING |
| HIGH | store/weibo/_store_impl.py | 0 | store comment to mongodb args: comment_item: comment data | STRING |
| HIGH | store/tieba/_store_impl.py | 0 | store comment to mongodb args: comment_item: comment data | STRING |
| HIGH | store/douyin/_store_impl.py | 0 | store comment to mongodb args: comment_item: comment data | STRING |
| HIGH | store/kuaishou/_store_impl.py | 0 | store comment to mongodb args: comment_item: comment data | STRING |
| HIGH | store/zhihu/_store_impl.py | 0 | store comment to mongodb args: comment_item: comment data | STRING |
| HIGH | store/weibo/_store_impl.py | 0 | calculate the prefix sorting number for data save files, supporting writing to different files for each run args: file_s | STRING |
| HIGH | store/tieba/_store_impl.py | 0 | calculate the prefix sorting number for data save files, supporting writing to different files for each run args: file_s | STRING |
| HIGH | store/kuaishou/_store_impl.py | 0 | calculate the prefix sorting number for data save files, supporting writing to different files for each run args: file_s | STRING |
| HIGH | store/zhihu/_store_impl.py | 0 | calculate the prefix sorting number for data save files, supporting writing to different files for each run args: file_s | STRING |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | recv_sms.py | 46 | def extract_verification_code(message: str) -> str: | CODE |
| LOW⚡ | main.py | 86 | async def _generate_wordcloud_if_needed() -> None: | CODE |
| LOW | database/db_session.py | 31 | async def create_database_if_not_exists(db_type: str): | CODE |
| LOW | tools/crawler_util.py | 66 | async def find_qrcode_img_from_canvas(page: Page, canvas_selector: str) -> str: | CODE |
| LOW | tools/crawler_util.py | 148 | async def convert_browser_context_cookies( | CODE |
| LOW | tools/crawler_util.py | 159 | def convert_str_cookie_to_dict(cookie_str: str) -> Dict: | CODE |
| LOW | tools/crawler_util.py | 177 | def match_interact_info_count(count_str: str) -> int: | CODE |
| LOW | tools/crawler_util.py | 226 | def extract_url_params_to_dict(url: str) -> Dict: | CODE |
| LOW | tools/async_file_writer.py | 62 | async def write_single_item_to_json(self, item: Dict, item_type: str): | CODE |
| LOW | tools/async_file_writer.py | 82 | async def generate_wordcloud_from_comments(self): | CODE |
| LOW | tools/words.py | 50 | async def generate_word_frequency_and_cloud(self, data, save_words_prefix): | CODE |
| LOW | tools/cdp_browser.py | 47 | def _register_cleanup_handlers(self): | CODE |
| LOW | tools/cdp_browser.py | 140 | async def _connect_existing_browser( | CODE |
| LOW | tools/cdp_browser.py | 288 | async def _get_browser_websocket_url(self, debug_port: int) -> str: | CODE |
| LOW | tools/time_util.py | 60 | def get_time_str_from_unix_time(unixtime): | CODE |
| LOW | tools/time_util.py | 71 | def get_date_str_from_unix_time(unixtime): | CODE |
| LOW | tools/time_util.py | 82 | def get_unix_time_from_time_str(time_str): | CODE |
| LOW | tools/file_header_manager.py | 67 | def generate_copyright_header(relative_path: str) -> str: | CODE |
| LOW | proxy/proxy_mixin.py | 57 | async def _refresh_proxy_if_expired(self) -> None: | CODE |
| LOW | test/test_utils.py | 38 | async def test_convert_browser_context_cookies_uses_url_filter(): | CODE |
| LOW | test/test_proxy_ip_pool.py | 94 | async def test_proxy_pool_auto_refresh(self): | CODE |
| LOW | test/test_proxy_ip_pool.py | 136 | async def test_ip_expiration_standalone(self): | CODE |
| LOW | test/test_proxy_ip_pool.py | 271 | async def test_create_ip_pool_unknown_provider_raises_value_error(self): | CODE |
| LOW | test/test_mongodb_integration.py | 222 | def test_xhs_store_implementation(self): | CODE |
| LOW | test/test_mongodb_integration.py | 274 | def test_douyin_store_implementation(self): | CODE |
| LOW | test/test_mongodb_integration.py | 319 | def test_concurrent_operations(self): | CODE |
| LOW | tests/test_weibo_no_user_info.py | 124 | def test_weibo_note_masks_user_info(): | CODE |
| LOW | tests/test_weibo_no_user_info.py | 164 | def test_weibo_comment_masks_user_info(): | CODE |
| LOW | tests/test_weibo_no_user_info.py | 203 | def test_weibo_store_end_to_end_sqlite(): | CODE |
| LOW | tests/test_cmd_arg_tieba.py | 10 | async def test_tieba_detail_cli_sets_specified_ids(): | CODE |
| LOW | tests/test_cmd_arg_tieba.py | 26 | async def test_tieba_creator_cli_sets_creator_urls(): | CODE |
| LOW | tests/test_cmd_arg_tieba.py | 45 | async def test_tieba_detail_reads_runtime_specified_ids(monkeypatch): | CODE |
| LOW | tests/test_store_factory.py | 67 | def test_create_mongodb_store(self): | CODE |
| LOW | tests/test_store_factory.py | 87 | def test_invalid_store_option(self): | CODE |
| LOW | tests/test_store_factory.py | 94 | def test_all_stores_registered(self): | CODE |
| LOW | tests/test_static_proxy_provider.py | 9 | def test_default_proxy_provider_remains_existing_provider(): | CODE |
| LOW | tests/test_static_proxy_provider.py | 16 | async def test_static_proxy_provider_parses_proxy_url(monkeypatch): | CODE |
| LOW | tests/test_static_proxy_provider.py | 32 | async def test_static_proxy_provider_rejects_invalid_url(monkeypatch): | CODE |
| LOW | tests/test_static_proxy_provider.py | 41 | async def test_static_proxy_pool_disables_validation(monkeypatch): | CODE |
| LOW | tests/test_api_limits.py | 12 | async def test_cmd_arg_crawler_max_notes_count(): | CODE |
| LOW | tests/test_api_limits.py | 29 | def test_crawler_manager_build_command(): | CODE |
| LOW | tests/test_api_limits.py | 65 | def test_api_start_crawler_with_limits(): | CODE |
| LOW | tests/test_api_limits.py | 90 | def test_api_start_crawler_without_limits(): | CODE |
| LOW | tests/test_api_limits.py | 123 | def test_api_rejects_invalid_limits(field_name, value): | CODE |
| LOW | tests/test_cdp_browser.py | 11 | async def test_existing_browser_connects_directly_to_devtools_browser(monkeypatch): | CODE |
| LOW | tests/test_cdp_browser.py | 37 | async def test_existing_browser_falls_back_to_discovered_websocket_url(monkeypatch): | CODE |
| LOW | tests/test_cdp_browser.py | 74 | async def test_launched_browser_uses_discovered_websocket_url(monkeypatch): | CODE |
| LOW | tests/test_no_user_info.py | 33 | def test_orm_has_no_forbidden_columns(): | CODE |
| LOW | tests/test_no_user_info.py | 47 | def test_creator_tables_removed(): | CODE |
| LOW | tests/test_no_user_info.py | 55 | def test_content_tables_have_creator_hash(): | CODE |
| LOW | tests/test_no_user_info.py | 99 | def test_xhs_note_extraction_masks_user_info(): | CODE |
| LOW | tests/test_no_user_info.py | 132 | def test_tieba_note_extraction_masks_user_info(): | CODE |
| LOW | tests/test_no_user_info.py | 148 | def test_tieba_comment_extraction_masks_user_info(): | CODE |
| LOW | tests/test_no_user_info.py | 164 | def test_zhihu_comment_extraction_masks_user_info(): | CODE |
| LOW | tests/test_no_user_info.py | 182 | def test_bilibili_video_dict_masks_user_info(): | CODE |
| LOW | tests/test_no_user_info.py | 216 | def test_store_no_forbidden_dict_keys(): | CODE |
| LOW | tests/test_no_user_info.py | 226 | def test_store_no_creator_orm_imports(): | CODE |
| LOW | tests/test_excel_store.py | 173 | def test_empty_sheets_removed(self, excel_store): | CODE |
| LOW | tests/test_excel_store.py | 191 | def test_excel_import_availability(): | CODE |
| LOW | tests/test_excel_store.py | 213 | def test_get_instance_returns_same_instance(self): | CODE |
| 123 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | recv_sms.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | var.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | main.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | database/db.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | database/models.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | database/mongodb_store_base.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | database/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | database/db_session.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/crawler_util.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/async_file_writer.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/easing.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/utils.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/slider_util.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/words.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/browser_launcher.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/cdp_browser.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/app_runner.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/time_util.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tools/file_header_manager.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/__init__.py | 21 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/types.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/proxy_mixin.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/proxy_ip_pool.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/base_proxy.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/providers/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/providers/kuaidl_proxy.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/providers/jishu_http_proxy.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | proxy/providers/wandou_http_proxy.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | cache/local_cache.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | cache/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | cache/abs_cache.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | cache/cache_factory.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | cache/redis_cache.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | test/test_utils.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | test/test_db_sync.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | test/test_db_sync.py | 221 | # [*] Changed tables: | COMMENT |
| LOW | test/test_db_sync.py | 241 | # >>> Manual confirmation required: Synchronize ORM model to MySQL database? (y/N): y | COMMENT |
| LOW | test/test_expiring_local_cache.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | test/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | test/test_redis_cache.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | test/test_proxy_ip_pool.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | test/test_mongodb_integration.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/zhihu_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/base_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/base_config.py | 41 | COMMENT | |
| LOW | config/base_config.py | 61 | # CDP 调试端口,用于与浏览器通信 | COMMENT |
| LOW | config/ks_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/dy_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/dy_config.py | 21 | PUBLISH_TIME_TYPE = 0 | COMMENT |
| LOW | config/tieba_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/xhs_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/db_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/weibo_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | config/bilibili_config.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tests/conftest.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tests/test_store_factory.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| LOW | tests/__init__.py | 1 | # -*- coding: utf-8 -*- | COMMENT |
| 102 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | main.py | 32 | CODE | |
| LOW | database/db.py | 23 | CODE | |
| LOW | database/models.py | 27 | CODE | |
| LOW | database/models.py | 29 | CODE | |
| LOW | tools/crawler_util.py | 27 | CODE | |
| LOW | tools/crawler_util.py | 35 | CODE | |
| LOW | tools/async_file_writer.py | 24 | CODE | |
| LOW | tools/utils.py | 24 | CODE | |
| LOW | tools/utils.py | 25 | CODE | |
| LOW | tools/utils.py | 26 | CODE | |
| LOW | tools/browser_launcher.py | 28 | CODE | |
| LOW | tools/browser_launcher.py | 29 | CODE | |
| LOW | tools/app_runner.py | 21 | CODE | |
| LOW | proxy/__init__.py | 25 | CODE | |
| LOW | proxy/proxy_mixin.py | 31 | CODE | |
| LOW | proxy/proxy_ip_pool.py | 29 | CODE | |
| LOW | proxy/providers/__init__.py | 25 | CODE | |
| LOW | proxy/providers/__init__.py | 26 | CODE | |
| LOW | proxy/providers/__init__.py | 27 | CODE | |
| LOW | proxy/providers/kuaidl_proxy.py | 29 | CODE | |
| LOW | proxy/providers/jishu_http_proxy.py | 28 | CODE | |
| LOW | proxy/providers/wandou_http_proxy.py | 28 | CODE | |
| LOW | test/test_db_sync.py | 27 | CODE | |
| LOW | config/base_config.py | 143 | CODE | |
| LOW | config/base_config.py | 144 | CODE | |
| LOW | config/base_config.py | 145 | CODE | |
| LOW | config/base_config.py | 146 | CODE | |
| LOW | config/base_config.py | 147 | CODE | |
| LOW | config/base_config.py | 148 | CODE | |
| LOW | config/base_config.py | 149 | CODE | |
| LOW | config/__init__.py | 21 | CODE | |
| LOW | config/__init__.py | 22 | CODE | |
| LOW | tests/test_store_factory.py | 24 | CODE | |
| LOW | tests/test_store_factory.py | 27 | CODE | |
| LOW | tests/test_excel_store.py | 25 | CODE | |
| LOW | tests/test_excel_store.py | 26 | CODE | |
| LOW | cmd_arg/__init__.py | 21 | CODE | |
| LOW | cmd_arg/arg.py | 21 | CODE | |
| LOW | media_platform/bilibili/client.py | 40 | CODE | |
| LOW | media_platform/bilibili/__init__.py | 26 | CODE | |
| LOW | media_platform/xhs/client.py | 23 | CODE | |
| LOW | media_platform/xhs/client.py | 36 | CODE | |
| LOW | media_platform/xhs/__init__.py | 21 | CODE | |
| LOW | media_platform/xhs/__init__.py | 22 | CODE | |
| LOW | media_platform/weibo/client.py | 43 | CODE | |
| LOW | media_platform/weibo/__init__.py | 25 | CODE | |
| LOW | media_platform/weibo/__init__.py | 26 | CODE | |
| LOW | media_platform/weibo/__init__.py | 27 | CODE | |
| LOW | media_platform/weibo/core.py | 29 | CODE | |
| LOW | media_platform/tieba/__init__.py | 22 | CODE | |
| LOW | media_platform/tieba/core.py | 24 | CODE | |
| LOW | media_platform/douyin/client.py | 38 | CODE | |
| LOW | media_platform/douyin/client.py | 39 | CODE | |
| LOW | media_platform/douyin/client.py | 40 | CODE | |
| LOW | media_platform/douyin/client.py | 36 | CODE | |
| LOW | media_platform/douyin/__init__.py | 21 | CODE | |
| LOW | media_platform/douyin/core.py | 24 | CODE | |
| LOW | media_platform/douyin/help.py | 29 | CODE | |
| LOW | media_platform/kuaishou/client.py | 27 | CODE | |
| LOW | media_platform/kuaishou/client.py | 37 | CODE | |
| 84 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | main.py | 82 | except Exception as e: | CODE |
| LOW⚡ | main.py | 96 | except Exception as e: | CODE |
| LOW | main.py | 129 | except Exception as e: | CODE |
| LOW | main.py | 137 | except Exception as e: | CODE |
| LOW | main.py | 158 | except Exception: | CODE |
| LOW | database/mongodb_store_base.py | 75 | except Exception as e: | CODE |
| LOW | database/mongodb_store_base.py | 111 | except Exception as e: | CODE |
| LOW | database/mongodb_store_base.py | 120 | except Exception as e: | CODE |
| LOW | database/mongodb_store_base.py | 132 | except Exception as e: | CODE |
| LOW | database/mongodb_store_base.py | 142 | except Exception as e: | CODE |
| LOW | database/db_session.py | 98 | except Exception as e: | CODE |
| LOW | tools/crawler_util.py | 61 | except Exception as e: | CODE |
| LOW | tools/async_file_writer.py | 146 | except Exception as e: | CODE |
| LOW | tools/browser_launcher.py | 187 | except Exception as e: | CODE |
| LOW | tools/browser_launcher.py | 206 | except Exception: | CODE |
| LOW | tools/browser_launcher.py | 238 | except Exception: | CODE |
| LOW | tools/browser_launcher.py | 288 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 135 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 246 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 309 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 330 | except Exception as direct_error: | CODE |
| LOW | tools/cdp_browser.py | 356 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 410 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 421 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 432 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 457 | except Exception as context_error: | CODE |
| LOW | tools/cdp_browser.py | 478 | except Exception as browser_error: | CODE |
| LOW | tools/cdp_browser.py | 507 | except Exception as e: | CODE |
| LOW | tools/cdp_browser.py | 533 | except Exception as e: | CODE |
| LOW | tools/app_runner.py | 82 | except Exception: | CODE |
| LOW | tools/app_runner.py | 102 | except Exception as e: | CODE |
| LOW | tools/time_util.py | 92 | except Exception as e: | CODE |
| LOW | tools/file_header_manager.py | 209 | except Exception as e: | CODE |
| LOW | proxy/proxy_ip_pool.py | 93 | except Exception as e: | CODE |
| LOW | proxy/proxy_ip_pool.py | 186 | except Exception as e: | CODE |
| LOW | proxy/base_proxy.py | 82 | except Exception as e: | CODE |
| LOW | test/test_db_sync.py | 196 | except Exception as e: | CODE |
| MEDIUM | test/test_db_sync.py | 197 | print(f"Error processing MySQL: {e}") | CODE |
| LOW | test/test_db_sync.py | 213 | except Exception as e: | CODE |
| MEDIUM | test/test_db_sync.py | 214 | print(f"Error processing SQLite: {e}") | CODE |
| LOW | test/test_mongodb_integration.py | 42 | except Exception as e: | CODE |
| LOW | test/test_mongodb_integration.py | 86 | except Exception as e: | CODE |
| MEDIUM | test/test_mongodb_integration.py | 36 | def setUpClass(cls): | CODE |
| LOW | media_platform/bilibili/client.py | 144 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 223 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 321 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 365 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 552 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 570 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 635 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 679 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 702 | except Exception as e: | CODE |
| LOW | media_platform/bilibili/core.py | 725 | except Exception as e: | CODE |
| LOW | media_platform/xhs/client.py | 256 | except Exception as e: | CODE |
| LOW | media_platform/xhs/client.py | 527 | except Exception as e: | CODE |
| LOW | media_platform/xhs/client.py | 532 | except Exception as e: | CODE |
| LOW | media_platform/xhs/core.py | 447 | except Exception as e: | CODE |
| LOW | media_platform/xhs/login.py | 68 | except Exception: | CODE |
| LOW | media_platform/xhs/login.py | 117 | except Exception as e: | CODE |
| LOW | media_platform/xhs/help.py | 368 | except Exception as e: | CODE |
| 50 more matches not shown… | ||||
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | main.py | 123 | CODE | |
| LOW | tools/async_file_writer.py | 82 | CODE | |
| LOW | tools/slider_util.py | 84 | CODE | |
| LOW | tools/browser_launcher.py | 191 | CODE | |
| LOW | tools/browser_launcher.py | 214 | CODE | |
| LOW | tools/browser_launcher.py | 241 | CODE | |
| LOW | tools/cdp_browser.py | 437 | CODE | |
| LOW | cache/redis_cache.py | 77 | CODE | |
| LOW | test/test_db_sync.py | 101 | CODE | |
| LOW | test/test_db_sync.py | 133 | CODE | |
| LOW | test/test_proxy_ip_pool.py | 271 | CODE | |
| LOW | cmd_arg/arg.py | 154 | CODE | |
| LOW | cmd_arg/arg.py | 160 | CODE | |
| LOW | media_platform/bilibili/client.py | 260 | CODE | |
| LOW | media_platform/bilibili/core.py | 70 | CODE | |
| LOW | media_platform/bilibili/core.py | 183 | CODE | |
| LOW | media_platform/bilibili/core.py | 240 | CODE | |
| LOW | media_platform/xhs/playwright_sign.py | 72 | CODE | |
| LOW | media_platform/xhs/client.py | 456 | CODE | |
| LOW | media_platform/xhs/core.py | 129 | CODE | |
| LOW | media_platform/weibo/core.py | 137 | CODE | |
| LOW | media_platform/weibo/help.py | 29 | CODE | |
| LOW | media_platform/tieba/client.py | 516 | CODE | |
| LOW | media_platform/tieba/core.py | 148 | CODE | |
| LOW | media_platform/tieba/core.py | 402 | CODE | |
| LOW | media_platform/tieba/help.py | 290 | CODE | |
| LOW | media_platform/douyin/client.py | 253 | CODE | |
| LOW | media_platform/douyin/core.py | 127 | CODE | |
| LOW | media_platform/douyin/core.py | 183 | CODE | |
| LOW | media_platform/zhihu/core.py | 150 | CODE | |
| LOW | media_platform/zhihu/help.py | 77 | CODE | |
| LOW | api/routers/data.py | 33 | CODE | |
| LOW | api/routers/data.py | 62 | CODE | |
| LOW | api/routers/data.py | 99 | CODE | |
| LOW | api/routers/data.py | 191 | CODE | |
| LOW | api/routers/websocket.py | 90 | CODE | |
| LOW | api/services/crawler_manager.py | 80 | CODE | |
| LOW | api/services/crawler_manager.py | 241 | CODE | |
| LOW | store/excel_store_base.py | 171 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | database/db_session.py | 46 | # Check if database exists | COMMENT |
| LOW | tools/browser_launcher.py | 97 | # Check if path exists and is executable | COMMENT |
| LOW | tools/cdp_browser.py | 448 | # Check if context is already closed | COMMENT |
| LOW | tools/cdp_browser.py | 472 | # Check if browser is still connected | COMMENT |
| LOW | tools/file_header_manager.py | 100 | # Check if contains Copyright keyword | COMMENT |
| LOW | tools/file_header_manager.py | 130 | # Check if first line is shebang | COMMENT |
| LOW | media_platform/bilibili/client.py | 70 | # Check if proxy has expired before each request | COMMENT |
| LOW | media_platform/bilibili/client.py | 307 | # Check if is_end and next exist | COMMENT |
| LOW | media_platform/bilibili/core.py | 177 | end_day = (start_day + timedelta(days=1) - timedelta(seconds=1)) # Set end_day to start_day + 1 day - 1 sec | CODE |
| LOW | media_platform/bilibili/core.py | 179 | end_day = (end_day + timedelta(days=1) - timedelta(seconds=1)) # Set end_day to end_day + 1 day - 1 second | CODE |
| LOW | media_platform/xhs/client.py | 127 | # Check if proxy is expired before each request | COMMENT |
| LOW | media_platform/xhs/client.py | 208 | # Check if proxy is expired before request | COMMENT |
| LOW | media_platform/weibo/client.py | 74 | # Check if proxy is expired before each request | COMMENT |
| LOW | media_platform/weibo/core.py | 435 | # Check if it's a long text | COMMENT |
| LOW | media_platform/tieba/client.py | 250 | # Check if proxy is expired before each request | COMMENT |
| LOW | media_platform/douyin/login.py | 150 | # Check if there is slider verification | COMMENT |
| LOW | media_platform/douyin/help.py | 121 | # Check if it's a short link (v.douyin.com) | COMMENT |
| LOW | media_platform/kuaishou/client.py | 67 | # Check if proxy is expired before each request | COMMENT |
| LOW | media_platform/zhihu/client.py | 97 | # Check if proxy is expired before each request | COMMENT |
| LOW | store/excel_store_base.py | 373 | # Check if there are any sheets left | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| HIGH | media_platform/tieba/core.py | 502 | get: () => undefined, | CODE |
| HIGH⚡ | media_platform/douyin/help.py | 95 | "([params, post_data, ua]) => window.bdms.init._v[2].p[42].apply(null, [0, 1, 8, params, post_data, ua])", | CODE |
| HIGH | api/main.py | 79 | "note": "WebUI not found, please build it first: cd webui && npm run build" | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | media_platform/tieba/core.py | 415 | # Step 1: Visit Baidu homepage | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 419 | # Step 2: Wait for page loading, using delay setting from config file | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 423 | # Step 3: Find and click "Tieba" link | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 449 | # Step 4: Click Tieba link (check if it will open in a new tab) | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 479 | # Step 5: Wait for page to stabilize, using delay setting from config file | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW⚡ | media_platform/tieba/core.py | 415 | # Step 1: Visit Baidu homepage | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 419 | # Step 2: Wait for page loading, using delay setting from config file | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 423 | # Step 3: Find and click "Tieba" link | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 449 | # Step 4: Click Tieba link (check if it will open in a new tab) | COMMENT |
| LOW⚡ | media_platform/tieba/core.py | 479 | # Step 5: Wait for page to stabilize, using delay setting from config file | COMMENT |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | test/test_mongodb_integration.py | 360 | CODE | |
| LOW | cmd_arg/arg.py | 160 | CODE | |
| LOW | media_platform/bilibili/client.py | 390 | CODE | |
| LOW | media_platform/tieba/client.py | 216 | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | api/routers/__init__.py | 23 | __all__ = ["crawler_router", "data_router", "websocket_router"] | CODE |
| LOW | api/schemas/__init__.py | 29 | __all__ = [ | CODE |
| LOW | api/services/__init__.py | 21 | __all__ = ["CrawlerManager", "crawler_manager"] | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| MEDIUM | package-lock.json | 3425 | "resolved": "https://registry.npmmirror.com/robust-predicates/-/robust-predicates-3.0.2.tgz", | CODE |
| Severity | File | Line | Snippet | Context |
|---|---|---|---|---|
| LOW | tools/time_util.py | 130 | # Example usage | COMMENT |