Commit Graph

54 Commits

Author SHA1 Message Date
lintsinghua 2df1b39e08 feat: Introduce Kunlun agent tool, add Docker and sandbox environment checks, and update agent services and frontend dialogs. 2025-12-15 02:00:34 +08:00
lintsinghua c64eddac7c feat(agent): 新增多语言代码测试和漏洞验证工具并增强错误处理
新增 PHP、Python、JavaScript 等多语言代码测试工具和命令注入、SQL 注入等专用漏洞验证工具
优化错误处理逻辑,提供更详细的错误信息和堆栈跟踪
增强 JSON 解析器,优先使用 json-repair 库处理复杂格式
改进 Agent 响应解析逻辑,更健壮地提取思考内容和操作指令
完善沙箱管理器的初始化和错误处理机制
2025-12-14 17:20:54 +08:00
lintsinghua a85cdb687d refactor(agent): 重构Agent架构并实现动态Agent树
feat(sandbox): 增强沙箱工具的错误处理和日志记录
test(verification): 添加沙箱验证工具的独立测试脚本
docs: 删除过时的文档和测试文件
2025-12-14 14:52:06 +08:00
lintsinghua e29231cc73 feat(安全扫描): 扩展高风险文件模式和危险模式库支持更多语言和文件类型
增加对Swift、Kotlin、Rust等语言的支持
添加iOS和Android相关配置文件类型
补充移动端开发中常见的安全风险模式
2025-12-14 14:00:04 +08:00
lintsinghua 95929a467b feat(沙箱): 强制使用沙箱验证并增强沙箱功能
- 移除 LLM 模拟验证工具,强制使用沙箱进行漏洞验证
- 增强沙箱容器配置,添加/tmp挂载点
- 扩展沙箱基础镜像,增加多种语言环境支持
- 添加RCE漏洞验证脚本用于测试沙箱功能
- 更新验证Agent的系统提示,明确必须使用沙箱工具
2025-12-14 12:43:44 +08:00
lintsinghua 189274fd56 feat(sandbox): 重构安全工具以使用沙箱环境执行
重构所有安全工具(Semgrep, Bandit等)使用Docker沙箱环境执行,提升安全隔离性
更新Dockerfile基础镜像并集成常用安全扫描工具
添加沙箱管理器的工具命令执行方法,支持资源限制和临时文件系统
2025-12-14 00:07:04 +08:00
lintsinghua 3d4f90c547 feat: Add `marked` for improved Markdown to HTML report generation and refined download handling. 2025-12-13 21:38:11 +08:00
lintsinghua a9a22b91c7 fix: Display aggregated findings for the orchestrator and task, and show sub-agent duration/status in UI. 2025-12-13 21:06:22 +08:00
lintsinghua 2a332d6eda feat: Improve report generation and statistics by normalizing severity comparisons and leveraging backend-calculated task metrics. 2025-12-13 20:42:10 +08:00
lintsinghua d449e2ba78 Fix: Improve agent stream stability by preventing unnecessary reconnections and correctly draining buffered events. 2025-12-13 20:21:30 +08:00
lintsinghua 507fe393d1 fix(agent): 改进沙盒工具初始化的日志记录
添加更详细的成功和失败日志信息,包括工具列表和完整的异常堆栈
2025-12-13 18:58:30 +08:00
lintsinghua 4e4dd05ddb feat(agent): 增强漏洞发现处理流程和前端兼容性
- 后端添加对旧事件类型'finding'的兼容支持
- 改进漏洞发现标准化和去重逻辑
- 新增PoC生成要求和相关字段
- 优化沙箱配置初始化流程
- 前端添加ADD_FINDING操作和状态管理
- 增强事件流处理和序列号过滤
- 改进历史事件加载和SSE连接逻辑
- 添加漏洞验证状态和PoC信息到报告
2025-12-13 18:45:05 +08:00
lintsinghua 6d98f29fa6 feat: 新增安全工具集成和漏洞知识库扩展
- 添加 Bandit 和 Safety 安全工具到依赖项
- 新增 CSRF、业务逻辑和开放重定向漏洞知识文档
- 实现安全工具一键安装脚本和文档
- 改进模式匹配工具支持直接文件扫描
- 增强遥测模块和 Agent 审计功能
- 修复验证节点中 findings 合并逻辑
- 优化前端 Agent 审计界面和状态展示
2025-12-13 12:35:03 +08:00
lintsinghua 3db20a3afb feat(agent): enhance error handling and project scope filtering
- Downgrade Python version from 3.13 to 3.11.12 for compatibility
- Improve empty LLM response handling with better diagnostics and retry logic in AnalysisAgent
- Add detailed logging for empty response retries with token count and iteration tracking
- Implement fallback result generation instead of immediate failure on consecutive empty responses
- Enhance stream error handling with partial content recovery and error message propagation
- Add comprehensive exception handling in stream_llm_call to prevent error suppression
- Implement project scope filtering to ensure consistent filtered views across Orchestrator and sub-agents
- Track filtered files and directories separately when target_files are specified
- Add scope_limited flag and scope_message to project structure for transparency
- Remove manual progress_percentage setting and rely on computed property for COMPLETED status
- Improve code comments with diagnostic markers (🔥) for critical sections
2025-12-12 16:36:39 +08:00
lintsinghua 31dc476015 feat: Improve streaming LLM token usage reporting by adding input estimation, requesting usage via `stream_options`, and providing fallback estimation. 2025-12-12 16:30:57 +08:00
lintsinghua b392e049e8 fix(agent_tasks): update task phase to REPORTING on completion
- Change task phase from COMPLETED to REPORTING when agent task finishes
- Ensures proper phase progression through task lifecycle
- Aligns with agent workflow state management requirements
2025-12-12 16:09:50 +08:00
lintsinghua eed111c04d chore: reduce logging verbosity and clean up file formatting
- Change logger.info to logger.debug in agent_tasks.py streaming and tree endpoints
- Disable SQLAlchemy echo mode in database session configuration
- Suppress uvicorn access logs and LiteLLM INFO level logging in main application
- Remove LogViewer component and LogsPage from frontend
- Add trailing newlines to multiple backend configuration and model files
- Update frontend routing to remove logs page reference
- Improve application startup logging clarity by filtering verbose third-party logs
2025-12-12 15:50:48 +08:00
lintsinghua f05c0073e1 feat(agent): implement comprehensive agent architecture with knowledge base and persistence layer
- Add database migrations for agent checkpoints and tree node tracking
- Implement core agent execution framework with executor, state management, and message handling
- Create knowledge base system with framework-specific modules (Django, FastAPI, Flask, Express, React, Supabase)
- Add vulnerability knowledge modules covering authentication, cryptography, injection, XSS, XXE, SSRF, path traversal, deserialization, and race conditions
- Introduce new agent tools: thinking tool, reporting tool, and agent-specific utilities
- Implement LLM memory compression and prompt caching for improved performance
- Add agent registry and persistence layer for checkpoint management
- Refactor agent implementations (analysis, recon, verification, orchestrator) with enhanced capabilities
- Remove legacy agent implementations (analysis_v2, react_agent)
- Update API endpoints for agent task creation and project management
- Add frontend components for agent task creation and enhanced audit UI
- Consolidate agent service architecture with improved separation of concerns
- This refactoring provides a scalable foundation for multi-agent collaboration with knowledge-driven decision making and state persistence
2025-12-12 15:27:12 +08:00
lintsinghua 147dfbaf5e feat(agent): enhance streaming with in-memory event manager and fallback polling
- Implement dual-mode streaming: prioritize in-memory EventManager for running tasks with thinking_token support
- Add fallback to database polling for completed tasks without thinking_token replay capability
- Introduce SSE event formatter utility for consistent event serialization across streaming modes
- Add 10ms micro-delay for thinking_token events to ensure proper TCP packet separation and frontend incremental rendering
- Refactor stream_agent_with_thinking endpoint to support both runtime and historical event streaming
- Update event filtering logic to handle both in-memory and database event sources
- Improve logging with debug markers for thinking_token tracking and stream mode selection
- Optimize polling intervals: 0.3s for running tasks, 2.0s for completed tasks
- Reduce idle timeout from 10 minutes to 1 minute for completed task streams
- Update frontend useAgentStream hook to handle unified event format from dual-mode streaming
- Enhance AgentAudit UI to properly display streamed events from both sources
2025-12-12 10:39:32 +08:00
lintsinghua 70776ee5fd feat: Introduce structured agent collaboration with `TaskHandoff` and `analysis_v2` agent, updating core agent logic, tools, and audit UI. 2025-12-11 23:29:04 +08:00
lintsinghua 8938a8a3c9 feat(agent): enhance agent functionality with LLM-driven decision-making and event handling
- Introduce LLM-driven decision-making across various agents, allowing for dynamic adjustments based on real-time analysis.
- Implement new event types for LLM thinking, decisions, actions, and observations to enrich the event streaming experience.
- Update agent task responses to include additional metrics for better tracking of task progress and outcomes.
- Refactor UI components to highlight LLM-related events and improve user interaction during audits.
- Enhance API endpoints to support new event structures and improve overall error handling.
2025-12-11 21:14:32 +08:00
lintsinghua 58c918f557 feat(agent): implement streaming support for agent events and enhance UI components
- Introduce streaming capabilities for agent events, allowing real-time updates during audits.
- Add new hooks for managing agent stream events in React components.
- Enhance the AgentAudit page to display LLM thinking processes and tool call details in real-time.
- Update API endpoints to support streaming event data and improve error handling.
- Refactor UI components for better organization and user experience during audits.
2025-12-11 20:33:46 +08:00
lintsinghua a43ebf1793 refactor(agent): update task completion fields and enhance status handling
- Rename 'finished_at' to 'completed_at' in AgentTaskResponse and related functions for clarity.
- Introduce 'RUNNING' status in AgentTaskStatus to better represent task states.
- Update task configuration handling in AgentRunner to improve initialization logic.
- Enhance UI to reflect new 'RUNNING' and 'REPORTING' statuses in the AgentAudit component.
2025-12-11 19:26:47 +08:00
lintsinghua 129112e4d7 refactor(agent): rename metadata fields and enhance task configuration
- Rename 'metadata' columns to 'event_metadata' and 'finding_metadata' for clarity in database schema.
- Update AgentRunner to construct task configuration from task fields, ensuring default values are applied.
- Improve UI to handle potential null values for progress percentage and security score, enhancing user experience.
2025-12-11 19:23:21 +08:00
lintsinghua a33f60bd44 refactor(agent_tasks): streamline task configuration parameters
- Simplify the creation of agent tasks by directly assigning parameters instead of using a config dictionary.
- Set default values for verification level, max iterations, and timeout seconds to enhance usability.
2025-12-11 19:11:09 +08:00
lintsinghua 9bc114af1f feat(agent): implement Agent audit module with LangGraph integration
- Introduce new Agent audit functionality for autonomous code security analysis and vulnerability verification.
- Add API endpoints for managing Agent tasks and configurations.
- Implement UI components for Agent mode selection and embedding model configuration.
- Enhance the overall architecture with a focus on RAG (Retrieval-Augmented Generation) for improved code semantic search.
- Create a sandbox environment for secure execution of vulnerability tests.
- Update documentation to include details on the new Agent audit features and usage instructions.
2025-12-11 19:09:10 +08:00
lintsinghua a995bef28c feat(audit): enhance file selection and exclude patterns UI
- Update demo data to mark running task as completed with audit results
- Add reset to defaults button for exclude patterns in CreateTaskDialog
- Implement quick-add buttons for common exclude patterns (.test., .spec., .min., coverage/, docs/, .md)
- Improve exclude patterns input with better placeholder text and visual feedback
- Refactor FileSelectionDialog with tree-based folder structure for better file organization
- Add file type icons (code files, config files, generic files) for improved visual distinction
- Implement folder expansion/collapse functionality with chevron indicators
- Add file filtering by extension and search capabilities
- Enhance UI with better spacing, visual hierarchy, and user guidance
- Improve accessibility with proper icon imports and component organization
2025-12-10 19:20:31 +08:00
lintsinghua b0861c1690 feat(file-selection): add custom exclude patterns for file filtering
- Add exclude_patterns parameter to get_project_files endpoint for custom file filtering
- Support JSON-formatted exclude patterns array in API requests
- Integrate custom exclude patterns into ZIP and repository file scanning workflows
- Update should_exclude and is_text_file functions to support user-defined patterns
- Pass exclude_patterns through scan configuration in both scan_zip and scan_stored_zip endpoints
- Add ScanRequest model field for exclude_patterns to support pattern specification
- Implement file filtering logic that respects both default and custom exclusion rules
- Add comprehensive unit and E2E tests for file selection and exclusion pattern functionality
- Enable users to customize which files are scanned by specifying glob patterns like ["node_modules/**", "*.log"]
2025-12-10 18:46:33 +08:00
lintsinghua a4b7efb1c9 fix(rules): fix unicode encoding error in rule set export filename
Use RFC 5987 filename*=UTF-8'' format for Content-Disposition header to support Chinese characters in exported filename
2025-12-10 17:43:56 +08:00
lintsinghua 11107ea455 feat(analysis): unify code analysis workflow with default template support
- Consolidate instant analysis to always use analyze_code_with_rules method
- Add use_default_template parameter to automatically select database default template when none specified
- Implement fallback logic to use hardcoded defaults only when no database template available
- Auto-select default prompt template on frontend page load
- Add visual labels for language and template selection dropdowns
- Improve template selector UX by removing empty default option and showing default indicator
- Simplifies analysis flow by removing conditional branching between analyze_code and analyze_code_with_rules
2025-12-09 23:13:06 +08:00
lintsinghua 4d71ed546a feat(prompts-scan): integrate prompt templates and audit rules into scan and analysis workflows
- Add user configuration retrieval with LLM API key decryption in prompt testing endpoint
- Support output language parameter in prompt template testing
- Integrate rule sets and prompt templates into ZIP file scanning process
- Add rule_set_id and prompt_template_id parameters to ScanRequest model
- Implement analyze_code_with_rules method for custom rule-based code analysis
- Add prompt_template_id support to instant analysis endpoint
- Update scan configuration to include rule set and prompt template selection
- Enhance error handling and logging in prompt testing with traceback output
- Extend InstantAnalysisRequest with optional prompt template ID parameter
- Add test code samples utility for prompt template validation
2025-12-09 23:03:08 +08:00
lintsinghua 357b9cc0a7 feat(prompts-rules): add prompt templates and audit rules management
- Add database migration (004) to create prompt_templates, audit_rule_sets, and audit_rules tables with proper indexes
- Create PromptTemplate and AuditRule models with relationships and validation
- Implement prompt template API endpoints for CRUD operations and testing
- Implement audit rules API endpoints for CRUD operations and rule set management
- Add prompt and rules schemas for request/response validation
- Create prompt template initialization service with default system templates
- Integrate LLM service with prompt template system for dynamic prompt selection
- Add frontend pages for PromptManager and AuditRules management
- Add API client utilities for prompts and rules endpoints
- Update API router to include new prompts and rules endpoints
- Update database initialization to seed default templates and rules
- Update sidebar navigation to include new management pages
- Update frontend routes to support new prompt and rules management pages
2025-12-09 21:42:00 +08:00
lintsinghua 4f0e8a2982 refactor(llm): simplify model name formatting logic in LiteLLMAdapter
- Remove redundant check for CUSTOM_BASE_URL_PROVIDERS
- Consolidate model name prefix logic into single code path
- Move prefix retrieval after model name validation
- Improve code clarity by eliminating unnecessary conditional branches
- Maintain backward compatibility with existing model name formats
2025-12-09 21:13:14 +08:00
lintsinghua 14b7c8cccc fix: 修复时间显示问题,使用带时区的UTC时间
- 将所有 datetime.utcnow() 替换为 datetime.now(timezone.utc)
- 修复 completed_at, started_at, updated_at, resolved_at 等时间字段
- 修复 JWT token 过期时间计算
- 修复数据导出和ZIP上传时间戳
- 调整README中项目管理和审计报告图片显示比例
2025-12-09 17:47:34 +08:00
lintsinghua 4eb81a5bca fix(scan): normalize file paths for cross-platform compatibility
- Add normalize_path() utility function to convert backslashes to forward slashes
- Ensure consistent path separators across Windows, Unix, and Mac platforms
- Apply path normalization to relative paths extracted from ZIP files
- Normalize target file paths before matching against scan configuration
- Update file filtering logic to use normalized path set for accurate matching
- Add inline comments explaining cross-platform path handling requirements
2025-12-09 15:38:17 +08:00
lintsinghua 3a297dda3f feat: add json-repair library as fallback for LLM response parsing
- Add json-repair>=0.30.0 to dependencies
- Add _repair_json_with_library method as final fallback in JSON parsing
- Update version to 2.0.0-beta.6
2025-12-08 21:45:58 +08:00
lintsinghua e856ad1a8d chore: rebrand XCodeReviewer to DeepAudit across all files
- Update project name from XCodeReviewer to DeepAudit in CI/CD workflows
- Replace Docker image names and release artifact naming conventions
- Update GitHub repository references in documentation files
- Modify database names and container names in setup instructions
- Update contributor badge and issue tracker links
- Rename frontend logo file from logo_deepaudit.png
- Update environment configuration examples and documentation
- Rebrand all references in CONTRIBUTING.md, DISCLAIMER.md, and README.md
- Update backend configuration and deployment documentation
- Ensure consistent naming across frontend and backend configurations
2025-12-08 21:35:09 +08:00
lintsinghua 40bd35ab08 feat(scanner): add file skipping logic and enhanced debug logging
- Add skipped_files counter to track empty and oversized files
- Implement file content validation to skip empty files
- Add file size check to skip files exceeding MAX_FILE_SIZE_BYTES limit
- Add detailed debug logging for file fetching, LLM analysis, and completion
- Add comprehensive error logging with traceback information for failed files
- Improve task status determination logic to distinguish between skipped files and LLM failures
- Mark tasks as completed when all files are empty/skipped instead of failing
- Update error messages to show actual failed file count instead of total files
- Enhance observability during code analysis workflow with emoji-prefixed status messages
2025-12-06 21:21:36 +08:00
lintsinghua cd80517f2d fix(llm): add debug logging for custom API base URL configuration
- Add debug print statement to log custom API base URL when configured
- Improves troubleshooting and visibility into LiteLLM adapter initialization
- Helps developers verify correct API endpoint configuration during runtime
2025-12-06 20:59:42 +08:00
lintsinghua 07810b309c feat(audit): refactor task creation with file selection and advanced options
- Add FileSelectionDialog component for granular file selection in audit tasks
- Extract task form logic into useTaskForm and useZipFile custom hooks
- Create modular components: BasicConfig, AdvancedOptions, ExcludePatterns, ProjectSelector, ZipFileSection
- Add file listing endpoint GET /projects/{id}/files with branch support
- Add branch listing endpoint GET /projects/{id}/branches for repository projects
- Implement ScanRequest model with file_paths, exclude_patterns, and branch_name fields
- Update scan endpoint to accept selective file scanning and exclude patterns
- Add branch_name and exclude_patterns fields to AuditTask model
- Enhance scanner service with GitHub and GitLab file/branch retrieval functions
- Improve CreateTaskDialog with better UX for repository and ZIP file scanning
- Support per-scan configuration storage in audit tasks
- Refactor repository scan services to handle file selection and branch parameters
2025-12-06 20:47:28 +08:00
lintsinghua 33c4df9645 refactor: 优化 Docker 部署配置
- 后端使用 uv 管理依赖,镜像内包含所有依赖
- 前端使用生产构建 + serve 提供静态文件
- 添加 WeasyPrint 完整系统依赖
- 修复 PDF 报告 Logo 显示问题
- 添加 .dockerignore 优化构建
- 更新部署文档和 GitHub Actions 工作流
- 前端端口从 5173 改为 3000
2025-12-05 20:51:22 +08:00
lintsinghua db3d8fd9f8 fix: 修复数据库初始化顺序问题
- docker-compose 启动时先运行 alembic upgrade head
- 优化数据库表不存在时的错误提示
2025-12-05 19:32:20 +08:00
lintsinghua f982d6a5e6 feat: 添加演示账户和演示数据,完善项目文档
- 添加演示账户 demo@example.com / demo123
- 创建 6 个演示项目和 15 个审计任务
- 完善 DEPLOYMENT.md、CONFIGURATION.md、LLM_PROVIDERS.md、FAQ.md 文档
- 更新 CONTRIBUTING.md、SECURITY.md、DISCLAIMER.md
- 前端版本号自动从 package.json 读取
- 仪表盘项目概览显示 6 个项目
- 版本号更新为 2.0.0-beta.3
2025-12-05 15:09:39 +08:00
lintsinghua 6fd305eac9 fix: 修复多个问题
- 修复创建ZIP项目时缺少source_type字段的问题
- 修复LiteLLM模型名称被错误解析为effort参数的问题
- 修复登录/注册页面无法正确显示验证错误的问题
- 添加Windows导出PDF需要GTK依赖的说明文档
2025-12-04 16:54:15 +08:00
lintsinghua 9054f0d2c5 feat: v2.0.0-test release
Major changes:
- Backend: Add report generator service with comprehensive analysis
- Backend: Enhanced scan and task endpoints
- Frontend: Refactor instant analysis page and export dialog
- Frontend: Optimize report export service
- Infrastructure: Simplify Dockerfile and update docker-compose
- Docs: Streamline README documentation
- Assets: Add logo with transparent background
2025-11-28 20:34:15 +08:00
lintsinghua c54212a8c9 refactor: 清理IndexedDB相关代码,统一使用后端ZIP存储
- 移除前端废弃的loadZipFile/saveZipFile函数
- ProjectDetail使用scanStoredZipFile替代loadZipFile
- RecycleBin移除手动删除ZIP文件逻辑
- 后端permanently_delete_project自动清理ZIP文件
2025-11-28 18:01:43 +08:00
lintsinghua f640bfbaba feat: 添加敏感信息加密存储功能
- 新增 encryption.py 加密服务,使用 Fernet 对称加密
- API Key、Token 等敏感字段在数据库中加密存储
- 读取时自动解密,兼容未加密的旧数据
- 优化配置保存后自动更新前端状态
2025-11-28 17:51:17 +08:00
lintsinghua bfef3b35a6 feat(projects): add ZIP file upload support and source type tracking
- Add source_type field to projects model to distinguish between repository and ZIP sources
- Implement ZIP file storage service with save, load, delete, and metadata operations
- Add database migration to populate source_type for existing projects
- Create ZIP upload endpoint with file handling and metadata tracking
- Add ZIP download endpoint for project file retrieval
- Implement project ZIP info endpoint to check file status and metadata
- Update project creation to support both repository and ZIP source types
- Add project type constants and utility functions for source type handling
- Update database export/import to include source_type field
- Extend frontend components to support ZIP file uploads in project creation
- Add instant analysis page for direct ZIP file scanning without project creation
- Update .gitignore to exclude uploaded ZIP files and metadata
- Enhance project detail and task detail pages with ZIP file management UI
2025-11-28 17:38:12 +08:00
lintsinghua 7091f891d1 feat(llm): enhance LLM connection testing with improved error handling and adapter instantiation
- Bypass LLMFactory cache during connection tests to ensure fresh API calls with latest configuration
- Directly instantiate native adapters (Baidu, Minimax, Doubao) and LiteLLM adapter based on provider type
- Add comprehensive error handling in LiteLLM adapter with specific exception catching for authentication, rate limiting, and connection errors
- Implement user-friendly error messages for common failure scenarios (invalid API key, authentication failure, timeout, connection issues)
- Add response validation to detect and report empty API responses
- Disable LiteLLM internal caching to guarantee actual API calls during testing
- Update available models list with 2025 latest models across all providers (Gemini, OpenAI, Claude, Qwen, DeepSeek, etc.)
- Improve error message clarity and debugging information in config endpoint
2025-11-28 16:53:01 +08:00
lintsinghua 22c528acf1 refactor(llm): consolidate LLM adapters with LiteLLM unified layer
- Replace individual adapter implementations (OpenAI, Claude, Gemini, DeepSeek, Qwen, Zhipu, Moonshot, Ollama) with unified LiteLLM adapter
- Keep native adapters for providers with special API formats (Baidu, MiniMax, Doubao)
- Update LLM factory to route requests through LiteLLM for supported providers
- Add test-llm endpoint to validate LLM connections with configurable timeout and token limits
- Add get-llm-providers endpoint to retrieve supported providers and their configurations
- Update config.py to ignore extra environment variables (VITE_* frontend variables)
- Refactor Baidu adapter to use new complete() method signature and improve error handling
- Update pyproject.toml dependencies to include litellm package
- Update env.example with new configuration options
- Simplify adapter initialization and reduce code duplication across multiple provider implementations
2025-11-28 16:41:39 +08:00