Research index
ImproBR Replication Package
Anonymus
Figshare1 Jan 2025Open access
NobleIDark:/48914/w1/20261005/818B7CF4DOI10.6084/m9.figshare.30086083.v5
ImproBR Replication Package
Anonymus · Figshare · 2025
NobleIDark:/48914/w1/20261005/818B7CF4DOI10.6084/m9.figshare.30086083.v5
Free to read at the publisher
You can read the abstract here, or the full text at the publisher.
Abstract
# ImproBR: AI-Powered Bug Report Improvement System ImproBR is an automated system for improving Minecraft bug reports using GPT 4o-mini, machine learning, natural language processing, and retrieval-augmented generation techniques. ### Dataset Information - **RQ1 Dataset**: 139 bug reports for improvement evaluation (stratified sampling) - **RQ2 Dataset**: 37 duplicate pairs for comparative analysis - **Statistical Validity**: ±3.04% margin of error at 95% confidence level ## Prerequisites - **Python 3.8 - 3.11.x**: Required for running the system (3.12+ not supported) - **Azure OpenAI API**: GPT-4o mini access required - **8GB+ RAM**: Recommended for vector embeddings - **Stable Internet**: For API calls and model downloads ## Setup Instructions 1. **Download and Extract** ```bash # Download ImproBR-Replication.zip from Figshare # Extract to your desired location cd ImproBR-Replication ``` 2. **Setup Virtual Environment** ```bash # Create virtual environment python -m venv improbr_env # Activate virtual environment source improbr_env/bin/activate # Linux/Mac # improbr_env\Scripts\activate # Windows ``` 3. **Install Dependencies** ```bash ` pip install -r requirements.txt ` python -m spacy download en_core_web_sm ``` 4. **Configure API Keys** ```bash cp config.py.template config.py # Edit config.py and fill in: # AZURE_API_KEY = "xxxxx" # AZURE_RESOURCE_NAME = "xxx" ``` 5. **Build Knowledge Base** ```bash python data/build_full_knowledge_base.py # This downloads and processes Minecraft Wiki pages for RAG (takes 10-15 minutes) # Creates: data/knowledge_base/vector_store/ ``` ## Usage **Process Sample Reports:** ```bash python improbr_pipeline.py --input-dir "RQ1-RQ2/RQ1/dataset/RQ1_139_raw_reports/raw/" --sample 5 --improve # Output: improbr_eval/ directory with subdirectories ``` **Improve All Raw Reports:** ```bash python improbr_pipeline.py --input-dir "RQ1-RQ2/RQ1/dataset/RQ1_139_raw_reports/raw/" --improve # Output: improbr_eval/ directory with subdirectories ``` **Fetch, Process and Improve New Reports:** ```bash python improbr_pipeline.py --count 10 --all # Output: improbr_eval/ directory with subdirectories ``` **Custom Output Location:** ```bash python improbr_pipeline.py --input-dir "RQ1-RQ2/RQ1/dataset/RQ1_139_raw_reports/raw/" --improve --output-dir "my_results" # Output: my_results/ directory with subdirectories ``` ### Output Structure All pipeline runs create the following directory structure: ``` improbr_eval/ # Default output directory (customize with --output-dir) ├── raw/ # Original fetched reports (if --fetch used) ├── preprocessed/ # Structured reports with extracted sections ├── detection/ # Quality issue detection results └── improved/ # Enhanced reports from ImproBR ``` ## Pipeline Options | Option | Description | |---------|-------------| | `--count N` | Fetch N new bug reports from Mojira | | `--input-dir PATH` | Use existing reports from directory | | `--sample N` | Randomly sample N reports from input directory | | `--output-dir PATH` | Set output directory for results | | `--fetch` | Run only the fetch step | | `--preprocess` | Run only the preprocess step | | `--improve` | Run only the improvement step | | `--all` | Run all steps: fetch, preprocess, and improve | | `--no-llm-validator` | Disable LLM preprocessing (faster, rule-based only) | ## System Architecture ImproBR processes reports through these steps: 1. **Fetch**: Download reports from Mojira API 2. **Preprocess**: Extract structured sections (LLM + heuristic fallback) 3. **Improve**: Enhance using knowledge retrieval and LLM generation ## Troubleshooting ### API Configuration Issues **LLM Connection Errors:** - Ensure `AZURE_API_KEY` is valid and not expired - Verify `AZURE_RESOURCE_NAME` matches your Azure OpenAI resource - Check that your deployment name is exactly "gpt-4o-mini" - Confirm you have sufficient quota/credits in your Azure account **Knowledge Base Missing:** ```bash # If you see "Vector store not found" error: python data/build_full_knowledge_base.py ``` ### Common Installation Issues **ChromaDB Installation Problems:** ```bash # If ChromaDB fails to install, try: pip install --upgrade pip pip install chromadb==0.4.15 --no-cache-dir # Alternative: Install without ChromaDB and use local embeddings pip install -r requirements.txt --exclude chromadb ``` **Requirements Installation Failures:** ```bash # If requirements.txt fails, install core dependencies first: pip install torch==2.8.0 transformers==4.30.2 sentence-transformers==2.2.2 pip install azure-ai-inference==1.0.0b1 openai==1.107.0 pip install requests==2.32.5 tqdm==4.67.1 colorama==0.4.6 # Then install remaining packages pip install -r requirements.txt ``` **Python Version Compatibility:** - **Minimum**: Python 3.8 - **Maximum**: Python 3.11.x - **Recommended**: Python 3.9-3.11 - **Known Issues**: Python 3.12+ not supported due to package conflicts ### Configuration Issues **Missing config.py:** ```bash # Copy template and edit with your API keys cp config.py.template config.py # Edit config.py with your Azure OpenAI credentials ``` **Import Errors:** ```bash # Make sure you're in the replication package directory cd ImproBR-Replication python improbr_pipeline.py --help ``` ## RQ2 Evaluation Pipeline To run the RQ2 evaluation pipeline that compares ImproBR-improved, ChatBR-improved, and raw bug reports against ground truth using similarity analysis on 37 duplicate pairs: ```bash cd Modified_ChatBR_Similarity_Codes/RQ2/ ``` Then follow the steps in: [`Modified_ChatBR_Similarity_Codes/RQ2/README.md`](Modified_ChatBR_Similarity_Codes/RQ2/README.md) ## Research Results & Evaluation Data ### RQ1: Bug Report Improvement Evaluation (139 reports) **Manual Evaluation Results:** - [`RQ1-RQ2/RQ1/manual_evaluation/Author 1 Responses.csv`](RQ1-RQ2/RQ1/manual_evaluation/Author 1 Responses.csv) - First evaluator assessments - [`RQ1-RQ2/RQ1/manual_evaluation/Author 2 Responses.csv`](RQ1-RQ2/RQ1/manual_evaluation/Author 2 Responses.csv) - Second evaluator assessments - [`RQ1-RQ2/RQ1/manual_evaluation/Final Results.csv`](RQ1-RQ2/RQ1/manual_evaluation/Final Results.csv) - Consolidated evaluation results **Inter-Rater Agreement (Cohen's Kappa):** - [`RQ1-RQ2/RQ1/cohen's_cappa_coefficient_matrices/confusion_matrix_s2r_label.png`](RQ1-RQ2/RQ1/cohen's_cappa_coefficient_matrices/confusion_matrix_s2r_label.png) - Steps to Reproduce κ scores - [`RQ1-RQ2/RQ1/cohen's_cappa_coefficient_matrices/confusion_matrix_ob_label.png`](RQ1-RQ2/RQ1/cohen's_cappa_coefficient_matrices/confusion_matrix_ob_label.png) - Observed Behavior κ scores - [`RQ1-RQ2/RQ1/cohen's_cappa_coefficient_matrices/confusion_matrix_eb_label.png`](RQ1-RQ2/RQ1/cohen's_cappa_coefficient_matrices/confusion_matrix_eb_label.png) - Expected Behavior κ scores **Algorithm Results:** - [`RQ1-RQ2/RQ1/algorithm_results/improbr_outputs/`](RQ1-RQ2/RQ1/algorithm_results/improbr_outputs/) - ImproBR improved reports - [`RQ1-RQ2/RQ1/algorithm_results/chatbr_outputs/`](RQ1-RQ2/RQ1/algorithm_results/chatbr_outputs/) - ChatBR baseline results - [`RQ1-RQ2/RQ1/algorithm_results/bee_analysis/`](RQ1-RQ2/RQ1/algorithm_results/bee_analysis/) - BEE tool structural analysis ### RQ2: Comparative Analysis vs ChatBR (37 pairs) **Similarity Score Results:** - [`RQ1-RQ2/RQ2/algorithm_results/similarity_scores/overall_tfidf.csv`](RQ1-RQ2/RQ2/algorithm_results/similarity_scores/overall_tfidf.csv) - TF-IDF similarity scores - [`RQ1-RQ2/RQ2/algorithm_results/similarity_scores/overall_word2vec.csv`](RQ1-RQ2/RQ2/algorithm_results/similarity_scores/overall_word2vec.csv) - Word2Vec similarity scores - [`RQ1-RQ2/RQ2/algorithm_results/similarity_scores/exact_string_comparisons.json`](RQ1-RQ2/RQ2/algorithm_results/similarity_scores/exact_string_comparisons.json) - Detailed comparisons - [`RQ1-RQ2/RQ2/algorithm_results/similarity_scores/word2vec_comparisons.json`](RQ1-RQ2/RQ2/algorithm_results/similarity_scores/word2vec_comparisons.json) - Semantic similarity data **Algorithm Outputs:** - [`RQ1-RQ2/RQ2/algorithm_results/ImproBR_results/`](RQ1-RQ2/RQ2/algorithm_results/ImproBR_results/) - ImproBR enhanced reports - [`RQ1-RQ2/RQ2/algorithm_results/ChatBR_results/`](RQ1-RQ2/RQ2/algorithm_results/ChatBR_results/) - ChatBR baseline outputs - [`RQ1-RQ2/RQ2/dataset/ground_truth/`](RQ1-RQ2/RQ2/dataset/ground_truth/) - High-quality reference reports ## Important Notes 1. **API Costs**: ~$0.10 per report processed 2. **Processing Time**: ~24 seconds per report average 3. **Knowledge Base**: Pre-built vector store included (248MB) 4. **Model Artifacts**: Fine-tuned DistilBERT included