Merge pull request #14711 from DefectDojo/release/2.57.2

Release: Merge release into master from: release/2.57.2

Author: rossops

CLAUDE.md

@@ -0,0 +1,249 @@
+# DefectDojo Development Guide
+
+## Project Overview
+
+DefectDojo is a Django application (`dojo` app) for vulnerability management. The codebase is undergoing a modular reorganization to move from monolithic files toward self-contained domain modules.
+
+## Module Reorganization
+
+### Reference Pattern: `dojo/url/`
+
+All domain modules should match the structure of `dojo/url/`. This is the canonical example of a fully reorganized module.
+
+```
+dojo/{module}/
+├── __init__.py       # import dojo.{module}.admin  # noqa: F401
+├── models.py         # Domain models, constants, factory methods
+├── admin.py          # @admin.register() for the module's models
+├── services.py       # Business logic (no HTTP concerns)
+├── queries.py        # Complex DB aggregations/annotations
+├── signals.py        # Django signal handlers
+├── [manager.py]      # Custom QuerySet/Manager if needed
+├── [validators.py]   # Field-level validators if needed
+├── [helpers.py]      # Async task wrappers, tag propagation, etc.
+├── ui/
+│   ├── __init__.py   # Empty
+│   ├── forms.py      # Django ModelForms
+│   ├── filters.py    # UI-layer django-filter classes
+│   ├── views.py      # Thin view functions — delegates to services.py
+│   └── urls.py       # URL routing
+└── api/
+    ├── __init__.py   # path = "{module}"
+    ├── serializer.py # DRF serializers
+    ├── views.py      # API ViewSets — delegates to services.py
+    ├── filters.py    # API-layer filters
+    └── urls.py       # add_{module}_urls(router) registration
+```
+
+### Architecture Principles
+
+
+**services.py is the critical layer**: Both `ui/views.py` and `api/views.py` call `services.py` for business logic. Services accept domain objects and primitives — never request/response objects, forms, or serializers.
+
+**Backward-compatible re-exports**: When moving code out of monolithic files (`dojo/models.py`, `dojo/forms.py`, `dojo/filters.py`, `dojo/api_v2/serializers.py`, `dojo/api_v2/views.py`), always leave a re-export at the original location:
+```python
+from dojo.{module}.models import {Model}  # noqa: F401 -- backward compat
+```
+Never remove re-exports until all consumers are updated in a dedicated cleanup pass.
+
+### Current State
+
+Modules in various stages of reorganization:
+
+| Module | models.py | services.py | ui/ | api/ | Status |
+|--------|-----------|-------------|-----|------|--------|
+| **url** | In module | N/A | Done | Done | **Complete** |
+| **location** | In module | N/A | N/A | Done | **Complete** |
+| **product_type** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work |
+| **test** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work |
+| **engagement** | In dojo/models.py | Partial (32 lines) | Partial (views at root) | In dojo/api_v2/ | Needs work |
+| **product** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work |
+| **finding** | In dojo/models.py | Missing | Partial (views at root) | In dojo/api_v2/ | Needs work |
+
+### Monolithic Files Being Decomposed
+
+These files still contain code for multiple modules. Extract code to the target module's subdirectory and leave a re-export stub.
+
+- `dojo/models.py` (4,973 lines) — All model definitions
+- `dojo/forms.py` (4,127 lines) — All Django forms
+- `dojo/filters.py` (4,016 lines) — All UI and API filter classes
+- `dojo/api_v2/serializers.py` (3,387 lines) — All DRF serializers
+- `dojo/api_v2/views.py` (3,519 lines) — All API viewsets
+
+---
+
+## Reorganization Playbook
+
+When asked to reorganize a module, follow these phases in order. Each phase should be independently verifiable.
+
+### Phase 0: Pre-Flight (Read-Only)
+
+Before any changes, identify all code to extract:
+
+```bash
+# 1. Model classes and line ranges in dojo/models.py
+grep -n "class {Model}" dojo/models.py
+
+# 2. Form classes in dojo/forms.py
+grep -n "class.*{Module}" dojo/forms.py
+grep -n "model = {Model}" dojo/forms.py
+
+# 3. Filter classes in dojo/filters.py
+grep -n "class.*{Module}\|class.*{Model}" dojo/filters.py
+
+# 4. Serializer classes
+grep -n "class.*{Model}" dojo/api_v2/serializers.py
+
+# 5. ViewSet classes
+grep -n "class.*{Model}\|class.*{Module}" dojo/api_v2/views.py
+
+# 6. Admin registrations
+grep -n "admin.site.register({Model}" dojo/models.py
+
+# 7. All import sites (to verify backward compat)
+grep -rn "from dojo.models import.*{Model}" dojo/ unittests/
+
+# 8. Business logic in current views
+# Scan dojo/{module}/views.py for: .save(), .delete(), create_notification(),
+# jira_helper.*, dojo_dispatch_task(), multi-model workflows
+```
+
+### Phase 1: Extract Models
+
+1. Create `dojo/{module}/models.py` with the model class(es) and associated constants
+2. Create `dojo/{module}/admin.py` with `admin.site.register()` calls (remove from `dojo/models.py`)
+3. Update `dojo/{module}/__init__.py` to `import dojo.{module}.admin  # noqa: F401`
+4. Add re-exports in `dojo/models.py`
+5. Remove original model code (keep re-export line)
+
+**Import rules for models.py:**
+- Upward FKs (e.g., Test -> Engagement): import from `dojo.models` if not yet extracted, or `dojo.{module}.models` if already extracted
+- Downward references (e.g., Product_Type querying Finding): use lazy imports inside method bodies
+- Shared utilities (`copy_model_util`, `_manage_inherited_tags`, `get_current_date`, etc.): import from `dojo.models`
+- Do NOT set `app_label` in Meta — all models inherit `dojo` app_label automatically
+
+**Verify:**
+```bash
+python manage.py check
+python manage.py makemigrations --check
+python -c "from dojo.{module}.models import {Model}"
+python -c "from dojo.models import {Model}"
+```
+
+### Phase 2: Extract Services
+
+Create `dojo/{module}/services.py` with business logic extracted from UI views.
+
+**What belongs in services.py:**
+- State transitions (close, reopen, status changes)
+- Multi-step creation/update workflows
+- External integration calls (JIRA, GitHub)
+- Notification dispatching
+- Copy/clone operations
+- Bulk operations
+- Merge operations
+
+**What stays in views:**
+- HTTP request/response handling
+- Form instantiation and validation
+- Serialization/deserialization
+- Authorization checks (`@user_is_authorized`, `user_has_permission_or_403`)
+- Template rendering, redirects
+- Pagination, breadcrumbs
+
+**Service function pattern:**
+```python
+def close_engagement(engagement: Engagement, user: User) -> Engagement:
+    engagement.active = False
+    engagement.status = "Completed"
+    engagement.save()
+    if jira_helper.get_jira_project(engagement):
+        dojo_dispatch_task(jira_helper.close_epic, engagement.id, push_to_jira=True)
+    return engagement
+```
+
+Update UI views and API viewsets to call the service instead of containing logic inline.
+
+### Phase 3: Extract Forms to `ui/forms.py`
+
+1. Create `dojo/{module}/ui/__init__.py` (empty)
+2. Create `dojo/{module}/ui/forms.py` — move form classes from `dojo/forms.py`
+3. Add re-exports in `dojo/forms.py`
+
+### Phase 4: Extract UI Filters to `ui/filters.py`
+
+1. Create `dojo/{module}/ui/filters.py` — move module-specific filters from `dojo/filters.py`
+2. Shared base classes (`DojoFilter`, `DateRangeFilter`, `ReportBooleanFilter`) stay in `dojo/filters.py`
+3. Add re-exports in `dojo/filters.py`
+
+### Phase 5: Move UI Views/URLs into `ui/`
+
+1. Move `dojo/{module}/views.py` -> `dojo/{module}/ui/views.py`
+2. Move `dojo/{module}/urls.py` -> `dojo/{module}/ui/urls.py`
+3. Update URL imports:
+   - product: update `dojo/asset/urls.py`
+   - product_type: update `dojo/organization/urls.py`
+   - others: update the include in `dojo/urls.py`
+
+### Phase 6: Extract API Serializers to `api/serializer.py`
+
+1. Create `dojo/{module}/api/__init__.py` with `path = "{module}"`
+2. Create `dojo/{module}/api/serializer.py` — move from `dojo/api_v2/serializers.py`
+3. Add re-exports in `dojo/api_v2/serializers.py`
+
+### Phase 7: Extract API Filters to `api/filters.py`
+
+1. Create `dojo/{module}/api/filters.py` — move `Api{Model}Filter` from `dojo/filters.py`
+2. Add re-exports
+
+### Phase 8: Extract API ViewSets to `api/views.py`
+
+1. Create `dojo/{module}/api/views.py` — move from `dojo/api_v2/views.py`
+2. Add re-exports in `dojo/api_v2/views.py`
+
+### Phase 9: Extract API URL Registration
+
+1. Create `dojo/{module}/api/urls.py`:
+   ```python
+   from dojo.{module}.api import path
+   from dojo.{module}.api.views import {ViewSet}
+
+   def add_{module}_urls(router):
+       router.register(path, {ViewSet}, path)
+       return router
+   ```
+2. Update `dojo/urls.py` — replace `v2_api.register(...)` with `add_{module}_urls(v2_api)`
+
+### After Each Phase: Verify
+
+```bash
+python manage.py check
+python manage.py makemigrations --check
+python -m pytest unittests/ -x --timeout=120
+```
+
+---
+
+## Cross-Module Dependencies
+
+The model hierarchy is: Product_Type -> Product -> Engagement -> Test -> Finding
+
+Extract in this order (top to bottom) so that upward FKs can import from already-extracted modules. The recommended order is: product_type, test, engagement, product, finding.
+
+For downward references (e.g., Product_Type's cached properties querying Finding), always use lazy imports:
+```python
+@cached_property
+def critical_present(self):
+    from dojo.models import Finding  # lazy import
+    return Finding.objects.filter(test__engagement__product__prod_type=self, severity="Critical").exists()
+```
+
+---
+
+## Key Technical Details
+
+- **Single Django app**: Everything is under `app_label = "dojo"`. Moving models to subdirectories does NOT require migration changes.
+- **Model discovery**: Triggered by `__init__.py` importing `admin.py`, which imports `models.py`. This is the same chain `dojo/url/` uses.
+- **Signal registration**: Handled in `dojo/apps.py` via `import dojo.{module}.signals`. Already set up for test, engagement, product, product_type.
+- **Watson search**: Uses `self.get_model("Product")` in `apps.py` — works via Django's model registry regardless of file location.
+- **Admin registration**: Currently at the bottom of `dojo/models.py` (lines 4888-4973). Must be moved to `{module}/admin.py` and removed from `dojo/models.py` to avoid `AlreadyRegistered` errors.

components/package.json

@@ -1,6 +1,6 @@
 {
   "name": "defectdojo",
-  "version": "2.57.1",
+  "version": "2.57.2",
   "license" : "BSD-3-Clause",
   "private": true,
   "dependencies": {

dojo/__init__.py

@@ -4,6 +4,6 @@
 # Django starts so that shared_task will use this app.
 from .celery import app as celery_app  # noqa: F401
 
-__version__ = "2.57.1"
+__version__ = "2.57.2"
 __url__ = "https://github.com/DefectDojo/django-DefectDojo"
 __docs__ = "https://documentation.defectdojo.com"

dojo/announcement/os_message.py

@@ -0,0 +1,119 @@
+import logging
+
+import bleach
+import markdown
+import requests
+from django.core.cache import cache
+
+logger = logging.getLogger(__name__)
+
+BUCKET_URL = "https://storage.googleapis.com/defectdojo-os-messages-prod/open_source_message.md"
+CACHE_SECONDS = 3600
+HTTP_TIMEOUT_SECONDS = 2
+CACHE_KEY = "os_message:v1"
+
+INLINE_TAGS = ["strong", "em", "a"]
+INLINE_ATTRS = {"a": ["href", "title"]}
+
+# Keep BLOCK_TAGS / BLOCK_ATTRS in sync with the DaaS publisher's
+# MARKDOWNIFY["default"]["WHITELIST_TAGS"] / WHITELIST_ATTRS so previews
+# on DaaS and rendering in OSS stay byte-identical.
+BLOCK_TAGS = [
+    "p", "ul", "ol", "li", "a", "strong", "em", "code", "pre",
+    "blockquote", "h2", "h3", "h4", "hr", "br", "b", "i",
+    "abbr", "acronym",
+]
+BLOCK_ATTRS = {
+    "a": ["href", "title"],
+    "abbr": ["title"],
+    "acronym": ["title"],
+}
+
+_MISS = object()
+
+
+def fetch_os_message():
+    cached = cache.get(CACHE_KEY, default=_MISS)
+    if cached is not _MISS:
+        return cached
+
+    try:
+        response = requests.get(BUCKET_URL, timeout=HTTP_TIMEOUT_SECONDS)
+    except Exception:
+        logger.debug("os_message: fetch failed", exc_info=True)
+        cache.set(CACHE_KEY, None, CACHE_SECONDS)
+        return None
+
+    if response.status_code != 200 or not response.text.strip():
+        cache.set(CACHE_KEY, None, CACHE_SECONDS)
+        return None
+
+    cache.set(CACHE_KEY, response.text, CACHE_SECONDS)
+    return response.text
+
+
+def _strip_outer_p(html):
+    stripped = html.strip()
+    if stripped.startswith("<p>") and stripped.endswith("</p>"):
+        return stripped[3:-4]
+    return stripped
+
+
+def parse_os_message(text):
+    lines = text.splitlines()
+
+    headline_source = None
+    body_start = None
+    for index, line in enumerate(lines):
+        if line.startswith("# "):
+            headline_source = line[2:].strip()
+            body_start = index + 1
+            break
+
+    if not headline_source:
+        return None
+
+    headline_source = headline_source[:100]
+    headline_rendered = markdown.markdown(headline_source)
+    headline_cleaned = bleach.clean(
+        headline_rendered,
+        tags=INLINE_TAGS,
+        attributes=INLINE_ATTRS,
+        strip=True,
+    )
+    headline_html = _strip_outer_p(headline_cleaned)
+
+    expanded_html = None
+    expanded_marker = "## Expanded Message"
+    expanded_body_lines = None
+    for offset, line in enumerate(lines[body_start:], start=body_start):
+        if line.strip() == expanded_marker:
+            expanded_body_lines = lines[offset + 1:]
+            break
+
+    if expanded_body_lines is not None:
+        expanded_source = "\n".join(expanded_body_lines).strip()
+        if expanded_source:
+            expanded_rendered = markdown.markdown(
+                expanded_source,
+                extensions=["extra", "fenced_code", "nl2br"],
+            )
+            expanded_html = bleach.clean(
+                expanded_rendered,
+                tags=BLOCK_TAGS,
+                attributes=BLOCK_ATTRS,
+                strip=True,
+            )
+
+    return {"message": headline_html, "expanded_html": expanded_html}
+
+
+def get_os_banner():
+    try:
+        text = fetch_os_message()
+        if not text:
+            return None
+        return parse_os_message(text)
+    except Exception:
+        logger.debug("os_message: get_os_banner failed", exc_info=True)
+        return None