fix(deploy): resolve explicit database credentials safely
This commit is contained in:
39
INSTALL.md
39
INSTALL.md
@@ -129,6 +129,42 @@ storage. Отказ для существующих данных или повр
|
||||
не доказывает готовность native evidence. `RESET_DATABASE_SCHEMA=false` сохраняет данные: неизвестная
|
||||
ревизия БД отказывает без автоматического сброса. Восстановление или reset требуют отдельного решения оператора.
|
||||
|
||||
### Явный режим подключения контейнера
|
||||
|
||||
Entrypoint принимает только один из двух режимов, без автоматического определения или fallback:
|
||||
|
||||
- `DATABASE_CONNECTION_MODE=postgres_components_v1`: обязательны `POSTGRES_HOST`, `POSTGRES_PORT`,
|
||||
`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`; `DATABASE_URL` должен отсутствовать, даже пустой.
|
||||
Full/light/enterprise Compose передают эти поля отдельно. Entrypoint строит URL через SQLAlchemy
|
||||
`URL.create`, сохраняя буквальный пароль, и экспортирует его до подключения и миграций.
|
||||
- `DATABASE_CONNECTION_MODE=external_url_v1`: обязателен полный заранее encoded `DATABASE_URL` с
|
||||
явными host/port/user/password/database; все пять `POSTGRES_*` полей должны отсутствовать.
|
||||
Существующие literal-URL current/master/e2e и build-smoke recipes выбирают этот режим явно.
|
||||
|
||||
Отсутствующий/неизвестный режим, пустые/конфликтующие поля отказывают до сертификатов, DB preflight и
|
||||
миграций. Допустимые user/database — ASCII identifiers до63 символов (буква/underscore первым,
|
||||
затем буквы, цифры, underscore или hyphen); port — десятичное1..65535. Никакие значения подключения
|
||||
не печатаются. Helper передаёт encoded URL только в приватный fd3 command-substitution pipe;
|
||||
stdout/stderr не содержат URL. Никакой shell eval/source для секрета не используется.
|
||||
|
||||
Для direct image run выберите режим явно, например в приватном env-файле:
|
||||
|
||||
```dotenv
|
||||
DATABASE_CONNECTION_MODE=external_url_v1
|
||||
DATABASE_URL=postgresql+psycopg2://your_user:your_percent_encoded_password@your_db_host:5432/your_database
|
||||
```
|
||||
|
||||
Этот URL-файл не должен содержать `POSTGRES_*` component-поля. Host `run.sh` запускает Uvicorn напрямую,
|
||||
не проходит container entrypoint и продолжает требовать корректный encoded `DATABASE_URL` по своей
|
||||
локальной конфигурации; Docker mode не меняет его выбранную БД.
|
||||
|
||||
Для Compose literal-паролей используйте process environment или single-quoted `.env` значения,
|
||||
например `POSTGRES_PASSWORD='your @/% # $ password'`. Literal quote/backslash требуют стандартного
|
||||
Compose escaping внутри quoted value. Unquoted/double-quoted `$` и inline `#` обрабатываются самим
|
||||
Compose; исходный текст до этого parser восстановить нельзя. Не shell-source `.env` и не публикуйте
|
||||
вывод resolved Compose с секретами. Изменение режима требует удалить поля другого режима,
|
||||
а не оставлять две якобы эквивалентные конфигурации.
|
||||
|
||||
## Локальная разработка
|
||||
|
||||
### Backend
|
||||
@@ -172,7 +208,8 @@ python src/scripts/create_admin.py --username admin --password '<temporary-secre
|
||||
| Категория | Переменные | Описание |
|
||||
|---|---|---|
|
||||
| **Security** | `AUTH_SECRET_KEY`, `ENCRYPTION_KEY`, `SERVICE_JWT` | JWT-подпись, шифрование данных, сервисный токен для сервисных принципалов |
|
||||
| **Database** | `DATABASE_URL` | Единственное PostgreSQL подключение |
|
||||
| **Database (контейнер)** | `DATABASE_CONNECTION_MODE` | Явный PostgreSQL component/URL contract до entrypoint preflight |
|
||||
| **Database (процесс backend)** | `DATABASE_URL` | Единственное итоговое подключение после preflight; local run.sh использует этот URL напрямую |
|
||||
| **Admin bootstrap** | `INITIAL_ADMIN_CREATE`, `INITIAL_ADMIN_USERNAME`, `INITIAL_ADMIN_PASSWORD`, `INITIAL_ADMIN_EMAIL` | Автосоздание admin при первом запуске |
|
||||
| **LLM** | `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL` | Провайдеры и модели |
|
||||
| **Features** | `FEATURES__DATASET_REVIEW`, `FEATURES__HEALTH_MONITOR` | Включение фич |
|
||||
|
||||
132
backend/src/scripts/resolve_database_connection.py
Normal file
132
backend/src/scripts/resolve_database_connection.py
Normal file
@@ -0,0 +1,132 @@
|
||||
# #region Scripts.ResolveDatabaseConnection [C:3] [TYPE Module] [SEMANTICS database,configuration,credentials,deployment]
|
||||
# @BRIEF Resolve one explicitly selected PostgreSQL connection contract before container startup.
|
||||
# @INVARIANT Missing/conflicting modes or fields refuse; application DATABASE_URL is never guessed from partial configuration.
|
||||
# @POST The CLI writes the encoded URL only to caller-owned fd3; stdout/logs never contain credentials.
|
||||
from collections.abc import Mapping
|
||||
import ipaddress
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
from sqlalchemy.engine import URL, make_url
|
||||
from sqlalchemy.exc import ArgumentError
|
||||
|
||||
MODE = 'DATABASE_CONNECTION_MODE'
|
||||
COMPONENTS = ('POSTGRES_HOST', 'POSTGRES_PORT', 'POSTGRES_USER', 'POSTGRES_PASSWORD', 'POSTGRES_DB')
|
||||
_NAME = re.compile(r'[A-Za-z_][A-Za-z0-9_-]{0,62}')
|
||||
_HOST = re.compile(r'[A-Za-z0-9](?:[A-Za-z0-9.-]{0,251}[A-Za-z0-9])?')
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Refusal [C:1] [TYPE Class]
|
||||
class DatabaseConnectionRefusal(ValueError):
|
||||
def __init__(self, code: str):
|
||||
self.code = code
|
||||
super().__init__(code)
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Refusal
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Required [C:2] [TYPE Function]
|
||||
def _required(values: Mapping, name: str) -> str:
|
||||
value = values.get(name)
|
||||
if not isinstance(value, str) or not value or '\x00' in value:
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_FIELD_REQUIRED')
|
||||
return value
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Required
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Identity [C:2] [TYPE Function]
|
||||
def _identity(value: str) -> str:
|
||||
if not _NAME.fullmatch(value):
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_IDENTITY_INVALID')
|
||||
return value
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Identity
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Host [C:2] [TYPE Function]
|
||||
def _host(value: str) -> str:
|
||||
try:
|
||||
ipaddress.ip_address(value)
|
||||
except ValueError:
|
||||
if not _HOST.fullmatch(value) or '..' in value:
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_HOST_INVALID') from None
|
||||
return value
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Host
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Port [C:2] [TYPE Function]
|
||||
def _port(value: str) -> int:
|
||||
if not re.fullmatch(r'[1-9][0-9]{0,4}', value) or not 1 <= int(value) <= 65535:
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_PORT_INVALID')
|
||||
return int(value)
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Port
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Components [C:2] [TYPE Function]
|
||||
def _components(values: Mapping) -> str:
|
||||
if 'DATABASE_URL' in values:
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_CONFLICT')
|
||||
parts = {name: _required(values, name) for name in COMPONENTS}
|
||||
url = URL.create('postgresql+psycopg2', username=_identity(parts['POSTGRES_USER']),
|
||||
password=parts['POSTGRES_PASSWORD'], host=_host(parts['POSTGRES_HOST']),
|
||||
port=_port(parts['POSTGRES_PORT']), database=_identity(parts['POSTGRES_DB']))
|
||||
return url.render_as_string(hide_password=False)
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Components
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.ExternalUrl [C:3] [TYPE Function]
|
||||
def _external_url(values: Mapping) -> str:
|
||||
if any(name in values for name in COMPONENTS):
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_CONFLICT')
|
||||
raw = _required(values, 'DATABASE_URL')
|
||||
if raw != raw.strip() or any(ord(character) < 32 or ord(character) == 127 for character in raw):
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_URL_INVALID')
|
||||
try:
|
||||
url = make_url(raw)
|
||||
if url.drivername not in {'postgresql', 'postgresql+psycopg2'} or url.password is None or not url.password:
|
||||
raise ValueError()
|
||||
_identity(url.username or '')
|
||||
_identity(url.database or '')
|
||||
_host(url.host or '')
|
||||
_port(str(url.port))
|
||||
except (ArgumentError, ValueError, TypeError):
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_URL_INVALID') from None
|
||||
return raw
|
||||
# #endregion Scripts.ResolveDatabaseConnection.ExternalUrl
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Resolve [C:2] [TYPE Function]
|
||||
# @POST Explicit external URL bytes are retained; components are encoded by URL.create exactly once.
|
||||
def resolve_database_connection(values: Mapping[str, str]) -> str:
|
||||
mode = values.get(MODE)
|
||||
if mode == 'postgres_components_v1':
|
||||
return _components(values)
|
||||
if mode == 'external_url_v1':
|
||||
return _external_url(values)
|
||||
raise DatabaseConnectionRefusal('DATABASE_CONNECTION_MODE_REQUIRED')
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Resolve
|
||||
|
||||
|
||||
# #region Scripts.ResolveDatabaseConnection.Main [C:2] [TYPE Function]
|
||||
def main() -> int:
|
||||
try:
|
||||
url = resolve_database_connection(os.environ)
|
||||
except DatabaseConnectionRefusal as exc:
|
||||
print(exc.code, file=sys.stderr)
|
||||
return 2
|
||||
try:
|
||||
remaining = memoryview(url.encode('utf-8'))
|
||||
while remaining:
|
||||
count = os.write(3, remaining)
|
||||
if count <= 0:
|
||||
raise OSError()
|
||||
remaining = remaining[count:]
|
||||
except (OSError, UnicodeError):
|
||||
print('DATABASE_CONNECTION_PRIVATE_CHANNEL_UNAVAILABLE', file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
# #endregion Scripts.ResolveDatabaseConnection.Main
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
# #endregion Scripts.ResolveDatabaseConnection
|
||||
@@ -138,6 +138,10 @@ class TestBackendImageCertPaths:
|
||||
"run",
|
||||
"--rm",
|
||||
"-e",
|
||||
"DATABASE_CONNECTION_MODE=postgres_components_v1",
|
||||
"-e",
|
||||
"POSTGRES_PORT=5432",
|
||||
"-e",
|
||||
"POSTGRES_HOST=does-not-exist",
|
||||
"-e",
|
||||
"POSTGRES_USER=test",
|
||||
@@ -218,6 +222,8 @@ class TestBackendContainerSmoke:
|
||||
"--network",
|
||||
"host",
|
||||
"-e",
|
||||
"DATABASE_CONNECTION_MODE=external_url_v1",
|
||||
"-e",
|
||||
f"DATABASE_URL={db_url}",
|
||||
"-e",
|
||||
"INITIAL_ADMIN_CREATE=true",
|
||||
|
||||
236
backend/tests/test_database_connection_contract.py
Normal file
236
backend/tests/test_database_connection_contract.py
Normal file
@@ -0,0 +1,236 @@
|
||||
# #region Test.DatabaseConnectionContract [C:3] [TYPE Module] [SEMANTICS database,credentials,configuration,privacy]
|
||||
# @TEST_INVARIANT Explicit modes are exclusive; missing/invalid configuration never becomes a default database connection.
|
||||
# @TEST_INVARIANT Legal password text survives URL construction; refusal/CLI output never contains supplied values.
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import json
|
||||
import yaml
|
||||
|
||||
import pytest
|
||||
from sqlalchemy.engine import make_url
|
||||
|
||||
from src.scripts.resolve_database_connection import DatabaseConnectionRefusal, resolve_database_connection
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
KEYS = ['DATABASE_CONNECTION_MODE', 'DATABASE_URL', 'POSTGRES_HOST', 'POSTGRES_PORT', 'POSTGRES_USER', 'POSTGRES_PASSWORD', 'POSTGRES_DB']
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.Fixtures [C:1] [TYPE Function]
|
||||
def _components(password='fixture-only-password'):
|
||||
return {'DATABASE_CONNECTION_MODE': 'postgres_components_v1', 'POSTGRES_HOST': 'db', 'POSTGRES_PORT': '5432',
|
||||
'POSTGRES_USER': 'analyst', 'POSTGRES_PASSWORD': password, 'POSTGRES_DB': 'ss_tools'}
|
||||
|
||||
|
||||
def _process_env(values):
|
||||
env = {key: value for key, value in os.environ.items() if key not in KEYS}
|
||||
return {**env, **values, 'PYTHONPATH': str(ROOT/'backend')}
|
||||
# #endregion Test.DatabaseConnectionContract.Fixtures
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.Password [C:2] [TYPE Function]
|
||||
@pytest.mark.parametrize('password', ['plain-fixture', 'p@ss/word%', 'at@slash/percent%colon:hash#',
|
||||
'literal$DOLLAR and # hash', ' leading and trailing ', 'quote\'double"back\\slash', 'пароль/€@%', 'line\nbreak'])
|
||||
def test_password_roundtrips_exactly_without_changing_connection_identity(password):
|
||||
raw = resolve_database_connection(_components(password))
|
||||
parsed = make_url(raw)
|
||||
assert parsed.password == password
|
||||
assert (parsed.host, parsed.port, parsed.username, parsed.database) == ('db', 5432, 'analyst', 'ss_tools')
|
||||
assert parsed.drivername == 'postgresql+psycopg2'
|
||||
assert parsed.query == {}
|
||||
# #endregion Test.DatabaseConnectionContract.Password
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.Strict [C:2] [TYPE Function]
|
||||
@pytest.mark.parametrize('mode', [None, '', 'guess', 'postgres_components_v0', 'external_url'])
|
||||
def test_mode_is_required_and_unknown_modes_refuse(mode):
|
||||
value = _components()
|
||||
if mode is None:
|
||||
del value['DATABASE_CONNECTION_MODE']
|
||||
else:
|
||||
value['DATABASE_CONNECTION_MODE'] = mode
|
||||
with pytest.raises(DatabaseConnectionRefusal):
|
||||
resolve_database_connection(value)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('field', ['POSTGRES_HOST', 'POSTGRES_PORT', 'POSTGRES_USER', 'POSTGRES_PASSWORD', 'POSTGRES_DB'])
|
||||
@pytest.mark.parametrize('change', [None, '', True])
|
||||
def test_components_cannot_be_missing_empty_or_untyped(field, change):
|
||||
value = _components()
|
||||
if change is None:
|
||||
del value[field]
|
||||
else:
|
||||
value[field] = change
|
||||
with pytest.raises(DatabaseConnectionRefusal):
|
||||
resolve_database_connection(value)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('field,change', [('POSTGRES_PORT', '0'), ('POSTGRES_PORT', '65536'), ('POSTGRES_PORT', ' 5432'),
|
||||
('POSTGRES_PORT', '5.432'), ('POSTGRES_USER', 'role\n'), ('POSTGRES_USER', 'x'*64), ('POSTGRES_DB', 'db/name'),
|
||||
('POSTGRES_DB', ' '), ('POSTGRES_HOST', 'user@db'), ('POSTGRES_HOST', 'db/path'), ('POSTGRES_PASSWORD', '\x00')])
|
||||
def test_invalid_component_identities_refuse_without_guessing(field, change):
|
||||
with pytest.raises(DatabaseConnectionRefusal):
|
||||
resolve_database_connection({**_components(), field: change})
|
||||
|
||||
|
||||
@pytest.mark.parametrize('url', ['', 'postgresql+psycopg2://analyst:fixture-only-password@db:5432/ss_tools'])
|
||||
def test_component_mode_refuses_any_url_field_even_if_equal(url):
|
||||
with pytest.raises(DatabaseConnectionRefusal, match='^DATABASE_CONNECTION_CONFLICT$'):
|
||||
resolve_database_connection({**_components(), 'DATABASE_URL': url})
|
||||
# #endregion Test.DatabaseConnectionContract.Strict
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.External [C:2] [TYPE Function]
|
||||
def test_explicit_external_url_is_retained_byte_for_byte():
|
||||
url = 'postgresql+psycopg2://analyst:quoted%40%2F%25@db:5432/ss_tools?sslmode=require'
|
||||
assert resolve_database_connection({'DATABASE_CONNECTION_MODE': 'external_url_v1', 'DATABASE_URL': url}) == url
|
||||
|
||||
|
||||
@pytest.mark.parametrize('field', ['POSTGRES_HOST', 'POSTGRES_PORT', 'POSTGRES_USER', 'POSTGRES_PASSWORD', 'POSTGRES_DB'])
|
||||
def test_external_mode_refuses_component_presence_even_if_empty(field):
|
||||
value = {'DATABASE_CONNECTION_MODE': 'external_url_v1',
|
||||
'DATABASE_URL': 'postgresql+psycopg2://analyst:fixture@db:5432/ss_tools', field: ''}
|
||||
with pytest.raises(DatabaseConnectionRefusal, match='^DATABASE_CONNECTION_CONFLICT$'):
|
||||
resolve_database_connection(value)
|
||||
|
||||
|
||||
@pytest.mark.parametrize('url', ['not-a-url', '', 'postgresql://analyst:fixture@db/ss_tools',
|
||||
'postgresql://analyst:fixture@user@db:5432/ss_tools', 'sqlite:///local.db', 'postgresql://analyst:fixture@db:5432/ss_tools?sslmode=require\n'])
|
||||
def test_malformed_external_url_is_a_categorical_refusal(url):
|
||||
with pytest.raises(DatabaseConnectionRefusal) as caught:
|
||||
resolve_database_connection({'DATABASE_CONNECTION_MODE': 'external_url_v1', 'DATABASE_URL': url})
|
||||
assert url not in str(caught.value) if url else True
|
||||
# #endregion Test.DatabaseConnectionContract.External
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.CliPrivacy [C:2] [TYPE Function]
|
||||
@pytest.mark.parametrize('values', [{'DATABASE_CONNECTION_MODE': 'external_url_v1', 'DATABASE_URL': 'not-a-url'},
|
||||
{'DATABASE_CONNECTION_MODE': 'external_url_v1', 'DATABASE_URL': 'private-fixture-invalid-url'}, {}])
|
||||
def test_cli_refusal_has_no_traceback_raw_value_or_stdout(values):
|
||||
result = subprocess.run([sys.executable, '-m', 'src.scripts.resolve_database_connection'], env=_process_env(values),
|
||||
cwd=ROOT/'backend', text=True, capture_output=True, timeout=10)
|
||||
assert result.returncode == 2
|
||||
assert result.stdout == ''
|
||||
assert 'DATABASE_CONNECTION_' in result.stderr
|
||||
assert 'Traceback' not in result.stderr
|
||||
for raw in values.values():
|
||||
assert raw not in result.stderr
|
||||
|
||||
|
||||
def test_cli_does_not_print_a_valid_url_without_a_private_fd():
|
||||
values = _components('private-fixture@/%')
|
||||
result = subprocess.run([sys.executable, '-m', 'src.scripts.resolve_database_connection'], env=_process_env(values),
|
||||
cwd=ROOT/'backend', text=True, capture_output=True, timeout=10, close_fds=True)
|
||||
assert result.returncode == 2
|
||||
assert result.stdout == ''
|
||||
assert result.stderr.strip() == 'DATABASE_CONNECTION_PRIVATE_CHANNEL_UNAVAILABLE'
|
||||
assert values['POSTGRES_PASSWORD'] not in result.stderr
|
||||
# #endregion Test.DatabaseConnectionContract.CliPrivacy
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.EntrypointPrivatePipe [C:3] [TYPE Function]
|
||||
def _shell_function(name):
|
||||
source = (ROOT/'docker/backend.entrypoint.sh').read_text()
|
||||
start = source.index(name+'() {')
|
||||
end = source.index('\n}\n', start)+len('\n}\n')
|
||||
return source[start:end]
|
||||
|
||||
|
||||
@pytest.mark.parametrize('password', ['private-fixture@/%:#$', ' leading literal # $ spaces ', 'quote\'double"back\\slash'])
|
||||
def test_actual_shell_preflight_exports_exact_password_via_private_fd_only(password):
|
||||
env = _process_env(_components(password))
|
||||
env['EXPECTED_PASSWORD'] = password
|
||||
# Pin the test interpreter without requiring a host-global SQLAlchemy installation.
|
||||
function = _shell_function('resolve_database_connection').replace('python3 -m ', '"$TEST_PYTHON" -m ')
|
||||
env['TEST_PYTHON'] = sys.executable
|
||||
program = "import os;from sqlalchemy.engine import make_url;u=make_url(os.environ['DATABASE_URL']);assert u.password==os.environ['EXPECTED_PASSWORD'];assert u.host=='db';print('CONFIGURED')"
|
||||
script = function+'\nresolve_database_connection || exit 1\n"$TEST_PYTHON" -c "$CHECK_PROGRAM"\n'
|
||||
env['CHECK_PROGRAM'] = program
|
||||
result = subprocess.run(['bash', '-c', script], env=env, text=True, capture_output=True, timeout=10)
|
||||
assert result.returncode == 0
|
||||
assert result.stdout == 'CONFIGURED\n'
|
||||
assert result.stderr == ''
|
||||
assert password not in result.stdout
|
||||
|
||||
|
||||
@pytest.mark.parametrize('values', [{}, {'DATABASE_CONNECTION_MODE': 'external_url_v1', 'DATABASE_URL': 'private-invalid-fixture'}])
|
||||
def test_actual_entrypoint_refuses_configuration_before_certificates_database_or_migrations(tmp_path, values):
|
||||
entrypoint = tmp_path/'entrypoint.sh'
|
||||
source = (ROOT/'docker/backend.entrypoint.sh').read_text().replace('python3 -m ', '"$TEST_PYTHON" -m ')
|
||||
entrypoint.write_text(source)
|
||||
# This source trap must remain untouched: connection validation precedes even sourcing certificates.
|
||||
marker = tmp_path/'unexpected-startup'
|
||||
(tmp_path/'certs.sh').write_text('touch "'+str(marker)+'"\nexit 98\n')
|
||||
env = _process_env(values)
|
||||
env['TEST_PYTHON'] = sys.executable
|
||||
result = subprocess.run(['bash', str(entrypoint)], env=env, text=True, capture_output=True, timeout=10)
|
||||
assert result.returncode == 1
|
||||
assert result.stdout == ''
|
||||
assert 'DATABASE_CONNECTION_CONFIGURATION_REFUSED' in result.stderr
|
||||
assert 'Traceback' not in result.stderr
|
||||
assert not marker.exists()
|
||||
for raw in values.values():
|
||||
assert raw not in result.stderr
|
||||
# #endregion Test.DatabaseConnectionContract.EntrypointPrivatePipe
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.Callers [C:2] [TYPE Function]
|
||||
@pytest.mark.parametrize('name,expected', [('docker-compose.yml', 'external_url_v1'),
|
||||
('docker-compose.e2e.yml', 'external_url_v1'), ('docker-compose.enterprise-clean.yml', 'postgres_components_v1'),
|
||||
('docker-compose.full-flow.yml', 'postgres_components_v1')])
|
||||
def test_current_compose_entrypoint_callers_select_one_complete_contract(name, expected):
|
||||
document = yaml.safe_load((ROOT/name).read_text())
|
||||
env = document['services']['backend']['environment']
|
||||
assert env['DATABASE_CONNECTION_MODE'] == expected
|
||||
if expected == 'postgres_components_v1':
|
||||
assert 'DATABASE_URL' not in env
|
||||
assert {'POSTGRES_HOST', 'POSTGRES_PORT', 'POSTGRES_USER', 'POSTGRES_PASSWORD', 'POSTGRES_DB'} <= set(env)
|
||||
assert isinstance(env['POSTGRES_PORT'], str)
|
||||
else:
|
||||
assert env['DATABASE_URL'].startswith('postgresql+psycopg2://')
|
||||
assert not set(env).intersection({'POSTGRES_HOST', 'POSTGRES_PORT', 'POSTGRES_USER', 'POSTGRES_PASSWORD', 'POSTGRES_DB'})
|
||||
if name == 'docker-compose.e2e.yml':
|
||||
for service in ['e2e-seed', 'e2e-reference-seed']:
|
||||
# The seed processes override ENTRYPOINT but retain their explicit URL-only connection input.
|
||||
candidates = [entry for entry in document['services'].values() if isinstance(entry.get('environment'), dict)
|
||||
and 'DATABASE_URL' in entry['environment']]
|
||||
assert len(candidates) == 3
|
||||
assert all(entry['environment']['DATABASE_CONNECTION_MODE'] == 'external_url_v1' for entry in candidates)
|
||||
|
||||
|
||||
def test_direct_docker_recipes_and_wrappers_keep_the_explicit_mode():
|
||||
build = (ROOT/'build.sh').read_text()
|
||||
assert '-e DATABASE_CONNECTION_MODE=external_url_v1' in build
|
||||
assert '-e DATABASE_URL="${database_url}"' in build
|
||||
integration = (ROOT/'backend/tests/integration/test_backend_container.py').read_text()
|
||||
assert '"DATABASE_CONNECTION_MODE=postgres_components_v1"' in integration
|
||||
assert '"DATABASE_CONNECTION_MODE=external_url_v1"' in integration
|
||||
wrapper = (ROOT/'backend/src/scripts/investigation_audit_entrypoint.py').read_text()
|
||||
assert 'os.execv("/app/entrypoint.sh"' in wrapper
|
||||
# Investigation wrapper inherits the selected base recipe mode; its secret loader does not overwrite DB fields.
|
||||
assert 'DATABASE_CONNECTION_MODE' not in wrapper
|
||||
# #endregion Test.DatabaseConnectionContract.Callers
|
||||
|
||||
|
||||
# #region Test.DatabaseConnectionContract.WaitPrivacy [C:3] [TYPE Function]
|
||||
def test_wait_program_logs_only_a_category_for_private_driver_exceptions(monkeypatch, capsys):
|
||||
import sqlalchemy
|
||||
source = _shell_function('wait_for_db')
|
||||
start = source.index('python3 -c "\n')+len('python3 -c "\n')
|
||||
end = source.index('\n" 2>&1)', start)
|
||||
program = source[start:end]
|
||||
secret = 'private-fixture@/%invalid'
|
||||
monkeypatch.setenv('DATABASE_URL', 'postgresql://analyst:'+secret+'@db:5432/ss_tools')
|
||||
def fail(*args, **kwargs):
|
||||
raise ValueError('invalid connection '+secret)
|
||||
monkeypatch.setattr(sqlalchemy, 'create_engine', fail)
|
||||
with pytest.raises(SystemExit) as caught:
|
||||
exec(compile(program, '<actual-entrypoint-wait-program>', 'exec'), {})
|
||||
assert caught.value.code == 99
|
||||
captured = capsys.readouterr()
|
||||
assert captured.out == ''
|
||||
assert captured.err.strip() == 'FATAL: DATABASE_CONNECTION_CONFIGURATION_INVALID'
|
||||
assert secret not in captured.err
|
||||
# #endregion Test.DatabaseConnectionContract.WaitPrivacy
|
||||
# #endregion Test.DatabaseConnectionContract
|
||||
@@ -96,6 +96,11 @@ def test_actual_full_generator_forwards_explicit_fresh_holds_and_refuses_implici
|
||||
assert backend["pull_policy"] == "never"
|
||||
assert backend["environment"]["NATIVE_RESULT_RETENTION_INIT"] == "${NATIVE_RESULT_RETENTION_INIT:-false}"
|
||||
assert backend["environment"]["STORAGE_ROOT_PATH"] == "/app/storage"
|
||||
assert backend["environment"]["DATABASE_CONNECTION_MODE"] == "postgres_components_v1"
|
||||
assert "DATABASE_URL" not in backend["environment"]
|
||||
assert backend["environment"]["POSTGRES_HOST"] == "db"
|
||||
assert backend["environment"]["POSTGRES_PORT"] == "5432"
|
||||
assert backend["environment"]["POSTGRES_PASSWORD"] == "${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}"
|
||||
assert backend["environment"]["RESET_DATABASE_SCHEMA"] == "${RESET_DATABASE_SCHEMA:-false}"
|
||||
env = _env_file(dist / ".env.example")
|
||||
assert env["NATIVE_RESULT_RETENTION_INIT"] == "true"
|
||||
@@ -116,6 +121,10 @@ def test_actual_light_generator_requires_its_exported_tag_and_explicit_deploymen
|
||||
assert app["environment"]["NATIVE_RESULT_RETENTION_INIT"] == "${NATIVE_RESULT_RETENTION_INIT:-false}"
|
||||
assert app["environment"]["RESET_DATABASE_SCHEMA"] == "${RESET_DATABASE_SCHEMA:-false}"
|
||||
assert app["environment"]["DB_SCHEMA"] == "${DB_SCHEMA:-public}"
|
||||
assert app["environment"]["DATABASE_CONNECTION_MODE"] == "postgres_components_v1"
|
||||
assert "DATABASE_URL" not in app["environment"]
|
||||
assert app["environment"]["POSTGRES_PORT"] == "5432"
|
||||
assert app["environment"]["POSTGRES_PASSWORD"] == "${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}"
|
||||
env = _env_file(dist / ".env.light.example")
|
||||
assert env["IMAGE"] == "superset-tools:release-light-fixture"
|
||||
assert env["NATIVE_RESULT_RETENTION_INIT"] == "true"
|
||||
|
||||
18
build.sh
18
build.sh
@@ -337,6 +337,7 @@ run_bundle_smoke() {
|
||||
|
||||
echo "[smoke] Verifying fresh migration, ORM schema initialization, and backend health..."
|
||||
docker run -d --name "${backend_container}" --network "${network}" \
|
||||
-e DATABASE_CONNECTION_MODE=external_url_v1 \
|
||||
-e DATABASE_URL="${database_url}" \
|
||||
-e AUTH_SECRET_KEY=release-smoke-auth-secret \
|
||||
-e ENCRYPTION_KEY="$(python3 -c 'import base64,os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())')" \
|
||||
@@ -494,7 +495,12 @@ services:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_URL: postgresql+psycopg2://\${POSTGRES_USER:-postgres}:\${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD}@db:5432/\${POSTGRES_DB:-ss_tools}
|
||||
DATABASE_CONNECTION_MODE: postgres_components_v1
|
||||
POSTGRES_HOST: db
|
||||
POSTGRES_PORT: "5432"
|
||||
POSTGRES_USER: \${POSTGRES_USER:-postgres}
|
||||
POSTGRES_PASSWORD: \${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
POSTGRES_DB: \${POSTGRES_DB:-ss_tools}
|
||||
STORAGE_ROOT_PATH: /app/storage
|
||||
NATIVE_RESULT_RETENTION_INIT: \${NATIVE_RESULT_RETENTION_INIT:-false}
|
||||
DB_SCHEMA: \${DB_SCHEMA:-public}
|
||||
@@ -638,7 +644,8 @@ generate_bundle_env() {
|
||||
# ======================================================================
|
||||
POSTGRES_DB=ss_tools
|
||||
POSTGRES_USER=postgres
|
||||
POSTGRES_PASSWORD=change-me
|
||||
# Quote literal .env values: unquoted/double-quoted $ and inline # follow Compose interpolation rules.
|
||||
POSTGRES_PASSWORD='change-me'
|
||||
POSTGRES_HOST_PORT=5432
|
||||
|
||||
# Схема, в которой живут все объекты приложения (по умолчанию public).
|
||||
@@ -872,7 +879,12 @@ services:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_URL: postgresql+psycopg2://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB:-ss_tools}
|
||||
DATABASE_CONNECTION_MODE: postgres_components_v1
|
||||
POSTGRES_HOST: db
|
||||
POSTGRES_PORT: "5432"
|
||||
POSTGRES_USER: ${POSTGRES_USER:-postgres}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
POSTGRES_DB: ${POSTGRES_DB:-ss_tools}
|
||||
DB_SCHEMA: ${DB_SCHEMA:-public}
|
||||
RESET_DATABASE_SCHEMA: ${RESET_DATABASE_SCHEMA:-false}
|
||||
STORAGE_ROOT_PATH: /app/storage
|
||||
|
||||
@@ -35,6 +35,7 @@ services:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_CONNECTION_MODE: external_url_v1
|
||||
DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
DB_SCHEMA: ${DB_SCHEMA:-public}
|
||||
RESET_DATABASE_SCHEMA: ${RESET_DATABASE_SCHEMA:-false}
|
||||
@@ -89,6 +90,7 @@ services:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_CONNECTION_MODE: external_url_v1
|
||||
DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
DB_SCHEMA: ${DB_SCHEMA:-public}
|
||||
STORAGE_ROOT_PATH: /app/storage
|
||||
@@ -112,6 +114,7 @@ services:
|
||||
backend:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_CONNECTION_MODE: external_url_v1
|
||||
DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
DB_SCHEMA: ${DB_SCHEMA:-public}
|
||||
STORAGE_ROOT_PATH: /app/storage
|
||||
|
||||
@@ -61,7 +61,12 @@ services:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_URL: postgresql+psycopg2://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@${POSTGRES_HOST:-db}:${POSTGRES_PORT:-5432}/${POSTGRES_DB:-ss_tools}
|
||||
DATABASE_CONNECTION_MODE: postgres_components_v1
|
||||
POSTGRES_HOST: ${POSTGRES_HOST:-db}
|
||||
POSTGRES_PORT: "${POSTGRES_PORT:-5432}"
|
||||
POSTGRES_USER: ${POSTGRES_USER:-postgres}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}
|
||||
POSTGRES_DB: ${POSTGRES_DB:-ss_tools}
|
||||
STORAGE_ROOT_PATH: /app/storage
|
||||
# Явная операция развёртывания: на свежем томе entrypoint создаёт метаданные удержаний
|
||||
# доказательств. false — пропустить (после первой установки флаг можно снять).
|
||||
|
||||
@@ -76,7 +76,12 @@ services:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_URL: postgresql+psycopg2://postgres:${POSTGRES_PASSWORD}@db:5432/ss_tools
|
||||
DATABASE_CONNECTION_MODE: postgres_components_v1
|
||||
POSTGRES_HOST: db
|
||||
POSTGRES_PORT: "5432"
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Generate full-flow env}
|
||||
POSTGRES_DB: ss_tools
|
||||
AUTH_SECRET_KEY: ${AUTH_SECRET_KEY:?Generate full-flow env}
|
||||
ENCRYPTION_KEY: ${ENCRYPTION_KEY:?Generate full-flow env}
|
||||
SERVICE_JWT: ${SERVICE_JWT:?Generate full-flow env}
|
||||
|
||||
@@ -32,6 +32,7 @@ services:
|
||||
db:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
DATABASE_CONNECTION_MODE: external_url_v1
|
||||
DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
|
||||
STORAGE_ROOT_PATH: /app/storage
|
||||
DB_SCHEMA: ${DB_SCHEMA:-public}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
# All-in-One Docker image (no Playwright, <200MB target)
|
||||
# ============================================================
|
||||
# Build: docker build -f docker/all-in-one.Dockerfile -t superset-tools:latest .
|
||||
# Run: docker run -p 8000:8000 --env-file backend/.env superset-tools:latest
|
||||
# Run: docker run -p 8000:8000 -e DATABASE_CONNECTION_MODE=external_url_v1 --env-file backend/.env superset-tools:latest
|
||||
#
|
||||
# Stages:
|
||||
# 1. frontend-builder — node:20-alpine, npm ci + build, discarded
|
||||
|
||||
@@ -6,7 +6,7 @@ set -euo pipefail
|
||||
# Alembic миграции, bootstrap admin, затем запуск backend (exec "$@").
|
||||
# @PRE Python runtime и backend source доступны в /app/backend.
|
||||
# Переменная CERTS_PATH указывает на директорию с .crt файлами (или пусто).
|
||||
# DATABASE_URL установлен и указывает на доступный PostgreSQL.
|
||||
# DATABASE_CONNECTION_MODE явно выбирает полный PostgreSQL component/URL contract.
|
||||
# @POST Сертификаты установлены в системное хранилище и NSS базу Chromium.
|
||||
# БД подтверждена как доступная. Alembic миграции применены.
|
||||
# Admin создан (если INITIAL_ADMIN_CREATE=true). Backend запущен через exec "$@".
|
||||
@@ -210,8 +210,8 @@ wait_for_db() {
|
||||
while [ $attempt -lt $max_attempts ]; do
|
||||
attempt=$((attempt + 1))
|
||||
# Capture stderr to detect permanent errors (KeyError, ArgumentError)
|
||||
local _err_output
|
||||
_err_output="$(python3 -c "
|
||||
local _err_output _rc
|
||||
if _err_output="$(python3 -c "
|
||||
import os, sys
|
||||
from sqlalchemy import create_engine, text
|
||||
try:
|
||||
@@ -227,11 +227,14 @@ except Exception as exc:
|
||||
# Permanent errors — no point retrying
|
||||
err_str = str(exc).lower()
|
||||
if 'database_url' in err_str or 'could not parse' in err_str or 'invalid' in err_str:
|
||||
print(f'FATAL: {exc}', file=sys.stderr)
|
||||
print('FATAL: DATABASE_CONNECTION_CONFIGURATION_INVALID', file=sys.stderr)
|
||||
sys.exit(99)
|
||||
sys.exit(1)
|
||||
" 2>&1)"
|
||||
local _rc=$?
|
||||
" 2>&1)"; then
|
||||
_rc=0
|
||||
else
|
||||
_rc=$?
|
||||
fi
|
||||
|
||||
if [ $_rc -eq 0 ]; then
|
||||
echo "[entrypoint] ✅ Database is reachable"
|
||||
@@ -248,11 +251,8 @@ except Exception as exc:
|
||||
sleep "$delay"
|
||||
done
|
||||
|
||||
# P0: Mask credentials — never print full DATABASE_URL to logs
|
||||
local _masked_url
|
||||
_masked_url="$(echo "$DATABASE_URL" | sed -E 's|://([^:]+):([^@]+)@|://\1:***@|')"
|
||||
# Connection values and driver exception strings can contain credentials; log only the outcome.
|
||||
echo "[entrypoint] ❌ Database not reachable after ${max_attempts} attempts" >&2
|
||||
echo "[entrypoint] DATABASE_URL=${_masked_url}" >&2
|
||||
return 1
|
||||
}
|
||||
# #endregion docker.backend.entrypoint.wait_for_db
|
||||
@@ -286,10 +286,26 @@ init_native_holds() {
|
||||
}
|
||||
# #endregion docker.backend.entrypoint.init_native_holds
|
||||
|
||||
# #region docker.backend.entrypoint.resolve_database_connection [C:2] [TYPE Function]
|
||||
# @PRE One explicit connection mode and its complete fields are supplied by deployment.
|
||||
# @POST Encoded URL is exported before certificates/DB/migrations; invalid configuration refuses without printing credentials.
|
||||
# @INVARIANT fd3 is a private command-substitution pipe, not stdout/stderr, argv or a shell assignment to evaluate.
|
||||
resolve_database_connection() {
|
||||
local resolved_url
|
||||
if ! resolved_url="$(python3 -m src.scripts.resolve_database_connection 3>&1 1>/dev/null)"; then
|
||||
echo "[entrypoint] DATABASE_CONNECTION_CONFIGURATION_REFUSED" >&2
|
||||
return 1
|
||||
fi
|
||||
export DATABASE_URL="${resolved_url}"
|
||||
}
|
||||
# #endregion docker.backend.entrypoint.resolve_database_connection
|
||||
|
||||
# ======================================================================
|
||||
# MAIN
|
||||
# ======================================================================
|
||||
|
||||
resolve_database_connection || exit 1
|
||||
|
||||
source "$(dirname "$0")/certs.sh"
|
||||
|
||||
download_llm_ca_certs
|
||||
|
||||
Reference in New Issue
Block a user