This commit is contained in:
2026-05-08 16:24:23 +00:00
parent 3d73f20c1f
commit 018f1f12d5
5 changed files with 188 additions and 236 deletions
+56 -63
View File
@@ -13,6 +13,9 @@ Created becasue github is so unusable and [unreliable](https://red-squares.cian.
Supported platforms: GitHub, GitLab, Gitea, Forgejo Supported platforms: GitHub, GitLab, Gitea, Forgejo
> [!NOTE]
> Meow
## Install ## Install
### Option 1. Install from source ### Option 1. Install from source
@@ -36,7 +39,9 @@ docker compose up -d --build
docker compose run --rm --entrypoint nano refray /data/config/refray/config.toml docker compose run --rm --entrypoint nano refray /data/config/refray/config.toml
``` ```
## Configure ## Usage
### 1. Configure
Run the interactive configuration wizard: Run the interactive configuration wizard:
@@ -44,7 +49,45 @@ Run the interactive configuration wizard:
refray config refray config
``` ```
## One-time Sync <details><summary>Example Config</summary>
```toml
[[sites]]
name = "github"
provider = "github"
base_url = "https://github.com"
token = { env = "GITHUB_TOKEN" }
[[sites]]
name = "gitea"
provider = "gitea"
base_url = "https://gitea.example.com"
token = { env = "GITEA_TOKEN" }
[[mirrors]]
name = "personal"
sync_visibility = "all"
repo_whitelist = ["^important-"]
repo_blacklist = ["-archive$"]
create_missing = true
visibility = "private"
allow_force = false
conflict_resolution = "auto_rebase_pull_request"
[[mirrors.endpoints]]
site = "github"
kind = "user"
namespace = "hykilpikonna"
[[mirrors.endpoints]]
site = "gitea"
kind = "user"
namespace = "azalea"
```
</details>
### 2. One-time Sync
Run all configured mirror groups: Run all configured mirror groups:
@@ -88,54 +131,40 @@ refray sync --jobs 8
</details> </details>
## Service & Webhooks ### 3. Service & Webhooks
You can run `refray` as a service that listens for webhook events and runs full sync periodically. This is the recommended way to run `refray`. You can run `refray` as a service that listens for webhook events and runs full sync periodically. This is the recommended way to run `refray`.
If you want to use webhooks, you need to expose port 8787 to a public URL that can be accessed by the git provider (e.g. using port forwarding, reverse proxy, or cloudflare tunnel). > [!NOTE]
> If you want to use webhooks, you need to expose port 8787 to a public URL that can be accessed by the git provider.<br>
> This can be done using e.g. port forwarding, reverse proxy, cloudflare tunnel, or tailscale funnel.
Start the receiver: Start the service:
```sh ```sh
refray serve refray serve
``` ```
Expose that listener with your reverse proxy or tunnel, then install repository webhooks. If `[webhook]` is configured, the URL and secret can come from config: Install webhooks on all repos:
```sh ```sh
refray webhook install refray webhook install
``` ```
Webhook install uses `[webhook].url` and `[webhook].secret` from your config.
Useful install filters:
```sh
refray webhook install \
--url https://mirror.example.com/webhook \
--secret-env REFRAY_WEBHOOK_SECRET \
--group personal \
--repo-pattern '^important-'
```
The receiver accepts `POST /` and `POST /webhook`. It verifies GitHub/Gitea HMAC SHA-256 signatures and GitLab webhook tokens, then queues `refray sync --group <group> --repo-pattern '^<repo>$'` internally. Duplicate events for the same group/repo are coalesced while a job is queued or running. Sync jobs are serialized inside the receiver so the local ref and failure caches stay consistent.
When `[webhook].install = true`, normal `refray sync` also checks webhook installation status and installs missing webhooks for repositories that have not been recorded yet. Installation status is stored in `webhook-state.toml` under the work directory.
To uninstall webhooks previously installed by `refray`: To uninstall webhooks previously installed by `refray`:
> [!WARNING]
> If you want to stop using `refray`, make sure you run this! Otherwise, all of your repos will keep trying to send webhooks to the URL.
```sh ```sh
refray webhook uninstall refray webhook uninstall
``` ```
Manual `webhook uninstall` checks repositories on the provider instead of trusting only local state. To uninstall one repository exactly:
```sh
refray webhook uninstall important-repo
```
To move installed hooks to a new public URL, use `webhook update`. It removes hooks matching the current configured `[webhook].url`, installs the new URL, updates `[webhook].url` in the config, and refreshes local webhook state: To move installed hooks to a new public URL, use `webhook update`. It removes hooks matching the current configured `[webhook].url`, installs the new URL, updates `[webhook].url` in the config, and refreshes local webhook state:
```sh ```sh
refray webhook update --url https://new.example.com/webhook refray webhook update https://new.example.com/webhook
``` ```
Serve can also run periodic full syncs. The interval can be configured in `[webhook].full_sync_interval_minutes` or overridden at startup: Serve can also run periodic full syncs. The interval can be configured in `[webhook].full_sync_interval_minutes` or overridden at startup:
@@ -183,42 +212,6 @@ Branch deletion follows the same rule at branch scope: if a branch existed on ev
Tags are fetched into provider-specific cache refs and pushed only when the tag object agrees across providers or exists on one side. Divergent tags are skipped and reported. Tag deletion is not propagated. Tags are fetched into provider-specific cache refs and pushed only when the tag object agrees across providers or exists on one side. Divergent tags are skipped and reported. Tag deletion is not propagated.
## Example Config
```toml
[[sites]]
name = "github"
provider = "github"
base_url = "https://github.com"
token = { env = "GITHUB_TOKEN" }
[[sites]]
name = "gitea"
provider = "gitea"
base_url = "https://gitea.example.com"
token = { env = "GITEA_TOKEN" }
[[mirrors]]
name = "personal"
sync_visibility = "all"
repo_whitelist = ["^important-"]
repo_blacklist = ["-archive$"]
create_missing = true
visibility = "private"
allow_force = false
conflict_resolution = "auto_rebase_pull_request"
[[mirrors.endpoints]]
site = "github"
kind = "user"
namespace = "hykilpikonna"
[[mirrors.endpoints]]
site = "gitea"
kind = "user"
namespace = "azalea"
```
## Testing ## Testing
Run the normal, non-destructive test suite: Run the normal, non-destructive test suite:
+30 -42
View File
@@ -89,50 +89,24 @@ enum WebhookCommand {
#[derive(Args, Debug)] #[derive(Args, Debug)]
struct WebhookInstallCommand { struct WebhookInstallCommand {
#[arg(value_name = "REPO", conflicts_with = "repo_pattern")]
repo: Option<String>,
#[arg(long, value_name = "URL")]
url: Option<String>,
#[arg(long, conflicts_with = "secret_env")]
secret: Option<String>,
#[arg(long, value_name = "ENV", conflicts_with = "secret")]
secret_env: Option<String>,
#[arg(long, value_name = "NAME")]
group: Option<String>,
#[arg(long, value_name = "REGEX")]
repo_pattern: Option<String>,
#[arg(long)] #[arg(long)]
dry_run: bool, dry_run: bool,
#[arg(long, value_name = "PATH")]
work_dir: Option<PathBuf>,
#[arg(long, default_value_t = DEFAULT_JOBS, value_name = "N")] #[arg(long, default_value_t = DEFAULT_JOBS, value_name = "N")]
jobs: usize, jobs: usize,
} }
#[derive(Args, Debug)] #[derive(Args, Debug)]
struct WebhookUninstallCommand { struct WebhookUninstallCommand {
#[arg(value_name = "REPO")]
repo: Option<String>,
#[arg(long, value_name = "URL")]
url: Option<String>,
#[arg(long, value_name = "NAME")]
group: Option<String>,
#[arg(long)] #[arg(long)]
dry_run: bool, dry_run: bool,
#[arg(long, value_name = "PATH")]
work_dir: Option<PathBuf>,
#[arg(long, default_value_t = DEFAULT_JOBS, value_name = "N")] #[arg(long, default_value_t = DEFAULT_JOBS, value_name = "N")]
jobs: usize, jobs: usize,
} }
#[derive(Args, Debug)] #[derive(Args, Debug)]
struct WebhookUpdateCommand { struct WebhookUpdateCommand {
#[arg(long, value_name = "URL")] #[arg(value_name = "URL")]
url: String, url: String,
#[arg(long, conflicts_with = "secret_env")]
secret: Option<String>,
#[arg(long, value_name = "ENV", conflicts_with = "secret")]
secret_env: Option<String>,
#[arg(long)] #[arg(long)]
dry_run: bool, dry_run: bool,
#[arg(long, value_name = "PATH")] #[arg(long, value_name = "PATH")]
@@ -199,33 +173,28 @@ fn main() -> Result<()> {
} }
Command::Webhook(WebhookCommand::Install(command)) => { Command::Webhook(WebhookCommand::Install(command)) => {
let config = load_config(&config_path)?; let config = load_config(&config_path)?;
let secret = resolve_webhook_secret(&config, command.secret, command.secret_env)?; let secret = resolve_config_webhook_secret(&config)?;
let url = resolve_webhook_url(&config, command.url)?; let url = resolve_config_webhook_url(&config)?;
install_webhooks( install_webhooks(
&config, &config,
WebhookInstallOptions { WebhookInstallOptions {
url, url,
secret, secret,
group: command.group,
repo: command.repo,
repo_pattern: command.repo_pattern,
dry_run: command.dry_run, dry_run: command.dry_run,
work_dir: command.work_dir, work_dir: None,
jobs: command.jobs, jobs: command.jobs,
}, },
) )
} }
Command::Webhook(WebhookCommand::Uninstall(command)) => { Command::Webhook(WebhookCommand::Uninstall(command)) => {
let config = load_config(&config_path)?; let config = load_config(&config_path)?;
let url = resolve_webhook_url(&config, command.url)?; let url = resolve_config_webhook_url(&config)?;
uninstall_webhooks( uninstall_webhooks(
&config, &config,
WebhookUninstallOptions { WebhookUninstallOptions {
url, url,
group: command.group,
repo: command.repo,
dry_run: command.dry_run, dry_run: command.dry_run,
work_dir: command.work_dir, work_dir: None,
jobs: command.jobs, jobs: command.jobs,
}, },
) )
@@ -239,7 +208,7 @@ fn main() -> Result<()> {
.ok_or_else(|| { .ok_or_else(|| {
anyhow::anyhow!("configure [webhook] before running webhook update") anyhow::anyhow!("configure [webhook] before running webhook update")
})?; })?;
let secret = resolve_webhook_secret(&config, command.secret, command.secret_env)?; let secret = resolve_config_webhook_secret(&config)?;
update_webhooks( update_webhooks(
&config, &config,
WebhookUpdateOptions { WebhookUpdateOptions {
@@ -261,9 +230,26 @@ fn main() -> Result<()> {
} }
fn load_config(path: &Path) -> Result<Config> { fn load_config(path: &Path) -> Result<Config> {
if !path.exists() {
anyhow::bail!(
"config not found at {}. Run `refray config` first to create one.",
path.display()
);
}
Config::load(path).with_context(|| format!("failed to load config at {}", path.display())) Config::load(path).with_context(|| format!("failed to load config at {}", path.display()))
} }
fn resolve_config_webhook_secret(config: &Config) -> Result<String> {
config
.webhook
.as_ref()
.map(|webhook| webhook.secret())
.transpose()?
.ok_or_else(|| {
anyhow::anyhow!("configure [webhook].secret before running webhook commands")
})
}
fn resolve_webhook_secret( fn resolve_webhook_secret(
config: &Config, config: &Config,
value: Option<String>, value: Option<String>,
@@ -283,10 +269,12 @@ fn resolve_webhook_secret(
} }
} }
fn resolve_webhook_url(config: &Config, value: Option<String>) -> Result<String> { fn resolve_config_webhook_url(config: &Config) -> Result<String> {
value config
.or_else(|| config.webhook.as_ref().map(|webhook| webhook.url.clone())) .webhook
.ok_or_else(|| anyhow::anyhow!("pass --url or configure [webhook].url")) .as_ref()
.map(|webhook| webhook.url.clone())
.ok_or_else(|| anyhow::anyhow!("configure [webhook].url before running webhook commands"))
} }
fn set_config_webhook_url(config: &mut Config, url: String) { fn set_config_webhook_url(config: &mut Config, url: String) {
-42
View File
@@ -39,9 +39,6 @@ pub struct ServeOptions {
pub struct WebhookInstallOptions { pub struct WebhookInstallOptions {
pub url: String, pub url: String,
pub secret: String, pub secret: String,
pub group: Option<String>,
pub repo: Option<String>,
pub repo_pattern: Option<String>,
pub dry_run: bool, pub dry_run: bool,
pub work_dir: Option<PathBuf>, pub work_dir: Option<PathBuf>,
pub jobs: usize, pub jobs: usize,
@@ -50,8 +47,6 @@ pub struct WebhookInstallOptions {
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct WebhookUninstallOptions { pub struct WebhookUninstallOptions {
pub url: String, pub url: String,
pub group: Option<String>,
pub repo: Option<String>,
pub dry_run: bool, pub dry_run: bool,
pub work_dir: Option<PathBuf>, pub work_dir: Option<PathBuf>,
pub jobs: usize, pub jobs: usize,
@@ -183,21 +178,8 @@ pub fn install_webhooks(config: &Config, options: WebhookInstallOptions) -> Resu
} }
let work_dir = options.work_dir.clone().unwrap_or_else(default_work_dir); let work_dir = options.work_dir.clone().unwrap_or_else(default_work_dir);
let state = Arc::new(Mutex::new(load_webhook_state(&work_dir)?)); let state = Arc::new(Mutex::new(load_webhook_state(&work_dir)?));
let repo_pattern = options
.repo_pattern
.as_deref()
.map(regex::Regex::new)
.transpose()
.with_context(|| "invalid --repo-pattern regex")?;
for mirror in &config.mirrors { for mirror in &config.mirrors {
if options
.group
.as_ref()
.is_some_and(|group| group != &mirror.name)
{
continue;
}
crate::logln!(); crate::logln!();
crate::logln!( crate::logln!(
"{} {}", "{} {}",
@@ -222,15 +204,6 @@ pub fn install_webhooks(config: &Config, options: WebhookInstallOptions) -> Resu
.filter(|repo| mirror.sync_visibility.matches_private(repo.private)) .filter(|repo| mirror.sync_visibility.matches_private(repo.private))
.filter(|repo| repo_filter.matches(&repo.name)) .filter(|repo| repo_filter.matches(&repo.name))
{ {
if options.repo.as_ref().is_some_and(|name| name != &repo.name) {
continue;
}
if repo_pattern
.as_ref()
.is_some_and(|pattern| !pattern.is_match(&repo.name))
{
continue;
}
tasks.push(WebhookInstallTask { tasks.push(WebhookInstallTask {
site: site.clone(), site: site.clone(),
group: mirror.name.clone(), group: mirror.name.clone(),
@@ -262,13 +235,6 @@ pub fn uninstall_webhooks(config: &Config, options: WebhookUninstallOptions) ->
let mut state = load_webhook_state(&work_dir)?; let mut state = load_webhook_state(&work_dir)?;
let mut tasks = Vec::new(); let mut tasks = Vec::new();
for mirror in &config.mirrors { for mirror in &config.mirrors {
if options
.group
.as_ref()
.is_some_and(|group| group != &mirror.name)
{
continue;
}
crate::logln!(); crate::logln!();
crate::logln!( crate::logln!(
"{} {}", "{} {}",
@@ -287,9 +253,6 @@ pub fn uninstall_webhooks(config: &Config, options: WebhookUninstallOptions) ->
.list_repos(endpoint) .list_repos(endpoint)
.with_context(|| format!("failed to list repos for {}", endpoint.label()))?; .with_context(|| format!("failed to list repos for {}", endpoint.label()))?;
for repo in repos { for repo in repos {
if options.repo.as_ref().is_some_and(|name| name != &repo.name) {
continue;
}
tasks.push(WebhookUninstallTask { tasks.push(WebhookUninstallTask {
group: mirror.name.clone(), group: mirror.name.clone(),
site: site.clone(), site: site.clone(),
@@ -329,8 +292,6 @@ pub fn update_webhooks(config: &Config, options: WebhookUpdateOptions) -> Result
config, config,
WebhookUninstallOptions { WebhookUninstallOptions {
url: options.old_url.clone(), url: options.old_url.clone(),
group: None,
repo: None,
dry_run: options.dry_run, dry_run: options.dry_run,
work_dir: options.work_dir.clone(), work_dir: options.work_dir.clone(),
jobs: options.jobs, jobs: options.jobs,
@@ -343,9 +304,6 @@ pub fn update_webhooks(config: &Config, options: WebhookUpdateOptions) -> Result
WebhookInstallOptions { WebhookInstallOptions {
url: options.new_url, url: options.new_url,
secret: options.secret, secret: options.secret,
group: None,
repo: None,
repo_pattern: None,
dry_run: options.dry_run, dry_run: options.dry_run,
work_dir: options.work_dir, work_dir: options.work_dir,
jobs: options.jobs, jobs: options.jobs,
+2 -22
View File
@@ -646,33 +646,13 @@ namespace = "{}"
self.seed_all_main(&repo, "webhook base", 1_700_001_601)?; self.seed_all_main(&repo, "webhook base", 1_700_001_601)?;
self.sync(["--repo-pattern", &exact_pattern(&repo)])?; self.sync(["--repo-pattern", &exact_pattern(&repo)])?;
self.refray([ self.refray(["webhook", "install", "--dry-run"])?;
"webhook", self.refray(["webhook", "uninstall", "--dry-run"])?;
"install",
"--dry-run",
"--repo-pattern",
&exact_pattern(&repo),
"--url",
"https://example.invalid/webhook",
"--secret",
WEBHOOK_SECRET,
])?;
self.refray([
"webhook",
"uninstall",
&repo,
"--dry-run",
"--url",
"https://example.invalid/webhook",
])?;
self.refray([ self.refray([
"webhook", "webhook",
"update", "update",
"--dry-run", "--dry-run",
"--url",
"https://example.invalid/new-webhook", "https://example.invalid/new-webhook",
"--secret",
WEBHOOK_SECRET,
])?; ])?;
let listener = TcpListener::bind("127.0.0.1:0")?; let listener = TcpListener::bind("127.0.0.1:0")?;
+100 -67
View File
@@ -85,98 +85,88 @@ fn cli_accepts_webhook_serve() {
#[test] #[test]
fn cli_accepts_webhook_install() { fn cli_accepts_webhook_install() {
let cli = Cli::try_parse_from([ let cli = Cli::try_parse_from(["refray", "webhook", "install", "--jobs", "6"]).unwrap();
"refray",
"webhook",
"install",
"repo-one",
"--url",
"https://mirror.example.test/webhook",
"--secret",
"secret",
"--group",
"sync-1",
"--jobs",
"6",
])
.unwrap();
let Command::Webhook(WebhookCommand::Install(args)) = cli.command else { let Command::Webhook(WebhookCommand::Install(args)) = cli.command else {
panic!("parsed unexpected command"); panic!("parsed unexpected command");
}; };
assert_eq!(
args.url,
Some("https://mirror.example.test/webhook".to_string())
);
assert_eq!(args.secret, Some("secret".to_string()));
assert_eq!(args.group, Some("sync-1".to_string()));
assert_eq!(args.repo, Some("repo-one".to_string()));
assert_eq!(args.repo_pattern, None);
assert_eq!(args.jobs, 6); assert_eq!(args.jobs, 6);
} }
#[test] #[test]
fn cli_accepts_webhook_install_repo_pattern() { fn cli_rejects_removed_webhook_install_args() {
let cli = Cli::try_parse_from([ for args in [
"refray", ["refray", "webhook", "install", "repo-one"].as_slice(),
"webhook", [
"install", "refray",
"--url", "webhook",
"https://mirror.example.test/webhook", "install",
"--secret", "--url",
"secret", "https://mirror.example.test/webhook",
"--repo-pattern", ]
"^repo$", .as_slice(),
]) ["refray", "webhook", "install", "--group", "sync-1"].as_slice(),
.unwrap(); ["refray", "webhook", "install", "--work-dir", "/tmp/refray"].as_slice(),
["refray", "webhook", "install", "--secret", "secret"].as_slice(),
let Command::Webhook(WebhookCommand::Install(args)) = cli.command else { [
panic!("parsed unexpected command"); "refray",
}; "webhook",
assert_eq!(args.repo, None); "install",
assert_eq!(args.repo_pattern, Some("^repo$".to_string())); "--secret-env",
"WEBHOOK_SECRET",
]
.as_slice(),
["refray", "webhook", "install", "--repo-pattern", "^repo$"].as_slice(),
] {
assert!(Cli::try_parse_from(args).is_err());
}
} }
#[test] #[test]
fn cli_accepts_webhook_uninstall() { fn cli_accepts_webhook_uninstall() {
let cli = Cli::try_parse_from([ let cli = Cli::try_parse_from(["refray", "webhook", "uninstall", "--dry-run", "--jobs", "3"])
"refray", .unwrap();
"webhook",
"uninstall",
"repo-one",
"--url",
"https://mirror.example.test/webhook",
"--group",
"sync-1",
"--dry-run",
"--jobs",
"3",
])
.unwrap();
let Command::Webhook(WebhookCommand::Uninstall(args)) = cli.command else { let Command::Webhook(WebhookCommand::Uninstall(args)) = cli.command else {
panic!("parsed unexpected command"); panic!("parsed unexpected command");
}; };
assert_eq!(
args.url,
Some("https://mirror.example.test/webhook".to_string())
);
assert_eq!(args.group, Some("sync-1".to_string()));
assert_eq!(args.repo, Some("repo-one".to_string()));
assert!(args.dry_run); assert!(args.dry_run);
assert_eq!(args.jobs, 3); assert_eq!(args.jobs, 3);
} }
#[test]
fn cli_rejects_removed_webhook_uninstall_args() {
for args in [
["refray", "webhook", "uninstall", "repo-one"].as_slice(),
[
"refray",
"webhook",
"uninstall",
"--url",
"https://mirror.example.test/webhook",
]
.as_slice(),
["refray", "webhook", "uninstall", "--group", "sync-1"].as_slice(),
[
"refray",
"webhook",
"uninstall",
"--work-dir",
"/tmp/refray",
]
.as_slice(),
] {
assert!(Cli::try_parse_from(args).is_err());
}
}
#[test] #[test]
fn cli_accepts_webhook_update() { fn cli_accepts_webhook_update() {
let cli = Cli::try_parse_from([ let cli = Cli::try_parse_from([
"refray", "refray",
"webhook", "webhook",
"update", "update",
"--url",
"https://new.example.test/webhook", "https://new.example.test/webhook",
"--secret-env",
"WEBHOOK_SECRET",
"--jobs", "--jobs",
"5", "5",
]) ])
@@ -186,10 +176,43 @@ fn cli_accepts_webhook_update() {
panic!("parsed unexpected command"); panic!("parsed unexpected command");
}; };
assert_eq!(args.url, "https://new.example.test/webhook"); assert_eq!(args.url, "https://new.example.test/webhook");
assert_eq!(args.secret_env, Some("WEBHOOK_SECRET".to_string()));
assert_eq!(args.jobs, 5); assert_eq!(args.jobs, 5);
} }
#[test]
fn cli_rejects_removed_webhook_update_secret_args() {
for args in [
[
"refray",
"webhook",
"update",
"--url",
"https://new.example.test/webhook",
]
.as_slice(),
[
"refray",
"webhook",
"update",
"https://new.example.test/webhook",
"--secret",
"secret",
]
.as_slice(),
[
"refray",
"webhook",
"update",
"https://new.example.test/webhook",
"--secret-env",
"WEBHOOK_SECRET",
]
.as_slice(),
] {
assert!(Cli::try_parse_from(args).is_err());
}
}
#[test] #[test]
fn cli_rejects_scoped_webhook_update() { fn cli_rejects_scoped_webhook_update() {
let result = Cli::try_parse_from([ let result = Cli::try_parse_from([
@@ -197,9 +220,19 @@ fn cli_rejects_scoped_webhook_update() {
"webhook", "webhook",
"update", "update",
"repo-one", "repo-one",
"--url",
"https://new.example.test/webhook", "https://new.example.test/webhook",
]); ]);
assert!(result.is_err()); assert!(result.is_err());
} }
#[test]
fn missing_config_error_asks_user_to_run_config() {
let temp = tempfile::tempdir().unwrap();
let path = temp.path().join("config.toml");
let error = load_config(&path).unwrap_err().to_string();
assert!(error.contains("config not found at"));
assert!(error.contains("Run `refray config` first"));
}