Running Commands on a Lab from the STS¶

Resend data, refresh, upgrade or roll back a connected lab from the STS, with nobody at the lab machine.

For how the plane works and why, see the remote command plane design.

Before starting, check both of these:

The STS account's role has Queue Lab Command, Cancel Lab Command and Lab Command History under Monitoring. Set them in ADMIN → Users & Roles → Roles. Without Queue Lab Command, the Queue button does not appear.
The lab is online and syncing with the STS.

Queue a command¶

Go to Admin → Monitoring → Lab Sync Status.
Find the lab's row. In the Command plane column, check that the courier chip shows a recent time. Upgrade, rollback and the other root commands also need a recent runner chip.

If the column shows only a dash

The lab has never reported the command plane. Either its release is too old, or remote commands are off on the lab. See Turn remote commands on or off for a lab.
Select Queue in the Actions column.

If Queue is greyed out

The lab has not polled for commands in the last 24 hours. Queueing stays off until it does. Check the lab machine is on, online and has remote commands turned on.

If the dialog warns about installations

More than one installation has reported for this lab recently. A queued command goes to whichever one polls first, and the other never sees it. Find out which machine is live before queueing an upgrade or a rollback.
Choose the Command. See Commands for what each one does. Greyed-out commands are ones the lab has not reported it can run.
If the upgrade and rollback commands are greyed out

The lab offers only the basic commands. One of these applies:
- Allow Remote Upgrade is off on the lab. See Turn remote commands on or off for a lab.
- The lab's database runs on another machine, or the lab machine has no systemd. Upgrade that lab the way it was installed.
- The lab has not sent a full capability report yet. Wait one sync, then reopen the dialog.

Fill in the fields the command shows:

Field	Shown for	What to enter
Module (optional)	Resend results, Resend requests	A module, or leave All enabled modules.
Resend data from last N days	Resend results, Resend requests	A number of days from 1 to 3650. Leave blank to send only records not yet synced. Resend requests ignores this field.
Prepared staging to apply	Apply a prepared upgrade	The staged release to install.
Not before (optional)	Upgrade, Prepare upgrade only, Apply a prepared upgrade	The earliest time the lab may start, by the STS clock. Leave blank to start at the next sync.
Show maintenance page to users during apply	Upgrade, Apply a prepared upgrade	Select it when the release runs database migrations or changes composer dependencies.

Select Queue command. The message Command queued. shows the command ID. A yellow badge such as upgrade: pending appears on the lab's row.

If it says a matching command is already in flight

The same command is already waiting or running for this lab. Wait for it to finish, or cancel it while it is still pending.
Wait for the lab's next sync. The lab syncs about every 5 minutes. The badge changes to picked or running, then disappears when the command finishes.

To cancel before the lab picks it up

Select × on the badge. This works only while the status is pending. Once the lab has picked the command up, it runs to the end.

To see how it ended, follow Check a command's result.

Commands¶

Command	What it does	Needs the runner
Ping (self-test, no side effects)	Confirms the lab picks up and reports commands. Changes nothing.	No
Resend results	Sends results to the STS again.	No
Resend requests	Pulls test requests from the STS again.	No
Metadata resync (force)	Pulls all reference data from the STS, ignoring the last sync time, and sends the lab's metadata.	No
Refresh cache	Clears the lab's file cache.	No
Rotate STS token	Drops the lab's STS token and fetches a new one.	No
Refresh permissions	Resets file ownership and permissions on the installation.	Yes
Restart Apache	Restarts the web server gracefully.	Yes
Upgrade (prepare + auto-apply)	Downloads the current release and installs it in one go.	Yes
Prepare upgrade only	Downloads and checks the current release. Does not install it.	Yes
Apply a prepared upgrade	Installs a release staged by Prepare upgrade only.	Yes
Roll back to the pre-upgrade snapshot (code only)	Restores the code from before the last upgrade. The database stays as it is.	Yes

Upgrade a lab¶

Choose one way, then follow its steps from top to bottom.

One step (upgrade)Staged (prepare, pilot, apply)

Use this for a routine release on one lab or a few.

Queue the upgrade¶

Go to Admin → Monitoring → Lab Sync Status.
On the lab's row, check that the runner chip in the Command plane column shows a recent time.
Select Queue.
Choose Upgrade (prepare + auto-apply).
To start at a set time, enter it in Not before. Pick a time when the lab is quiet.
If the release runs database migrations or changes composer dependencies, select Show maintenance page to users during apply. Users then see an "upgrade in progress" page during the install instead of errors.
Select Queue command.
Wait. Users keep working while the release downloads. The install itself is short.

Check the upgrade¶

When the upgrade badge disappears, check that the Version column shows the new release.
Select Lab Command History. Find the upgrade row for the lab and select Details. Exit code shows 0 and succeeded.
If the status is failed

Read the output in Details.
- If the smoke check failed, the updater already restored the previous code. The lab runs the old release on the migrated database.
- If the database migrations failed, the new code stays in place and is not rolled back. With the maintenance page selected, the lab keeps showing it. Someone at the lab machine must fix the cause and run intelis update again.

Use this for a risky release or many labs. Each lab downloads the release first, and installs it only when an apply is queued.

Stage the release¶

Go to Admin → Monitoring → Lab Sync Status.
On each lab's row, check that the runner chip in the Command plane column shows a recent time.
On each lab's row, select Queue.
Choose Prepare upgrade only.
To start at a set time, enter it in Not before.
Select Queue command.
Wait for a blue Staged: vX.Y.Z badge on each lab's row. Users are not affected while a lab prepares.

If a lab shows two Staged badges

Each prepare replaces the older staging on the lab machine. Apply the newest one. The older one fails with staging dir invalid or READY sentinel missing.

Apply on pilot labs¶

Pick two or three pilot labs.
On a pilot lab's row, select Queue.
Choose Apply a prepared upgrade.
In Prepared staging to apply, choose the staged release.
To start at a set time, enter it in Not before.
If the release runs database migrations or changes composer dependencies, select Show maintenance page to users during apply.
Select Queue command.
When the upgrade-apply badge disappears, check that the Version column shows the new release.
Select Lab Command History. Find the upgrade-apply row for the lab and select Details. Exit code shows 0 and succeeded.
If the status is failed

Read the output in Details.
- If the smoke check failed, the updater already restored the previous code. The lab runs the old release on the migrated database.
- If the database migrations failed, the new code stays in place and is not rolled back. With the maintenance page selected, the lab keeps showing it. Someone at the lab machine must fix the cause and run intelis update again.
If the Staged badge stays after a successful apply

The badge does not clear by itself. Go by the Version column. Do not apply the same staging again. It fails with no prepared record.
Repeat steps 9 to 16 for the other pilot labs.
Watch the pilots for a day or two.

Apply on the other labs¶

If the pilots are healthy, repeat steps 9 to 16 for each remaining lab.

Roll back a lab¶

Roll back when an upgrade went through but the lab misbehaves on the new release. A rollback restores the code from the snapshot taken right before the last upgrade. It does not touch the database. Migrations only run forward, so the previous release then runs against the newer database.

Choose one way, then follow its steps from top to bottom.

From the STSOn the lab machine

Queue the rollback¶

Go to Admin → Monitoring → Lab Sync Status.
On the lab's row, check that the runner chip in the Command plane column shows a recent time.
Select Queue.
Choose Roll back to the pre-upgrade snapshot (code only). A warning explains that the database is not rolled back.

If the rollback command is greyed out

The lab has not reported it can run rollbacks. Its courier is older than the rollback command, or Allow Remote Upgrade is off on the lab. Use the On the lab machine tab instead.
Select Queue command.

If it reports Unknown command

An STS older than this release does not accept rollbacks from the queue. Use the On the lab machine tab instead.
Wait for the lab's next sync, about 5 minutes. The rollback badge disappears when the command finishes.

Check the rollback¶

Select Lab Command History. Find the rollback row for the lab and select Details.
Check that Exit code shows 0 and the output ends with Rolled back and the installation path. The output also names the snapshot it used, after Most recent snapshot:.

If the output says No rollback snapshot found

There is nothing to go back to. The last upgrade ran without a snapshot, or none was ever taken on this machine.

If the output says composer install failed during rollback

The code is restored but vendor/ is missing, so the lab does not open. The lab machine needs internet access. Someone at the lab machine runs intelis update --rollback again.
After the next sync, check that the Version column shows the previous release.
Ask the lab to open InteLIS and confirm the pages they use work.

This way always works, even when the lab cannot reach the STS.

Run the rollback¶

Open a terminal on the lab machine.
Run:
```
intelis update --rollback
```
Enter the password when asked.
If intelis is not recognised

The install is older. Run the updater directly:
```
sudo intelis-update -p /var/www/intelis --rollback
```
On older installs, use /var/www/vlsm in place of /var/www/intelis.
Read the line starting with Most recent snapshot:. It names the snapshot being restored.
Wait for Rolled back and the installation path.

If it says No rollback snapshot found

There is nothing to go back to. The last upgrade ran without a snapshot, or none was ever taken on this machine.

If it says composer install failed during rollback

The code is restored but vendor/ is missing, so the lab does not open. Connect the machine to the internet and run the same command again.

Check the rollback¶

Open InteLIS in the browser.
Confirm the pages the lab uses work.

Do not copy the snapshot back by hand

The snapshot leaves out uploads, runtime data and vendor/. Copying it over the installation with rsync --delete deletes the lab's uploads and leaves no vendor/. The rollback command skips the same folders and reinstalls vendor/.

After a rollback, fix the cause, then upgrade again with a corrected release.

Check a command's result¶

Go to Admin → Monitoring → Lab Sync Status.
Select Lab Command History. It opens in a new tab and lists the 200 most recent commands.
To find an older command, narrow the list by Lab, Command, Status or Date range, then select Search.
On the command's row, select Details.
Read Status. For a finished command, also read Exit code and the output below it.

Status	Meaning
`pending`	Queued. The lab has not picked it up yet.
`picked`	The lab has picked it up.
`running`	The lab machine is running it.
`prepared`	A Prepare upgrade only finished. The release waits for an apply.
`completed`	Finished. Exit code is `0`.
`failed`	Finished with an error. Read the output.
`expired`	Passed its deadline before the lab ran it.
`cancelled`	Cancelled while `pending`.

To run a finished command again with the same settings, select Replay on its row.

If a command stays pending

The lab is not polling. Check the courier chip on Lab Sync Status. If it is old or missing, the lab is offline or has remote commands turned off. A command with Not before also stays pending until that time.

If a command stays picked or running

The lab took it but has not reported back. On the lab machine, read the runner log for root commands:

sudo tail -n 100 /var/log/intelis-runner/runner-$(date +%Y%m%d).log
systemctl status intelis-runner.timer

For other commands, read /var/log/apache2/error.log and the application logs.

If it failed with Stale: no status report received within 2 hours of pick-up

The STS stopped waiting after 2 hours. The command may still have finished on the lab. For an upgrade, check the Version column before queueing it again.

If the result says runner disabled on this instance

Allow Remote Upgrade is off on the lab, or the lab's database is on another machine, or the machine has no systemd. The runner then refuses root commands. See Turn remote commands on or off for a lab.

Turn remote commands on or off for a lab¶

Two settings on the lab control remote commands. Both are on by default.

Setting	When off
`remote_commands_enabled`	The lab ignores all remote commands. Queued commands stay `pending`.
`allow_remote_upgrade`	The lab refuses root commands: upgrades, rollback, Refresh permissions and Restart Apache. The other commands still run.

No screen changes these settings. Change them in the lab's database, on the lab machine.

Open a terminal on the lab machine.
Open the database:
```
sudo mysql vlsm
```
If the database has another name, use the name in configs/config.production.php.

Run the statement for the change needed:

To	Run
Turn all remote commands off	`UPDATE global_config SET value = 'no' WHERE name = 'remote_commands_enabled';`
Turn only root commands off	`UPDATE global_config SET value = 'no' WHERE name = 'allow_remote_upgrade';`
Turn all remote commands back on	`UPDATE global_config SET value = 'yes' WHERE name = 'remote_commands_enabled';`
Turn root commands back on	`UPDATE global_config SET value = 'yes' WHERE name = 'allow_remote_upgrade';`

Check the result:

SELECT name, value FROM global_config
 WHERE name IN ('remote_commands_enabled', 'allow_remote_upgrade');

Type exit to leave the database.
Clear the cache so the lab reads the new value now, not within the hour:
```
intelis purge-cache
```
Wait for the lab's next sync, about 5 minutes.

A forced metadata sync turns a setting back on

Both settings come down from the STS with the lab's reference data. A forced metadata sync, including the Metadata resync (force) command, copies the STS value over the lab's own. Check the setting again after one.

Check the change¶

On Lab Sync Status, open the lab's Queue dialog.
- With root commands off, the upgrade and rollback commands are greyed out.
- With all remote commands off, the courier chip stops updating. After 24 hours, the Queue button is greyed out.
- With them back on, the commands are offered again after the next sync.