Check if command can be used directly.

[tomasnorre]

from https://twitter.com/teh_plague/status/1449804058550804493

Why not simply invoke the respective Command class directly?

[helhum]

That might be an option, but might not lead to expected results either.

I have no clue in what context you want to process the queue, but I'd rather refactor that code to not expect that the queue is processed at all, but allows for async processing of the queue (which most of the time is the whole point of a queue anyway)

[tomasnorre]

Thanks for your input @helhum.

There are 3 options to process the Queue in the Crawler

1. cli typo3 crawler:processQueue
2. with BE Scheduler Module
3. From the Crawler itself with an "Add Process" button.

The "problem" that I'm trying to solve here is the "Add Process" button from the backend.
This can be handy if you don't want to wait for the scheduler, and want a batch processes a little faster.

[helhum]

There are 3 options to process the Queue in the Crawler

1. cli typo3 crawler:processQueue
2. with BE Scheduler Module
3. From the Crawler itself with an "Add Process" button.

Do all do exactly the same? I'm asking because "add process" sounds different to "process queue"

I will now comment under the assumption that alle three do the same.

I would suggest to only implement a command and document the following options:

1. The command can be executed manually vi cli
2. The command can be executed regularly by adding a scheduler task for the command (a generic task for any command is offered by TYPO3). How often can be configured in the task
3. The scheduler task can also be executed manually in the scheduler UI, bypassing its schedule

The "problem" that I'm trying to solve here is the "Add Process" button from the backend. This can be handy if you don't want to wait for the scheduler, and want a batch processes a little faster.

From what I have written above, what is missing to solve your problem?

If it is to have a button in a different place (UI) that does what I have suggested as 3., the I would suggest to apply the following:

• Add a warning instead of a button, when no scheduler task exists, with the porcessQueue command
• When the task exists, make the "Add Process" a link to the Scheduler executing the task manually

Here are the benefits when doing so:

• It works exactly the same as other crawling solutions (e.g. like in solr extension)
• You don't have to code a solution for a problem that has already be code, which means you can't introduce bugs and don't need to maintain the code
• If TYPO3 implements a better solution to execute CLI tasks from a web request, you benefit from it without changing anything

[tomasnorre]

Thanks again for your input, which is all valid. They are all 3 doing the same job: Processing entries in the Queue.

The reason for the "add process" from the backend, is in case that something needs to happen faster.
We have quite some time ago introduced the "Multi Process" function, https://docs.typo3.org/p/tomasnorre/crawler/master/en-us/Features/MultiprocessSupport/Index.html#multi-process-support

Which allows multiple processes to run at the same time, but at the same time with a limit of let's say 4 processes at ones.
If I remove that and do it over the scheduler with am "allow parallel executions" I lose control of how many processes that can be executed at ones, and therefore I risk putting too much load on the server by the Queue Processing.

This is the reason for the differentiation right now. I haven't looked into your idea yet, of triggering it over the scheduler, so I don't know yet if I can add a limit for that as well.

I really value your input for this, as this will help improve the decision that I make regarding the refactoring.
Thanks.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomasnorre]

Since we are on TYPO3 v12. I think I would see if I could come up with a setup that enables the Message Bus [1] instead of the Crawler Queue.

This will remove some logic from the TYPO3 Crawler, and ease the maintenance.

[1] https://docs.typo3.org/m/typo3/reference-coreapi/main/en-us/ApiOverview/MessageBus/Index.html