How can I avoid 504 Gateway Timeout when an LLM response takes more than 240 seconds in Azure App Service?

I have deployed a FastAPI application on Azure App Service. The application processes large tender/RFP documents using Azure Document Intelligence and then sends the extracted content to an LLM (GPT-4o) for response generation.

Current workflow:

1. User uploads a tender document.

2. Azure Document Intelligence extracts the text.

3. The document is split into chunks (~10,000 tokens each).

4. Chunk-level LLM processing is performed.

5. An aggregation prompt combines the chunk outputs.

6. The final response can be around 16,000 output tokens.

The issue is that the complete processing sometimes takes more than 240 seconds, and the client receives a 504 Gateway Timeout from Azure App Service.

Constraints:

• Azure App Service deployment

• GPT-4o model

• Large prompts and large outputs

• Current implementation is mostly sequential

• The request is processed synchronously and the API waits for the final LLM response before returning

Questions:

1. What are the recommended architectural patterns for handling long-running LLM workloads in Azure App Service?

2. Is moving the LLM processing to a background job (Azure Functions, WebJobs, Service Bus, etc.) the preferred solution?

3. Would streaming responses prevent the gateway timeout, or does the backend request still need to complete within the App Service timeout limit?

4. What are the best practices for reducing end-to-end latency when processing large documents with GPT-4o?

5. Has anyone implemented asynchronous job-based processing for similar RFP/tender document generation workflows?

Any guidance on avoiding gateway timeouts and designing a scalable architecture for long-running LLM requests would be appreciated.