docs: Refine docs (#200)

* Update docs

* update one demo video

* update docs

* remove the video

* move configuration file

* fix demo title

README.md

@@ -17,14 +17,14 @@
 # 📰 News
 | 🗞️News        | 📝Description                 |
 | --            | ------                        |
-| First release | RDAgent are release on Github |
+| First release | RDAgent is release on Github |
 
 
 # 🌟 Introduction
 
 ![Our focused scenario](docs/_static/scen.jpg)
 
-RDAgent aims to automate the most critical and valuable aspects of the industrial R&D process, and we begins with focusing on the data-driven scenarios to streamline the development of models and data. 
+RDAgent aims to automate the most critical and valuable aspects of the industrial R&D process, and we begin with focusing on the data-driven scenarios to streamline the development of models and data. 
 Methodologically, we have identified a framework with two key components: 'R' for proposing new ideas and 'D' for implementing them.
 We believe that the automatic evolution of R&D will lead to solutions of significant industrial value.
 
@@ -38,15 +38,21 @@ R&D is a very general scenario. The advent of RDAgent can be your
 
 You can click the [🎥link](https://rdagent.azurewebsites.net) above to view the demo. More methods and scenarios are being added to the project to empower your R&D processes and boost productivity.
 <!-- 
+- TODO: Demo: it fails to display the video in the README.md.
 We have a quick 🎥demo for one use case of RDAgent.
-- TODO: Demo
+[![Demo Video](https://img.youtube.com/vi/5275fcb75803ad2bb9541c3abd86dedfd578a28fa32b46fa28917b33/0.jpg)](https://rdagent.azurewebsites.net:443/media/5275fcb75803ad2bb9541c3abd86dedfd578a28fa32b46fa28917b33.mp4)
  -->
 
+
+
 # ⚡Quick start
-You can try our demo by running the following command:
+You can try above demos by running the following command:
+
+### 🐳Docker installation.
+Users must ensure Docker is installed before attempting most scenarios. Please refer to the [official 🐳Docker page](https://docs.docker.com/engine/install/) for installation instructions.
 
 ### 🐍 Create a Conda Environment
-- Create a new conda environment with Python (3.10 and 3.11 are well tested in our CI):
+- Create a new conda environment with Python (3.10 and 3.11 are well-tested in our CI):
   ```sh
   conda create -n rdagent python=3.10
   ```
@@ -60,27 +66,20 @@ You can try our demo by running the following command:
   ```sh
   pip install rdagent
   ```
-
-
-### 🛠️ Run Make Files
-
-- If you want to use our project in development mode. **Navigate to the directory containing the MakeFile** and set up the development environment:
-  ```sh
-  make dev
-  ```
 
+### ⚙️ Configuration
+You have to config your GPT model in the `.env`
+```bash
+cat << EOF  > .env
+OPENAI_API_KEY=<your_api_key>
+# EMBEDDING_MODEL=text-embedding-3-small
+CHAT_MODEL=gpt-4-turbo
+EOF
+```
 
-### ⚙️ Environment Configuration
-- Place the `.env` file in the same directory as the `.env.example` file.
-  - The `.env.example` file contains the environment variables required for users using the OpenAI API (Please note that `.env.example` is an example file. `.env` is the one that will be finally used.)
-  - please refer to [Configuration](docs/build/html/installation.html#azure-openai) for the detailed explanation of the `.env`
 ### 🚀 Run the Application
-- Start web app to show log traces:
-  ```sh
-  rdagent ui --port 80 --log_dir <your log folder>
-  ```
 
-The [🎥demo](https://rdagent.azurewebsites.net) is implemented by the following commands:
+The [🎥demo](https://rdagent.azurewebsites.net) is implemented by the following commands(each item represents one demo, you can select the one you prefer):
 
 - Run the **Automated Quantitative Trading & Iterative Factors Evolution**:  Qlib self-loop factor proposal and implementation application
   ```sh
@@ -107,10 +106,10 @@ The [🎥demo](https://rdagent.azurewebsites.net) is implemented by the followin
   rdagent general_model  <Your paper url>
   ```
 
-### 🚀 Monitor the Application
+### 🚀 Monitor the Application Results
 - You can serve our demo app to monitor the RD loop by running the following command:
   ```sh
-  rdagent ui --port 80 --log_dir <your log folder>
+  rdagent ui --port 80 --log_dir <your log folder like "log/2024-07-16_11-21-46-612120/">
   ```
 
 # Scenarios
@@ -144,8 +143,12 @@ The supported scenarios are listed below:
 
 Different scenarios vary in entrance and configuration. Please check the detailed setup tutorial in the scenarios documents.
 
-TODO: Scenario Gallary
-- map(scenario) => knowledge list;
+Here is a gallery of successful explorations. You can download the source code and view the execution trace using the command below:
+
+```bash
+rdagent ui --port 80 --log_dir gallary/
+```
+
 
 # ⚙️Framework
 
@@ -206,7 +209,7 @@ For more detail, please refer to our [Demos page](https://rdagent.azurewebsites.
 
 # Contributing
 
-More documents can be found in the [📚readthedocs](). TODO: add link
+More documents can be found in the [📚readthedocs](https://rdagent.readthedocs.io/).
 
 ## Guidance
 This project welcomes contributions and suggestions.

docs/development.rst

@@ -73,4 +73,4 @@ File Naming Convention
    * - `conf.py`
      - The configuration for the module, app, and project.
 
-<!-- TODO: renaming files -->
+.. <!-- TODO: renaming files -->

docs/installation_and_configuration.rst

@@ -143,3 +143,25 @@ Configuration List
 | max_past_message_include     | Maximum number of past messages to include       | 10                      |
 +------------------------------+--------------------------------------------------+-------------------------+
 
+
+
+
+Loading Configuration
+---------------------
+
+For users' convenience, we provide a CLI interface called `rdagent`, which automatically runs `load_dotenv()` to load environment variables from the `.env` file.
+However, this feature is not enabled by default for other scripts. We recommend users load the environment with the following steps:
+
+
+- ⚙️ Environment Configuration
+    - Place the `.env` file in the same directory as the `.env.example` file.
+        - The `.env.example` file contains the environment variables required for users using the OpenAI API (Please note that `.env.example` is an example file. `.env` is the one that will be finally used.)
+
+    - Export each variable in the .env file:
+
+      .. code-block:: sh
+
+          export $(grep -v '^#' .env | xargs)
+    
+    - If you want to change the default environment variables, you can refer to the above configuration and edith the `.env` file.
+

docs/project_framework_introduction.rst

@@ -23,19 +23,5 @@ We have established a basic method framework that continuously proposes hypothes
 The figure above shows the main classes and how they fit into the workflow for those interested in the detailed code.
 
 
-Detailed Design
-=========================
-
-
-Configuration
--------------
-
-You can manually source the `.env` file in your shell before running the Python script:
-Most of the workflow are controlled by the environment variables.
-```sh
-# Export each variable in the .env file; Please note that it is different from `source .env` without export
-export $(grep -v '^#' .env | xargs)
-# Run the Python script
-python your_script.py
-```
-
+.. Detailed Design
+.. ===============

docs/scens/data_agent_fin.rst

@@ -17,9 +17,17 @@ By leveraging these factors, quantitative traders can develop sophisticated stra
 The ability to systematically analyze and apply these factors is what separates ordinary trading from truly strategic market outmaneuvering.
 And this is where the **Finance Model Agent** comes into play.
 
-🎥 Demo
-~~~~~~~~~~
-TODO: Here should put a video of the demo.
+🎥 `Demo <https://rdagent.azurewebsites.net/factor_loop>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. https://rdagent.azurewebsites.net:443/media/54d59c2bf85211de6a7cd2bd3c57d3b56c36cf9795a31111a9111ddb.mp4
+
+.. raw:: html
+
+    <video width="600" controls>
+      <source src="https://rdagent.azurewebsites.net:443/media/54d59c2bf85211de6a7cd2bd3c57d3b56c36cf9795a31111a9111ddb.mp4" type="video/mp4">
+      Your browser does not support the video tag.
+    </video>
 
 
 🌟 Introduction
@@ -75,6 +83,8 @@ Here's an enhanced outline of the steps:
 ⚡ Quick Start
 ~~~~~~~~~~~~~~~~~
 
+Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency.
+
 You can try our demo by running the following command:
 
 - 🐍 Create a Conda Environment
@@ -97,12 +107,6 @@ You can try our demo by running the following command:
 
           pip install rdagent
 
-- ⚙️ Environment Configuration
-    - Place the `.env` file in the same directory as the `.env.example` file.
-        - The `.env.example` file contains the environment variables required for users using the OpenAI API (Please note that `.env.example` is an example file. `.env` is the one that will be finally used.)
-
-    - If you want to change the default environment variables, you can refer to `Env Config`_ below
-
 - 🚀 Run the Application
     .. code-block:: sh
 
@@ -126,4 +130,4 @@ The following environment variables can be set in the `.env` file to customize t
     :settings-show-field-summary: False
     :members: coder_use_cache, data_folder, data_folder_debug, cache_location, enable_execution_cache, file_based_execution_timeout, select_method, select_threshold, max_loop, knowledge_base_path, new_knowledge_base_path
     :exclude-members: Config, fail_task_trial_limit, v1_query_former_trace_limit, v1_query_similar_success_limit, v2_query_component_limit, v2_query_error_limit, v2_query_former_trace_limit, v2_error_summary, v2_knowledge_sampler
-    :no-index:
+    :no-index:

docs/scens/data_copilot_fin.rst

@@ -20,9 +20,16 @@ Therefore, there is an urgent need for a systematic approach to design a framewo
 And this is where the **Finance Data Copilot** steps in.
 
 
-🎥 Demo
-~~~~~~~~~~
-TODO: Here should put a video of the demo.
+🎥 `Demo <https://rdagent.azurewebsites.net/report_factor>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <video width="600" controls>
+      <source src="https://rdagent.azurewebsites.net:443/media/42a3280d592fb9e3d227608170860c426fc2bf5c63eea5a66ed94ca7.mp4" type="video/mp4">
+      Your browser does not support the video tag.
+    </video>
+
 
 
 🌟 Introduction
@@ -76,6 +83,8 @@ Here's an enhanced outline of the steps:
 ⚡ Quick Start
 ~~~~~~~~~~~~~~~~~
 
+Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency.
+
 You can try our demo by running the following command:
 
 - 🐍 Create a Conda Environment
@@ -98,12 +107,6 @@ You can try our demo by running the following command:
 
           pip install rdagent
 
-- ⚙️ Environment Configuration
-    - Place the `.env` file in the same directory as the `.env.example` file.
-        - The `.env.example` file contains the environment variables required for users using the OpenAI API (Please note that `.env.example` is an example file. `.env` is the one that will be finally used.)
-
-    - If you want to change the default environment variables, you can refer to `Env Config`_ below
-
 - 🚀 Run the Application
     - Store the factors you want to extract from the financial reports in your desired folder. Then, save the paths of the reports in the `report_result_json_file_path`. The format should be as follows:
 
@@ -139,4 +142,4 @@ The following environment variables can be set in the `.env` file to customize t
     :settings-show-field-summary: False
     :members: coder_use_cache, data_folder, data_folder_debug, cache_location, enable_execution_cache, file_based_execution_timeout, select_method, select_threshold, max_loop, knowledge_base_path, new_knowledge_base_path
     :exclude-members: Config, python_bin, fail_task_trial_limit, v1_query_former_trace_limit, v1_query_similar_success_limit, v2_query_component_limit, v2_query_error_limit, v2_query_former_trace_limit, v2_error_summary, v2_knowledge_sampler
-    :no-index:
+    :no-index:

docs/scens/model_agent_fin.rst

@@ -16,9 +16,20 @@ The effectiveness of a quantitative strategy depends not only on the factors use
 However, the process of developing and optimizing these models can be labor-intensive and complex, requiring continuous refinement and adaptation to ever-changing market conditions. 
 And this is where the **Finance Model Agent** steps in.
 
-🎥 Demo
-~~~~~~~~~~
-TODO: Here should put a video of the demo.
+
+🎥 `Demo <https://rdagent.azurewebsites.net/model_loop>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <video width="600" controls>
+      <source src="https://rdagent.azurewebsites.net:443/media/5275fcb75803ad2bb9541c3abd86dedfd578a28fa32b46fa28917b33.mp4" type="video/mp4">
+      Your browser does not support the video tag.
+    </video>
+
+
+
+
 
 
 🌟 Introduction
@@ -74,6 +85,8 @@ Here's an enhanced outline of the steps:
 ⚡ Quick Start
 ~~~~~~~~~~~~~~~~~
 
+Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency.
+
 You can try our demo by running the following command:
 
 - 🐍 Create a Conda Environment
@@ -96,18 +109,6 @@ You can try our demo by running the following command:
 
           pip install rdagent
 
-- ⚙️ Environment Configuration
-    - Place the `.env` file in the same directory as the `.env.example` file.
-        - The `.env.example` file contains the environment variables required for users using the OpenAI API (Please note that `.env.example` is an example file. `.env` is the one that will be finally used.)
-
-    - Export each variable in the .env file:
-
-      .. code-block:: sh
-
-          export $(grep -v '^#' .env | xargs)
-    
-    - If you want to change the default environment variables, you can refer to `Env Config`_ below
-
 - 🚀 Run the Application
     .. code-block:: sh
 
@@ -143,4 +144,4 @@ The following environment variables can be set in the `.env` file to customize t
         - **batch_size**: The batch size, set to `2000`.
         - **metric**: The evaluation metric, set to `loss`.
         - **loss**: The loss function, set to `mse`.
-        - **n_jobs**: The number of parallel jobs, set to `20`.
+        - **n_jobs**: The number of parallel jobs, set to `20`.

docs/scens/model_agent_med.rst

@@ -12,9 +12,15 @@ Medical Model Agent
 In this scenario, we consider the problem of risk prediction from patients' ICU monitoring data. We use the a public EHR dataset - MIMIC-III and extract a binary classification task for evaluating the framework.
 In this task, we aim at predicting the whether the patients will suffer from Acute Respiratory Failure (ARF) based their first 12 hours ICU monitoring data. 
 
-🎥 Demo
-~~~~~~~~~~
-TODO: Here should put a video of the demo.
+🎥 `Demo <https://rdagent.azurewebsites.net/dmm>`_
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. raw:: html
+
+    <video width="600" controls>
+      <source src="https://rdagent.azurewebsites.net:443/media/0fbd26bb297369a4ca52409ff65a3f7cb941c6af6429894367a6afcd.mp4" type="video/mp4">
+      Your browser does not support the video tag.
+    </video>
 
 
 🌟 Introduction
@@ -58,6 +64,8 @@ Here's an enhanced outline of the steps:
 ⚡ Quick Start
 ~~~~~~~~~~~~~~~~~
 
+Please refer to the installation part in :doc:`../installation_and_configuration` to prepare your system dependency.
+
 You can try our demo by running the following command:
 
 - 🐍 Create a Conda Environment
@@ -85,17 +93,6 @@ You can try our demo by running the following command:
     - Request access to FIDDLE preprocessed data: `FIDDLE Dataset <https://physionet.org/content/mimic-eicu-fiddle-feature/1.0.0/>`_.
     - Place your username and password in `.rdagent.app.data_mining.conf`.
 
-- ⚙️ Environment Configuration
-    - Place the `.env` file in the same directory as the `.env.example` file.
-        - The `.env.example` file contains the environment variables required for users using the OpenAI API (Please note that `.env.example` is an example file. `.env` is the one that will be finally used.)
-
-    - Export each variable in the .env file:
-
-      .. code-block:: sh
-
-          export $(grep -v '^#' .env | xargs)
-    
-    - If you want to change the default environment variables, you can refer to `Env Config`_ below
 
 - 🚀 Run the Application
     .. code-block:: sh
@@ -113,4 +110,4 @@ The following environment variables can be set in the `.env` file to customize t
 
 .. autopydantic_settings:: rdagent.app.data_mining.conf.PropSetting
     :settings-show-field-summary: False
-    :exclude-members: Config
+    :exclude-members: Config