Installing the tools

For the functioning of the system you should install the following tools

Installing the tools

Installing the tools

1. ClickHouse

ClickHouse is an open-source column-oriented DBMS (columnar database management system) for online analytical processing (OLAP) that allows users to generate analytical reports using SQL queries in real-time.

Deploy ClickHouse in the local machine using Docker

This is docker-compose file that you can use for deploying of ClickHouse.

6KB

clickhouse.zip

archive

Open

There is also README.md where you can see all commands to deploy ClickHouse.

Important:

• Users (users.d):

  • Admin:

    • login: super_admin;

    • password - super_admin.

  • Default user:

    • login: default;

    • password - no (empty).

  • User with access to DB deals_history:

    • login: deals_history;

    • password - deals_history.

• DataBase deals_history and tables for working with deals_history. Catalog docker-entrypoint-initdb.d.

Example of docker-copmpose:

version: "3"
services:
  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: clickhouse
    volumes:
      - clickhouse_db_data:/var/lib/clickhouse
      - clickhouse_db_logs:/var/log/clickhouse-server
      - ./sql:/sql:ro
      - ./users.d:/etc/clickhouse-server/users.d
      - ./docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
    ports:
      - "8123:8123"
      - "9000:9000"
volumes:
  clickhouse_db_data:
  clickhouse_db_logs:

You can see the official doumentation for ClickHouse here:

Install ClickHouse | ClickHouse Docsclickhouse.com

For the installation of ClickHouse use the following:

clickhouse/clickhouse-server - Docker Imagehub.docker.com

2. NATS

NATS is a connective technology that powers modern distributed systems. A connective technology is responsible for addressing, discovery and exchanging of messages that drive the common patterns in distributed systems; asking and answering questions, aka services/microservices, and making and processing statements, or stream processing.

Software applications and services need to exchange data. NATS is an infrastructure that allows such data exchange, segmented in the form of messages. We call this a "message-oriented middleware.

This is docker-compose file for installing NATS where you should define login nats and password nats for the connection.

1KB

nats.zip

archive

Open

There is also README.md where you can see all commands to deploy NATS.

Example of docker-copmpose:

version: '3'
services:
  nats:
    container_name: nats
    image: nats
    entrypoint: /nats-server
    command: --js --user nats --pass nats
    ports:
      - 4222:4222
    volumes:
      - nats_data:/data

volumes:
  nats_data:

You can see the official doumentation for NATS here:

nats - Official Image | Docker Hubhub.docker.com

For the installation of NATS Server use the following:

Installing a NATS Server | NATS Docsdocs.nats.io

2. Deploy NUTS in Kubernetes Cluster Use Helm chart.

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm install my-nats nats/nats

Get more information in Official Documentation:

NATS and Kubernetes | NATS Docsdocs.nats.io

k8s/helm/charts/nats at main · nats-io/k8sGitHub

Creation of Stream in NATS

You can see the officail documentation about working with JetStream:

JetStream | NATS Docsdocs.nats.io

or you can use the following command for creation of JetStream in NATS:

nats --user=nats --password=nats --server=nats://localhost:4222 stream add \
  --subjects="RW-MANAGER.MT5-DEAL.>" \
  --storage=file \
  --replicas=1 \
  --retention=limits \
  --discard=old \
  --max-age="1h" \
  --max-bytes=-1 \
  --max-consumers=-1 \
  --max-msg-size=5120 \
  --max-msgs=1000000 \
  --max-msgs-per-subject=-1 \
  --dupe-window="2m0s" \
  --no-allow-rollup \
  --no-deny-delete \
  --no-deny-purge \
  DEALS

If you need you can replace user and password on what you use.

If you need you can change localhost to ip where nats-server is situated.

While the command executing you will be asked questions about the JetStream settings. You can use the following:

? Stream Name DEALS ? Subjects RW-MANAGER.MT5-DEAL.> ? Storage file ? Replication 1 ? Retention Policy Limits ? Discard Policy Old ? Stream Messages Limit 1000000 ? Per Subject Messages Limit -1 ? Total Stream Size -1 ? Message TTL 1h ? Max Message Size -1 ? Duplicate tracking time window 2m0s ? Allow message Roll-ups No ? Allow message deletion Yes ? Allow purging subjects or the entire stream Yes

Let's mention the main parameters:

• Stream Name - the name of JetStream;

• Subjects - subjects that will be in JetStream. You can use special symbol > for specifing any other symbols.

As confirmation of the creation of the JetStream, the following information will be displayed:

Stream DEALS was created

Information for Stream DEALS created 2023-06-20 17:59:01

             Subjects: RW-MANAGER.MT5-DEAL.>
             Replicas: 1
              Storage: File

Options:

            Retention: Limits
     Acknowledgements: true
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: 1,000,000
  Maximum Per Subject: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 1h0m0s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 

To make sure that JetStream was created you can execute the following command:

nats --user=nats --password=nats stream ls

The result of the command:

╭────────────────────────────────────────────────────────────────────────────╮
│                                  Streams                                   │
├───────┬─────────────┬─────────────────────┬──────────┬──────┬──────────────┤
│ Name  │ Description │ Created             │ Messages │ Size │ Last Message │
├───────┼─────────────┼─────────────────────┼──────────┼──────┼──────────────┤
│ DEALS │             │ 2023-06-21 12:00:37 │ 0        │ 0 B  │ never        │
╰───────┴─────────────┴─────────────────────┴──────────┴──────┴──────────────╯

After RWManager sends deals to JetStream the information will change:

╭──────────────────────────────────────────────────────────────────────────────╮
│                                   Streams                                    │
├───────┬─────────────┬─────────────────────┬──────────┬────────┬──────────────┤
│ Name  │ Description │ Created             │ Messages │ Size   │ Last Message │
├───────┼─────────────┼─────────────────────┼──────────┼────────┼──────────────┤
│ DEALS │             │ 2023-06-21 12:00:37 │ 57       │ 41 KiB │ 0.44s        │
╰───────┴─────────────┴─────────────────────┴──────────┴────────┴──────────────╯

To get information from JetStream you can execute the following command:

nats --user=nats --password=nats stream view

The result will display the JetStream list in which we want to view the message, select the DEALS we created:

? Select a Stream DEALS

[1] Subject: RW-MANAGER.MT5-DEAL.> Received: 2023-06-21T12:43:56+06:00

{"server_uri":"test","server_code":"test_1","group_name":"test_group","deal_id":1687329836529978001,"order_id":1687329836529979002,"position_id":1687329836529979003,"login":"test_user_1","time":1687329836529,"action":"buy","symbol":"test_1_symbol","price":0,"tp":0,"sl":0,"volume":0,"profit":0,"profit_raw":0,"swap":0,"commission":0,"fee":0,"profit_rate":0,"margin_rate":0,"comment":"test_comment","balance":0,"equity":0,"floating":0,"free_margin":0,"margin":0,"credit":0,"leverage":0,"gateway":"","reason":0,"dealer":0,"agent":0,"account":"","execution":0,"contract_size":0,"calc_mode":0,"time_from_mt":1687329836,"time_sent_nats":1687329836,"time_received_deals":0,"time_sent_db":0}