lyqtRpc

中文

A lightweight C++20 RPC framework built on muduo + Protobuf. Dual TCP / shared-memory zero-copy transport, etcd registry, circuit breaker, token bucket, distributed tracing, Prometheus metrics, and an HTTP-to-RPC API gateway — all in one repo.

Author: lczllx · Language: C++20 · Network: muduo · Transport: TCP & SHM zero-copy · Serialization: Protobuf, JSON, FlatBuffers · Build: CMake

Highlights

• Dual transport: TCP (LV variable-length framing) and SHM zero-copy (per-client ring buffer + Protobuf SerializeToArray direct write)
• Service governance: etcd registry, heartbeat keep-alive, CAS leader election, load reporting, three-state circuit breaker, token bucket rate limiting
• Call models: synchronous, std::future async, and callback-based
• Pluggable serialization: ISerializer abstraction, default Protobuf, JSON for debugging, FlatBuffers for zero-copy reads
• Distributed tracing: trace_id + span_id end-to-end propagation
• Multi-client concurrency: SHM path with muduo EventLoopThread worker pool, per-client independent ring buffers
• Observability: built-in Prometheus /metrics endpoint (text format 0.0.4), covering request count / latency histogram / concurrency / error count / connection count / circuit breaker state / token bucket / process-level metrics — mirrors brpc /vars
• API Gateway: HTTP→RPC gateway on the same muduo foundation, with route matching, rate limiting, circuit breaker, Prometheus metrics, and distributed tracing — deployed as a separate process, zero changes to the RPC framework

Performance: lyqtRpc vs brpc

Test environment: 4C8G cloud VM, Ubuntu 22.04, g++ 11.4.0, all Protobuf, echo payload. brpc 1.17.0.

Single-thread latency & throughput

Payload	brpc TCP	lyqtRpc TCP Proto	lyqtRpc SHM Proto ZC
16B QPS	17,859	13,556	26,725
16B P90	61μs	83μs	30μs
16B P99	74μs	100μs	38μs
64KB QPS	6,268	2,751	12,735
64KB P90	171μs	397μs	71μs
64KB P99	237μs	483μs	137μs

4-thread concurrency

Metric	brpc TCP	lyqtRpc TCP Proto	lyqtRpc SHM Proto ZC
QPS	44,534	38,834	153,916
P90	129μs	125μs	23μs
P99	233μs	189μs	39μs

SHM latency is well below brpc: single-thread P99 is ≈51% of brpc, dropping to ≈17% at 4 threads. TCP QPS is ≈76% of brpc for 16B payloads but falls to ≈44% for 64KB, stemming from bthread coroutines, IOBuf zero-copy chains, and baidu_std multiplexing in brpc.

Quick Start

Requirements: g++ 11 or newer (C++20), CMake ≥ 3.16.

git clone https://github.com/lczllx/lyqtRpc.git
cd lyqtRpc
git submodule update --init --recursive
bash autobuild/quick_build.sh

Docker

bash autobuild/docker.sh doctor
bash autobuild/docker.sh setup

Examples

# TCP RPC (requires etcd)
docker compose up -d etcd
bash demosh/demo.sh etcd

# SHM Proto zero-copy (no extra dependencies)
cd rpc/build
./bin/shm_proto_server &
./bin/shm_proto_client

# Full-path benchmark
cd example/shm && bash run_shm_benchmark.sh all

Prometheus metrics

cd rpc/build
./bin/benchmark_server 8889 0 8080 0 &
./bin/benchmark_client single echo 2000 1 1
curl localhost:9090/metrics

Add the address to prometheus.yml scrape_configs. Percentiles are computed query-side: histogram_quantile(0.99, rate(rpc_request_duration_us_bucket[1m])).

API Gateway

cd rpc/build
./bin/benchmark_server 8889 &      # 1. start RPC backend
./bin/gateway_server &             # 2. start gateway (HTTP :8080, metrics :9091)

curl -d '{"data":"hello"}' localhost:8080/api/echo   # → RPC echo backend
curl localhost:8080/api/health                        # → health check
curl localhost:8080/diagnose                          # → rate limiter + breaker status
curl localhost:9091/metrics | grep gateway            # → gateway metrics

Architecture

Provider ──REGISTER/HEARTBEAT──> Registry(etcd) <──DISCOVER── Consumer
   │                                                          │
   └── RPC call ────────────────────────────────────────────>─┘

• LV framing: | 4B total_len | 4B msg_type | 4B id_len | id | body |
• SHM zero-copy: SerializeToArray → ring buffer → eventfd → ParseFromArray, bypassing TCP
• Registry backend: LCZ_ETCD env var switches between Memory / Etcd; defaults to in-memory
• Circuit breaker: three-state (CLOSED→OPEN→HALF_OPEN), method×host granularity, memory/etcd persistence
• Rate limiter: TokenBucket + BACKOFF automatic backoff and retry
• Thread pool: muduo EventLoopThread, per-client ring buffer, lock-free SPSC

See docs/en/architecture-en.md for detailed flowcharts.

Directory

lyqtRpc/
├── rpc/
│   ├── src/
│   │   ├── client/           # RpcClient, ClientDiscover, CircuitBreaker, ShmClient
│   │   ├── server/           # RpcServer, Registry, LeaderElection, ShmServer
│   │   └── general/          # ShmChannel, LVProtocol, MessageFactory, Serializer, Logger
│   ├── tests/                # 76 GTest cases
│   ├── example/              # Examples + benchmarks
│   ├── proto/                # Protobuf definitions
│   └── muduo/                # Git submodule
├── gateway/
│   ├── src/                  # HttpServer, HttpRouter, GatewayHandler, DiagnoseHandler
│   └── example/              # gateway_server entry point
├── autobuild/                # Build + Docker scripts
├── demosh/                   # Demo scripts
├── docs/                     # Design docs (cn + en)
├── grafana/dashboards/       # Grafana dashboard JSON
├── Dockerfile
└── docker-compose.yml

Build

Two ways to build — system deps (fast) or vcpkg (zero-setup, cross-platform).

1. System deps (apt) — recommended

Dependencies (one apt install):

sudo apt-get install -y build-essential cmake g++ make \
  libboost-dev libjsoncpp-dev libcurl4-openssl-dev \
  protobuf-compiler libprotobuf-dev \
  flatbuffers-compiler libflatbuffers-dev    # optional, for SHM FlatBuf path

Compile (CMake presets):

cd lyqtRpc/rpc
cmake --preset system -B build -DLCZ_RPC_BUILD_EXAMPLES=ON
cmake --build build -j$(nproc)

Unit tests: add -DLCZ_RPC_BUILD_TESTS=ON (needs libgtest-dev), then run ./build/bin/lcz_rpc_unit_tests.

2. vcpkg (zero apt deps, cross-platform)

cd lyqtRpc/rpc
cmake --preset vcpkg -B build-vcpkg -DLCZ_RPC_BUILD_EXAMPLES=ON
cmake --build build-vcpkg -j$(nproc)

First build downloads and compiles all deps (protobuf / curl / jsoncpp / flatbuffers / gtest / benchmark / fmt) from source — ~30min. Subsequent builds are cached.

Roadmap

• GitHub Actions CI (build + unit tests + SHM smoke test)
• Docker image + docker-compose
• etcd registry backend (Lease + LCZ_ETCD env-var switch)
• Three-state circuit breaker (method×host, configurable)
• Registry multi-instance HA (etcd lease + CAS election)
• Token bucket rate limiting + BACKOFF
• Distributed tracing (trace_id/span_id)
• Pluggable serializer (ISerializer + ProtobufSerializer)
• Prometheus /metrics (QPS / latency histogram / error codes / process metrics)

Known Limitations

• TCP path single-connection send() holds a lock; connection pool + protocol multiplexing planned
• etcd heartbeat re-registers on every keepalive failure (lease TTL too short), write amplification under load
• SHM payloads >64KB copy twice through the ring buffer; throughput worse than TCP zero-copy equivalents
• No auth / encryption; no streaming RPC; Topic has no persistence
• Unit tests cover core modules only; registry / circuit breaker / network layer untested