Skip to main content
Supermicro NVMesh Storage Solutions

Low-latency distributed block storage for web-scale data centers

Solution Overview

New application workloads are driving the need for new storage architectures: Cloud, Edge, AI and IoT bring new latency requirements – therefore companies need to rethink their storage strategies.

Supermicro and Excelero have joined forces to deliver a high-performance, low-latency distributed block storage solution leveraging NVMe flash. The Supermicro NVMesh solution enables unmodified applications to utilize pooled NVMe storage devices across a network at local speeds and latencies. It was designed to combine virtually unlimited Supermicro servers with NVMe to scale capacity and performance.

Solution Key Benefits

  • Maximize the utilization of your NVMe SSDs
  • Predictable application performance
  • Supports standard hardware & protocols
  • Use as block storage or with any file system
  • Easy to manage & monitor

Use Cases

Databases

Benefit from the low latency, high throughput and IO/s of a local NVMe device while enjoying the advantages of centralized, redundant storage

AI/ML/DL

Scalable disaggregated shared storage enables working on huge data sets, and reduces training time from weeks to days

Data analytics

Scalable data-protected storage satisfies the high bandwidth requirements of modern data analytics applications at any scale, with improved cost-efficiency

High-performance computing

Drastically shortens processing time and gives supercomputing centers the ability to maximize their availability

Product Brief

The Supermicro NVMesh Reference Architecture implements the next-generation of intelligent infrastructure platforms designed to provide applications with all of the raw performance you can expect from local storage — and all of the flexibility, manageability and scalability typically experienced with traditional storage area network-based solutions. Leveraging the latest NVMe and PCIe standards, this reference architecture uses a low-latency, high-bandwidth networking to connect compute and storage together in a flexible way that will fit almost any application’s architecture requirements. Deploy this RA in storage-centric, compute-centric or mixed storage and compute configurations, and integrate it with existing data center application servers as desired.

Scale Out diagram

Supermicro NVMesh Reference Architecture – Gen 4 Systems

System Generation X12 (Intel Gen4) H12 (AMD Gen4)
Image 1x SYS-220BT-HNTR 4x AS -1114S-WN10RT
Supermicro SKU 1x SYS-220BT-HNTR 4x AS -1114S-WN10RT
Minimum Form Factor Per Cluster One 2U 4-node system Four 1U Systems
Storage Media Up to 24x 15.36TB NVMe SSD devices Up to 40x 15.36TB NVMe SSD devices
CPU 8x ICX 4314 2P 16C/32T 2.4G 4x Rome 7452 32C/64T 2.35G
Memory 16x 16GB DDR4 3200 per node 16x 16GB DDR4 3200 per node
Network Up to 2x Mellanox ConnectX 6 VPI NIC per node Up to 2x Mellanox ConnectX 6 VPI NIC per node
RAW Capacity Up to 369 TB Up to 614 TB
IOPS Performance* Up to 34M random 4k Up to 39M random 4k
Throughput Performance* Up to 162GB/s Up to 184GB/s
I/O Latency* < 95μs < 95μs
Performance Density 81GB/s or 17M IOPs per RU 46GB/s or 9.7M IOPs per RU

* Performance number is estimated. Reference: https://www.excelero.com/storage-configurator/

Reference Architecture – Intel Gen 3 Systems

Platform SuperStorage EDSFF Server BigTwin Server Ultra Server
Platform Form Factor 1U 1 node 2U 4 nodes 1U 1 node or
2U 1node
Storage Media 32x 4TB or 16TB NVMe SSD devices Up to 24x 15.3TB NVMe SSD devices Up to 10 or 24x 15.3TB NVMe SSD devices
CPU (two per node) Intel® Xeon® Scalable Processors for X11 SuperStorage 2nd Generation Intel® Xeon® Scalable Processors for X11 BigTwin Intel® Xeon® processor E5-2600 v4 family for X10 Ultra
or
Intel® Xeon® Scalable Processors for X11 Ultra
Memory suggested 192GB DDR4 2933 per node suggested 192GB DDR4 2933 per node suggested 192GB DDR4 2933 per node
Network up to 2 Mellanox ConnectX 5 VPI NIC up to 8 Mellanox ConnectX 5 VPI NIC up to 2 Mellanox Connect 5 VPI NIC
RAW Capacity up to 512 TB per node up to 92TB per node (367TB per BigTwin) up to 367TB per node
Available Data Protection Fully Distributed RAID, enabling linear scale of performance and capacity
Infrastructure Support TCP/IP; RoCEv2; Infiniband
IOPs Performance up to 10M random 4k up to 20M random 4k up to 5M random 4k
Throughput Performance up to 24GB/s up to 76GB/s up to 24GB/s
IO Latency Alike the local NVMe SSD Latency
Performance Density 24GB/s or 10M IOPs per RU 38GB/s or 10M IOPs per RU 12GB/s or 2.5M IOPs per RU
Management CLI, GUI, REST API CLI, GUI, REST API CLI, GUI, REST API
Operating System Support Linux
File System Support POSIX like as ext4 and xfs, plus GPFS, BeeGFS & Lustre
Excelero Software NVMesh Client
NVMesh Target
Available Excelero Service NVMesh 3y premium support
NVMesh Install Service, On-site
Available Supermicro Service 3-year onsite 24x7x4 service / 3-year onsite next business day service
Onsite Integration Services Onsite Survey Onsite logistics Racking & Cabling (with 3rd party switches) Racking & Cabling Engineering Drawing Releasing (Test included)

GPU Storage Reference Architecture

Configuration with HA (high availability)

A 2U-4-Node server with 24 NVMe drives that enables a HA solution with MeshProtect-0, 1, 10 & 6 @ 8+2

2U-4-Node server with 24 NVMe drives
  Supermicro P/N Description Qty
Server SYS-2029BZ-HNR BigTwin 3UPI 2U 4-Node, 6x 2.5" NVMe, X11DPT-BH, 217BHQ+ 1
CPU P4X-CLX4216-SRFBB CLX 4216 2P 16C/32T 2.1G 22M 9.6GT 100W 3647 L1 8
Memory MEM-DR416L-CL02-ER29 16GB DDR4-2933 1RX4 LP ECC RDIMM 48
M.2 Carrier AOC-SMG3-2H8M2-B-O 2x Hybrid NVMe/SATA M.2 Carrier for Big Twin,HF,RoHS 4
Boot Drive HDS-IMT0-SSDSCKKB960G8 Intel D3 S4510 960GB M.2 SATA 6Gb/s 3D TLC 22x80mm 1DWPD 8
NVMe** HDS-MUN-MTFDHAL15T3TDP1A Micron 9300 PRO 15.3TB NVMe PCIe 3.0 3D TLC U.2 15mm 24
NIC Card* AOC-MCX556A-ECAT MCX556A-ECAT, CX-5 VPI, EDR IB, 100GbE, 2p, QSFP28, PCIe3 x16 8
SIOM NIC AOC-MGP-I4M-O SIOM 4-port GbE RJ45, Intel i350 with 1U bracket 4
Supermicro System Management Software SFT-DCMS-SINGLE Supermicro System Managment Software Suite node license, HF, RoHS/REACH, PBF 4
NVMesh SFT-EX-SWCCI24-3YR NVMesh per-chassis subscription license, up to 24 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. 1
NVMesh SVC-EX-PSINSTALL On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes 1

* Infiniband or RoCE

** The architecture supports NVMe drives from all major manufacturers, including Micron, Intel, Kioxia, Samsung and Western Digital. Different drive sizes are supported. For Erasure Coded configurations, NVMe drive families, including Western Digital SN200, Samsung 1725b & 1733, Kioxia CM5 & CM6, with 3 DWPD or more endurance are supported.

Configuration without high availability (non-HA)

A 1U 10-drive-bay server with 10 NVMe drives that allows a non-HA solution with MeshProtect-0 & 6 @ 8+2

1U 10-drive-bay server with 10 NVMe drives
  Supermicro P/N Description Qty
Server SYS-1029U-TN10RT X11DPU, 119UTS-R1K02P-N10T, AOC-URN6-I2XT 1
CPU P4X-CLX4216-SRFBB CLX 4216 2P 16C/32T 2.1G 22M 9.6GT 100W 3647 L1 2
Memory MEM-DR416L-CL01-ER29 16GB DDR4-2933 2RX8 LP ECC RDIMM 6
Boot Drive HDS-IMT0-SSDSCKKB960G8 Intel D3 S4510 960GB M.2 SATA 6Gb/s 3D TLC 22x80mm 1DWPD 1
NVMe** HDS-MUN-MTFDHAL15T3TDP1A Micron 9300 PRO 15.3TB NVMe PCIe 3.0 3D TLC U.2 15mm 10
NIC Card* AOC-MCX556A-ECAT MCX556A-ECAT, CX-5 VPI, EDR IB, 100GbE, 2p, QSFP28, PCIe3 x16 2
Supermicro System Management Software SFT-DCMS-SINGLE Supermicro System Managment Software Suite node license, HF, RoHS/REACH, PBF 1
NVMesh SFT-EX-SWCCI12-3YR NVMesh per-chassis subscription license, up to 12 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. 1
NVMesh SVC-EX-PSINSTALL On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes 1

EDSFF-Short E1.S Server

EDSFF-Short E1.S Server
  Supermicro P/N Description Qty
Server SSG-1029P-NES32R CSE-121EF + X11DSF-E-P 1
CPU P4X-CLX4210-SRFBL CLX 4210 2P 10C/20T 2.2G 13.75M 9.6GT 85W 3647 R1 2
Memory MEM-DR416L-SL04-ER26 16GB DDR4-2666 1Rx4 LP ECC REG DIMM,HF,RoHS 12
Boot Drive HDS-MMT-MTFDDAV960TDS1AW Micron 5300 PRO 960GB, SATA, M.2, 22x80mm,3D TLC,1.5DWPD 2
AOC AOC-SLG2-2TM2-T-O OEM LP PCIe2 2x 7pin SATA3 RAID 1 1
NIC Card* AOC-MCX556A-ECAT Mellanox ConnectX-5 VPI adapter card, EDR IB(100Gb/s) & 100GbE, dual port QSFP28 2
NVMe HDS-IEN0-SSDPEYKX040T8 Intel DC P4511 EDSFF E1.S TLC 5.9mm 4TB 32
Supermicro System Management Software SFT-DCMS-SINGLE Supermicro System Managment Software Suite node license,HF,RoHS/REACH,PBF 1
NVMesh SFT-EX-SWCCI32-3YR NVMesh per-chassis subscription license, up to 32 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. 1
NVMesh SVC-EX-PSINSTALL On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes 1

EDSFF-Long E1.L Server

EDSFF-Short E1.S Server
  Supermicro P/N Description Qty
Server SSG-1029P-NEL32R 1U DP 32 EDSFF Long 1
CPU P4X-CLX4210-SRFBL CLX 4210 2P 10C/20T 2.2G 13.75M 9.6GT 85W 3647 R1 2
Memory MEM-DR416L-SL04-ER26 16GB DDR4-2666 1Rx4 LP ECC REG DIMM,HF,RoHS 12
Boot Drive HDS-MMT-MTFDDAV960TDS1AW Micron 5300 PRO 960GB, SATA, M.2, 22x80mm,3D TLC,1.5DWPD 2
NIC Card* AOC-MCX556A-ECAT Mellanox ConnectX-5 VPI adapter card, EDR IB(100Gb/s) & 100GbE, dual port QSFP28 2
NVMe HDS-IEN0-SSDPEWNV153T8 Intel D5-P4326 E1.L PCIe3.1x4QLC9.5mm15.36T 3D2 <0.5DWPD 32
Supermicro System Management Software SFT-DCMS-SINGLE Supermicro System Managment Software Suite node license,HF,RoHS/REACH,PBF 1
NVMesh SFT-EX-SWCCI32-3YR NVMesh per-chassis subscription license, up to 32 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. 1
NVMesh SVC-EX-PSINSTALL On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes 1

* Infiniband or RoCE

** The architecture supports NVMe drives from all major manufacturers, including Micron, Intel, Kioxia, Samsung and Western Digital. Different drive sizes are supported. For Erasure Coded configurations, NVMe drive families, including Western Digital SN200, Samsung 1725b & 1733, Kioxia CM5 & CM6, with 3 DWPD or more endurance are supported.

Configure your NVMesh Setup
Certified Servers for NVMesh Reference Architecture
AS -1114S-WN10RT

AS -1114S-WN10RT

1U 1N WIO

AMD Gen4 System

AS -1114S-WN10RT

  • Single AMD EPYC™ 7003/7002 Series Processor
  • 16 DIMMs; up to 4TB 3DS ECC DDR4-3200MHz RDIMM/LRDIMM
  • 2 PCI-E 4.0 x16 (FHHL) slots, 1 PCI-E 4.0 x16 (LP) slot
  • 10 Hot-swap NVMe/SATA3 hybrid drive bays
  • 750W Redundant Platinum Level Power Supplies
SYS-220BT-HNTR

SYS-220BT-HNTR

2U 4N BigTwin®

Intel Gen4 System

SYS-220BT-HNTR

  • 4 hot-pluggable nodes in 2U; each node supports:
  • Dual 3rd Gen Intel® Xeon® Scalable processors
  • 20 DIMMs; Up to 6TB DDR4-3200: Intel® Optane™ Persistent Memory 200 Series; Up to 4TB ECC DDR4-3200: RDIMM
  • 2 PCI-E 4.0 x16 (LP) slots
  • 6 Hot-swap 2.5" NVMe/SATA drive bays
  • 2600W Redundant Titanium Level Power Supplies
SYS-2029BZ-HNR

SYS-2029BZ-HNR

2U 4N BigTwin®

Intel Gen3 System

SYS-2029BZ-HNR

  • 4 hot-pluggable nodes in 2U; each node supports:
  • Dual 2nd Gen Intel® Xeon® Scalable processors
  • 24 DIMMs; up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; Supports Intel® Optane™ DCPMM
  • 2 PCI-E 3.0 x16 (LP) slots
  • 6 Hot-swap 2.5" NVMe drive bays
  • 2200W Redundant Titanium Level Power Supplies
AS -2123BT-HNR

AS -2123BT-HNR

2U 4N BigTwin®

AMD Gen3 System

AS -2123BT-HNR

  • 4 hot-pluggable nodes in 2U; each node supports:
  • Dual AMD EPYC™ 7002/7001 Series Processors
  • 16 DIMMs; up to 4TB 3DS ECC DDR4-3200MHz SDRAM with 7002 series processors
  • 2 PCI-E 3.0 x16 (LP) slots
  • 6 Hot-swap 2.5" NVMe drive bays
  • 2600W Redundant Titanium Level Power Supplies
SYS-1029U-TN10RT

SYS-1029U-TN10RT

1U 1N Ultra

Intel Gen3 System

SYS-1029U-TN10RT

  • Dual 2nd Gen Intel® Xeon® Scalable processors
  • 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; supports Intel® Optane™ DCPMM
  • 2 PCI-E 3.0 x16 slots (FH, 10.5" L)
  • 10 Hot-swap 2.5" NVMe drive bays
  • 1000W Redundant Titanium Level Power Supplies
SYS-2029U-TN24R4T

SYS-2029U-TN24R4T

2U 1N Ultra

Intel Gen3 System

SYS-2029U-TN24R4T

  • Dual 2nd Gen Intel® Xeon® Scalable processors
  • 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM, supports Intel® Optane™ DCPMM
  • 2 PCI-E 3.0 x16 slots (FH, 10.5" L)
  • 24 front Hot-swap 2.5" NVMe drive bays, 2 rear Hot-swap 2.5" drive bays
  • 1600W Redundant Titanium Level Power Supplies
SSG-1029P-NES32R

SSG-1029P-NES32R

1U 1N Petascale E1.S Storage Server

Intel Gen3 System

SSG-1029P-NES32R

  • Dual 2nd Gen Intel® Xeon® Scalable processors
  • 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; supports Intel® Optane™ DCPMM
  • 2 PCI-E 3.0 x16 (FHHL) slots, 1 PCI-E 3.0 x4 (LP) slot
  • 32 Hot-swap EDSFF Short (E1.S) drive slots
  • 1600W Redundant Titanium Level Power Supplies
SSG-1029P-NEL32R

SSG-1029P-NEL32R

1U 1N Petascale E1.L Storage Server

Intel Gen3 System

SSG-1029P-NEL32R

  • Dual 2nd Gen Intel® Xeon® Scalable processors
  • 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; supports Intel® Optane™ DCPMM
  • 2 PCI-E 3.0 x16 (LP) slots
  • 32 Hot-swap EDSFF Long (E1.L) drive slots
  • 1600W Redundant Titanium Level Power Supplies
Certified Network Adapters
AOC-653106A-HDAT

AOC-653106A-HDAT

Standard Low-profile Mellanox HDR IB (200Gb/s) and 200GbE card with 2 QSFP56 ports
AOC-653105A-HDAT

AOC-653105A-HDAT

Standard Low-profile Mellanox HDR IB (200Gb/s) and 200GbE card with 1 QSFP56 port
AOC-MCX556A-ECAT

AOC-MCX556A-ECAT

Standard Low-profile Mellanox EDR IB (100Gb/s) and 100GbE card with 2 QSFP28 ports
AOC-MCX555A-ECAT

AOC-MCX555A-ECAT

Standard Low-profile Mellanox EDR IB (100Gb/s) and 100GbE card with 1 QSFP28 port