Low-latency distributed block storage for web-scale data centers
Solution Overview
New application workloads are driving the need for new storage architectures: Cloud, Edge, AI and IoT bring new latency requirements – therefore companies need to rethink their storage strategies.
Supermicro and Excelero have joined forces to deliver a high-performance, low-latency distributed block storage solution leveraging NVMe flash. The Supermicro NVMesh solution enables unmodified applications to utilize pooled NVMe storage devices across a network at local speeds and latencies. It was designed to combine virtually unlimited Supermicro servers with NVMe to scale capacity and performance.
Solution Key Benefits
- Maximize the utilization of your NVMe SSDs
- Predictable application performance
- Supports standard hardware & protocols
- Use as block storage or with any file system
- Easy to manage & monitor
Use Cases
Databases
Benefit from the low latency, high throughput and IO/s of a local NVMe device while enjoying the advantages of centralized, redundant storage
AI/ML/DL
Scalable disaggregated shared storage enables working on huge data sets, and reduces training time from weeks to days
Data analytics
Scalable data-protected storage satisfies the high bandwidth requirements of modern data analytics applications at any scale, with improved cost-efficiency
High-performance computing
Drastically shortens processing time and gives supercomputing centers the ability to maximize their availability
Product Brief
The Supermicro NVMesh Reference Architecture implements the next-generation of intelligent infrastructure platforms designed to provide applications with all of the raw performance you can expect from local storage — and all of the flexibility, manageability and scalability typically experienced with traditional storage area network-based solutions. Leveraging the latest NVMe and PCIe standards, this reference architecture uses a low-latency, high-bandwidth networking to connect compute and storage together in a flexible way that will fit almost any application’s architecture requirements. Deploy this RA in storage-centric, compute-centric or mixed storage and compute configurations, and integrate it with existing data center application servers as desired.
GPU Storage Reference Architecture
Configuration with HA (high availability)
A 2U-4-Node server with 24 NVMe drives that enables a HA solution with MeshProtect-0, 1, 10 & 6 @ 8+2
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SYS-2029BZ-HNR | BigTwin 3UPI 2U 4-Node, 6x 2.5" NVMe, X11DPT-BH, 217BHQ+ | 1 |
| CPU | P4X-CLX4216-SRFBB | CLX 4216 2P 16C/32T 2.1G 22M 9.6GT 100W 3647 L1 | 8 |
| Memory | MEM-DR416L-CL02-ER29 | 16GB DDR4-2933 1RX4 LP ECC RDIMM | 48 |
| M.2 Carrier | AOC-SMG3-2H8M2-B-O | 2x Hybrid NVMe/SATA M.2 Carrier for Big Twin,HF,RoHS | 4 |
| Boot Drive | HDS-IMT0-SSDSCKKB960G8 | Intel D3 S4510 960GB M.2 SATA 6Gb/s 3D TLC 22x80mm 1DWPD | 8 |
| NVMe** | HDS-MUN-MTFDHAL15T3TDP1A | Micron 9300 PRO 15.3TB NVMe PCIe 3.0 3D TLC U.2 15mm | 24 |
| NIC Card* | AOC-MCX556A-ECAT | MCX556A-ECAT, CX-5 VPI, EDR IB, 100GbE, 2p, QSFP28, PCIe3 x16 | 8 |
| SIOM NIC | AOC-MGP-I4M-O | SIOM 4-port GbE RJ45, Intel i350 with 1U bracket | 4 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license, HF, RoHS/REACH, PBF | 4 |
| NVMesh | SFT-EX-SWCCI24-3YR | NVMesh per-chassis subscription license, up to 24 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSGENREM | Remote, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
* Infiniband or RoCE
** The architecture supports NVMe drives from all major manufacturers, including Micron, Intel, Kioxia, Samsung and Western Digital. Different drive sizes are supported. For Erasure Coded configurations, NVMe drive families, including Western Digital SN200, Samsung 1725b & 1733, Kioxia CM5 & CM6, with 3 DWPD or more endurance are supported.
Configuration without high availability (non-HA)
A 1U 10-drive-bay server with 10 NVMe drives that allows a non-HA solution with MeshProtect-0 & 6 @ 8+2
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SYS-1029U-TN10RT | X11DPU, 119UTS-R1K02P-N10T, AOC-URN6-I2XT | 1 |
| CPU | P4X-CLX4216-SRFBB | CLX 4216 2P 16C/32T 2.1G 22M 9.6GT 100W 3647 L1 | 2 |
| Memory | MEM-DR416L-CL01-ER29 | 16GB DDR4-2933 2RX8 LP ECC RDIMM | 12 |
| Boot Drive | HDS-IMT0-SSDSCKKB960G8 | Intel D3 S4510 960GB M.2 SATA 6Gb/s 3D TLC 22x80mm 1DWPD | 1 |
| NVMe** | HDS-MUN-MTFDHAL15T3TDP1A | Micron 9300 PRO 15.3TB NVMe PCIe 3.0 3D TLC U.2 15mm | 10 |
| NIC Card* | AOC-MCX556A-ECAT | MCX556A-ECAT, CX-5 VPI, EDR IB, 100GbE, 2p, QSFP28, PCIe3 x16 | 2 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license, HF, RoHS/REACH, PBF | 1 |
| NVMesh | SFT-EX-SWCCI12-3YR | NVMesh per-chassis subscription license, up to 12 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSGENREM | Remote, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
EDSFF-Short E1.S Server
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SSG-1029P-NES32R | CSE-121EF + X11DSF-E-P | 1 |
| CPU | P4X-CLX4210-SRFBL | CLX 4210 2P 10C/20T 2.2G 13.75M 9.6GT 85W 3647 R1 | 2 |
| Memory | MEM-DR416L-SL04-ER26 | 16GB DDR4-2666 1Rx4 LP ECC REG DIMM,HF,RoHS | 12 |
| Boot Drive | HDS-MMT-MTFDDAV960TDS1AW | Micron 5300 PRO 960GB, SATA, M.2, 22x80mm,3D TLC,1.5DWPD | 2 |
| AOC | AOC-SLG2-2TM2-T-O | OEM LP PCIe2 2x 7pin SATA3 RAID 1 | 1 |
| NIC Card* | AOC-MCX556A-ECAT | Mellanox ConnectX-5 VPI adapter card, EDR IB(100Gb/s) & 100GbE, dual port QSFP28 | 2 |
| NVMe | HDS-IEN0-SSDPEYKX040T8 | Intel DC P4511 EDSFF E1.S TLC 5.9mm 4TB | 32 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license,HF,RoHS/REACH,PBF | 1 |
| NVMesh | SFT-EX-SWCCI32-3YR | NVMesh per-chassis subscription license, up to 32 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSGENREM | Remote, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
EDSFF-Long E1.L Server
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SSG-1029P-NEL32R | 1U DP 32 EDSFF Long | 1 |
| CPU | P4X-CLX4210-SRFBL | CLX 4210 2P 10C/20T 2.2G 13.75M 9.6GT 85W 3647 R1 | 2 |
| Memory | MEM-DR416L-SL04-ER26 | 16GB DDR4-2666 1Rx4 LP ECC REG DIMM,HF,RoHS | 12 |
| Boot Drive | HDS-MMT-MTFDDAV960TDS1AW | Micron 5300 PRO 960GB, SATA, M.2, 22x80mm,3D TLC,1.5DWPD | 2 |
| NIC Card* | AOC-MCX556A-ECAT | Mellanox ConnectX-5 VPI adapter card, EDR IB(100Gb/s) & 100GbE, dual port QSFP28 | 2 |
| NVMe | HDS-IEN0-SSDPEWNV153T8 | Intel D5-P4326 E1.L PCIe3.1x4QLC9.5mm15.36T 3D2 <0.5DWPD | 32 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license,HF,RoHS/REACH,PBF | 1 |
| NVMesh | SFT-EX-SWCCI32-3YR | NVMesh per-chassis subscription license, up to 32 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSGENREM | Remote, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
* Infiniband or RoCE
** The architecture supports NVMe drives from all major manufacturers, including Micron, Intel, Kioxia, Samsung and Western Digital. Different drive sizes are supported. For Erasure Coded configurations, NVMe drive families, including Western Digital SN200, Samsung 1725b & 1733, Kioxia CM5 & CM6, with 3 DWPD or more endurance are supported.
Reference Architecture – Intel Gen 3 Systems
| Platform | SuperStorage EDSFF Server | BigTwin Server | Ultra Server |
|---|---|---|---|
| Platform Form Factor | 1U 1 node | 2U 4 nodes | 1U 1 node or 2U 1node |
| Storage Media | 32x 4TB or 16TB NVMe SSD devices | Up to 24x 15.3TB NVMe SSD devices | Up to 10 or 24x 15.3TB NVMe SSD devices |
| CPU (two per node) | Intel® Xeon® Scalable Processors for X11 SuperStorage | 2nd Generation Intel® Xeon® Scalable Processors for X11 BigTwin | Intel® Xeon® processor E5-2600 v4 family for X10 Ultra or Intel® Xeon® Scalable Processors for X11 Ultra |
| Memory | suggested 192GB DDR4 2933 per node | suggested 192GB DDR4 2933 per node | suggested 192GB DDR4 2933 per node |
| Network | up to 2 Mellanox ConnectX 5 VPI NIC | up to 8 Mellanox ConnectX 5 VPI NIC | up to 2 Mellanox Connect 5 VPI NIC |
| RAW Capacity | up to 512 TB per node | up to 92TB per node (367TB per BigTwin) | up to 367TB per node |
| Available Data Protection | Fully Distributed RAID, enabling linear scale of performance and capacity | ||
| Infrastructure Support | TCP/IP; RoCEv2; Infiniband | ||
| IOPs Performance | up to 10M random 4k | up to 20M random 4k | up to 5M random 4k |
| Throughput Performance | up to 24GB/s | up to 76GB/s | up to 24GB/s |
| IO Latency | Alike the local NVMe SSD Latency | ||
| Performance Density | 24GB/s or 10M IOPs per RU | 38GB/s or 10M IOPs per RU | 12GB/s or 2.5M IOPs per RU |
| Management | CLI, GUI, REST API | CLI, GUI, REST API | CLI, GUI, REST API |
| Operating System Support | Linux | ||
| File System Support | POSIX like as ext4 and xfs, plus GPFS, BeeGFS & Lustre | ||
| Excelero Software | NVMesh Client | ||
| NVMesh Target | |||
| Available Excelero Service | NVMesh 3y premium support | ||
| NVMesh Install Service, On-site | |||
| Available Supermicro Service | 3-year onsite 24x7x4 service / 3-year onsite next business day service | ||
| Onsite Integration Services Onsite Survey Onsite logistics Racking & Cabling (with 3rd party switches) Racking & Cabling Engineering Drawing Releasing (Test included) | |||
GPU Storage Reference Architecture
Configuration with HA (high availability)
A 2U-4-Node server with 24 NVMe drives that enables a HA solution with MeshProtect-0, 1, 10 & 6 @ 8+2
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SYS-2029BZ-HNR | BigTwin 3UPI 2U 4-Node, 6x 2.5" NVMe, X11DPT-BH, 217BHQ+ | 1 |
| CPU | P4X-CLX4216-SRFBB | CLX 4216 2P 16C/32T 2.1G 22M 9.6GT 100W 3647 L1 | 8 |
| Memory | MEM-DR416L-CL02-ER29 | 16GB DDR4-2933 1RX4 LP ECC RDIMM | 48 |
| M.2 Carrier | AOC-SMG3-2H8M2-B-O | 2x Hybrid NVMe/SATA M.2 Carrier for Big Twin,HF,RoHS | 4 |
| Boot Drive | HDS-IMT0-SSDSCKKB960G8 | Intel D3 S4510 960GB M.2 SATA 6Gb/s 3D TLC 22x80mm 1DWPD | 8 |
| NVMe** | HDS-MUN-MTFDHAL15T3TDP1A | Micron 9300 PRO 15.3TB NVMe PCIe 3.0 3D TLC U.2 15mm | 24 |
| NIC Card* | AOC-MCX556A-ECAT | MCX556A-ECAT, CX-5 VPI, EDR IB, 100GbE, 2p, QSFP28, PCIe3 x16 | 8 |
| SIOM NIC | AOC-MGP-I4M-O | SIOM 4-port GbE RJ45, Intel i350 with 1U bracket | 4 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license, HF, RoHS/REACH, PBF | 4 |
| NVMesh | SFT-EX-SWCCI24-3YR | NVMesh per-chassis subscription license, up to 24 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSINSTALL | On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
* Infiniband or RoCE
** The architecture supports NVMe drives from all major manufacturers, including Micron, Intel, Kioxia, Samsung and Western Digital. Different drive sizes are supported. For Erasure Coded configurations, NVMe drive families, including Western Digital SN200, Samsung 1725b & 1733, Kioxia CM5 & CM6, with 3 DWPD or more endurance are supported.
Configuration without high availability (non-HA)
A 1U 10-drive-bay server with 10 NVMe drives that allows a non-HA solution with MeshProtect-0 & 6 @ 8+2
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SYS-1029U-TN10RT | X11DPU, 119UTS-R1K02P-N10T, AOC-URN6-I2XT | 1 |
| CPU | P4X-CLX4216-SRFBB | CLX 4216 2P 16C/32T 2.1G 22M 9.6GT 100W 3647 L1 | 2 |
| Memory | MEM-DR416L-CL01-ER29 | 16GB DDR4-2933 2RX8 LP ECC RDIMM | 6 |
| Boot Drive | HDS-IMT0-SSDSCKKB960G8 | Intel D3 S4510 960GB M.2 SATA 6Gb/s 3D TLC 22x80mm 1DWPD | 1 |
| NVMe** | HDS-MUN-MTFDHAL15T3TDP1A | Micron 9300 PRO 15.3TB NVMe PCIe 3.0 3D TLC U.2 15mm | 10 |
| NIC Card* | AOC-MCX556A-ECAT | MCX556A-ECAT, CX-5 VPI, EDR IB, 100GbE, 2p, QSFP28, PCIe3 x16 | 2 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license, HF, RoHS/REACH, PBF | 1 |
| NVMesh | SFT-EX-SWCCI12-3YR | NVMesh per-chassis subscription license, up to 12 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSINSTALL | On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
EDSFF-Short E1.S Server
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SSG-1029P-NES32R | CSE-121EF + X11DSF-E-P | 1 |
| CPU | P4X-CLX4210-SRFBL | CLX 4210 2P 10C/20T 2.2G 13.75M 9.6GT 85W 3647 R1 | 2 |
| Memory | MEM-DR416L-SL04-ER26 | 16GB DDR4-2666 1Rx4 LP ECC REG DIMM,HF,RoHS | 12 |
| Boot Drive | HDS-MMT-MTFDDAV960TDS1AW | Micron 5300 PRO 960GB, SATA, M.2, 22x80mm,3D TLC,1.5DWPD | 2 |
| AOC | AOC-SLG2-2TM2-T-O | OEM LP PCIe2 2x 7pin SATA3 RAID 1 | 1 |
| NIC Card* | AOC-MCX556A-ECAT | Mellanox ConnectX-5 VPI adapter card, EDR IB(100Gb/s) & 100GbE, dual port QSFP28 | 2 |
| NVMe | HDS-IEN0-SSDPEYKX040T8 | Intel DC P4511 EDSFF E1.S TLC 5.9mm 4TB | 32 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license,HF,RoHS/REACH,PBF | 1 |
| NVMesh | SFT-EX-SWCCI32-3YR | NVMesh per-chassis subscription license, up to 32 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSINSTALL | On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
EDSFF-Long E1.L Server
| Supermicro P/N | Description | Qty | |
|---|---|---|---|
| Server | SSG-1029P-NEL32R | 1U DP 32 EDSFF Long | 1 |
| CPU | P4X-CLX4210-SRFBL | CLX 4210 2P 10C/20T 2.2G 13.75M 9.6GT 85W 3647 R1 | 2 |
| Memory | MEM-DR416L-SL04-ER26 | 16GB DDR4-2666 1Rx4 LP ECC REG DIMM,HF,RoHS | 12 |
| Boot Drive | HDS-MMT-MTFDDAV960TDS1AW | Micron 5300 PRO 960GB, SATA, M.2, 22x80mm,3D TLC,1.5DWPD | 2 |
| NIC Card* | AOC-MCX556A-ECAT | Mellanox ConnectX-5 VPI adapter card, EDR IB(100Gb/s) & 100GbE, dual port QSFP28 | 2 |
| NVMe | HDS-IEN0-SSDPEWNV153T8 | Intel D5-P4326 E1.L PCIe3.1x4QLC9.5mm15.36T 3D2 <0.5DWPD | 32 |
| Supermicro System Management Software | SFT-DCMS-SINGLE | Supermicro System Managment Software Suite node license,HF,RoHS/REACH,PBF | 1 |
| NVMesh | SFT-EX-SWCCI32-3YR | NVMesh per-chassis subscription license, up to 32 NVMe devices. Includes unlimited Clients. 3 year subscription including Premium Support. | 1 |
| NVMesh | SVC-EX-PSINSTALL | On-site, NVMesh Installation Services, per day, per 100 Client and/or Target Nodes | 1 |
* Infiniband or RoCE
** The architecture supports NVMe drives from all major manufacturers, including Micron, Intel, Kioxia, Samsung and Western Digital. Different drive sizes are supported. For Erasure Coded configurations, NVMe drive families, including Western Digital SN200, Samsung 1725b & 1733, Kioxia CM5 & CM6, with 3 DWPD or more endurance are supported.
AS -1114S-WN10RT
1U 1N WIO
AMD Gen4 System
AS -1114S-WN10RT
- Single AMD EPYC™ 7003/7002 Series Processor
- 16 DIMMs; up to 4TB 3DS ECC DDR4-3200MHz RDIMM/LRDIMM
- 2 PCI-E 4.0 x16 (FHHL) slots, 1 PCI-E 4.0 x16 (LP) slot
- 10 Hot-swap NVMe/SATA3 hybrid drive bays
- 750W Redundant Platinum Level Power Supplies
SYS-220BT-HNTR
2U 4N BigTwin®
Intel Gen4 System
SYS-220BT-HNTR
- 4 hot-pluggable nodes in 2U; each node supports:
- Dual 3rd Gen Intel® Xeon® Scalable processors
- 20 DIMMs; Up to 6TB DDR4-3200: Intel® Optane™ Persistent Memory 200 Series; Up to 4TB ECC DDR4-3200: RDIMM
- 2 PCI-E 4.0 x16 (LP) slots
- 6 Hot-swap 2.5" NVMe/SATA drive bays
- 2600W Redundant Titanium Level Power Supplies
SYS-2029BZ-HNR
2U 4N BigTwin®
Intel Gen3 System
SYS-2029BZ-HNR
- 4 hot-pluggable nodes in 2U; each node supports:
- Dual 2nd Gen Intel® Xeon® Scalable processors
- 24 DIMMs; up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; Supports Intel® Optane™ DCPMM
- 2 PCI-E 3.0 x16 (LP) slots
- 6 Hot-swap 2.5" NVMe drive bays
- 2200W Redundant Titanium Level Power Supplies
AS -2123BT-HNR
2U 4N BigTwin®
AMD Gen3 System
AS -2123BT-HNR
- 4 hot-pluggable nodes in 2U; each node supports:
- Dual AMD EPYC™ 7002/7001 Series Processors
- 16 DIMMs; up to 4TB 3DS ECC DDR4-3200MHz SDRAM with 7002 series processors
- 2 PCI-E 3.0 x16 (LP) slots
- 6 Hot-swap 2.5" NVMe drive bays
- 2600W Redundant Titanium Level Power Supplies
SYS-1029U-TN10RT
1U 1N Ultra
Intel Gen3 System
SYS-1029U-TN10RT
- Dual 2nd Gen Intel® Xeon® Scalable processors
- 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; supports Intel® Optane™ DCPMM
- 2 PCI-E 3.0 x16 slots (FH, 10.5" L)
- 10 Hot-swap 2.5" NVMe drive bays
- 1000W Redundant Titanium Level Power Supplies
SYS-2029U-TN24R4T
2U 1N Ultra
Intel Gen3 System
SYS-2029U-TN24R4T
- Dual 2nd Gen Intel® Xeon® Scalable processors
- 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM, supports Intel® Optane™ DCPMM
- 2 PCI-E 3.0 x16 slots (FH, 10.5" L)
- 24 front Hot-swap 2.5" NVMe drive bays, 2 rear Hot-swap 2.5" drive bays
- 1600W Redundant Titanium Level Power Supplies
SSG-1029P-NES32R
1U 1N Petascale E1.S Storage Server
Intel Gen3 System
SSG-1029P-NES32R
- Dual 2nd Gen Intel® Xeon® Scalable processors
- 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; supports Intel® Optane™ DCPMM
- 2 PCI-E 3.0 x16 (FHHL) slots, 1 PCI-E 3.0 x4 (LP) slot
- 32 Hot-swap EDSFF Short (E1.S) drive slots
- 1600W Redundant Titanium Level Power Supplies
SSG-1029P-NEL32R
1U 1N Petascale E1.L Storage Server
Intel Gen3 System
SSG-1029P-NEL32R
- Dual 2nd Gen Intel® Xeon® Scalable processors
- 24 DIMMs; Up to up to 6TB 3DS ECC DDR4-2933MHz RDIMM/LRDIMM; supports Intel® Optane™ DCPMM
- 2 PCI-E 3.0 x16 (LP) slots
- 32 Hot-swap EDSFF Long (E1.L) drive slots
- 1600W Redundant Titanium Level Power Supplies
AOC-653106A-HDAT
AOC-653105A-HDAT
AOC-MCX556A-ECAT
Request More Information