Skip to main content

Advertisement

SpringerLink
Log in

ByShard: sharding in a Byzantine environment

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

The emergence of blockchains has fueled the development of resilient systems that deal with Byzantine failures due to crashes, bugs, or even malicious behavior. Recently, we have also seen the exploration of sharding in these resilient systems, this to provide the scalability required by very large data-based applications. Unfortunately, current sharded resilient systems all use system-specific specialized approaches toward sharding that do not provide the flexibility of traditional sharded data management systems. To improve on this situation, we fundamentally look at the design of sharded resilient systems. We do so by introducing ByShard, a unifying framework for the study of sharded resilient systems. Within this framework, we show how two-phase commit and two-phase locking—two techniques central to providing atomicity and isolation in traditional sharded databases—can be implemented efficiently in a Byzantine environment, this with a minimal usage of costly Byzantine resilient primitives. Based on these techniques, we propose eighteen multi-shard transaction processing protocols. Finally, we practically evaluate these protocols and show that each protocol supports high transaction throughput and provides scalability while each striking its own trade-off between throughput, isolation level, latency, and abort rate. As such, our work provides a strong foundation for the development of ACID-compliant general-purpose and flexible sharded resilient data management systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. We provide a practical definition of consensus. In practice, decisions will be made on external requests (guaranteeing non-triviality) if such requests are available to non-faulty replicas (guaranteeing termination). Theoretical definitions typically have more abstract requirements for termination and non-triviality.

  2. The full C++ implementation of these experiments and the raw measurements are available at https://www.jhellings.nl/projects/byshard/.

  3. The transaction duration includes waiting times (e.g., waiting for locks to be released, waiting for votes to arrive, waiting for a next shard-step to be executed). As many transactions can be active in parallel (even at a single shard due to waiting), the cumulative duration can be much higher than the product of the number of shards and the total runtime.

  4. All other processed transactions are aborted (either due to constraint failure or, when non-blocking locks are used, the inability to acquire locks).

  5. Linear orchestration can only have a single constraint failure per transaction (which will lead to an abort for that transaction), while both centralized and distributed orchestration can have many constraint failures per transaction (which will lead to a single abort for that transaction).

References

  1. Al-Bassam, M., Sonnino, A., Bano, S., Hrycyszyn, D., Danezis, G.: Chainspace: A sharded smart contracts platform (2017). arxiv:1708.03778

  2. Amiri, M.J., Agrawal, D., El Abbadi, A.: SharPer: sharding permissioned blockchains over network clusters. In: Proceedings of the 2021 International Conference on Management of Data, pp. 76–88. ACM (2021). https://doi.org/10.1145/3448016.3452807

  3. Amiri, M.J., Agrawal, D., Abbadi, A.E.: CAPER: a cross-application permissioned blockchain. Proc. VLDB Endow. 12(11), 1385–1398 (2019). https://doi.org/10.14778/3342263.3342275

    Article  Google Scholar 

  4. Atluri, V., Bertino, E., Jajodia, S.: A theoretical formulation for degrees of isolation in databases. Inform. Software Tech. 39(1), 47–53 (1997). https://doi.org/10.1016/0950-5849(96)01109-3

    Article  Google Scholar 

  5. Berenson, H., Bernstein, P., Gray, J., Melton, J., O’Neil, E., O’Neil, P.: A critique of ANSI SQL isolation levels. SIGMOD Rec. 24(2), 1–10 (1995). https://doi.org/10.1145/568271.223785

    Article  Google Scholar 

  6. Berger, C., Reiser, H.P.: Scaling byzantine consensus: a broad analysis. In: Proceedings of the 2nd Workshop on Scalable and Resilient Infrastructures for Distributed Ledgers, pp. 13–18. ACM (2018). https://doi.org/10.1145/3284764.3284767

  7. Cachin, C., Vukolic, M.: Blockchain consensus protocols in the wild (keynote talk). In: 31st International Symposium on Distributed Computing, vol. 91, pp. 1:1–1:16. Schloss Dagstuhl (2017). https://doi.org/10.4230/LIPIcs.DISC.2017.1

  8. Casey, M., Crane, J., Gensler, G., Johnson, S., Narula, N.: The impact of blockchain technology on finance: a catalyst for change. Tech. Rep., International Center for Monetary and Banking Studies (2018). https://www.cimb.ch/uploads/1/1/5/4/115414161/geneva21_1.pdf

  9. Castro, M., Liskov, B.: Practical byzantine fault tolerance and proactive recovery. ACM Trans. Comput. Syst. 20(4), 398–461 (2002). https://doi.org/10.1145/571637.571640

    Article  Google Scholar 

  10. Correia, M., Veronese, G.S., Neves, N.F., Verissimo, P.: Byzantine consensus in asynchronous message-passing systems: a survey. Int. J. Crit. Comput.-Based Syst. 2(2), 141–161 (2011)

    Article  Google Scholar 

  11. Dang, H., Dinh, T.T.A., Loghin, D., Chang, E.C., Lin, Q., Ooi, B.C.: Towards scaling blockchain systems via sharding. In: Proceedings of the 2019 International Conference on Management of Data, pp. 123–140. ACM (2019). https://doi.org/10.1145/3299869.3319889

  12. Dinh, T.T.A., Liu, R., Zhang, M., Chen, G., Ooi, B.C., Wang, J.: Untangling blockchain: a data processing view of blockchain systems. Trans. Knowl. Data Eng. 30(7), 1366–1385 (2018). https://doi.org/10.1109/TKDE.2017.2781227

    Article  Google Scholar 

  13. Dolev, D.: Unanimity in an unknown and unreliable environment. In: 22nd Annual Symposium on Foundations of Computer Science, pp. 159–168. IEEE (1981). https://doi.org/10.1109/SFCS.1981.53

  14. Dolev, D.: The byzantine generals strike again. J. Algorithms 3(1), 14–30 (1982). https://doi.org/10.1016/0196-6774(82)90004-9

    Article  MathSciNet  MATH  Google Scholar 

  15. El-Hindi, M., Binnig, C., Arasu, A., Kossmann, D., Ramamurthy, R.: BlockchainDB: a shared database on blockchains. Proc. VLDB Endow. 12(11), 1597–1609 (2019). https://doi.org/10.14778/3342263.3342636

  16. Ethereum Foundation: BTC Relay: a bridge between the bitcoin blockchain & ethereum smart contracts (2017). http://btcrelay.org

  17. Gordon, W.J., Catalini, C.: Blockchain technology for healthcare: facilitating the transition to patient-driven interoperability. Comput. Struct. Biotechnol. J. 16, 224–230 (2018). https://doi.org/10.1016/j.csbj.2018.06.003

    Article  Google Scholar 

  18. Gray, J.: Notes on data base operating systems. In: Operating Systems, An Advanced Course, pp. 393–481. Springer-Verlag (1978). https://doi.org/10.1007/3-540-08755-9_9

  19. Gupta, S., Hellings, J., Sadoghi, M.: Fault-tolerant distributed transactions on blockchain. In: Synthesis Lectures on Data Management. Morgan & Claypool (2021). https://doi.org/10.2200/S01068ED1V01Y202012DTM065

  20. Gupta, S., Rahnama, S., Sadoghi, M.: Permissioned blockchain through the looking glass: architectural and implementation lessons learned. In: 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pp. 754–764. IEEE (2020). https://doi.org/10.1109/ICDCS47774.2020.00012

  21. Gupta, S., Rahnama, S., Hellings, J., Sadoghi, M.: ResilientDB: global scale resilient blockchain fabric. Proc. VLDB Endow. 13(6), 868–883 (2020). https://doi.org/10.14778/3380750.3380757

    Article  Google Scholar 

  22. Haerder, T., Reuter, A.: Principles of transaction-oriented database recovery. ACM Comput. Surv. 15(4), 287–317 (1983). https://doi.org/10.1145/289.291

    Article  MathSciNet  Google Scholar 

  23. Hassanzadeh-Nazarabadi, Y., Taheri-Boshrooyeh, S.: a consensus protocol with deterministic finality. In: INFOCOM 2021–IEEE Conference on Computer Communications Workshops, pp. 1–2. IEEE (2021). https://doi.org/10.1109/INFOCOMWKSHPS51825.2021.9484527

  24. Hassanzadeh-Nazarabadi, Y., Küpçü, A., Özkasap, Ö.: LightChain: scalable DHT-based blockchain. Parallel Distrib. Syst. 32(10), 2582–2593 (2021). https://doi.org/10.1109/TPDS.2021.3071176

    Article  Google Scholar 

  25. Hellings, J., Hughes, D.P., Primero, J., Sadoghi, M.: Cerberus: Minimalistic multi-shard byzantine-resilient transaction processing (2020). arxiv:2008.04450

  26. Hellings, J., Sadoghi, M.: Byzantine cluster-sending in expected constant communication (2021). arxiv:2108.08541

  27. Hellings, J., Sadoghi, M.: The fault-tolerant cluster-sending problem. In: Foundations of Information and Knowledge Systems, pp. 168–186. Springer (2022). https://doi.org/10.1007/978-3-031-11321-5_10

  28. Hellings, J., Sadoghi, M.: Byshard: sharding in a byzantine environment. Proc. VLDB Endow. 14(11), 2230–2243 (2021). https://doi.org/10.14778/3476249.3476275

    Article  Google Scholar 

  29. Hentschel, A., Hassanzadeh-Nazarabadi, Y., Seraj, R., Shirley, D., Lafrance, L.: Flow: Separating consensus and compute–block formation and execution (2020). arxiv:2002.07403

  30. Herlihy, M.: Atomic cross-chain swaps. In: Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing, pp. 245–254. ACM (2018). https://doi.org/10.1145/3212734.3212736

  31. Herlihy, M.: Blockchains from a distributed computing perspective. Commun. ACM 62(2), 78–85 (2019). https://doi.org/10.1145/3209623

    Article  Google Scholar 

  32. Herlihy, M., Liskov, B., Shrira, L.: Cross-chain deals and adversarial commerce. VLDB J. (2021). https://doi.org/10.1007/s00778-021-00686-1

    Article  Google Scholar 

  33. Herzog, T.N., Scheuren, F.J., Winkler, W.E.: Data quality and record linkage techniques. Springer (2007). https://doi.org/10.1007/0-387-69505-2

  34. Kamel Boulos, M.N., Wilson, J.T., Clauson, K.A.: Geospatial blockchain: promises, challenges, and scenarios in health and healthcare. Int. J. Health. Geogr 17(1), 1211–1220 (2018). https://doi.org/10.1186/s12942-018-0144-x

    Article  Google Scholar 

  35. Kamilaris, A., Fonts, A., Prenafeta-Boldú, F.X.: The rise of blockchain technology in agriculture and food supply chains. Trends Food Sci. Technol. 91, 640–652 (2019). https://doi.org/10.1016/j.tifs.2019.07.034

    Article  Google Scholar 

  36. Kwon, J., Buchman, E.: Cosmos whitepaper: a network of distributed ledgers (2019). https://cosmos.network/cosmos-whitepaper.pdf

  37. Lamport, L.: Paxos made simple. ACM SIGACT News 32(4), 51–58 (2001). https://doi.org/10.1145/568425.568433

    Article  Google Scholar 

  38. Lao, L., Li, Z., Hou, S., Xiao, B., Guo, S., Yang, Y.: A survey of IoT applications in blockchain systems: architecture, consensus, and traffic modeling. ACM Comput. Surv. (2020). https://doi.org/10.1145/3372136

    Article  Google Scholar 

  39. Nakamoto, S.: Bitcoin: a peer-to-peer electronic cash system. https://bitcoin.org/en/bitcoin-paper

  40. Narayanan, A., Clark, J.: Bitcoin’s academic pedigree. Commun. ACM 60(12), 36–45 (2017). https://doi.org/10.1145/3132259

    Article  Google Scholar 

  41. Nathan, S., Govindarajan, C., Saraf, A., Sethi, M., Jayachandran, P.: Blockchain meets database: design and implementation of a blockchain relational database. Proc. VLDB Endow. 12(11), 1539–1552 (2019). https://doi.org/10.14778/3342263.3342632

  42. Nawab, F., Sadoghi, M.: Consensus in data management: from distributed commit to blockchain. Found. Trends Datab. 12(4), 221–364 (2023). https://doi.org/10.1561/1900000075

    Article  MATH  Google Scholar 

  43. Özsu, M.T., Valduriez, P.: Principles of Distributed Database Systems. Springer (2020). https://doi.org/10.1007/978-3-030-26253-2

  44. Pisa, M., Juden, M.: Blockchain and economic development: Hype vs. reality. Tech. rep., Center for Global Development (2017). https://www.cgdev.org/publication/blockchain-and-economic-development-hype-vs-reality

  45. Reinsel, D., Gantz, J., Rydning, J.: Data age 2025: the digitization of the world, from edge to core. Tech. rep., IDC (2018). https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf

  46. Skeen, D.: A Quorum-Based Commit Protocol, Tech. rep. Cornell University (1982)

  47. Tel, G.: Introduction to Distributed Algorithms, 2nd edn. Cambridge University Press (2001)

  48. The Hyperledger White Paper Working Group: An introduction to Hyperledger. Tech. rep, The Linux Foundation (2018)

  49. Treiblmaier, H., Beck, R. (eds.): Business Transformation through Blockchain. Springer (2019). https://doi.org/10.1007/978-3-319-98911-2

  50. van Steen, M., Tanenbaum, A.S.: Distributed Systems, 3th edn. Maarten van Steen (2017). https://www.distributed-systems.net/

  51. Wood, G.: Ethereum: A Secure Decentralised Generalised Transaction Ledger. https://gavwood.com/paper.pdf. EIP-150 revision

  52. Wood, G.: Polkadot: vision for a heterogeneous multi-chain framework (2016). https://polkadot.network/PolkaDotPaper.pdf

  53. Wu, M., Wang, K., Cai, X., Guo, S., Guo, M., Rong, C.: A comprehensive survey of blockchain: from theory to IoT applications and beyond. Internet Things J. 6(5), 8114–8154 (2019). https://doi.org/10.1109/JIOT.2019.2922538

  54. Xiao, Y., Zhang, N., Lou, W., Hou, Y.T.: A survey of distributed consensus protocols for blockchain networks. Commun. Surv. Tutor 22(2), 1432–1465 (2020). https://doi.org/10.1109/COMST.2020.2969706

  55. Yin, M., Malkhi, D., Reiter, M.K., Gueta, G.G., Abraham, I.: HotStuff: BFT consensus with linearity and responsiveness. In: Proceedings of the ACM Symposium on Principles of Distributed Computing, pp. 347–356. ACM (2019). https://doi.org/10.1145/3293611.3331591

  56. Zakhary, V., Agrawal, D., El Abbadi, A.: Atomic commitment across blockchains. Proc. VLDB Endow. 13(9), 1319–1331 (2020). https://doi.org/10.14778/3397230.3397231

Download references

Author information

Authors and Affiliations

  1. Department of Computing and Software, McMaster University, 1280 Main Street West, Hamilton, ON, Canada

    Jelle Hellings

  2. Exploratory Systems Lab, Department of Computer Science, University of California, Davis, One Shields Avenue, Davis, CA, USA

    Mohammad Sadoghi

Authors
  1. Jelle Hellings
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Sadoghi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jelle Hellings.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

An extended abstract of this work appeared at the 47th International Conference on Very Large Data Bases (VLDB 2021) [26].

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hellings, J., Sadoghi, M. ByShard: sharding in a Byzantine environment. The VLDB Journal 32, 1343–1367 (2023). https://doi.org/10.1007/s00778-023-00794-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-023-00794-0

Keywords

Search

Navigation