2017上半年四川教师公招笔试备考_学霸速成秘籍

旧金山湾区
2万 位关注者 500+ 位好友

加入领英，查看档案

关于

We are hiring!

I am an Engineering Lead at Databricks. Our engineering teams build…

动态

立即加入，查看全部动态

工作经历和教育背景

  • Databricks

查看Xiao的完整工作经历

查看他们的职位头衔、任职时间等。

或

点击“继续加入或登录”，即表示您同意遵守领英的《用户协议》、《隐私政策》及《Cookie 政策》。

出版作品

  • Databricks Lakeguard: Supporting Fine-grained Access Control and Multi-user Capabilities for Apache Spark Workloads

    SIGMOD

    Today, upgrading Apache Spark versions typically involves significant effort, with unclear investment requirements, including trial and error. This is mainly because in Spark there is no clear separation between the application code and the engine code. Apart from changes in the public API, any Spark internal changes may affect user workloads as users may rely on Spark internals: bug fixes in the Spark engine, changes to internal APIs, library upgrades, or language upgrades may affect customer…

    Today, upgrading Apache Spark versions typically involves significant effort, with unclear investment requirements, including trial and error. This is mainly because in Spark there is no clear separation between the application code and the engine code. Apart from changes in the public API, any Spark internal changes may affect user workloads as users may rely on Spark internals: bug fixes in the Spark engine, changes to internal APIs, library upgrades, or language upgrades may affect customer workloads. As a result, Spark users are often reluctant to upgrade. The downside is that performance improvements, bug fixes, and new features take significantly longer to adopt, preventing customers from quickly benefiting from these improvements. In addition, it increases engineering complexity to manage a large number of different Spark versions. For Databricks serverless jobs and notebooks, we fundamentally transformed and simplified the user experience when using Spark. We shifted user focus from managing Spark runtime versions to managing the stable API that they integrate with - we created client-versioned workloads with a versionless Spark server. Decoupling the client from the Spark engine using Spark Connect has enabled Databricks to automatically upgrade the Spark server, providing users faster access to the latest features while minimizing disruptions from both intentional and unintentional breaking changes, all without compromising workload compatibility and with zero code changes needed from the user. This approach also offers significant benefits to Databricks by streamlining the release process, consolidating usage onto fewer server versions, and reducing engineering overhead from needing to backport changes.

    查看作品

  • Adaptive and Robust Query Execution for Lakehouses At Scale

    VLDB

    Many organizations have embraced the "Lakehouse" data man- agement paradigm, which involves constructing structured data warehouses on top of open, unstructured data lakes. This approach stands in stark contrast to traditional, closed, relational databases and introduces challenges for performance and stability of dis- tributed query processors. Firstly, in large-scale, open Lakehouses with uncurated data, high ingestion rates, external tables, or deeply nested schemas, it is often costly or…

    Many organizations have embraced the "Lakehouse" data man- agement paradigm, which involves constructing structured data warehouses on top of open, unstructured data lakes. This approach stands in stark contrast to traditional, closed, relational databases and introduces challenges for performance and stability of dis- tributed query processors. Firstly, in large-scale, open Lakehouses with uncurated data, high ingestion rates, external tables, or deeply nested schemas, it is often costly or wasteful to maintain perfect and up-to-date table and column statistics. Secondly, inherently imperfect cardinality estimates with conjunctive predicates, joins and user-defined functions can lead to bad query plans. Thirdly, for the sheer magnitude of data involved, strictly relying on static query plan decisions can result in performance and stability issues such as excessive data movement, substantial disk spillage, or high memory pressure. To address these challenges, this paper presents our design, implementation, evaluation and practice of the Adaptive Query Execution (AQE) framework, which exploits natural execu- tion pipeline breakers in query plans to collect accurate statistics and re-optimize them at runtime for both performance and robust- ness. In the TPC-DS benchmark, the technique demonstrates up to 25× per query speedup. At Databricks, AQE has been successfully deployed in production for multiple years. It powers billions of queries and ETL jobs to process exabytes of data per day, through key enterprise products such as Databricks Runtime, Databricks SQL, and Delta Live Tables.

    查看作品

  • Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores

    VLDB

    Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer…

    Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer over cloud object stores initially developed at Databricks. Delta Lake uses a transaction log that is compacted into Apache Parquet format to provide ACID properties, time travel, and significantly faster metadata operations for large tabular datasets (e.g., the ability to quickly search billions of table partitions for those relevant to a query). It also leverages this design to provide high-level features such as automatic data layout optimization, upserts, caching, and audit logs. Delta Lake tables can be accessed from Apache Spark, Hive, Presto, Redshift and other systems. Delta Lake is deployed at thousands of Databricks customers that process exabytes of data per day, with the largest instances managing exabyte-scale datasets and billions of objects.

    查看作品

  • Spark SQL

    Encyclopedia of Big Data Technologies

    SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark, which is a unified engine for distributed data processing (Zaharia et al. 2012). Spark SQL can process, integrate, and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka, and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). The common use cases include ad hoc analysis, logical warehouse, query federation,…

    SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark, which is a unified engine for distributed data processing (Zaharia et al. 2012). Spark SQL can process, integrate, and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka, and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). The common use cases include ad hoc analysis, logical warehouse, query federation, and ETL processing. It also powers the other Spark libraries, including structured streaming for stream processing (Michael et al. 2018), MLlib for machine learning (Meng et al. 2016), GraphFrame for graph-parallel computation (Dave et al. 2016), and TensorFrames for TensorFlow binding. These libraries and Spark SQL can be seamlessly combined in the same application with holistic optimization by Spark SQL.

    其他作者
    查看作品

  • Optimizing Inter-data-center Large-Scale Database Parallel Replication with Workload-Driven Partitioning

    Transactions on Large-Scale Data- and Knowledge-Centered Systems XXIV

    Geographically distributed data centers are deployed for non-stop business operations by many enterprises. In case of disastrous events, ongoing workloads must be failed over from the current data center to another active one within just a few seconds to achieve continuous service availability. Software-based parallel database replication techniques are designed to meet very high throughput with near-real-time latency. Understanding workload characteristics is one of the key factors for…

    Geographically distributed data centers are deployed for non-stop business operations by many enterprises. In case of disastrous events, ongoing workloads must be failed over from the current data center to another active one within just a few seconds to achieve continuous service availability. Software-based parallel database replication techniques are designed to meet very high throughput with near-real-time latency. Understanding workload characteristics is one of the key factors for improving replication performance. In this paper, we propose a workload-driven method to optimize database replication latency and minimize transaction splits with a minimum of parallel replication consistency groups. Our two-phased approach includes (1) a log-based mechanism for workload pattern discovery; (2) a history-based algorithm on pattern analysis, database partitioning and partition adjustment. The experimental results from a real banking batch workload and a benchmark OLTP workload demonstrate the effectiveness of the solution even for partitioning 1000 s of database tables in very large workloads. Finally, the algorithm to automate the cyclic flow of workload profile capturing and partitioning readjustment is developed and verified.

    其他作者
    查看作品

  • Inter-Data Center Large-scale Database Replication Optimization - a Workload Driven Partitioning Approach

    25th International Conference on Database and Expert Systems Applications (DEXA) 2014: 417-432

    Inter-data-center asynchronous middleware replication between active-active databases has become essential for achieving continuous business availability. Near real-time replication latency is expected despite intermittent peaks in transaction volumes. Database tables are divided for replication across multiple parallel replication consistency groups; each having a maximum throughput capacity, but doing so can break transaction integrity. It is often not known which tables can be updated by a…

    Inter-data-center asynchronous middleware replication between active-active databases has become essential for achieving continuous business availability. Near real-time replication latency is expected despite intermittent peaks in transaction volumes. Database tables are divided for replication across multiple parallel replication consistency groups; each having a maximum throughput capacity, but doing so can break transaction integrity. It is often not known which tables can be updated by a common transaction. Independent replication also requires balancing resource utilization and latency objectives. Our work provides a method to optimize replication latencies, while minimizing transaction splits among a minimum of parallel replication consistency groups. We present a two-staged approach: a log-based workload discovery and analysis and a history-based database partitioning. The experimental results from a real banking batch workload and a benchmark OLTP workload demonstrate the effectiveness of our solution even for partitioning 1000s of database tables for very large workloads.

    其他作者
    查看作品

  • System Monitoring in a Parallel Database Replication Apply Processing

    IBM Technical Disclosure Bulletin

    Disclosed is an efficient monitoring method for health and performance monitoring as well as troubleshooting in a parallel replication apply processing. This method allows near-real-time collection of a large amount of data with a negligible performance impact.

    其他作者

  • Mapping Reuse for Meta-Querier Customization.

    Ph.D. Dissertation. University of Florida, Gainesville, FL, USA.

专利发明

  • Replication Group Partitioning

    颁发日期: US11157518B2

    Systems for replication group partitioning include a workload profiling module configured to analyze historical workload data for a plurality of data elements to identify and categorize one or more transaction patterns; and a recommendation module configured to generate a recommended partitioning of the plurality of data elements into one or more replication groups, based on the one or more transaction patterns, that are optimized toward a partitioning goal.

    其他发明人
    查看专利

  • Parallel Replication of Data Table Partitions

    颁发日期: US10902015B2

    Several replication subscriptions are defined for a table in a database management system. The table is divided into partitions. Each replication subscription replicates transactions to a range of partitions. Subscriptions are assignable to different consistency groups. Transaction consistency is preserved at the consistency group level. A persistent delay table is created for each of several apply functions. Each apply function processes replication subscriptions for one consistency group…

    Several replication subscriptions are defined for a table in a database management system. The table is divided into partitions. Each replication subscription replicates transactions to a range of partitions. Subscriptions are assignable to different consistency groups. Transaction consistency is preserved at the consistency group level. A persistent delay table is created for each of several apply functions. Each apply function processes replication subscriptions for one consistency group, to replicate the table to a target table in parallel. Transactions for a given range of a partition are executed in parallel. When an apply function upon a row of the target table results in an error, the row is stored in the delay table. Application of each row in the delay table is repeatedly retried, and if successful, the row is removed from the delay table.

    其他发明人
    查看专利

  • Automatically restoring data replication consistency without service interruption during parallel apply

    颁发日期: US 10229152B2

    A data replication method can begin with the detection of an inconsistency between records of a target table and corresponding records of a source table of a relational database management system (RDBMS) performing a parallel apply replication by an improved data replication manager. The target table can be a copy of the source table, both of which include multiple unique constraints and indexes. A timeframe that encompasses the records of the target table having the inconsistency can be…

    A data replication method can begin with the detection of an inconsistency between records of a target table and corresponding records of a source table of a relational database management system (RDBMS) performing a parallel apply replication by an improved data replication manager. The target table can be a copy of the source table, both of which include multiple unique constraints and indexes. A timeframe that encompasses the records of the target table having the inconsistency can be determined. The timeframe can utilize a commit timestamp or a log sequence number. Consistency between the target table and the source table can be automatically restored for the determined timeframe through use of a reactive-apply process. Data suppression for updates is automatically restored once the copy is consistent. Transactions performed upon the target table by the reactive-apply process can be performed in parallel. Service at the source table and the target table can be uninterrupted.

    其他发明人
    查看专利

  • Recovery Log Analytics With A Big Data Management Platform

    颁发日期: US 10216584B2

    Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using…

    Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using the consistent change records.

    其他发明人
    查看专利

  • System And Method For Transferring Data Between RDBMS And Big Data Platform

    颁发日期: US 10169409B2

    A system for transferring data from a Relational Database Management System (“RDBMS”) to a big data platform and methods for making and using the same. The system can acquire a partitioning execution scheme of a selected table from the RDBMS and submitting partitioned queries from the big data platform to each mapper of partitions. The partitioned queries are generated based on the partitioning execution scheme. The partitioning execution scheme can be acquired by submitting a query explain…

    A system for transferring data from a Relational Database Management System (“RDBMS”) to a big data platform and methods for making and using the same. The system can acquire a partitioning execution scheme of a selected table from the RDBMS and submitting partitioned queries from the big data platform to each mapper of partitions. The partitioned queries are generated based on the partitioning execution scheme. The partitioning execution scheme can be acquired by submitting a query explain request to an optimizer of the RDBMS to generating a parallel query plan. The partitioning execution scheme can also be acquired by querying statistics from a statistics catalog of the RDBMS or by user inputs. The system can use RDBMS capabilities and statistics for parallel data fetching. Thereby, the system can increase efficiency of the fetching and can avoid straggling when target data is not evenly distributed and can avoid table query-in-serial.

    其他发明人
    查看专利

  • Data Caching In Hybrid Data Processing And Integration Environment

    颁发日期: US 10169429B2

    A caching mechanism is disclosed for enhancing performance in an integrated data processing environment, a.k.a. a unified data processing system, that is composed of two or more heterogeneous data processing system.

    其他发明人
    查看专利

  • Lightweight Table Comparison

    颁发日期: US 9928281B2

    A system, method and computer program product is provided for enabling light weight table comparison with high-accuracy (high confidence) of tables where one is a copy of the other, which copy may be maintained synchronized by replication. The method performs database comparison using a sample-based, statistics-based, or materialized query tables-based approaches. The method first identifies a block comprising a sub-set of rows of data of a source database table and a corresponding block from…

    A system, method and computer program product is provided for enabling light weight table comparison with high-accuracy (high confidence) of tables where one is a copy of the other, which copy may be maintained synchronized by replication. The method performs database comparison using a sample-based, statistics-based, or materialized query tables-based approaches. The method first identifies a block comprising a sub-set of rows of data of a source database table and a corresponding block from a target database table, and obtains a statistical value associated with each block. Then the statistical values for the corresponding source and target block are compared and a consistency evaluation of source and target database is determined based on comparing results. Further methods enable a determination of the data as being persistent or not in manner that accounts for real-time data modifications to underlying source and target database tables while identified blocks are being compared.

    其他发明人
    查看专利

  • Database Table Comparison

    颁发日期: US 9,600,513

    Techniques are disclosed for comparing database tables. In one embodiment, the database tables are partitioned. Queries are generated for retrieving each partition. For each generated query, a stored procedure is invoked, using the respective generated query as an input parameter to the stored procedure. The stored procedure is configured to generate a checksum based on the partition retrieved from executing the respective query. The application compares the generated checksums to determine if…

    Techniques are disclosed for comparing database tables. In one embodiment, the database tables are partitioned. Queries are generated for retrieving each partition. For each generated query, a stored procedure is invoked, using the respective generated query as an input parameter to the stored procedure. The stored procedure is configured to generate a checksum based on the partition retrieved from executing the respective query. The application compares the generated checksums to determine if the partitions and/or tables are consistent.

    其他发明人
    查看专利

  • Verifying Data Consistency

    颁发日期: US9542406B1

    A method for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures is provided. The method includes loading data from an update-in-place data structure to a first set of hash buckets in a processing platform, loading data from append-only data structures to a second set of hash buckets in the processing platform, performing a bucket-level comparison between the data in…

    A method for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures is provided. The method includes loading data from an update-in-place data structure to a first set of hash buckets in a processing platform, loading data from append-only data structures to a second set of hash buckets in the processing platform, performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of has buckets, and generating a report based on the bucket-level comparison.

    其他发明人
    查看专利

  • Difference Determination in a Database Environment

    颁发日期: US 9,529,881

    Techniques are disclosed to determine differences between a source table and a target table in a database environment, as being persistent or transient. A first set of differences between the source table and the target table is determined at a first point in time. A second set of differences between the source table and the target table is determined at a second point in time subsequent to the first point in time. At least one of a set of persistent differences and a set of transient…

    Techniques are disclosed to determine differences between a source table and a target table in a database environment, as being persistent or transient. A first set of differences between the source table and the target table is determined at a first point in time. A second set of differences between the source table and the target table is determined at a second point in time subsequent to the first point in time. At least one of a set of persistent differences and a set of transient differences is determined. The set of persistent differences includes a set intersection of the first and second sets of differences, the set intersection being filtered based on matching non-key values of the differences. The set of transient differences includes a relative complement of the second set of differences in the first set of differences.

    其他发明人
    查看专利

  • Transaction Consistency Query Support For Replicated Data From Recovery Log To External Data Stores

    申请日期: US 11455217B2

    Provided are techniques for transaction consistency query support for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a…

    Provided are techniques for transaction consistency query support for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

    其他发明人
    查看专利

荣誉奖项

  • ACM SIGMOD Systems Award

    SIGMOD

    The 2022 ACM SIGMOD Systems Award goes to “Apache Spark“, an innovative, widely-used, open-source, unified data processing system encompassing relational, streaming, and machine-learning workloads. The award recognizes the contributions of Michael Armbrust, Tathagata Das, Ankur Dave, Wenchen Fan, Michael J. Franklin, Huaxin Gao Maxim Gekk, Ali Ghodsi, Joseph Gonzalez, Liang-Chi Hsieh, Dongjoon Hyun, Hyukjin Kwon, Xiao Li, Cheng Lian, Yanbo Liang, Xiangrui Meng, Sean Owen, Josh Rosen, Kousuke…

    The 2022 ACM SIGMOD Systems Award goes to “Apache Spark“, an innovative, widely-used, open-source, unified data processing system encompassing relational, streaming, and machine-learning workloads. The award recognizes the contributions of Michael Armbrust, Tathagata Das, Ankur Dave, Wenchen Fan, Michael J. Franklin, Huaxin Gao Maxim Gekk, Ali Ghodsi, Joseph Gonzalez, Liang-Chi Hsieh, Dongjoon Hyun, Hyukjin Kwon, Xiao Li, Cheng Lian, Yanbo Liang, Xiangrui Meng, Sean Owen, Josh Rosen, Kousuke Saruta, Scott Shenker, Ion Stoica, Takuya Ueshin, Shivaram Venkataraman, Gengliang Wang, Yuming Wang, Patrick Wendell, Reynold Xin, Takeshi Yamamuro, Kent Yao, Matei Zaharia, Ruifeng Zheng, and Shixiong Zhu.

  • Master Inventor Certificate

    IBM

  • Plateau Invention Achievement Award #2

    IBM

  • Plateau Invention Achievement Award #1

    IBM

  • Invention Achievement Award

    IBM

  • National First Prize

    China Post-Graduate Mathematical Contest in Modeling

  • Special-class Scholarship

    NUST

    Rank 1/420, one of the highest academic honor in NUST

Xiao的更多动态

查看Xiao的完整档案

  • 浏览共同好友
  • 请求引荐
  • 直接联系Xiao
加入领英，查看完整档案

其他相似会员

美国中其他姓名为Xiao Li的会员

领英上有美国的其他 491 位姓名为Xiao Li的会员

查看其他姓名为Xiao Li的会员

学习在线课程，新技能轻松 get！

查看全部课程