Machine Learning and Big Data With Kdb+/q - Q, High Frequency Financial Data and Algorithmic Trading

Upgrade your programming language to more effectively handle high-frequency data
 
Machine Learning and Big Data with KDB+/Q offers quants, programmers and algorithmic traders a practical entry into the powerful but non-intuitive kdb+ database and q programming language. Ideally designed to handle the speed and volume of high-frequency financial data at sell- and buy-side institutions, these tools have become the de facto standard; this book provides the foundational knowledge practitioners need to work effectively with this rapidly-evolving approach to analytical trading.
 
The discussion follows the natural progression of working strategy development to allow hands-on learning in a familiar sphere, illustrating the contrast of efficiency and capability between the q language and other programming approaches. Rather than an all-encompassing "bible"-type reference, this book is designed with a focus on real-world practicality -to help you quickly get up to speed and become productive with the language.
* Understand why kdb+/q is the ideal solution for high-frequency data
* Delve into "meat" of q programming to solve practical economic problems
* Perform everyday operations including basic regressions, cointegration, volatility estimation, modelling and more
* Learn advanced techniques from market impact and microstructure analyses to machine learning techniques including neural networks
 
The kdb+ database and its underlying programming language q offer unprecedented speed and capability. As trading algorithms and financial models grow ever more complex against the markets they seek to predict, they encompass an ever-larger swath of data -- more variables, more metrics, more responsiveness and altogether more "moving parts."
 
Traditional programming languages are increasingly failing to accommodate the growing speed and volume of data, and lack the necessary flexibility that cutting-edge financial modelling demands. Machine Learning and Big Data with KDB+/Q opens up the technology and flattens the learning curve to help you quickly adopt a more effective set of tools.

List of contents

Preface xvii
 
About the Authors xxiii
 
Part One Language Fundamentals
 
Chapter 1 Fundamentals of the q Programming Language 3
 
1.1 The (Not So Very) First Steps in q 3
 
1.2 Atoms and Lists 5
 
1.2.1 Casting Types 11
 
1.3 Basic Language Constructs 14
 
1.3.1 Assigning, Equality and Matching 14
 
1.3.2 Arithmetic Operations and Right-to-Left Evaluation: Introduction to q Philosophy 17
 
1.4 Basic Operators 19
 
1.5 Difference between Strings and Symbols 31
 
1.5.1 Enumeration 31
 
1.6 Matrices and Basic Linear Algebra in q 33
 
1.7 Launching the Session: Additional Options 35
 
1.8 Summary and How-To's 38
 
Chapter 2 Dictionaries and Tables: The q Fundamentals 41
 
2.1 Dictionary 41
 
2.2 Table 44
 
2.3 The Truth about Tables 48
 
2.4 Keyed Tables are Dictionaries 50
 
2.5 From a Vector Language to an Algebraic Language 51
 
Chapter 3 Functions 57
 
3.1 Namespace 59
 
3.1.0.1 .quantQ. Namespace 60
 
3.2 The Six Adverbs 60
 
3.2.1 Each 60
 
3.2.1.1 Each 61
 
3.2.1.2 Each-left : 61
 
3.2.1.3 Each-right /: 62
 
3.2.1.4 Cross Product /: : 62
 
3.2.1.5 Each-both ' 63
 
3.2.2 Each-prior ': 66
 
3.2.3 Compose (') 67
 
3.2.4 Over and Fold / 67
 
3.2.5 Scan 68
 
3.2.5.1 EMA: The Exponential Moving Average 69
 
3.2.6 Converge 70
 
3.2.6.1 Converge-repeat 70
 
3.2.6.2 Converge-iterate 71
 
3.3 Apply 72
 
3.3.1 @ (apply) 72
 
3.3.2 . (apply) 73
 
3.4 Protected Evaluations 75
 
3.5 Vector Operations 76
 
3.5.1 Aggregators 76
 
3.5.1.1 Simple Aggregators 76
 
3.5.1.2 Weighted Aggregators 77
 
3.5.2 Uniform Functions 77
 
3.5.2.1 Running Functions 77
 
3.5.2.2 Window Functions 78
 
3.6 Convention for User-Defined Functions 79
 
Chapter 4 Editors and Other Tools 81
 
4.1 Console 81
 
4.2 Jupyter Notebook 82
 
4.3 GUIs 84
 
4.3.1 qStudio 85
 
4.3.2 Q Insight Pad 88
 
4.4 IDEs: IntelliJ IDEA 90
 
4.5 Conclusion 92
 
Chapter 5 Debugging q Code 93
 
5.1 Introduction to Making It Wrong: Errors 93
 
5.1.1 Syntax Errors 94
 
5.1.2 Runtime Errors 94
 
5.1.2.1 The Type Error 95
 
5.1.2.2 Other Errors 98
 
5.2 Debugging the Code 100
 
5.3 Debugging Server-Side 102
 
Part Two Data Operations
 
Chapter 6 Splayed and Partitioned Tables 107
 
6.1 Introduction 107
 
6.2 Saving a Table as a Single Binary File 108
 
6.3 Splayed Tables 110
 
6.4 Partitioned Tables 113
 
6.5 Conclusion 119
 
Chapter 7 Joins 121
 
7.1 Comma Operator 121
 
7.2 Join Functions 125
 
7.2.1 ij 125
 
7.2.2 ej 126
 
7.2.3 lj 126
 
7.2.4 pj 127
 
7.2.5 upsert 128
 
7.2.6 uj 129
 
7.2.7 aj 131
 
7.2.8 aj0 134
 
7.2.8.1 The Next Valid Join 135
 
7.2.9 asof 138
 
7.2.10 wj 140
 
7.3 Advanced Example: Running TWAP 144
 
Chapter 8 Parallelisation 151
 
8.1 Parallel Vector Operations 152
 
8.2 Parallelisation over Processes 155
 
8.3 Map-Reduce 155
 
8.4 Advanced Topic: Parallel File/Directory Access 158
 
Chapter 9 Data Cleaning and Filtering 161
 
9.1 Predicate Filtering 161
 
9.1.1 The Where Clause 161
 
9.1.2 Aggregation Filt

About the author

JAN NOVOTNY is an eFX quant trader at Deutsche Bank. Previously, he worked at the Centre for Econometric Analysis on high-frequency econometric models. He holds a PhD from CERGE-EI, Charles University, Prague. PAUL A. BILOKON is CEO and founder of Thalesians Ltd and an expert in algorithmic trading. He previously worked at Nomura, Lehman Brothers, and Morgan Stanley. Paul was educated at Christ Church College, Oxford, and Imperial College. ARIS GALIOTOS is the global technical lead for the eFX kdb+ team at HSBC, where he helps develop a big data installation processing billions of real-time records per day. Aris holds an MSc in Financial Mathematics with Distinction from the University of Edinburgh. FRÉDÉRIC DÉLÈZE is an independent algorithm trader and consultant. He has designed automated trading strategies for hedge funds and developed quantitative risk models for investment banks. He holds a PhD in Finance from Hanken School of Economics, Helsinki.

Summary

Upgrade your programming language to more effectively handle high-frequency data

Machine Learning and Big Data with KDB+/Q offers quants, programmers and algorithmic traders a practical entry into the powerful but non-intuitive kdb+ database and q programming language. Ideally designed to handle the speed and volume of high-frequency financial data at sell- and buy-side institutions, these tools have become the de facto standard; this book provides the foundational knowledge practitioners need to work effectively with this rapidly-evolving approach to analytical trading.

The discussion follows the natural progression of working strategy development to allow hands-on learning in a familiar sphere, illustrating the contrast of efficiency and capability between the q language and other programming approaches. Rather than an all-encompassing "bible"-type reference, this book is designed with a focus on real-world practicality -to help you quickly get up to speed and become productive with the language.
* Understand why kdb+/q is the ideal solution for high-frequency data
* Delve into "meat" of q programming to solve practical economic problems
* Perform everyday operations including basic regressions, cointegration, volatility estimation, modelling and more
* Learn advanced techniques from market impact and microstructure analyses to machine learning techniques including neural networks

The kdb+ database and its underlying programming language q offer unprecedented speed and capability. As trading algorithms and financial models grow ever more complex against the markets they seek to predict, they encompass an ever-larger swath of data -- more variables, more metrics, more responsiveness and altogether more "moving parts."

Traditional programming languages are increasingly failing to accommodate the growing speed and volume of data, and lack the necessary flexibility that cutting-edge financial modelling demands. Machine Learning and Big Data with KDB+/Q opens up the technology and flattens the learning curve to help you quickly adopt a more effective set of tools.

Product details

• Authors: Paul Bilokon, Paul A Bilokon, Paul A. Bilokon, Paul A. Novotny Bilokon, Frederic Deleze, Aris Galiotos, Aris et al Galiotos, J Novotny, Ja Novotny, Jan Novotny, Jan Bilokon Novotny
• Publisher: Wiley, John and Sons Ltd
• Languages: English
• Product format: Hardback
• Released: 31.01.2018
• EAN: 9781119404750
• ISBN: 978-1-119-40475-0
• No. of pages: 640
• Series: Wiley Finance; Wiley Finance Editions; Wiley Finance; Wiley Finance Editions
• Subjects: Social sciences, law, business > Business > Business administration; Big Data, Financial Engineering, Finance & Investments, Finanz- u. Anlagewesen, Finanztechnik