Comprehensive Data Analysis Process

This document outlines a thorough, step-by-step process for analyzing marketing risk data with a focus on data lineage and transparency. The guide will provide you with instructions on how data is sourced, processed, and validated, ensuring that it reaches its optimal state for analytical purposes. The process involves automated workflows, data normalization, validation, and transformation, ultimately leading to the selection of a golden copy for use. We will also explore the ways in which users can interact with this data for deeper insights and analysis.

Step 1

Understand the importance of lineage and transparency in marketing risk data. This involves knowing the data's origin and the transformations it undergoes to reach its final state.

Step 2

Normalize the data and apply validation rules, derivations, or transformations. Capture all these processes to make the information accessible to end users.

Step 3

Conduct a demonstration focusing on accessing reference data such as curve data and individual tenor points. This includes detailed information about issuers, classifications, or ratings.

Step 4

Review information about contributors, including liquidity metrics and validation lineages. Lastly, examine time series and analytical datasets for further insights.

Step 5

Create a trustworthy time series dataset to enhance transparency and lineage. Initiate the process by setting up your screen for analysis.

Step 6

Examine an interest rate curve representation for a specific date. Automated workflows have compiled this golden curve from various sources, ensuring smooth consistency.

Step 7

Each line on the curve represents a vendor, with one selected as the golden curve. Utilize the visual tools available to investigate the data further with ease.

Step 8

Quickly examine the visual lineage of the data and explore individual tenors in detail.

Step 9

Select a tenor point to view the extensive data captured, including price types, quote types, and associated commentary or changes.

Step 10

Access extensive data, including actual values and the rules applied, from the origin point to the vendor file.

Step 11

View the status and results of rules applied to the data, including deviations and thresholds. Explore liquidity measures such as bid-offer spreads and contributor numbers.

Step 12

Gain visibility into the trustworthiness of the market data. Review the contributors and vendors involved in creating the golden copy.

Step 13

Rank vendors according to the rules applied to select the golden rule.

Step 14

Access various options by right-clicking on an individual point, including exception resolution methods and contributor analysis for visual scaling.

Step 15

Identify the golden candidate and its placement on the visual scale within the screen interface.

Step 16

Ensure lineage and audit information capture manual and system-generated changes, derivations, and store them in the database.

Step 17

Export the audit report and parameterize it by specific dates. A pre-downloaded file can be used for reference.

Step 18

Review the summary screen in an Excel spreadsheet for a specific date, detailing system and user activities by asset types.

Step 19

Drill down into details for manual changes and metadata for maker and checker users, including associated commentary.

Step 20

Access underlying reference data through right-click, navigating to the curve's definition for detailed insights.

Step 21

Explore the construction details of a curve, such as CHF curve, with its underlying index and specific points.

Step 22

Navigate to specific tenor points to view terms, conditions, calendars, and other related information.

Step 23

Utilize this data for downstream systems, aiding in processes like bootstrapping and surface smoothing.

Step 24

Review the daily market data process and how changes are recorded, exploring deeper into the time series dataset.

Step 25

Access the time series screen to view CHF IRS curve details and past data over a decade, with a color-coded Excel grid view.

Step 26

Investigate additional price details, including status and data source, for a thorough analysis.

Step 27

Capture changes in price and details of failed validations within the system's data columns.

Step 28

Visualize and correlate data points, performing ad hoc checks and leveraging analytical functions.

Step 29

Utilize system-generated checks or parameterize them, conduct standard deviation or outlier checks as needed.

Step 30

Choose from various analytical functions, like percent return, to analyze liquidity horizon impacts.

Step 31

View generated series and plot them on a chart using the chosen analytical function for insights on data changes.

Step 32

Analyze the system-generated data for time series shock impacts over the curve, identifying outliers and data consistency.

Step 33

Implement advanced calibrations using Python integration for analysis like PCA and other visual features.

Step 34

Plot data points to calculate correlations, highlighting any outliers for better understanding of the dataset.

Step 35

Customize Python scripts to extend analysis to other curves, ensuring comprehensive data handling.

Step 36

Explore additional analyses like PCA or bootstrapping with available tools for more in-depth evaluation.

Step 37

Conclude the agenda and open the floor for any questions or further clarifications needed on the process.