IDEA 1R algoritm One rule false up true Apriori algoritm false up true Association rules false up true Average, median and mode false up true Bar, Pie and Line chart false up true Binning false up true Box and Whisker diagram false up true Classification false up true Cluster-based local outlier factors (CBLOF) false up true Clustering false up true Correlation false up true Cross tab false up true Data analysis visualisations false up true Data analytics false up true Data reduction false up true Decision tree false up true Feature extraction false up true Forward selection and backward elimination See https://en.wikipedia.org/wiki/Stepwise_regression false up true Histogram false up true Inter Quartile Range false up true Inverted document frequency false up true K-means false up true K-nearest neighbors false up true Liniaire regression false up true Logistic regression false up true n-grams false up true Naive Bayes false up true Outlier detection false up true Pattern recognizion false up true Regression false up true Rule based classification false up true Scatter plot Scatterplot https://en.wikipedia.org/wiki/Scatter_plot false up true Standard deviation false up true Statististical functions false up true Term Frequency false up true Text analysis false up true Timeseries false up true Z-Score false up true Hadoop Open source suite voor big data oplossingen Hive Hadoop component voor SQL achtige bevraging Mahoot Oozie Pig Sqoop Storm Dataverwerking voor met name realtime verwerking bij een hoge velocity Big Data SQL Behandelen van big data als een tabelstructuur en werkt daarbij met een SQL achtige querytaal false up true Data filtering and selection false up true Data processing and transport ETL achtige tool voor transport van Hadoop naar gestructureerde dataopslag zoals RDBMS en DWH false up true Data Workflow false up true Machine Learning false up true Scriptng false up true Data visualisations false up true Geo viewers false up true Graphs and statistical visualisations false up true Heatmaps false up true Hybrid visualisations false up true Networks graphs and trees false up true Time series and trends false up true Data acquisition Selecting data sources on information content and data qualities false up true Data analysis Analysing the cleaned, aggregated and filtered datasets false up true Data governance Data governance activities and application functions layers for maintaining qualities of the datasets false up true Data management false up true Data processing Processing activities for filtering, cleaning and transformation of datasets false up true Data security Application functions and activities for securing datasets and storage functionalities false up true Data sources Not part of a data architecture but the input of the architecture, it is the raw material of a data pipe and therefore an essential part false up true Data storage Storage of primary and temporary datasets for usage in higher layers of the architecture false up true Data utilisation Usage of the processed data in work processes in the consumer organisation false up true Data visualisation Visualisation of the data or information for interpretation by analists and users false up true Consumer Consumer of data, in most situation the consumer gets access to processed data (cleaning, filtering, transforming) based on a standardized model Data source Description of the data sources in general. Is an aggregation of specific data source types. This generic datasource is added to the model to make associations possible with generic requirements and principles Big Data Platform Logical application composite for implementation of Big Data functionalities false up true Enterprise Data WareHouse (EDWH) This is the logical application of a data warehouse not the component. It describes the functionality of storing, modeling updating and retrieving enterprise data based on relational data including the history of the data false up true Serial Integration of DWH and BDP Serial implementation of Big Data and EDWH functionality where the Big Data functionality is consumed by the EDWH functionality false up true Virtualisation Logical application for virtualization of data implemented in a standardized object or data model false up true Virtualisation Integration of DWH and BDP Implementation of EDWH and Big Data functionality extracted via a virtualisation function. This virtualization acts as an encapsulation layer and API for the consumer applications false up true Analytics Engine Functionality for analysing the transformed and standardized data in a data analytics tool. false up true Appliance Integration of DWH and BDP Appliance configuration where the EDWH and Big Data functionality is tightly integrated and accessible via various API functions false up true Cluster Manager Management of clusters in a distributed or cloud based infrastructure false up true Compression Engine Big data sets can be voluminous so compressing has advantages for storage and/or processing false up true Coordination Engine Coordination of automated tasks for example in batch processes and mapreduce. false up true Data Governance Manager Register function for the governance of data sets in relation to consumers, business owners and qualities. false up true Data Transfer Engine Transformation and transfer functionality from data sources to storage, processing and usage. false up true Parallel Integration of DWH and BDP Parallel integration of EDWH and Big Data functionality. Often between these functions an interconnect function is implemented false up true Processing Engine Processing of data for transformation, cleaning and filtering of data false up true Productivity Portal Portal functionality to give users and administrator access to the other logical functionalities in a Big Data environment. false up true Query Engine Retrieval and querying function on the transformed and standardized datasets false up true Resource Manager Management function for the various resources in a Big Data environment false up true Security Manager Administration and registration of security concepts in a Big Data environment. false up true Serialization Engine Serialization and deserialization of proprietary datasets to standardized stored datasets false up true Storage Device Storage function of big data sets. false up true Visualization Engine Visualisation of query and analytics results to data scientists and end users. false up true Workflow Engine Functionality to design workflows to automatically perform transformations and cleaning of datasets. false up true Analytical Sandbox* The Analytical Sandbox is a standalone solution environment capable of capturing and processing large quantities of data from multiple sources in order to perform analytics in isolation the enterprise data warehouse. false up true Application Enhancement* The Application Enhancement compound pattern represents a solution environment where the Big Data platform is used to ingest large amounts of data in order to calculate certain statistics or execute a machine learning and then to feed results to downstream systems. false up true Automated Data Replication and Reconstruction How can large amounts of data be stored in a fault tolerant manner such that the data remains available in the face of hardware failures? false up true Automated Data Sharding How can very large amounts of data be stored without degrading the access performance of the underlying storage technology? false up true Automated Dataset Execution How can the execution of a number of data processing activities starting from data ingress to egress be automated? false up true Automated Processing Metadata Insertion How can confidence be instilled in results whose computation involves applying a series of processing steps in a Big Data environment? false up true Batch Big Data Processing* The Batch Data Processing compound pattern represents a solution environment capable of ingesting large amounts of structured data for the sole purpose of offloading existing enterprise systems from having to process this data. false up true Big Data Pipeline* The Big Data pipeline compound pattern generally comprises multiple stages whose objectives are to divide complex processing operations into down into modular steps for easier understanding and debugging and to be amenable to future data processing requirements. false up true Big Data Processing Environment* The Big Data Processing Environment represents an environment capable of handling the range of distinct requirements of large-scale dataset processing. false up true Big Data Transformation* Data transformation represents a solution environment where the Big Data platform is exclusively used for transforming large amounts of data obtained from a variety of sources. false up true Big Data Warehouse* The Big Data Warehouse represents a solution environment where a Big Data platform is used as a data warehouse capable of storing both structured and unstructured data online. false up true Canonical Data Format How can the same dataset be consumed by disparate client programs? false up true Centralized Access Management How can access to resources within a Big Data platform be managed efficiently and consistently? false up true Centralized Dataset Governance How can a variety of datasets stored in a Big Data platform be governed efficiently and in a standardized manner? false up true Cloud Based Big Data Processing How can large amounts of data be processed without investing in any Big Data processing infrastructure and only paying for the amount of time the processing resources are actually used? false up true Cloud Based Big Data Storage How can large amounts of data be stored without investing in any Big Data storage infrastructure and only paying for the used storage space? false up true Complex Logic Decomposition How can complex processing tasks be carried out in a manageable fashion when using contemporary processing techniques? false up true Confidential Data Storage How can data stored in a Big Data solution environment be kept private so that only the intended client is able to read it? false up true Data Size Reduction How can the size of the data be reduced to enable more cost effective storage and increased data movement mobility when faced with very large amounts of data? false up true Dataset Decomposition How can a large dataset be made amenable to distributed data processing in a Big Data solution environment? false up true Dataset Denormalisation How can a dataset, where related attributes are spread across more than one record, be stored in a way that lends itself to distributed data processing techniques that process data on a record-by-record basis? false up true Direct Data Access How can large amounts of raw data be analyzed in place by contemporary data analytics tools without having to export data? false up true Fan-in ingress How can data arriving from multiple instances of a data source be collected in an efficient manner? false up true Fan-out ingress How can streaming data arriving from a single source be simultaneously forwarded to multiple destinations without any delay? false up true File-based Sink How can processed data be ported from a Big Data platform to systems that use proprietary, non-relational storage technologies? false up true File-based Source How can large amounts of unstructured data be imported into a Big Data platform from a variety of different sources in a reliable manner? false up true High Velocity Realtime Processing Storing large amounts of data, arriving at fast pace, as a dataset and processing it in a batch manner incurs processing latency, causing a delay before analysis results become available. false up true High Volume Binary Storage How can a variety of unstructured data be stored in a scalable manner such that it can be randomly accessed based on a unique identifier? false up true High Volume Hierarchical Storage How can large amounts of non-relational data that conforms to a nested structure be stored in a scalable manner so that the data retains its internal structure and sub-sections of a data unit can be accessed? false up true High Volume Linked Storage How can very large datasets comprising entities that are connected together be stored in a way that enables efficient analysis of such connected entities? false up true High Volume Tabular Storage How can large amounts of non-relational data be stored in a table-like form where each record may consist of a very large number of fields or related groups of fields? false up true Indirect Data Access How can traditional BI tools access data stored in Big Data storage technologies without having to make separate connections to these technologies? false up true Integrated Access How can users seamlessly access enterprise IT systems and a Big Data platform’s resources without having to authenticate twice? false up true Intermediate Results Storage How can the complete re-execution of a series of processing steps be avoided in case an error occurs partway through? false up true Large Scale Batch Processing How can very large amounts of data be processed with maximum throughput? false up true Large Scale Graph Processing How can algorithms that require iterative processing of a large dataset that may or may not contain connected entities be processed in an efficient and timely manner? false up true Online Data Repository* The Online Data Repository compound pattern represents a solution environment where the Big Data platform’s inexpensive storage is used to store data from internal and external data sources in its raw form available for consumption by any downstream application. false up true Operational Big Data Store* The Operational Data Store (ODS) compound pattern represents a solution environment such that the Big Data platform’s inexpensive NoSQL storage can be utilized as a traditional ODS where transactional data from operational systems across the enterprise is collected for operational BI and reporting. false up true Poly Sink* The Poly Sink compound pattern represents a part of a Big Data platform capable of egressing high-volume, high-velocity and high-variety data out to downstream enterprise systems. false up true Poly Source* The Poly Source compound pattern represents a part of a Big Data platform capable of ingesting high-volume and high-velocity data from a range of structured, unstructured and semi-structured data sources. false up true Poly Storage* The Poly Storage compound pattern represents a part of a Big Data platform capable of storing high-volume, high-velocity and high-variety data. false up true Processing Abstraction How can different distributed processing frameworks be used to process large amounts of data without having to learn the programmatic intricacies of each framework? false up true Random Access Storage* The Random Access Storage compound pattern represents a part of a Big Data platform capable storing high-volume and high-variety data and making it available for random access. false up true Realtime Access Storage How can large amounts of data be accessed instantaneously without any delay? false up true Realtime Data Processing* false up true Relational Sink How can large amounts of processed data be ported from a Big Data platform directly to a relational database? false up true Relational Source How can large amounts of data be imported into a Big Data platform from a relational database? false up true Streaming Access Storage* false up true Streaming Egress How can processed data be exported in realtime from a Big Data platform to other systems? false up true Streaming Source How can high velocity data be imported reliably into a Big Data platform in realtime? false up true Streaming Storage How can large datasets be accessed in a way that lends itself to efficient processing of data in batch mode? false up true Unstructured Data Store* false up true Map MapReduce https://en.wikipedia.org/wiki/MapReduce Reduce Data processing false up true Amazon DynamoDB Amazon SimpleDB Cassandra Clustrix CouchDB H-Store HBase InfinitGraph MongoDB Neo4J NuoDB ObjectDB Objectstore OrientDB Redis Riak Terrastore TokuDB Column family https://en.wikipedia.org/wiki/Column_family false up true Data storage false up true Distributed Filesystem false up true Document store false up true Document stores Graph database false up true Key Value store false up true NewSql false up true NoSQL https://en.wikipedia.org/wiki/NoSQL false up true Object database false up true RDBMS false up true Dublin Core JSON LOB CLOB, BLOB etc Metadata OWMS Productspecific metadata Denk aan de eigenschappen in docs, of image headers Semi structured data SQL tabellen Structured data Tekst Toepassingsprofiel UBL Unstructured data XBRL XML XML obv XSD TEZ YARN Data processing false up true Resourcemanagement false up true DataObject1 AFAS AFAS HRM Microsoft Dynamics Moodle Human Resource Management Register for training Trainer selection and accreditation Training planning Datatype evaluation relational database Introduce Geo extension in database Map application datamodels to logical datamodel Connection requirements Requirements for consumers when connection to a service or service implementation (interface) Data Qualities See package with the description of these DaMa qualities External consumers External consumers of the (standardized) data and master data produced by the TDP solutions. For external consumers extra requirements are necessary for example with aspects like security, privacy and governance. File Data saved in a file for example semi structured like: XML, XLS, JSon, edifact etc Unstructured like Word, Text etc. Internal consumers Consumers in the TenneT organisation of the standardized TDP data products Message XML message as a stream or file imported in a message transformation handler with the transformation function. The message should be structured and described for example with a XSD definition NoSql NoSQL is a new dataplatform for the implementation of semi structured data. There are various NoSQL database types like column, name value, document or graph databases Relational databases Data entities saved in a relational (staging) database Society general Extraction of data to the society (unknown consumers like customers, citizens etc) Standardized object or datamodel Standardized data or business object. Models are based on (open) standards like CIM Streaming data Data that is received as a data stream. For example smart meter data or data received from push webservices in a continuous manner Application integration Different application integration types for application - application integration Well known integration types ares: - Webservices - Filetransfer and FTP etc See for details the description in the TenneT architecture provided Data access Generic description of the interface for the extraction of standardized data for the various consumers. provided Database integration Integration from data storage to data storage. For example the relational database integrations like - views, - materialized views - database logic in packages and stored procedures. - ETL: implementations provided ETL processing Complex transformation between different database structures provided Portals and widgets Web user interface for the display and maintenance of data within a portal application (for example Sharepoint) provided User interface Different graphical user interfaces like: - Reports - Analytical tools - Geospatial viewers - Forms - Portals and widgets e.g. provided File transformation Transformation of a datafile (mostly semi structured) for example an XML file with an internal datamodel that has to be transformed to the standardized model. false up true Generic Dataset Access This is a logical service for the publication of a certain standardized dataset. In the current implementation of TDP plateau 1 list of differences. This logical dataservice is implemented in one or more technical interfaces like user interfaces or XML webservices Governance processes Data governance processes, mainly focused on the realization of a data target with acceptable data qualities false true Implementation processes Implentation and operations processes for developing and maintaining the implemented data pipes false true Message transformation Transformation of a XML stream (mostly semi structured) with an internal or specific datamodel that has to be transformed to the standardized model in the data target. false up true NoSql transformation Transformation of a NoSQL datasource to a standardized data target false up true Register extraction Function for the registration and extractions of governance aspects of the datasets published over the logical services. For example data qualities, connection requirements and the standardized object or information model false up true Relational transformation Transformation of relational data entities (stored in a relational database) to the standardized data target. false up true Streaming transformation false up true Transformation Transformation function from the specific source data model to the standardized target data model. This can be a model or a protocol transformation. Often this transformation is divided in sub steps and these sub steps will be analyzed and modeled in the architecture of a specific data pipe implementation and components false up true XML Transformation Transformation of an XML stream or file to a relational model false up true Stream transformation false up true Availablity Availability means that information must be available when it is needed Low (A1) The disruption of access to or use of information will do no or very little harm to TenneT as a company, and no harm to the continuous and secure supply of electricity to society Medium (A2) The disruption of access to or use of information could clearly harm TenneT, but do no harm to the continuous and secure supply of electricity to society High (A3) The disruption of access to or use of information could do considerable harm to TenneT, but not directly to the continuous and secure supply of electricity to society Very High (A4) The disruption of access to or use of information could (very) seriously and acutely harm TenneT as a company and/or directly harm the continuous and secure supply of electricity to society Confidentiality This means the prevention of the disclosure of information to unauthorised individuals and / or systems Low (C1) Unauthorized disclosure of information will do no or very little harm to TenneT as a company, and no harm to the continuous and secure supply of electricity to society Medium (C2) Unauthorized disclosure of information could clearly harm TenneT, but do no harm to the continuous and secure supply of electricity to society High (C3) Unauthorized disclosure of information could do considerable harm to TenneT, but not directly to the continuous and secure supply of electricity to society Very High (C4) Unauthorized disclosure of information could (very) seriously and acutely harm TenneT as a company and/or directly harm the continuous and secure supply of electricity to society Integrity Integrity means that information is reliable to the degree necessary Low (11) The unauthorized modification of information will do no or very little harm to TenneT as a company, and no harm to the continuous and secure supply of electricity to society Medium (I2) The unauthorized modification of information could clearly harm TenneT, but do no harm to the continuous and secure supply of electricity to society High (I3) The unauthorized modification of information could do considerable harm to TenneT, but not directly to the continuous and secure supply of electricity to society Very high (I4) The unauthorized modification of information could (very) seriously and acutely harm TenneT as a company and/or directly harm the continuous and secure supply of electricity to society Accurateness Accurateness refers to the degree of which a data entity displays reality. Accurateness can be decided by comparing a data entity with the entity in reality. An example is a a difference between a mailing list of clients and the true clients of an organisation.. 101 Actuality Degree of which a data entity display the current situation of reality. Good examples are deceased people that received a letter from an older data set. Replication of data is often a source of low actuality 101 Completeness This refers to the degree in which certain attributes are present within a data entity. In addition to that the completeness also counts for a certain set of entities (rows) within a data set always being present. For example a person could only have the quality name, yet also consist of nickname, first names, surname and maiden name. In the last case there is a higher completeness 101 Consistancy This refers to the fact that the one data set of a certain entity is equal to another data set. In other words a data entity is always the same regardless of the source. An example of a low consistency is when there are differences between data sets of the same entities originating from different sources. Replication of data is often a source of low consistency. 101 Precision Degree of detail in which a data entity displays reality. For example this refers to the precision of numbers and such. Storage of numbers and dates can be insufficiently accurate because rounding is needed in storage. Domains in features can also have insufficient precision (such as a Dutch postal code in an international data storage.) 101 Privacy For some data entities access control (authorisation and authentication) or monitoring of use is needed. Take for example requirements that are placed on the access of confidential data. In the GBA there are multiple levels of confidentiality. As such queries of officials are logged and displayed to the civilian yet for investigating officers they are logged and not displayed. 101 Reasonableness Mainly refers to expectations within a certain operational context. Take for example the accepting of a lower performance during peak loads or having to wait a long time on a result set of archived data entities. 101 Referential integrity This is the situation where referrals from one data entity always correctly refer to the related data entities. Examples are double keys in a data set which makes the connected entities unable to decide which entity is older. Also dangling references or floating references where the parent no longer exists belong to this. 101 Timeliness Is a data set available on time within the set expectations. It is the difference between the moment of need and availability. For example requesting data in a Call Center. In this situation waiting five minutes for a response is not acceptable. 101 Uniqueness Uniqueness of a data entity is focused on the fact that there are no other data entities with the same data. An example from practice was a twin with the same initials, surname and birth date. A distinction could not be made due to the completeness being to low. 101 Validity This is the degree in which a data entity meets the desired format in storage and exchange. Take for example the domain yet also the datatype of the attributes of a data entity. Within chain exchange for example this is of the highest importance. Nobody wants to find out that at the end of the chain the data is not valid. This needs to be detected in an early stage. Everyone knows the examples from the past with web applications where after sending a message “invalid data” showed up without further explanation. 101 Backoffice Management Trainee Trainer Classroom training Employee An employee is a person that has a function for supporting the trainees in our training company Online training Person Registration A registration is a record in which is described that a trainee particates in a training Trainee A trainee is a person participating in one or more trainings of our company Trainer A trainer is an employee who participates in the education activities of our company Training A training is an offer to the market of knowledge transfer about a specific area or subject Training Session A training session is part of a training in which the trainer and trainee exchange information and knowledge Asset management Clients and Markets Contractors Government agencies Local and (inter) national government agencies that use standardized datasets Sector representation Other (collegue) System Operations Data target Data target based on a standardized data model. Based on this model and the included data entities one or more interfaces can be implemented for the consumers. DataMart Registration Training XML based webservice XML File File with a validated XML content that can be processed to a structured model and pessed to an ETL process Compliancy with legislations Strategic Complliancy with GDPR Employee satisfaction Quality Quality of registration Quality of reporting Stakeholder satisfaction Trainee satisfaction Trainer Satisfaction A central design authority governs the data sources and the data integration layers A central design authority governs the data sources and the data integration layers and design and technology decisions on these layers are with this design authority only Ability to abstract on the data sources via implementation of specific connectors Connection and integration of data sources of different types and ability to abstract on the data sources via implementation specific connectors, e.g. for smart meters, wind craft, etc. Cloud and on-premise deployment options Cloud and on-premise deployment options with ability to migrate from cloud to on-premise (cloud is preferred for short term temporary solution) Compliancy with law and legislation Compliancy with privacy legislation (GDPR) Customizable self-defined API to access platform services Data analytics based on data integration layer provides real-time performance and scalability next to batch processing Data analytics based on data integration layer provides real-time performance and scalability next to batch processing Data entities have a common definition Data is a valuable asset Data ownership/stewardship and control of the data in the data integration layer. Data ownership/stewardship and control of the data in the data integration layer. Central management around gathering, integration, and control of data is with the organisation. The single source of truth should include the most possible finest granularity of data. Data sources on heterogeneous environments Data sources can be run on heterogeneous environments, several deployment models (on-premise, cloud, etc.) and can be internal and/or external Dedicated access via authorization and authentication mechanisms on data integration layer Distinguish and decouple classic BI, near real-time BI and real-time OI and prioritize OI real-time requirements. Distinguish and decouple classic BI, near real-time BI (Business Intelligence), and real-time OI (Operations Intelligence) and prioritize OI real-time requirements. Flexibility of integrating different parallel data sources and providing scalable real time data Flexibility in terms of integrating different parallel data sources and providing integrated real time data in a scalable way Flexible customizable analytics functions Flexible customizable analytics functions (self-defined and customizable algorithms) covering rules-based analytics and machine learning For all data entities is meta data available When data is described in multiple dimensions data becomes more reuasbale and generic. Therefore context must be added to the data entities Functional modules and data integration layer can be hosted on different physical environments Functional modules and data integration layer can be hosted on different physical environments, e.g. the data integration layer is on-premise while the functionalities are plugged in analytics cloud solutions that work on top of the data Functionalities plugged in on top of dedicated data integration layer Functionalities can be developed and plugged in on top of dedicated data integration layer as functional modules or can be sourced from 3rd parties (make or buy decision) Layered Architecture Layered architecture that separates data sources, data integration, data analytics, and presentation layers Prioritized capabilities are incubated in a learning environment and OTAP Prioritized capabilities are incubated in a learning environment to be transitioned effectively in a development, testing, and production environment at a later stage Processes are supported by data fit for purpose Scalability with the number of connected devices to provide a basis for real-time analytics The functional data analytics layer and the data integration layer are coupled by a standard information model The functional data analytics layer and the data integration layer are coupled by a standard information model that is binding and under governance of the central design authority Virtualized infrastructure Virtualized infrastructure to allow easy migration in on-premise and also cloud environments Accurateness This refers to the correctness of the functions, an ATM may provide a cash dispensing function but is the amount correct? Adaptability Characterizes the ability of the system to change to new specifications or operating environments. Analysability Characterizes the ability to identify the root cause of a failure within the software. Attractivity Satisfaction of latent user desires and preferences, through services, behavior and presentation beyond actual demands. Availability Amount of time the product is available to the user at the time it is needed. Changability Characterizes the amount of effort to change a system. Clarity Clarity of making the user aware of the functions the software can perform. Compliance Where appropriate certain industry (or government) laws and guidelines need to be complied with, i.e. SOX. This subcharacteristic addresses the compliant capability of software. Conformance Similar to compliance for functionality, but this characteristic relates to portability. One example would be Open SQL conformance which relates to portability of database used. Degradability Effort needed to reestablish the essential functionality after a breakdown. Efficiency This characteristic is concerned with the system resources used when providing the required functionality. The amount of disk space, memory, network etc. provides a good indication of this characteristic. As with a number of these characteristics, there are overlaps. For example the usability of a system is influenced by the system''s Performance, in that if a system takes 3 hours to respond the system would not be easy to use although the essential issue is a performance or efficiency characteristic. Explicitness Clarity of the software product’s status (progression bars, etc.). Fault tolerance The ability of software to withstand (and recover) from component, or environmental, failure. Functionality Existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs. Generic requirements Description of generic requirements for all the transformation functionalities Helpfullness Availability of instructions for the user on how to interact with the software Installability Characterizes the effort required to install the software. Interoperability A given software component or system does not typically function in isolation. This subcharacteristic concerns the ability of a software component to interact with other components or systems. Learnability Learning effort for different users, i.e. novice, expert, casual etc. Maintainability The ability to identify and fix a fault within a software component is what the maintainability characteristic addresses. In other software quality models this characteristic is referenced as supportability. Maintainability is impacted by code readability or complexity as well as modularization. Anything that helps with identifying the cause of a fault and then fixing the fault is the concern of maintainability. Also the ability to verify (or test) a system, i.e. testability, is one of the subcharacteristics of maintainability. Managability Effort needed to (re)establish the software’s running status. Maturity This subcharacteristic concerns frequency of failure of the software. Operability Ability of the software to be easily operated by a given user in a given environment. Portability This characteristic refers to how well the software can adopt to changes in its environment or with its requirements. The subcharacteristics of this characteristic include adaptability. Object oriented design and implementation practices can contribute to the extent to which this characteristic is present in a given system. Recoverability Ability to bring back a failed system to full operation, including data and network connections. Reliability Once a software system is functioning, as specified, and delivered the reliability characteristic defines the capability of the system to maintain its service provision under defined conditions for defined periods of time. One aspect of this characteristic is fault tolerance that is the ability of a system to withstand component failure. For example if the network goes down for 20 seconds then comes back the system should be able to recover and continue functioning. Replacability Characterizes the plug and play aspect of software components, that is how easy is it to exchange a given software component within a specified environment. Resource behaviour Characterizes resources used, i.e. memory, cpu, disk and network usage. Reusability Potential for complete or partial reuse in another software product. Security This subcharacteristic relates to unauthorized access to the software functions. Specific requirements Specific requirments for specific application functions like specific transformations Stability Characterizes the sensitivity to change of a given system that is the negative impact that may be caused by system changes. Suitability This is the essential Functionality characteristic and refers to the appropriateness (to specification) of the functions of the software. Testability Characterizes the effort needed to verify (test) a system change. Time behaviour Characterizes response times for a given thru put, i.e. transaction rate. Tracability Ease of verifying correctness of data processing on required points. Understandability Determines the ease of which the systems functions can be understood, relates to user mental models in Human Computer Interaction methods. Usability Usability only exists with regard to functionality and refers to the ease of use for a given function. For example a function of an ATM machine is to dispense cash as requested. Placing common amounts on the screen for selection, i.e. $20.00, $40.00, $100.00 etc, does not impact the function of the ATM but addresses the Usability of the function. The ability to learn how to use a system (learnability) is also a major subcharacteristic of usability. User friendliness User-friendliness in sytem usage Employee Legislator Trainee Trainer Application integration for publication Application interfaces for application to application integration. This includes the fully automated system integration processes, ETL implementations and the automated and managed file transfer. provided Application integration for registration Application integration interfaces for machine 2 machine integration provided Change Data Capture Dashboard provided Data entry screens Modification of data via (webbased) screens. This includes geo based screens provided Database link Specific implementation of an interface based on a database link concept provided Export Office documents Unmanaged and user driven export of asset data to various office documents including CSV and XML. provided File transfer File transfer for example via FTP or the usage of a file share, mostly applied for batch data processing provided Fileshare Forms and pages Interface to standardized data via (web) forms and pages provided Forms and reports Forms and reports interfaces in which users can retrieve asset data in an interactive manner provided FTP GEO ETL ETL implementations specific for the transfer of geo data (for example FME) provided Geo ETL to consumers provided Geo viewer Geo viewer intereface for the retireval of asset data in combination with geographical selection and representation functionality provided Geo webservices Message based transfer of geo data for example based on WMS/WFS/WMTS or GML over webservices provided Geo webservices to consumers provided Graphs provided Import Office documents Import and manual transformation of office documents like excel sheets etc. provided JSon/REST Managed File Transfer Fully automated file transfer without manual handling of the data provided Managed File Transfer to consumers provided Packages and stored procedures Database functionality for the transformation and extraction of data via an interface provided Relational ETL ETL processes for the transformation of data from relational to relational databases or structured files provided Relational ETL to consumers provided Report Reporting interface provided SFTP SOAP-XML webservices SOAP/XML webservices including eventually REST services provided SOAP-XML webservices to consumers provided Unmanaged File Transfer Transfer of files that are manually handled and eventually modified before implementation in the data storage provided User Interfaces for Publication Publication of asset data to various user interface types provided User Interfaces for registration User interfaces for human interaction with the data stored in the data registration provided Views and materialized views Specific relational database interface implementation provided Webservice Webservice interface for application to application integration. This can be implemented in a service bus environment. Mostly applied in event based data processing provided XML/SOAP Consuiming systems KA DWH This is a DWH database that acts as data target for the various datapipes. Currently this is the KA-DWH component but for different project implementations there are also other targets possible. Oracle Workflow and Scheduler (11) Tool for workflow and scheduling of BI tasks within datapipes Automated Data Validation Machine based validation of the data entered in the system but also based on timely intervals within the stored data. This is implemented in a kind of rule engine. false up true Data Model Transformation Transformation of the data as stored in the asset data registration and transformation to a model for messages (CGMES, data marts or file formats). false up true Data Processing Processing of data before it is stored in the data storage and when the data is retrieved from the storage false up true Data Protocol Transformation Transformation of data to various protocols, for example for the implementation of webservices, REST but also to a format readable for reports false up true Data Publication Functionality to publish the data as stored in the data registration for use of various consumer types. false up true Data Register MDM like function that aggregates all the data in a standardized data model false up true Data Registration Actual registration function with storage and processing of the master data false up true Data Storage Physical storage of the data in a (relational) database. false up true Data Validation Validation of data based on the data entered in the system and eventually on the existing data stored in the master data registration. It is essential that all the data received from the acquisition/registration service is processed in these functions before it is stored in the data register false up true ERP ERP (or EAM) functionality for the registration of data within a number of highly standardized processes using asset data. false up true Extract Extract functionality for receiving data from the source systems. This has a XML format and the source is either a webservice or a XML file. The data is extracted and prepared to be transformed false up true Geo Geo functionality and register in combination with the extraction of asset data in combination with geo data and maps false up true Load Load functionality for transferring the data to a data target and load it there. This can either be an ETL process or the implementation in a message handler false up true Logical modeling The logfical or conceptual model is abstract in nature and is therefore independent from the implementation platform. In most cases this model must be readable for non IT stakeholderrs. false up true Manual Data Validation Validations when data is entered by hand (via screens etc) false up true Master Data Management and Governance Application functionality that support data management and governance processes. Think about data quality processes, data ownership and data security policies etc. false up true Master Data Modelling and Meta Modelling Data modeling and meta modeling for example models based on UBL and CIM. These models are used for the storage of the data but also for the data integration, - transformation and - validation false up true Modeling For the transformation of source to target datamodels a description of the data elements has to be modeled. false up true Other Data Producers Other MDM data producing application functions like the delivery of office files etc. false up true Physical modeling Physical model for the description of a technical implementatin model. This is platform specfic and is used for the creation of SQL DDL and ETL scripts and eventually for technical documentation. false up true Register Data Consumption Abstract architectural entity for the register data consumers, all user interfaces and data integrations are an master data consumer false up true Register Data Consumption via Integration Application and database integration like webservices, file transfer, database links, views etc. In a later phase we will model the various integration methods and model these in detail false up true Register Data Consumption via User Interface All kinds of user interfaces in which an user can consume data via a (graphical) user interface. Examples of user interfaces are reports, forms, portals graphs, geoviews etc. false up true Register Data Production Logical application function for the storage and transformation of master data in various source function and the data register function false up true Transform Transformation functionality for transforming the data content from a (application) specific datamodel to a generic datamodel based on a standardized datamodel (see the service diagrams). This can be impllemented either in an ETL transform process or in a message handler for example in a service bus. false up true Workflow management Workflow management for the scheduling and iteration of transformation tasks false up true Data Acquisition/Registration Service Logical service for the import of data from other systems and sources Register Data Publication and Integration Service Logical services that publish the data from the register to various register data consumers Access Distribution Access Security Authentication Authorisation Data Zone Protection Facility Monitoring Identification Permission management Data owner Data owner is the one who decides on the data as it is applied within business objects. The data owner makes decisions about the data quality, the security aspects, but also about the structure of the data within a business object. Data steward Is the person who is delegated by the data owner to take over certain responsibilities and tasks of the owner. These tasks are often at a more detailed level, hence the link with data objects within ArchiMate instead of Business objects Data user Different user types that have a relationship with business and data objects within the ArchiMate modeling domain Kwaliteit Qualities are worked out in the IDEA project as a requirement (that is a relatively arbitrary choice because a constraint is of course also an option) Business object Is an ArchiMate concept that is used to visualize the different links between the business objects at the business level. That is actually building a so-called business glossary. In addition, these entities can be used to visualize the links with other concepts within ArchiMate. For example, the links to business processes, functions and services are examples of this. 73 Business role ArchiMate concept for describing (logical) actors) in a business layer that are involved in, for example, business processes and functions. Please keep in mind that roles can be performed by multiple persons an a person can perform multiple roles. 73 Data object Data object is an ArchiMate concept that on the one hand can be linked to a number of ArchiMate concepts so that aspects such as quality, requirements. But also where is the dataset used and by whom, which application has a relation with this dataset etc. If necessary, you could link to the infrastructure from the data object. At this moment, this is not yet chosen in this example. As soon as there is a need in the community for this, this will be worked out further 73 Stakeholder General understanding of a person or organization within or outside the modeling domain that influences the data, for example in the role of user, steward or owner. Component or information system Physical implementation or component within the ArchiMate domain that is used to describe applications that implement certain logical applications or application functions. For example the application process collaborate is implemented in the component SharePoint. Note that this can be a relatively complex elaboration in organizations with a fragmented application landscape. 73 Application function Description of a logical application that has a relationship with a certain data object. Logical applications are independent of the implementation and are often modeled with verbs or with terms ending on system. For example collaborate or collaboration system. 73 Business process Business process that uses a certain business object, roof can be both consult and mutation use 73 Score 1..10 Score 1..10 RACI RACI CRUD CRUD RA CI RCI RA CI CI RA CI RA CI CI RA CI CI RA CI CI CI RA CI CI RA CI RA CI CI 3 3 CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRUD CRU R R R R R R R R CRU R R R R R R R 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 1 3 3 3 3 3 3 3 3 3 3 1 1 1 1 1 2 2 1 1 1 1 1 1 1 atomic rotation showdecoration interfacekind originalpackage MeDaMa_DataEntityType MeDaMa_GoalLevel MeDaMa_Implications MeDaMa_Rationale IDEARationale IDEA RACI IDEA_Ranking IDEA_CRUD Navigatie algemeen Data Governance owners and stakeholders Data Governance goals and stakeholders Data Governance Stakeholder Matrix Data analytics en -science Data analytics en -science main categories Data filtering and selection Data filtering and selection including components Data visualisations for business users Big Data Blueprint Layered or tiered architecture fortransformation of data from sources to utilisation. It includes three architectural columns that influence all layers Serial BDP-DWH ABB Serial integration is implemented by introducing a big data platform for the transformation and extraction of unstructured and semi structured data as source for the EDWH functionality. Characteristics - Introduction of the big data platform is relatively easy since it is an extra layer added to the DWH functionality - Relative easy big data patterns are available because the source is always the datawarehouse - Introducing big data solutions for other functionalities than DHW is not possible. Appliance BDP-DWH ABB In the appliance integration of a big data platform with DWH functionality the appliance acts like a black box in which all functionality is integrated in a (proprietary) solution. This solution is configured for optimal performance of transformation and analysis. Characteristics - Appliance is developed, configured and often maintained by an external supplier - It is introduced as a fully integrated solution therefore existing implementations of the DWH have to migrate to this solution - Appliances are often introduced when a cloud solution is selected for the data platform Parallel BDP-DWH ABB The parallel integration is an extension of the DWH functiionality with the Big Data Platform. This extension makes it possible to use both functionalities side by side. Characteristics - Easy (incremental) introduction of the Big Data functionality - Integration of both functionalities requires attention for the introduction of the interconnect functionality because this can become a bottleneck in performance and configuration - Virtualisation BDP-DWH ABB This integration pattern has a close relation with the parallel integration, however there is an extra layer introduced for the virtualisation and standardisation of data extraction to consumers of the data. Characteristics - Virtualisation layer encapsulate the internal confuguration of the two platforms - The virtualisation layer requires a standardized data or objectmodel for the extraction by the consumers - The virtualisation can become a bottleneck in a number of qualities like performance, integratability e.g. - Big Data Mechanisms in Big Data Blueprint Plot of the described mechanisms on the Big Data Blueprint Big Data Mechanisms overview Technology mechanisms represent well-defined IT artifacts that are established within an IT industry. Additional Big Data Patterns Additional Big Data Patterns not linked in a compound pattern Analytical Sandbox Application enhancement Batch Data Processing Big Data Pipeline Big Data Processing Environment Big Data Transformation Big Data Warehouse Online Data Repository Operational Big Data Store Poly Sink Poly Source Poly Storage Random Access Storage Realtime Data Processing Streaming Access Storage Unstructured Data Store Data processing Data storage Data storage NewSql implementations Data storage NoSQL implementations Definition Big Data Types Hadoop components Big Data Overview Academy Principles Data Source Landscape Data Usage Landscape Sample Principles Overview of the principles from the DT Reference architecture. The implications have to be added in a later phase Data Quality Measures Data qualities This is an overview of the DaMa data qualities. It can be extended with the implications for a specific project Software qualities This is an overview of the Quint 2 software qualities. It can be extended with the implications for a specific project Scenario models consumer perspective In this master data consumer model a limited view of the relevant architectural entities are displayed. Here you see a high level model of register data consumption. Via different application functions data is consumed. For example via user interfaces like reports, portals, geo viewers etc data is directly consumed by various end users. A detailed inventarization of these end users and user interfaces will be modeled. Scenario model data collaboration In this scenario the master data register collaborates with the various data producing application functions. This means that when data is modified in one of the systems these modifications are shared between all collaborating functions. Therefore the integration between these data producers is essential in this scenario An interesting scenario in this is where the Data Register is only used as a key store or key cabinet and the detailed data is kept in the other source systems Advantages - Data is gathered directly from source systems and thus it is always accurate and real time. - Data can be stored within the source systems in a specific format supporting the business processes within these systems - Differences in availability between consumers and sources can be handled by the Data Register - Reuse of screens, workflows and validations in the source systems - Data standardization within the Data Register - Introduction of a key store or key cabinet. Disadvantages - Managing the synchronization between systems is an extra piece of work and complexity. - Replication of data - Complex data transformations from sources to register and back Scenario model data collector In this scenario data is collected form the various data producing applications and combined and standardized in the master data register. This means that data is modified in one of the data producing applications and eventually enriched in the data register. The data register is mainly a data replication with a standardized data model of the other data producers. An example is a datawarehouse Advantages - All data directly integrated at hand. - Standardization of data is possible within the Data Register - There is a possibility to enhance data by intelligently combing it to new information. - High availability only for the data register when consumers need a high availability - Data validation can be implemented in the system where it is most advantageous/efficient - Reuse of screens, validations, existing data integrations and workflows - Supports a iterative migration to a more centralized (register) scenario Disadvantages - When integration of data is asynchronous the data is not the same as in source systems on every moment. This will not be a problem if timing is not an issue. - When synchronization of data is synchronous high availability requirements for the registry systems is necessary - Data replication and need for extra storage - Fetching and distributing data back and forth is equally much work as with an MDM solution - Possible very complex data transformations necessary Scenario model data registry In this scenario there is only one master data producing application. That is the data register. It can also be one of the existing source systems. All other applications are consuming this data from the data register and use this in their application processes. This includes the application functions for ERP and geo etc. Advantages - The service design is directly mapped in the data register. - Possibility to standardize the information model and service interfaces - Verification and business rules are implemented only in the dasset data register. - Real time alignment of the data only upon read/request - High availability only necessary for the data register. - Eventually no replication of data (depends on the maturity of the consuming systems) Disadvantages - Any change in data model in consumers leads to change in service, this should be aligned or require a large standardized datamodel in the service interface. - Information provisioning to applications needs to be redesigned which is a lot of work - Redesign of the full application landscape - High demand in performance and availability for the data register - Introduction of a single point of failure so extra non functional requirements in AIC Scenario model data service In this scenario there is no data register but all master data is stored within the data producers like ERP and geo functions. However for the data consumers the data is available via the asset data services in a standardized manner. This means that when a consumer needs asset data this is requested via the data services and collected from the various data producing applications. The implementation of the data services handles the standardization of the master data model and the data exchange protocol Advantages - Real time alignment of the data. - Single Point of Truth and Maintenance - No replication of data (and the involved complexity) - Reuse of existing user interfaces, validations and (hidden) integrations Disadvantages - The service design should not enhance the data so the application might have to be redesigned. - Any change in data model in sources leads to change in service, this should be aligned. - Verification and business rules are implemented in source systems. - High availability and performance needs for all the producing systems - Complex model transformations within the service layer to transform for specific producer system model to the required model by the consumers - Releases of the source systems become more complex because of the new dependencies in the data services Logical Application Model Data Modeling Conceptual Data Modeling Hybrid Data Modeling Objectmodel BI and DataWareHousing ETL process AICP classification Basic diagram for an AIC classification, for a project this diagram can be extended with the implication for the project of these security requirements Data Security Technology Services Data Operations Incident File transformation overview Overview of the transformation from file based data to a data target Relation transformation overview Overview of the transformation from a relational database to a data target Servicemodel ABB Detailed ABB for describing the logical service interface in combination with extra governance functionalities Transformation generic ABB Description of the generic aspectis of the transformation application function. This has the following elements - Data sources - Transformation functions - Data targets - Data or object modeling functions XML transformation overview In this diagram a description of the datapipe from an (webservice) or XML file source to the target datamodel is described. This is based on the transformation of a XML model to an intermediate tabular or relational model and this is then processed in an ETL process to transform the source data model in a number of steps to the required target model. Data Integration General view The hourglass model is a specific model for the transformation of data sources to a standardized model in a target datastore. It is the simplified implementation of a layered Big Data architecture. The hourglass model can be used to medel specific implementations of transformation of data in a pattern called the datapipe. In a number of other diagrams a detail view is given of these implementations in projects like Digital Transformation, TDP, MaxLimit and others. Data Integration Consumers overview Overview of internal and external consumers of the standardized datasets. Data Integration Overview interfaces Specialized model of the various interfaces available for the consumers of the standardized datasets. Data Integration Transformation detailview In the context of TDP is the logical application of transformation of data a characteristic pattern.. This pattern includes aspects like extract data from the source, transformation of the model or the protocol and selection of the source and the target model. Transformation exists at two levels in the hourglass. The first is transformation from the source data to the target data in a standardized model. The second is the transformation of the generic model to a specific model in use by the consumer. The last one is not explicitly modeled in the hourglass model because this is considered as an implementation of the consumer. Meta Data Overview Navigation DaMAcademy Data Governance beleid Data Governance eigenaren and stewards Data Governance RACI matrix Data Kwaliteit Score Matrix Data Kwaliteit Maatregelen Master Data Logisch Register Model Master Data Scenario Model Data Architectuur Landschap Data Architectuur Patroon catalogus Data Architectuur Principe register Data Modelleren Conceptueel Datamodel BI and DataWareHousing ETL process Data Security CIA Data Infrastructuur Model Data Operations Maatregel model Data Integratie Patronen Data Integratie Register Primary data quality viewpoint This viewpoint connects three archimate concepts. These are the information entities and a requirement that is subdivided into a number of data qualities. In the example, the DAMA data quality list is used. Within the associations, a score is given to the data within the entities (business object and data object). Keep in mind that this score can be an ist but also a soll or desired score. This can be further elaborated in the matrix by including a double score. In this viewpoint an example of the matrix is included including the example qualities of DaMa Primary data ownership viewpoint This viewpoint describes which concepts are used for modeling data ownership. Data ownership and stewardship are important aspects of after-management. In the viewpoint, use is made of four ArchiMate concepts, three on the business layer and one on the application layer. It is important that for the ownership a number of associations have been extended on the basis of a RACI classification. These can be displayed in a RACI matrix. Primary data usage viewpoint This viewpoint indicates which business and data objects are used by which users and perhaps even more importantly in which way. This is done by giving the associations between the user and the data objects a CRUD score. This can of course also be displayed in a matrix view Secundary data usage viewpoint This secondary viewpoint describes which data and business objects are used within which processes and functions. This can easily be extended with, for example, the application where the data is used. However, this will usually not be within the scope of a data architect and should be cooperated with other architecture disciplines. Primary conceptual datamodel viewpoint The primary conceptual model consists of two ArchiMate concepts from the passive structure part. Namely data objects and business objects. Often one of these two entities is used in the models. This depends on the further modeling, finds this place on application level then you can use the data objects, otherwise the business objects For both entities you can use specialization and aggregation as an association. In addition, a data object realizes a business object. If desired, you can show cardinalities at the associations, but that is not common. In addition, in a complex model or in many synonyms and homonyms, the modeling of two layers of abstraction can have added value Secundary conceptual model viewpoint In the secondary conceptual model it is modeled where, how and by which business or application entities these information entities are used or more precisely which entities have access to the information entities. For this, use is made of the behavioral tendencies in the application and the business layer of ArchiMate depending on the modeling level. Optionally, you can extend the model even further with active structure elements, however the question is whether this is the domain of the data modeller. For the link between the information entities and the behavioral entities, the access association (access) is used. Please note that this association can go two ways (or both). Primary Logical data model viewpoint Conceptueel - Logisch In this viewpoint we see the relationship of the logical model with the conceptual model. An association is therefore made between two modeling languages and that is a specific extension for IDEA and is not part of both standards. Within EA there are three standard association types for modeling between modeling languages. Within IDEA we opt for the trace association for an association between the three modeling layers. Navigation Project Template