Many financial applications require aggregating a consumer’s data from banks and financial institutions to deliver their services to market. Financial data such as account balances, transactions, and other information is used for a wide range of use cases across areas such as lending, investment, Buy Now Pay Later (BNPL) and many other fintech services. Open Banking and Web scraping are two different data connectivity methods to access financial data, with both methods using application programming interfaces (APIs) to extract the data.

Web scraping (or screen scraping)

Web scraping (often referred to as Digital Data Capture or screen scraping), have been a widely used practice by third party service providers for decades to access and aggregate financial data from banks and financial institutions. It uses an institution’s web based interface (like their online banking platform) to login and acquire a customer’s data, with consent, using their username and password.  

From a consumer perspective, the lack of control and security are often seen as downsides in the use of web scraping. Because a user hands over credentials in the process, it’s unclear who accepts liability if something goes wrong. 

Furthermore, as web scraping rely on the institution’s interface, any updates made can impact a service provider’s ability to access this data, even with consent, making it difficult to deliver their service to market. It is why many countries, including Australia, have moved towards Open Banking as the preferred option.

Open Banking

Open Banking has been implemented as part of the Government’s rollout of the Consumer Data Right (CDR) program. The CDR is an initiative aimed at driving competition and the development of new financial products and services. It gives consumers greater control over their data and the ability to securely share their data for specific purposes, such as using a personal finance management app or applying for a loan. 

Open Banking uses open APIs to enable approved service providers to access and aggregate data from banks and financial institutions. It uses an authentication process that does not require the consumer to disclose their login and password details. 

One of the major advantages of using Open Banking APIs is that the data received is more structured, thus reducing time-to-development (especially when retrieving data from multiple financial institutions). It also provides prescribed guidance on the handling of data and reduces the potential of misinterpretation.

Other benefits of Open Banking include:

• More secure authentication process that does not require the consumer to disclose their login and password combination
• Access to a greater number of financial institutions
• Ability to retrieve financial data up to 20 times per day per consumer
• Consumers have a greater level of control over their data. Open Banking enables consumers to select which accounts they wish to share, and they can turn on or off data sharing via the bank or fintech application
• Parties that provide Open Banking services have been accredited to ensure that they comply with appropriate security practices

Comparing Open Banking and Web scraping

The following tables compare Open Banking and Web scraping in Australia across a number of features.

Differentiator	Open Banking	Web scraping
Regulation	* Open Banking is a regulated system. Treasury writes the CDR Rules, which are monitored and enforced by the ACCC and OAIC. * CDR restricts who can access Open Banking data. Any business using Open Banking data must: - meet eligibility for a CDR access model - adhere to strict privacy and security rules.	* Web scraping is unregulated. * Businesses can access consumer financial data via scraping without restrictions. * The Australian Government is considering sunsetting web scraping due to its unregulated nature.
ADIs (Authorised Deposit-taking Institutions) available	* All entities matching the regulated definition of an ADI must provide data or seek limited-period exemptions. * 111 ADIs are currently available with Open Banking. * APIs are stable, and any changes are mandated and carefully planned, making Open Banking more reliable.	* ADIs available depends on the data aggregator. * Banks control the ease of data access via scraping and can block scraping attempts anytime without warning.
Data-sharing	* Open Banking data can only be shared with eligible businesses outlined under CDR’s access models and with the consumer's permission as outlined in the ‘data connection process.’	* There are no restrictions on how and with whom the data can be shared. * There's nothing to prevent third parties from sharing consumer data with other organisations, meaning consumers' data can be duplicated and stored in various places.
Consumer consent	* The consumer must provide consent before data is shared. * Consumers can withdraw their consent at any time.	* Because web scraping is unregulated, there is no consistency regarding when or how customer consent is captured. * Third-party apps determine if and how they obtain and store consent, often embedded in a business’s Terms and Conditions.
Authentication process	* Consumers are directed to their banking app (Data Holder) to log in securely. * Consumers are NOT asked to share their online banking username and password.	* Consumers share their username and password with a third-party app to authenticate and connect their bank account, which logs in on the consumer’s behalf.
Data collection	* Open Banking uses purpose-built, open APIs to collect data. * Data points for collection are specified. * Consumers select the accounts and data shared.	* Web scraping collects data using an institution’s ‘unofficial’ API or web interface (like their online banking platform). * Once authenticated, the third-party app can collect any data accessible via online banking * Consumers have no control over the accounts and data shared.
Data availability	* More data is available. Each API has many mandatory and optional fields, including user, accounts and transaction data.	* It can only return data made available on online banking screens and is limited to what a scraper can realistically extract.
Data usage	* Open Banking data can only be used for the purpose specified during the ‘data connection process.’	* There are no restrictions on how scraping data can be used.
Data collection frequency	* Open Banking data can be refreshed on demand up to 20 times daily (capped by the CDR standards).	* Web scrapers usually only refresh data once per day. This can be increased but comes at the risk of banks blocking access.
Data deletion and retention	* There are strict rules for data deletion. * Unless the law requires, data must be deleted once consumer consent expires or is revoked. A de-identified version of the data can be kept with consumer permission.	* There are no requirements about when and how consumer data must be deleted. The decision is made by the third-party app. * Data can be held indefinitely.
Issue resolution	* Because Open Banking is regulated, there are established roles and responsibilities regarding liability.	* It’s unclear who is liable if there is an incident using web scraping data.

Find out more about Open Banking

Basiq’s Open Banking Hub has a library of relevant resources or visit the website of the Consumer Data Right homepage for more information.